Apache Gravitino

Overview

Apache Gravitino describes itself as a federated metadata lake: rather than crawling other systems and holding a copy of what it finds, it connects to them and manages their metadata directly, so a change made through Gravitino appears in the underlying system and a change made there appears in Gravitino.

The object model is a three-level namespace inside a container called a metalake. A metalake holds catalogues, each with a connector to one metadata source; a catalogue holds schemas; and a schema holds tables, views, filesets, models, topics or user-defined functions. That spread is the point — one namespace covers relational tables, files on object storage, Kafka topics and registered ML models rather than tables alone.

Sources are reached through connectors. Lakehouse catalogues cover Iceberg, Hudi and Paimon plus a generic option; a Hive catalogue fronts a metastore; JDBC catalogues cover MySQL, PostgreSQL, ClickHouse, Doris, StarRocks, OceanBase and Hologres; fileset catalogues cover HDFS and cloud object storage; and Kafka and model catalogues cover topics and ML models.

In the other direction Gravitino serves a standards-compliant Iceberg REST catalogue, which is how engines reach it without bespoke integration, alongside connectors for Trino, Spark, Flink and Daft, Java and Python clients, a CLI and an MCP server for LLM tools.

Access control uses two models together: role-based, where privileges go to roles and roles to users and groups, and discretionary, where every object has an owner who controls management operations on it. Built-in authorisation is opt-in through configuration, and privileges can instead be pushed down to a source's own permissions or to Apache Ranger.

It began at Datastrato, entered the Apache Incubator and is now a top-level project under Apache 2.0, with 1.3.0 the current release. Datastrato still employs many of its contributors and sells a commercial distribution.

Key features and capabilities

The same headings are used for every table & technical catalogues entry, so two tools can be read side by side.

What it catalogues

• Metalake as the tenant or container, exposing a three-level catalogue.schema.table namespace
• Objects: tables, views, filesets, models, topics and user-defined functions
• Metadata managed directly in the source, so changes flow both ways rather than being crawled
• Tags, policies and statistics attach to objects; jobs are tracked as first-class entities

Table formats

• Lakehouse catalogues: Apache Iceberg, Apache Hudi, Apache Paimon, plus a generic option
• Hive catalogue fronting an existing metastore
• JDBC catalogues: MySQL, PostgreSQL, ClickHouse, Doris, StarRocks, OceanBase, Hologres
• Filesets for unstructured data on HDFS and cloud object storage; Kafka topics; registered ML models

Engines that can use it

• Connectors for Trino, Apache Spark, Apache Flink and Daft
• Any engine that speaks the Iceberg REST protocol, via the built-in catalogue service
• Java and Python clients, a REST API and a CLI
• An MCP server, so LLM tooling can read and manage metadata

Access control and auditing

• Role-based access control combined with discretionary control through per-object ownership
• Built-in authorisation is opt-in by configuration, with a pass-through implementation to disable it
• Privileges can be pushed down to a source's native permissions or to Apache Ranger
• Column-level lineage through OpenLineage, collected from Spark, across fileset, Iceberg, Hudi, Paimon, Hive and model catalogues

Interoperability

• Serves a standards-compliant Iceberg REST catalogue rather than a proprietary protocol
• A Lance REST service for multimodal datasets
• Geo-distributed deployment: a local Iceberg REST catalogue proxies a remote one for a cross-region view
• REST is the current interface; Thrift and JDBC are stated as future work

Table maintenance and operations

• Table Maintenance Service, the optimizer, is alpha: built-in strategy covers Iceberg compaction only
• Compaction limited to Iceberg tables with identity partition transforms
• Statistics and metrics collection feed rule evaluation, which submits Spark jobs from templates
• Server metrics are exposed for every service; job status is readable over REST

How it runs

• Self-hosted only; no managed service from the foundation
• Binary distribution or Docker images, on Java 17
• Server and Iceberg REST server run together or independently
• A playground is published for evaluation; entities persist in a relational backend

Pricing

Open source

Free under the Apache 2.0 licence, with no paid edition from the foundation. Commercial support and added operational features come from Datastrato Enterprise, a separate distribution.

Vendor pricing page →

Demos and videos

• Apache Gravitino documentation Tutorial · gravitino.apache.org (the project)

About The Apache Software Foundation

Gravitino is an Apache Software Foundation top-level project, licensed Apache 2.0 and governed by its own project management committee. The code originated at Datastrato, the company founded to build it, which donated it to the foundation; Datastrato continues to employ many of its contributors and maintains a commercial distribution, Datastrato Enterprise, on top of it. The foundation itself offers no managed service or paid edition. Development is active, with the repository seeing commits on the day this was checked.

gravitino.apache.org