OpenLineage

Overview

OpenLineage is a specification, not a product you run. It defines a common way for data tools to describe what ran, what it read and what it wrote, so lineage can be collected once at the source instead of being reconstructed by every catalogue separately.

Its object model is deliberately small: datasets, jobs and runs, extended by facets, which are named pieces of extra metadata attached to any of the three. Facets cover schema, SQL, source code, data quality assertions and column-level lineage, and anyone can define their own.

Lineage is captured by instrumentation rather than scanning. The Spark integration attaches a listener to the Spark context; Airflow support ships as a native provider from Airflow 2.7; a Rust SQL parser extracts lineage from statements; and Python and Java clients let you emit events from anything else.

There is no user interface, no server and nothing to host. Events are sent over HTTP, Kafka, files or cloud transports to a consumer that stores and displays them. Marquez is the project's reference implementation, and catalogues including DataHub, Atlan, Collibra, Amundsen and Microsoft's ecosystem consume the same events.

It graduated at the LF AI & Data Foundation in September 2023 and is released frequently; version 1.53.0 arrived in September 2026. First-party integrations number around nine, covering Spark, Airflow, dbt, Flink, Hive, Trino, Presto, Great Expectations and Feast.

Key features and capabilities

The same headings are used for every lineage entry, so two tools can be read side by side.

How lineage is captured

• Runtime instrumentation, such as a Spark listener reading the job's execution plan
• Native Airflow provider from Airflow 2.7 onwards
• A Rust SQL parser, also available as Java and Python libraries
• Python and Java clients for emitting events from anything not covered

Level of detail

• Three core objects: dataset, job and run, each identified within a namespace
• Column-level lineage facet, distinguishing direct and indirect transformations
• Facets for schema, SQL, source code, quality metrics and assertions
• Static design-time lineage as well as run events

Systems covered

• Around nine first-party integrations: Spark, Airflow, dbt, Flink, Hive, Trino, Presto, Great Expectations, Feast
• Spark sources include JDBC, S3, GCS, Azure Blob and ADLS, BigQuery and Synapse
• dbt support covers Snowflake and BigQuery

Analysis and visualisation

• None built in; the project ships no user interface or storage
• Analysis happens in the consumer, for example Marquez or a catalogue

Standards and APIs

• The specification itself, published as JSON Schema and extensible through custom facets
• Strict facet naming, and each custom facet must point at an immutable versioned schema
• Custom facets can be promoted into the core specification
• Documentation is versioned per specification release

Integrations

• Consumed by catalogues: DataHub, Atlan, Collibra, Amundsen, Egeria, Collate, Ataccama
• Cloud services: AWS including DataZone, Google Cloud, IBM; Datadog for observability
• Transports include HTTP, Kafka, files, S3, Google Cloud Storage and a proxy for routing

How it runs

• Nothing to host; you add a client or listener to existing jobs
• Java client published to Maven Central, Python client to PyPI
• Apache 2.0 licence

Pricing

Open source

Free and open source under the Apache 2.0 licence. There is no commercial edition of the specification itself.

Vendor pricing page →

Demos and videos

• Demo: cross-platform lineage for Flink and Airflow jobs Product demo · YouTube
• OpenLineage Project channel Channel or playlist · YouTube

About LF AI & Data Foundation

OpenLineage is a Linux Foundation project, hosted by LF AI & Data, and has no vendor behind it. It graduated to top-level status in September 2023, which the foundation tied to meeting OpenSSF Gold standards for governance, code quality and security. A technical steering committee runs it, with members drawn from Airflow, Iceberg, Parquet, dbt, Marquez and Snowflake; Astronomer is named a key contributor. The code is Apache 2.0 licensed.

openlineage.io