Marquez

Overview

Marquez is a metadata server: a running service that collects, stores and displays lineage. Where OpenLineage defines the format, Marquez is the thing that receives those events, and the project describes itself as the specification's reference implementation.

It exposes an OpenLineage-compatible HTTP endpoint, so any conformant producer can write to it, and a REST API for querying datasets, jobs, runs and namespaces directly. A web interface draws the lineage graph and lets you search across namespaces.

Because it stores runs as well as datasets, it keeps a history: dataset versions, job versions and run states from start through completion or failure. That is what makes it useful for tracing a bad output back to the run that produced it, or working out what to backfill.

In practice it is used with the OpenLineage integrations for Airflow, Spark, Flink, dbt and Dagster. Version 0.50.0 added a dashboard of event statistics.

It is a graduated LF AI & Data project, Apache 2.0 licensed, and runs on Java 17 with PostgreSQL as its only datastore, which keeps operations light. One thing to weigh: the repository is still being committed to, but the most recent tagged release, 0.50.0, dates from October 2024.

Key features and capabilities

The same headings are used for every lineage entry, so two tools can be read side by side.

How lineage is captured

• Receives OpenLineage events pushed to its HTTP endpoint; it does not instrument sources itself
• Metadata can also be written directly through the REST API
• Collection normally comes from the OpenLineage integrations for Airflow, Spark, Flink, dbt and Dagster

Level of detail

• Datasets, jobs and runs, with run states for start, complete, fail and abort
• Column-level lineage exposed through the API alongside the dataset graph
• Facets retrievable individually, and field-level tags
• Dataset and job metadata is versioned

Systems covered

• Anything that emits valid OpenLineage, since the endpoint follows the specification
• Airflow, Spark, Flink, dbt and Dagster in practice
• Physical sources such as database tables and message topics are modelled explicitly

Analysis and visualisation

• Web interface with an interactive lineage graph
• Search across datasets and jobs in all namespaces
• Run history and dataset versions, for root-cause work and backfills
• Event statistics dashboard added in 0.50.0

Standards and APIs

• Consumes OpenLineage, and is its reference implementation
• Used as the backend in OpenLineage's own getting-started tutorial
• Emitting OpenLineage events onward to other systems is not published

Integrations

• Java and Python HTTP clients ship with the project
• A published guide for running on AWS with managed PostgreSQL
• Integration with commercial catalogues is not published

How it runs

• Self-hosted only; Docker Compose quickstart, or a Helm chart for Kubernetes
• Requires Java 17 and PostgreSQL 14; PostgreSQL is the only datastore
• API on port 5000 and web interface on 3000; images on Docker Hub

Pricing

Open source

Free and open source under the Apache 2.0 licence. No paid tier or hosted service is published.

Vendor pricing page →

Demos and videos

• Demo: Marquez with dbt Product demo · YouTube
• Marquez Project channel Channel or playlist · YouTube

About LF AI & Data Foundation

Marquez was built and open-sourced by WeWork, and is now a graduated project of the LF AI & Data Foundation rather than a company product. It is Apache 2.0 licensed and run as a community project, with a code of conduct, monthly community meetings and a public Slack. No commercial vendor, funding or head office applies. Its founding year is not published on the project site.

marquezproject.ai