Data tooling landscape

Pipelines and Dataflow Gen2 for ingesting and transforming data in Microsoft Fabric.

• Cloud service

Lakehouse platform built on Apache Spark and open table formats, governed through Unity Catalog.

• Commercial

Defines Databricks jobs, pipelines and other workspace assets as YAML and deploys them with the Databricks CLI. Renamed Declarative Automation Bundles in 2026.

• Commercial

Monitoring and observability SaaS for infrastructure, applications, logs and data pipelines.

• Commercial

Data catalogue, documentation, lineage and quality platform pitched as lightweight, priced per editor with unlimited viewers and published list prices rather than quote-only.

• Commercial

Datafold

Control Plane · DataOps & CI/CD

Data diffing and automated testing that shows how code changes affect data before deployment.

• Commercial

Google Cloud service for developing and orchestrating SQL transformations in BigQuery.

• Cloud service

Data catalogue and governance platform positioning itself as a context engine, covering discovery, business meaning, lineage and AI-ready data context for both people and agents.

• Commercial

DataHub

Metadata Plane · Data Catalogues & Discovery

Open-source metadata platform for discovery, lineage and governance, with a managed cloud offering.

• Open core

Commercial distribution of Apache Gravitino, adding SCIM provisioning, role-based access control across federated catalogues, an administration UI, audit logging and a supported Helm chart.

• Commercial

dbt

Integration & ETL Plane · Transformation & Modelling

SQL-based transformation framework with testing and documentation, plus a managed platform.

• Open core

dbt Semantic Layer

Metadata Plane · Semantic Layer & Metrics

Metrics defined centrally in dbt and queryable from BI and other data tools.

• Commercial

Debezium

Integration & ETL Plane · Change Data Capture & Replication

Open-source change data capture platform that streams database changes into Kafka and other systems.

• Open source

Library from AWS Labs for defining unit tests for data on Apache Spark.

• Open source

Delta Lake

Data Plane · Open Table Formats

Open table format that adds ACID transactions, time travel and schema enforcement to data lakes.

• Open source

dlt

Integration & ETL Plane · ELT & Ingestion

Open-source Python library for loading data from APIs and files into warehouses and lakes.

• Open source

Cloud analytics platform spanning connectors, transformation, dashboards, workflow automation and AI, priced by consumption credits with unlimited users rather than per seat.

• Commercial

Dremio

Data Plane · Lakehouse Platforms

Lakehouse query engine and semantic layer built on Apache Iceberg and Apache Arrow.

• Open core

DuckDB

Data Plane · Query & Processing Engines

In-process analytical SQL database, popular for local analytics and embedded workloads.

• Open source

Lakehouse compute engine that allocates compute per query rather than by fixed warehouse size, aimed at high-concurrency SQL and AI workloads with no data movement.

• Commercial

Elementary

Observability Plane · Data Observability

dbt-native data observability, with an open-source package and a cloud platform.

• Open core

Idera's enterprise data modelling suite for logical and physical models, with a shared team repository, business glossary and web portal for metadata.

• Commercial

Long-established data modelling tool, now from Quest, for conceptual, logical and physical models, with forward and reverse engineering of database schemas.

• Commercial

Evidence

Visualisation & BI Plane · Notebooks & Data Apps

Code-based BI that builds reports from SQL and Markdown.

• Open source