Data tooling landscape

Data and AI observability across pipelines, data quality and the platforms underneath, including Hadoop and hybrid estates, sold as products on one control plane.

• Commercial

Catalogue and governance platform built on a federated knowledge graph, covering discovery, lineage, data products and contracts, with a marketplace and agent-facing access to its metadata.

• Commercial

Airbyte

Integration & ETL Plane · ELT & Ingestion

Data integration platform with a large connector catalogue, self-hosted or as a cloud service.

• Open core

Enterprise data catalogue for search, discovery, stewardship and governance.

• Commercial

AWS services for collecting and processing real-time streaming data.

• Cloud service

Managed cloud data warehouse on AWS, with provisioned and serverless options.

• Cloud service

Object storage service from AWS, commonly used as the foundation of data lakes.

• Cloud service

AWS service for building, training and deploying machine learning models.

• Cloud service

Amundsen

Metadata Plane · Data Catalogues & Discovery

Data discovery tool built at Lyft, with search ranked by usage. Archived in September 2026 and no longer maintained; listed for reference and migration planning.

• Open source

Automated data quality monitoring that uses machine learning to detect anomalies.

• Commercial

Open-source platform for authoring, scheduling and monitoring workflows defined as Python DAGs.

• Open source

Apache Flink

Data Plane · Streaming & Stream Processing

Stateful stream processing framework for real-time computation over unbounded data.

• Open source

Federated metadata catalogue for tables, files, streams and models, managed in place across sources and served to Spark, Trino and Flink through one namespace.

• Open source

Apache Hudi

Data Plane · Open Table Formats

Open data lakehouse format focused on incremental processing, upserts and deletes.

• Open source

Apache Iceberg

Data Plane · Open Table Formats

Open table format for large analytic tables, supported by many engines and platforms.

• Open source

Distributed event streaming platform for high-throughput data pipelines and streaming applications.

• Open source

Open-source catalogue that implements the Apache Iceberg REST catalogue API.

• Open source

Apache Ranger

Data Governance Plane · Access Policy Enforcement

Framework for centrally managing fine-grained access policies across Hadoop and related data services.

• Open source

Distributed engine for large-scale data processing, SQL, streaming and machine learning.

• Open source

Open-source data exploration and visualisation platform.

• Open source

Managed Apache Airflow platform for running and observing data pipelines at scale.

• Commercial

Atlan

Metadata Plane · Data Catalogues & Discovery

Active metadata platform for data cataloguing, discovery and governance.

• Commercial

AtScale

Metadata Plane · Semantic Layer & Metrics

Semantic layer for consistent business metrics across BI tools and data platforms.

• Commercial

Serverless data integration service on AWS for discovering, preparing and moving data.

• Cloud service