Data tooling landscape

Kestra

Control Plane · Orchestration & Scheduling

Declarative, event-driven orchestration platform with workflows defined in YAML.

• Open core

Keycloak

Security Plane · Identity & Access

Open-source identity and access management with single sign-on, OpenID Connect and SAML.

• Open source

Kubeflow

AI & ML Plane · ML Platforms & Lifecycle

Open-source toolkit for running machine learning workflows on Kubernetes.

• Open source

lakeFS

Control Plane · DataOps & CI/CD

Git-like version control for data lakes, with branches, commits and rollbacks on object storage.

• Source available

BI built natively on dbt, taking metrics from the dbt project as the semantic layer. Open source to self-host, with a cloud tier priced per month rather than per seat.

• Open core

BI platform on Google Cloud with a modelling layer, LookML, for governed metrics.

• Cloud service

Marquez

Metadata Plane · Lineage

Open-source metadata service that collects and visualises OpenLineage data.

• Open source

Matillion

Integration & ETL Plane · ELT & Ingestion

Cloud data integration and transformation platform for building pipelines on cloud data platforms.

• Commercial

Metabase

Visualisation & BI Plane · BI & Dashboards

Open-source BI tool for questions, dashboards and embedded analytics.

• Open core

Microsoft's cloud identity and access management service, used for single sign-on to Azure and Fabric.

• Cloud service

SaaS analytics platform that unifies data engineering, warehousing, real-time analytics and Power BI on OneLake.

• Cloud service

T-SQL data warehouse in Microsoft Fabric that stores its tables in Delta format in OneLake.

• Cloud service

Microsoft's portfolio for data governance, security and compliance, including a data catalogue.

• Cloud service

Milvus

AI & ML Plane · Vector Databases & Search

Open-source vector database built for large-scale similarity search.

• Open source

MinIO

Data Plane · Object Storage

High-performance, S3-compatible object storage for on-premises or any cloud. The AGPL community edition is archived and no longer maintained; the supported product is the commercially licensed AIStor.

• Commercial

MLflow

AI & ML Plane · ML Platforms & Lifecycle

Open-source platform for experiment tracking, model registry and GenAI evaluation, created by Databricks.

• Open source

Monte Carlo

Observability Plane · Data Observability

Data observability platform that monitors freshness, volume, schema and quality across the data stack. Now trading as Monte Carlo AI, with montecarlodata.com redirecting, and extended to monitoring AI agents.

• Commercial

MotherDuck

Data Plane · Cloud Data Warehouses

Serverless cloud warehouse built on DuckDB, with dual execution splitting a query between laptop and cloud, and zero-copy sharing between accounts.

• Cloud service

Okta

Security Plane · Identity & Access

Identity platform for single sign-on, multi-factor authentication and user lifecycle management.

• Commercial

Omni

Visualisation & BI Plane · BI & Dashboards

BI that pairs a shared data model with raw SQL, building the model progressively out of the analysis people actually do rather than requiring it up front.

• Commercial

Privacy, consent and data governance platform for regulatory compliance programmes.

• Commercial

Open specification from the Bitol project for describing data contracts in YAML.

• Open source

OpenBao

Security Plane · Secrets & Key Management

Open-source, community-governed fork of HashiCorp Vault for secrets management.

• Open source

OpenLineage

Metadata Plane · Lineage

Open standard for collecting lineage metadata from pipelines and engines.

• Open source