Control Plane

Open-source platform for authoring, scheduling and monitoring workflows defined as Python DAGs.

• Open source

Managed Apache Airflow platform for running and observing data pipelines at scale.

• Commercial

Serverless AWS service for orchestrating multi-step workflows across AWS services.

• Cloud service

Dagster

Control Plane · Orchestration & Scheduling

Data orchestrator built around software-defined assets, with lineage and observability built in.

• Open core

Kestra

Control Plane · Orchestration & Scheduling

Declarative, event-driven orchestration platform with workflows defined in YAML.

• Open core

Prefect

Control Plane · Orchestration & Scheduling

Python-native workflow orchestration framework with a managed cloud service.

• Open core

Microsoft's domain-specific language for declaratively deploying Azure resources.

• Open source

Defines Databricks jobs, pipelines and other workspace assets as YAML and deploys them with the Databricks CLI. Renamed Declarative Automation Bundles in 2026.

• Commercial

OpenTofu

Control Plane · Infrastructure as Code

Open-source fork of Terraform under the Linux Foundation, compatible with Terraform providers.

• Open source

Pulumi

Control Plane · Infrastructure as Code

Infrastructure as code written in general-purpose languages such as TypeScript, Python and Go.

• Open core

Infrastructure as code tool from HashiCorp, with providers for clouds and data platforms such as Databricks and Snowflake.

• Source available

Microsoft's suite of repos, boards and CI/CD pipelines.

• Commercial

Datafold

Control Plane · DataOps & CI/CD

Data diffing and automated testing that shows how code changes affect data before deployment.

• Commercial

CI/CD automation built into GitHub, widely used to test and deploy data pipelines.

• Commercial

lakeFS

Control Plane · DataOps & CI/CD

Git-like version control for data lakes, with branches, commits and rollbacks on object storage.

• Source available