Apache Airflow

Overview

Apache Airflow is the default answer to "what orchestrates our pipelines?" and has been for years. Workflows are written as Python code, which gives version control, code review and testing, and it schedules, retries and monitors them across whatever systems they touch.

The project is refreshingly honest about who it does not suit: its own documentation says that if you prefer clicking to coding, Airflow may not be the right fit.

Airflow 3, the largest release in its history, changed the architecture rather than the surface. Task execution was decoupled from the scheduler through a task execution API, which allows remote execution; the interface was rebuilt; and DAG versioning became native, so you can see how a pipeline looked when a past run happened.

The other shift is that it now understands data, not only tasks. Assets are first-class objects identified by a URI: a task declares what it produces, and another DAG can be scheduled to run when those assets update, with AND and OR conditions across several.

Version 3.3 adds stateful tasks, tasks written in Java or Go, asset partitioning and pluggable retry policies.

Its real moat is the ecosystem: over 100 provider packages, including dbt Cloud, Spark, Snowflake and the major clouds, plus an official Helm chart and Docker image. It is Apache 2.0, with no commercial edition from the foundation; managed Airflow is sold by vendors such as Astronomer.

Key features and capabilities

The same headings are used for every orchestration & scheduling entry, so two tools can be read side by side.

How pipelines are written

• Python DAGs, either classic operators or the decorator-based TaskFlow API
• Local testing with commands that run a DAG or a single task
• A stable task SDK, with tasks in Java or Go from version 3.3

Scheduling and triggers

• Cron and interval schedules declared per DAG
• Asset-based scheduling, with AND and OR expressions across several assets
• Event-driven triggers, by pushing asset events or watching an external source
• Scheduler-managed backfills and selective rerun

Execution and scaling

• Executors: local, Celery, Kubernetes, Edge, AWS Batch and ECS
• Containerised executors isolate each task, at the cost of start-up latency
• A task execution API decouples execution from the scheduler, enabling remote workers
• Pluggable retry policies replace a fixed retry count

Monitoring and recovery

• Rebuilt web interface with version-aware graph and grid views
• Logging, metrics, traces, callbacks and health checks, with Sentry support
• Deadline alerts for time-based task targets
• Native DAG versioning records structure changes for historical inspection

Data awareness

• Assets are first-class, identified by URI, declared as task outlets
• An asset counts as updated only when the producing task succeeds
• Consuming DAGs can read the triggering asset events
• OpenLineage ships as an official provider package

Integrations

• Over 100 active provider packages
• dbt Cloud, Spark, Snowflake, AWS, Google Cloud and Azure among them
• A CLI, Python API client and task SDK alongside the core

How it runs

• Official Helm chart for Kubernetes, and an official Docker image
• Self-hosted from the project; managed options come from vendors
• Apache 2.0 licence

Pricing

Open source

Free and open source under Apache 2.0. The foundation sells nothing; managed Airflow comes from vendors.

Vendor pricing page →

Demos and videos

• Airflow monthly town hall Webinar or talk · YouTube
• Apache Airflow channel Channel or playlist · YouTube

About The Apache Software Foundation

Airflow is an Apache Software Foundation project, Apache 2.0 licensed, governed by a project management committee and committers rather than a company. Releases follow the ASF release policy and are voted on by the PMC, and significant changes go through a public improvement proposal process on the mailing list. There is no vendor owner, no commercial edition and no pricing: companies such as Astronomer sell managed Airflow and employ many of its committers, but hold no governance authority over the project.

airflow.apache.org