AWS Step Functions

Overview

AWS Step Functions is a managed workflow service built on state machines. It is a general-purpose orchestrator for distributed applications rather than a data pipeline tool, and that distinction matters when comparing it with Airflow or Dagster: it models states and tasks, not datasets.

Workflows are written in a JSON-based states language, or drawn in Workflow Studio, which offers design, code and configuration views and exports to JSON or YAML. Retries, catches, parallel branches and map states are built in rather than bolted on.

There are two workflow types with different economics. Standard workflows run up to a year, guarantee exactly-once execution and are billed per state transition. Express workflows run up to five minutes, are at-least-once, and are billed by request count and duration, which suits high-volume event processing.

Distributed map is the feature data teams tend to use: it fans a workflow across large datasets in S3, defaulting to 10,000 parallel child executions, with failure tolerance thresholds.

Two gaps are worth stating plainly. It has no scheduler of its own, so recurring runs come from EventBridge Scheduler. And it has no data awareness at all: no assets, no lineage, no OpenLineage, and no built-in data quality checks.

Its strength is reach inside AWS: SDK integrations to over 200 services, and optimised integrations for Glue, EMR, Athena, Lambda, SageMaker and Bedrock. There is no third-party plugin ecosystem, and no dbt integration.

Key features and capabilities

The same headings are used for every orchestration & scheduling entry, so two tools can be read side by side.

How pipelines are written

• Amazon States Language, a JSON-based definition, or the visual Workflow Studio
• Workflow Studio has design, code and configuration modes, exporting JSON or YAML
• Two query languages, JSONPath and JSONata, with variables for passing data between states
• Local testing exists but is unsupported; AWS recommends its test state API

Scheduling and triggers

• No built-in scheduler; recurring runs use EventBridge Scheduler
• EventBridge supports cron and rate expressions, and one-off invocations
• Event-driven starts through EventBridge rules, for example on S3 events
• Workflow-to-workflow dependencies by one state machine invoking another; no backfill

Execution and scaling

• Serverless and fully managed; no executors, workers or clusters to size
• Built-in retries, catches, parallel and map states
• Distributed map fans out over S3 data, defaulting to 10,000 parallel child executions
• Work can also run on external activity workers outside the service

Monitoring and recovery

• Console visualisation with step-by-step debugging
• Standard workflows keep execution history in the service; Express workflows do not
• Express logging goes to CloudWatch Logs, which AWS notes is best-effort
• X-Ray tracing per state machine, and a map run details view

Data awareness

• None; it models tasks and states, not datasets or assets
• No OpenLineage emission is published
• No built-in data quality checks
• The nearest equivalents are S3 item readers and result writers in distributed map

Integrations

• SDK integrations to over 200 AWS services and thousands of API actions
• Optimised integrations for Glue, EMR, Athena, Lambda, SageMaker, Bedrock, ECS and more
• Three patterns: request-response, run-a-job and wait-for-callback
• AWS only; no third-party plugin ecosystem and no dbt integration

How it runs

• Managed AWS service only; no self-hosted, Docker or Kubernetes option
• Infrastructure as code through CloudFormation, SAM or CDK
• Local version for development only, marked unsupported

Pricing

Usage-based4,000 state transitions a month free

Standard workflows cost $0.025 per 1,000 state transitions, with 4,000 free every month indefinitely, not just for the first year. Express workflows cost $1.00 per million requests plus duration billed per GB-hour, from $0.06 falling to $0.016 at volume, with memory billed in 64 MB chunks and no separate free tier. Rates vary by region. CloudWatch Logs and the services a workflow calls are billed separately.

Vendor pricing page →

Demos and videos

• Serverless 101: AWS Step Functions Tutorial · YouTube (Serverless Land, an AWS property)
• Advanced serverless workflow patterns and best practices Webinar or talk · YouTube (AWS Events)

About Amazon Web Services

Step Functions is a managed AWS service, launched in December 2016, consumed under the AWS Customer Agreement. There is no source code, no self-hosted licence and no community governance: the roadmap is entirely AWS's. A local version exists for development, but AWS marks it unsupported and without feature parity, recommending its test API instead. Regional availability follows the AWS region table.

aws.amazon.com