Apache Flink

Overview

Apache Flink belongs in a different column from Kafka, and confusing the two is the most common mistake in this category. Flink is a processing engine with no storage layer: it computes over streams that Kafka, Kinesis or Event Hubs hold.

Its tagline is stateful computations over data streams, and state is the point. Applications keep large working state locally, and Flink guarantees exactly-once state consistency by checkpointing that state to durable storage periodically and asynchronously.

It handles both unbounded and bounded streams, the second being streams with a defined end, which is how one engine serves both streaming and batch work.

APIs are layered, and the choice is a real trade-off: SQL and the Table API for conciseness, the DataStream API for control, and ProcessFunctions for fine-grained handling of time and state.

Event-time processing with watermarks is the mechanism for correctness over late data, letting you trade latency against completeness explicitly rather than hoping records arrive in order.

Flink 2.0 in March 2025 was the architectural reset, introducing disaggregated state through the ForSt backend for cloud-native deployments, and removing the DataSet API, the Scala DataStream API and the old source and sink interfaces.

Plan upgrades around that: 2.0 raised the minimum to Java 11 and removed APIs that older jobs depend on, while the 1.20 line remains the long-term support release.

Operationally, savepoints are the feature that earns its keep, used for version upgrades, cluster migration, rescaling, and pausing and resuming applications.

One security default to check before exposing anything: TLS is not enabled by default, and the REST endpoint does not authenticate clients unless you configure it.

The current direction is Flink Agents, a streaming agent runtime with Python and Java APIs, still a preview with experimental interfaces.

Key features and capabilities

The same headings are used for every streaming & stream processing entry, so two tools can be read side by side.

What it does

• Stateful stream processing over unbounded and bounded streams, with no storage tier of its own
• State primitives for values, lists and maps, held in a pluggable backend, in memory or RocksDB
• Jobs are dataflow graphs of operators, with per-operator parallelism
• Flink 2.0 added disaggregated state through the ForSt backend for cloud-native deployment

Delivery and ordering guarantees

• Exactly-once state consistency through asynchronous, incremental checkpointing
• End-to-end exactly-once depends on transactional sinks; the engine guarantees its own state
• Event-time semantics with watermarks, trading latency against completeness
• Savepoints support application evolution, migration, version upgrades, rescaling, and pause and resume

Scale and retention

• Published user-reported ceilings of trillions of events a day and multi-terabyte state on thousands of cores
• No partition or retention quotas, since retention is the source system's concern
• Rescaling up or down is done from a savepoint rather than in place
• ForSt targets scaling very large state without local disk limits

Processing and transformation

• Four abstraction levels, Flink SQL, the Table API, the DataStream API and ProcessFunction
• Java and Scala are the core languages, with Python available and used by Flink Agents
• Materialized tables and changelog conversion operators unify streaming and batch
• Flink Agents 0.3 adds a streaming agent runtime, as a preview with experimental interfaces

Security and governance

• Internal connections can use mutual TLS authentication and encryption
• TLS is not enabled by default, and must be switched on explicitly
• The REST endpoint does not authenticate clients by default
• No built-in role-based access control, schema registry, audit subsystem or certifications

Connectors and ecosystem

• Connectors release separately from the engine, including Kafka, JDBC, AWS, Elasticsearch and CDC
• Metrics reporters for JMX, Prometheus, Graphite, StatsD, Datadog and others
• Flink ML and Stateful Functions as companion projects
• An experimental native S3 filesystem arrived in 2026 for lake-facing work

Where it runs and what it costs

• Self-hosted standalone, or on Hadoop YARN or Kubernetes
• An official Kubernetes operator, currently 1.16.1, supporting Flink 1.19 through 2.3
• Managed by Confluent at $0.21 a Flink unit hour, or by AWS at $0.11 a processing unit hour
• Version support runs on published policy, with 1.20 as the long-term support line

Pricing

Open sourceFree, Apache 2.0

Flink itself is free under Apache 2.0, with no paid tier and nothing sold by the foundation. Real figures exist only for the managed services: Confluent Cloud for Apache Flink at $0.21 a unit hour, metered per minute with a one-minute minimum per statement and regional multipliers, and Amazon Managed Service for Apache Flink at $0.11 per processing unit hour in US East, billed by the second, plus $0.10 a GB a month for running storage. Notebook use on AWS adds two processing units.

Vendor pricing page →

Demos and videos

• The Apache Software Foundation channel Channel or playlist · YouTube
• What is Flink? Architecture and concepts Tutorial · flink.apache.org

About Apache Software Foundation

Flink entered Apache incubation in April 2014 and graduated in December of the same year, one of the faster graduations in the foundation's history, and is licensed under Apache 2.0. It is stewarded by the Apache Software Foundation, a 501(c)(3) nonprofit, with no company in control and no managed service of its own, so commercial Flink comes from others, principally Confluent Cloud for Apache Flink and Amazon Managed Service for Apache Flink. Flink 2.3.0 arrived in June 2026, with 1.20.5 as the long-term support line and an official Kubernetes operator supporting both.

Founded 2014 · Wilmington, Delaware · flink.apache.org