Apache Kafka

Overview

Apache Kafka is the durable log that most of the streaming world is built on. It stores and transports an append-only, partitioned commit log, letting applications publish events, subscribe to them, and keep them for as long as they want.

It is worth being precise about what it is not. Kafka is a streaming platform, not a processing engine: it holds and moves data, while Flink, Spark or Kafka Streams compute over it.

Events go into topics split into partitions. Events with the same key land in the same partition, and consumers always read a partition in exactly the order it was written, which is where the ordering guarantee comes from.

The problem it actually solves is decoupling producers from consumers with durable, replayable, ordered retention, rather than the transient delivery a queue gives you. Production clusters typically replicate three ways.

ZooKeeper has gone. From Kafka 4.0 it runs entirely without it, in KRaft mode by default, with the control plane integrated into Kafka itself using Raft, which removes a whole separate cluster from operations.

Tiered storage is documented and generally available: completed log segments move to external storage such as S3 or HDFS, so retention is no longer bounded by broker disks.

The newest capability closes an old gap. Queues, implemented as share groups, became production-ready in 4.2.0, which brings competing-consumer semantics to the same log.

Delivery semantics are configurable rather than assumed. At-most-once, at-least-once and exactly-once are all documented, with an idempotent producer and transactions across multiple partitions.

Security is present but basic by the standards of the commercial distributions: TLS, SASL mechanisms including Kerberos and OAuth, and pluggable ACLs, with no schema registry, audit subsystem or certifications.

Plan for Java versions when upgrading: from 4.0 brokers, Connect and tools need Java 17, while clients and Streams need Java 11, and 4.2.0 added Java 25 support.

Key features and capabilities

The same headings are used for every streaming & stream processing entry, so two tools can be read side by side.

What it does

• A distributed, partitioned, replicated commit log, with keys routing events to partitions
• Consume as a stream through consumer groups, or as a queue through share groups
• Log compaction retains at least the last state for each key, for changelog-style topics
• Brokers form the storage layer, with clients talking to them over a TCP protocol

Delivery and ordering guarantees

• Three documented semantics, at-most-once, at-least-once and exactly-once
• An idempotent producer guarantees resends do not duplicate entries in the log
• Transactions send messages atomically across multiple topic partitions
• Ordering is always maintained per partition, and a write is not committed until all in-sync replicas have it

Scale and retention

• Published claims of latencies as low as 2ms and trillions of messages a day across thousands of brokers
• No hard partition ceiling published; retention defaults to seven days with no size limit
• Tiered storage offloads completed segments to S3 or HDFS, separating retention from broker disk
• Tiered storage does not support compacted topics, and cannot be disabled without deleting tiered topics first

Processing and transformation

• Kafka Streams is a client library, needing no separate processing cluster, with exactly-once semantics
• Streams is Java and Scala only, deployed to containers, virtual machines, bare metal or cloud
• ksqlDB is not part of Apache Kafka; it is Confluent software under the Confluent Community License
• Heavier processing means an external engine such as Flink or Spark

Security and governance

• Authentication through SSL or SASL, with GSSAPI, PLAIN, SCRAM and OAUTHBEARER mechanisms
• Encryption in transit with SSL between clients, brokers and tools
• Pluggable authorisation of read and write operations through ACLs
• No schema registry, audit subsystem or compliance certifications ship with the project

Connectors and ecosystem

• Kafka Connect streams data to and from other systems, in distributed or standalone mode, with a REST API
• Five core APIs, admin, producer, consumer, Streams and Connect
• A large community ecosystem of clients and tools indexed by the project
• The Kafka wire protocol is the de facto standard, re-implemented by Redpanda and Azure Event Hubs

Where it runs and what it costs

• Self-hosted on any Java platform, with official container images documented
• From 4.0, brokers, Connect and tools need Java 17 while clients and Streams need Java 11
• Kubernetes operators are all third-party rather than project deliverables
• No managed service from the foundation, so hosting means a vendor or your own operations

Pricing

Open sourceFree, Apache 2.0

Kafka has no price. It is distributed free under Apache 2.0 with no paid tier, no free-tier concept and nothing sold by the foundation, which also means no managed service and no support contract from the project itself. The real cost of self-hosting is operational, brokers to run and upgrade, storage to size, and the monitoring and security tooling the distribution does not include. Managed Kafka pricing lives with the vendors, and Confluent publishes the most complete rate card of them.

Vendor pricing page →

Demos and videos

• Apache Kafka community videos Channel or playlist · kafka.apache.org
• The Apache Software Foundation channel Channel or playlist · YouTube

About Apache Software Foundation

Kafka entered Apache incubation in July 2011 and graduated to a top-level project in October 2012. It is licensed under Apache 2.0 and stewarded by the Apache Software Foundation, a 501(c)(3) nonprofit operating since 1999, funded by donations and corporate sponsors and run almost entirely by volunteers. No company controls it and the foundation runs no managed service, so every hosted Kafka is a third-party product. Its origin at LinkedIn is well known but is not stated on the project's own pages. Releases 4.3.0 and 4.3.1 landed in May and June 2026.

Founded 2012 · Wilmington, Delaware · kafka.apache.org