Amazon Kinesis

Overview

Amazon Kinesis Data Streams is AWS's own streaming platform, serverless and managed, for capturing and processing data streams at scale. It does the same job as Kafka but with its own API rather than a compatible one.

The unit of capacity and parallelism is the shard, which plays the role Kafka's partition does. Records carry a partition key and a sequence number, and shards are what you either size or let AWS scale.

Provisioned mode gives explicit arithmetic: each shard supports 1 MB/s or 1,000 records a second in, and 2 MB/s or 2,000 records a second out.

On-demand mode removes that planning. New streams start at 4 MB/s write and 8 MB/s read, scaling to 10 GB/s write and 20 GB/s read in the three largest regions, and a twentieth of that elsewhere.

Latency is documented as typically under a second from put to get, which is the number to hold against Kafka's low-millisecond claims.

The guarantee is the most important difference from Kafka. Kinesis is at-least-once, not exactly-once: AWS documents that producer and consumer retries both cause redelivery and tells you to expect processing a record more than once.

Its advice follows from that, embedding a primary key in the record so duplicates can be removed downstream, with consumer position handled by checkpointing in the client library.

Retention defaults to 24 hours and extends to 365 days, so replay is possible but costs extra beyond the first day and again beyond seven.

There is no processing engine in the stream itself. That role went to Amazon Managed Service for Apache Flink, which supports Java, Scala, Python and SQL and handles provisioning, failover, scaling and checkpoints.

The current direction is direct delivery into the lakehouse, streaming tables in Amazon S3 Tables and Iceberg tables, with AWS claiming up to 50% lower end-to-end ingestion cost, and a cheaper On-Demand Advantage capacity mode.

Note there is no Kafka protocol compatibility at all; AWS sells Amazon MSK separately for that.

Key features and capabilities

The same headings are used for every streaming & stream processing entry, so two tools can be read side by side.

What it does

• A sharded, ordered, append-only stream, with records grouped into shards by partition key
• Two capacity modes, provisioned where you size shards and on-demand where AWS scales them
• Multiple applications can consume the same stream concurrently and independently
• Mode can be switched twice within 24 hours

Delivery and ordering guarantees

• At-least-once, not exactly-once, and AWS says applications must expect to process a record more than once
• Redelivery comes from both producer retries and consumer retries
• Duplicate writes get unique sequence numbers, so AWS advises a primary key in the record for deduplication
• Consumer position is checkpointed by the client library, restarting from the last checkpoint

Scale and retention

• Provisioned, 1 MB/s or 1,000 records a second in, 2 MB/s or 2,000 out, per shard
• Default quota 20,000 shards per account in the three largest regions, 1,000 or 6,000 elsewhere
• On-demand scales to 10 GB/s write and 20 GB/s read in the largest regions, 200 MB/s and 400 MB/s elsewhere
• Retention from 24 hours to 365 days, with payloads up to 10 MiB and 500 records per batched request

Processing and transformation

• No processing engine in the stream itself
• Amazon Managed Service for Apache Flink is the successor to Kinesis Data Analytics
• It supports Java, Scala, Python and SQL, with provisioning, failover, scaling and checkpoints handled
• Lightweight processing through AWS Lambda, and delivery into S3 and Iceberg tables

Security and governance

• Server-side encryption at rest using AWS KMS keys, including customer-generated keys, with FIPS endpoints
• TLS 1.2 required and 1.3 recommended in transit, with forward-secret cipher suites
• Private networking through interface VPC endpoints
• IAM access control, resource policies and dry-run permission testing
• No schema registry ships with the service; AWS Glue Schema Registry is separate

Connectors and ecosystem

• Client, producer and connector libraries, plus AWS SDKs across languages
• Native integration with Lambda, S3 including S3 Tables, Iceberg tables and managed Flink
• An EMR connector for Hadoop and Spark workloads
• No Kafka protocol compatibility; AWS sells Amazon MSK for that

Where it runs and what it costs

• Managed AWS service only, with no self-hosting and no Kubernetes operator
• Available across AWS regions, though on-demand ceilings differ between the largest three and the rest
• Streaming tables and S3 delivery require on-demand capacity mode
• Fully published pricing, unusual in this category

Pricing

Usage-basedProvisioned at $0.015 a shard-hour

Provisioned mode costs $0.015 a shard-hour plus $0.014 per million 25 KB payload units, with extended retention at $0.020 a shard-hour to seven days, then $0.023 a GB a month with $0.021 a GB retrieval, and enhanced fan-out at $0.015 a consumer-shard-hour. On-Demand Standard is $0.040 a stream-hour with $0.08 a GB in and $0.040 out. The newer On-Demand Advantage drops that to $0.032 and $0.016 a GB with no stream-hour charge and free fan-out, above a 25 MB/s account minimum. There is no free tier.

Vendor pricing page →

Demos and videos

• Amazon Web Services channel Channel or playlist · YouTube
• Kinesis Data Streams resources and sessions Webinar or talk · aws.amazon.com

About Amazon Web Services

Kinesis Data Streams is a first-party AWS service with no separate ownership or funding. Amazon announced Kinesis in November 2013 and made it generally available that December, so it is one of the older managed streaming services still in wide use. It is proprietary, consumed under the AWS customer agreement, with no open-source edition, no self-hosting and no downloadable binary; only the client libraries are open source. AWS publishes a footprint of 124 availability zones across 39 regions. It sits in a family with Firehose, Video Streams and Managed Service for Apache Flink.

aws.amazon.com