Amazon S3

Overview

Amazon S3 is the object store that defined the category, and its API is now the de facto standard that everything else implements. For most data platforms it is not a component so much as the floor the rest is built on.

The published guarantees are the reason it is trusted with everything: eleven nines of durability, 99.99% availability by default, and objects stored across at least three availability zones.

Consistency was the old complaint and no longer is. S3 delivers strong read-after-write consistency automatically, with no performance or availability penalty for it.

The current direction is the interesting part. S3 Tables is purpose-built storage for structured data in Apache Iceberg format, with table buckets as a distinct bucket type and AWS performing table maintenance automatically.

Those tables are reachable from any engine that speaks the Iceberg REST catalog specification, as well as through the AWS Glue Data Catalog, so this is the cloud's clearest bet on Iceberg as the default table format.

S3 Vectors adds native vector storage for semantic search, which puts embedding storage in the same place as the data it was derived from.

Performance is documented rather than vague: at least 3,500 writes or 5,500 reads per second per partitioned prefix, no limit on prefixes, up to 100 Gb/s on a single instance, and roughly 100 to 200 ms latency on small objects.

For low latency there is S3 Express One Zone, a directory-bucket class with single-digit millisecond latency, at roughly five times the storage price.

Encryption is now on by default. Since January 2023 all new uploads are encrypted automatically, with KMS, dual-layer, customer-provided and client-side options available.

One deprecation to note: S3 Select is no longer available to new customers, so in-place querying now means Athena or an engine over S3 Tables rather than a storage feature.

Key features and capabilities

The same headings are used for every object storage entry, so two tools can be read side by side.

Storage model

• Buckets with a flat namespace, default quota 10,000 general purpose buckets per account and no bucket size limit
• Three bucket types, general purpose, directory buckets for Express One Zone, and table buckets for Iceberg
• Objects from 0 bytes to 50 TB, with a 5 GB limit on a single upload and multipart advised above 100 MB
• Strong read-after-write consistency, with versioning a prerequisite for Object Lock

Performance and scale

• At least 3,500 writes or 5,500 reads a second per partitioned prefix, with no limit on prefixes
• Up to 100 Gb/s on a single instance, aggregating to multiple terabits a second
• Roughly 100 to 200 milliseconds latency on small objects, or single-digit milliseconds on Express One Zone
• Parallelism through multipart upload and byte-range fetches; caching means CloudFront or ElastiCache

Storage classes and lifecycle

• Nine classes, from Standard and Intelligent-Tiering through Standard-IA and One Zone-IA to three Glacier tiers
• Minimum durations, 30 days for infrequent access, 90 for Glacier, 180 for Deep Archive
• Intelligent-Tiering moves data automatically for $0.0025 per 1,000 objects a month, with no retrieval charge
• Retrieval fees range from $0.01 a GB on infrequent access to $0.03 on Glacier Instant Retrieval

Security and governance

• IAM identity and resource policies; table buckets are private and cannot be made public
• Encryption on by default since January 2023, with KMS, dual-layer, customer-provided and client-side options
• Object Lock gives write-once-read-many retention in governance or compliance mode, plus legal holds
• Assessed by Cohasset Associates for SEC 17a-4, CFTC and FINRA environments
• Visibility through CloudTrail, S3 Inventory and Storage Lens

Analytics features

• S3 Tables stores Iceberg tables as first-class resources, with maintenance performed automatically
• AWS-managed table buckets also hold system tables such as S3 Metadata inventory and journal tables
• S3 Vectors adds native vector storage for semantic search
• S3 Select is closed to new customers; Object Lambda remains available

Ecosystem

• The S3 REST API is the reference implementation that other object stores emulate
• S3 Tables are readable by any engine supporting the Iceberg REST catalog specification
• AWS analytics integrate through the Glue Data Catalog
• Open-source engines connect through the S3 Tables Iceberg REST endpoint or the S3 Tables catalog library

Where it runs and what it costs

• 39 AWS regions and 124 availability zones
• On-premises only through S3 on Outposts
• Standard storage $0.023 a GB a month in US East, down to $0.00099 on Deep Archive
• Requests charged per thousand, with Express One Zone adding per-GB upload and retrieval charges

Pricing

Usage-basedStandard at $0.023 a GB a month

Pay as you go with no minimum. In US East, Standard storage is $0.023 a GB a month for the first 50 TB, falling with volume, against $0.0125 for Standard-IA, $0.004 Glacier Instant Retrieval and $0.00099 Deep Archive; Express One Zone is $0.11. Requests on Standard cost $0.005 per thousand writes and $0.0004 per thousand reads, and deletes are free. S3 Tables storage is $0.0265 a GB a month plus compaction at $0.005 a GB and monitoring at $0.025 per 1,000 objects. New accounts get up to $200 in credits for six months.

Vendor pricing page →

Demos and videos

• Getting started with Amazon S3 Tables Tutorial · YouTube
• Understanding S3 Tables, architecture and performance Webinar or talk · YouTube

About Amazon Web Services

S3 is a first-party AWS service, announced on the AWS blog in March 2006, which makes it the oldest service in this category and the origin of the object-storage API everyone else copies. It is proprietary and closed source, with no community edition or self-hostable build, sold pay-as-you-go with no minimum charge under the AWS customer agreement. AWS publishes a footprint of 124 availability zones across 39 regions, with further regions announced in Saudi Arabia and Chile. The only on-premises option is S3 on Outposts.

aws.amazon.com