Apache Iceberg

Overview

Apache Iceberg is the open table format that has become the interoperability layer of the modern data stack. Its own description is a high-performance format for huge analytic tables, and it is now read or written by almost every warehouse and engine in this directory.

Metadata is a tree rather than a log. A table metadata file points to snapshots, each snapshot to a manifest list, each manifest list to manifest files, and those to the data files themselves.

Manifests are deliberately reused between snapshots to avoid rewriting slow-changing metadata, and a commit is an atomic swap of one table metadata file for another, using optimistic concurrency with retries.

Hidden partitioning is the feature people notice first. Iceberg produces partition values itself, so queries stop depending on the table's physical layout and you cannot write a query that silently scans everything.

Schema changes are metadata-only. Add, drop, rename, reorder and type widening all happen without rewriting data files, and sort orders can change too.

Partition evolution is the capability its rivals do not match. Change the partition spec and old data stays as it was written, with no eager rewrite, so a layout mistake is no longer permanent.

Format versions matter when choosing engines: v1 covers analytic tables, v2 added row-level deletes, v3 adds extended types, row lineage and binary deletion vectors, and v4 is in development and not yet adopted.

Maintenance is explicit and documented: rewrite data files by bin-pack or sort, cluster with Z-order or Hilbert curves, compact manifests, expire snapshots and delete orphan files.

The REST catalog specification is arguably its most strategic piece. One client works against any compliant server, which is why Snowflake, Databricks, Google, AWS and a dozen startups now speak the same catalogue API.

Non-JVM access is real, not theoretical: PyIceberg reads and writes tables from Python with no JVM at all.

Key features and capabilities

The same headings are used for every open table formats entry, so two tools can be read side by side.

How the format works

• Immutable data files in Parquet, Avro or ORC, with an Avro-based manifest layer above them
• A metadata tree of table metadata, snapshots, manifest lists and manifest files
• Manifest lists carry partition ranges so whole manifests can be pruned before reading
• A snapshot log records how the current snapshot changed over time

Transactions and concurrency

• Optimistic concurrency with retries, committing by an atomic metadata file swap
• Multi-table atomic commits are defined in the REST catalog API
• Time travel and rollback to any retained snapshot
• Row-level deletes from v2, with binary deletion vectors in v3

Schema and partition evolution

• Add, drop, rename, reorder and widen columns as metadata changes, with no data rewrite
• Partition evolution keeps data written under the old spec unchanged, with no eager rewrite
• Hidden partitioning transforms include identity, bucket, truncate, year, month, day, hour and void
• Sort order can evolve as well as the partition spec

Performance features

• Rewrite data files to combine small files, reducing metadata overhead and file-open cost
• Bin-pack or sort strategies, plus Z-order and Hilbert clustering in Spark procedures
• Manifest compaction alongside data compaction
• Expire snapshots, delete orphan files and rewrite position delete files as documented housekeeping

Engine support

• Project-maintained connectors for Spark 3.5, 4.0 and 4.1, and Flink 1.20 through 2.3
• A Kafka Connect sink is maintained in the repository
• Third-party engines include Trino, Presto, DuckDB, ClickHouse, Dremio, StarRocks, Athena, Redshift, BigQuery and Snowflake
• PyIceberg gives Python access with no JVM; Hive 2 and 3 connectors are deprecated

Catalogues and interoperability

• The REST catalog specification lets one client work against any compliant catalogue server
• Authentication through basic, OAuth2, AWS SigV4 or Google Cloud credentials
• Also supported, Hive Metastore, Hadoop tables, JDBC, Nessie, AWS Glue, DynamoDB and Dell ECS
• PyIceberg adds BigQuery and SQL catalogues over PostgreSQL or SQLite

Project governance

• An Apache top-level project since May 2020, governed by its project management committee
• Apache 2.0 licensed, with no single company in control
• Engine support is formally staged as beta, maintained, deprecated or end-of-life
• No fixed release cadence is published; 1.11.0 arrived in May 2026

Pricing

Open sourceFree, Apache 2.0

There is no licence fee. The format, the specification, the REST catalog API and the reference libraries are all Apache 2.0 and free to use, and the Apache Software Foundation sells nothing. Costs come entirely from the engines, catalogues and storage you choose to run against it, and the project keeps a vendor page listing the commercial products that support it. If you are comparing total cost, compare those rather than the format.

Vendor pricing page →

Demos and videos

• Apache Iceberg channel Channel or playlist · YouTube
• v3 and beyond, Iceberg's ongoing evolution Webinar or talk · YouTube

About Apache Software Foundation

Iceberg entered Apache incubation in November 2018 and graduated to a top-level project in May 2020, with initial mentors including Ryan Blue, Julien Le Dem and Owen O'Malley. It is governed by a project management committee under the Apache Software Foundation, a nonprofit run largely by volunteers, and licensed under Apache 2.0. No company controls it, which is a genuine differentiator in a category where the alternatives originated inside vendors. The project also formalises engine support as beta, maintained, deprecated or end-of-life, so you can see what is actually being kept working.

Founded 2018 · Wilmington, Delaware · iceberg.apache.org