Delta Lake

Overview

Delta Lake is a table format that turns a directory of Parquet files into a table with transactions. Its own description is a storage framework for building a format-agnostic lakehouse, and it is the format Databricks created and still ships by default.

The mechanism is a transaction log. Data lives in Parquet files and every change is recorded in a _delta_log directory beside the data, as newline-delimited JSON commits named by a zero-padded version number.

Reading a table means replaying that log. Checkpoints periodically consolidate it into a single file holding the complete replay up to that version, so readers do not have to walk thousands of commits.

That linear log is the clearest difference from Iceberg, which builds a tree of manifests per snapshot, and from Hudi, which keeps base and log files per file group with a record-level index.

Guarantees are ACID with serialisable isolation, the strongest level, plus idempotent writes so a retried job does not double-append, and time travel by version or timestamp with 30 days of retention by default.

Schema changes are handled well. New columns can be added automatically during a write, and column mapping lets you rename or drop a column without rewriting data.

Partitioning is where it diverges from Iceberg. There is no partition evolution; instead liquid clustering, available since 3.1.0, replaces both partitioning and ZORDER and lets clustering columns be redefined without rewriting existing data.

Housekeeping is explicit: OPTIMIZE bin-packs small files and is idempotent, Z-ordering colocates related data and is not, and deletion vectors mark rows deleted without rewriting the files they sit in.

Interoperability is handled by UniForm, which lets Delta tables be read by Iceberg and Hudi clients, so choosing Delta no longer forecloses the others.

Key features and capabilities

The same headings are used for every open table formats entry, so two tools can be read side by side.

How the format works

• Parquet data files plus a JSON transaction log in a _delta_log directory at the table root
• Table state is the replay of log actions in commit order, consolidated periodically into checkpoints
• Actions cover metadata, file add and remove, change data, transactions, protocol and domain metadata
• Reader and writer table features are negotiated through a protocol action

Transactions and concurrency

• ACID with serialisable isolation, the strongest level the project offers
• Idempotent writes, so a retried job does not append the same data twice
• Time travel by version or timestamp, retained 30 days by default
• A full audit history of changes to the table

Schema and partition evolution

• New columns can be added automatically during a write when schema evolution is enabled
• Column mapping allows rename and drop without rewriting data files
• Liquid clustering redefines clustering columns without rewriting existing data, up to four columns
• There is no partition evolution; liquid clustering replaces partitioning and ZORDER

Performance features

• OPTIMIZE bin-packs small files and is idempotent; auto compaction and optimised write are also available
• Z-ordering colocates related data but is explicitly not idempotent
• Data skipping from per-column minimum and maximum statistics
• Deletion vectors mark rows deleted without rewriting the data files

Engine support

• Engines include Spark, Flink, Trino, PrestoDB, Hive, Druid, ClickHouse, StarRocks and RisingWave
• Managed services include Athena, EMR, Glue, BigQuery, Fabric, Power BI, Redshift, Snowflake and Starburst
• APIs in Scala, Java, Rust and Python, including a Delta Rust implementation with no JVM
• Delta Kernel underpins newer connectors, including an experimental Flink connector

Catalogues and interoperability

• Catalog-managed tables through Unity Catalog, using a Delta REST catalog API
• Hive Metastore and AWS Glue interoperate through the listed integrations
• Delta Sharing is a separate open protocol for sharing Delta data across organisations
• UniForm exposes Delta tables to Iceberg and Hudi readers

Project governance

• Hosted by the Linux Foundation as a series LLC, with a charter adopted in September 2019
• Technical oversight sits with a Technical Steering Committee of maintainers
• Apache 2.0 licensed, with documented contributor, committer and maintainer roles
• No formal release cadence is published; 4.4.0 landed in August 2026, supporting Spark 4.2.0

Pricing

Open sourceFree, Apache 2.0

There is no licence fee. The format and its reference implementations are Apache 2.0 and free to use, and the project charges nothing for support or certification. Cost arises only from whichever platform runs it, whether that is Databricks compute priced in DBUs, a cloud provider's managed Spark, or your own infrastructure. If you are comparing total cost, compare the engines rather than the format.

Vendor pricing page →

Demos and videos

• Delta Lake channel Channel or playlist · YouTube
• Intro to Delta Lake Tutorial · YouTube

About Delta Lake Project

Delta Lake was created at Databricks and donated to the Linux Foundation in 2019, where it is hosted as a series LLC under LF Projects with a charter adopted in September 2019. The charter names Apache 2.0 as the project licence and gives technical oversight to a Technical Steering Committee drawn from the project's maintainers. The site states it is an independent open-source project not controlled by any single company, and reports more than 190 developers from over 70 organisations. Databricks remains the principal originating vendor and ships Delta in its platform.

Founded 2019 · Linux Foundation, United States · delta.io