Apache Hudi

Overview

Apache Hudi is the table format built around incremental processing. Where the other two started from "give a data lake ACID tables", Hudi started from "stop reprocessing the whole table", and its design follows from that.

It describes itself as an open data lakehouse platform, and it is more than a file layout: it ships writers, indexes, table services and query types rather than only a specification.

The first choice is table type. Copy-on-write rewrites base files when records change, giving slower writes and faster reads. Merge-on-read appends changes to log files and merges them at query time, which reverses that trade-off.

Query types are unusually rich. Beyond snapshot reads there is incremental query for the latest state since a point in time, incremental CDC with before and after images, read-optimised queries on merge-on-read tables, and time travel.

The timeline is the source of truth: a log of every action on the table, stored as an LSM tree.

Record-level indexing is the real differentiator. Hudi treats indexing as integral, with a record-level index mapping keys to file locations, so an upsert can find the affected file rather than scanning.

Concurrency offers a genuinely different option, non-blocking concurrency control, where locks are needed only for timeline metadata writes, alongside conventional optimistic control with external locks.

Read the guarantees carefully: Hudi states upserts will never produce duplicates, but with multiple writers under optimistic control, plain inserts might, even with deduplication enabled.

Schema evolution covers adding nullable columns, dropping, renaming and type promotion, but column reordering is not supported and partition evolution is not published, which is the clearest gap against Iceberg.

Maintenance is where operational effort lands: compaction merges log files into base files, and clustering rewrites the layout, each inline or asynchronous with several strategies to choose from.

Key features and capabilities

The same headings are used for every open table formats entry, so two tools can be read side by side.

How the format works

• Base files plus row-based log files, organised into file groups and file slices
• Copy-on-write writes only base files; merge-on-read adds log files merged at query time
• A timeline of instants covering commits, compaction, clustering, cleaning and rollback
• An active timeline plus archived LSM history stored as Parquet

Transactions and concurrency

• Optimistic concurrency with external locks, plus non-blocking concurrency control needing locks only for metadata
• Multi-version concurrency control gives snapshot isolation between writers, readers and table services
• Upserts are guaranteed never to produce duplicates
• Plain inserts under multi-writer optimistic control might duplicate, even with deduplication enabled

Schema and partition evolution

• Add a nullable root-level column, drop columns, rename columns and promote types
• Type demotion is not allowed
• Column reordering is not supported on either table type
• Partition evolution is not published

Performance features

• Compaction merges log files into base files, inline or asynchronous, triggered by commits or elapsed time
• Clustering rewrites file layout with size-based, sort, consistent-bucket or commit-based strategies
• Writer-side indexes include bloom, simple, global variants, HBase and bucket indexes
• Metadata-table indexes cover bloom filters, column and partition statistics, secondary and expression indexes

Engine support

• Queried by Hive, Spark SQL, Flink, Trino and PrestoDB once the bundle is installed
• Also listed, Athena, BigQuery, Apache Doris, StarRocks and Impala
• Processing through Spark, Flink, Databricks and EMR
• Ingest integrations for Kafka, Pulsar, Debezium and Flink CDC

Catalogues and interoperability

• Hive Metastore sync makes tables visible to Presto and Trino as well as Hive
• Also documented, AWS Glue Data Catalog, BigQuery and DataHub
• Apache XTable and Apache Polaris, both incubating, provide interoperability
• Storage on HDFS, S3, Google Cloud Storage, Azure Blob, MinIO and several other clouds

Project governance

• An Apache top-level project since May 2020, run by its project management committee
• Apache 2.0 licensed, with no single company in control
• Community on Apache mailing lists, Slack and GitHub discussions
• No formal cadence published; 1.2.0 landed in May 2026 with 0.14 still maintained

Pricing

Open sourceFree, Apache 2.0

There is no licence fee. The format, the writers, the indexes and the table services are all Apache 2.0 and free to use, and the Apache Software Foundation sells nothing. Cost comes from the engine and storage you run it on, and in Hudi's case also from the compute spent on compaction and clustering, which are ongoing jobs rather than one-off setup. Budget for those table services when comparing it with a format that leaves maintenance to the query engine.

Vendor pricing page →

Demos and videos

• Apache Hudi channel Channel or playlist · YouTube
• Apache Hudi community call with Amazon engineering Webinar or talk · YouTube

About Apache Software Foundation

Hudi entered Apache incubation in January 2019 and graduated to a top-level project in May 2020, with committers spread across five or more organisations at graduation. It is governed by a project management committee under the Apache Software Foundation and licensed under Apache 2.0, with no company in control, and community activity runs on Apache mailing lists plus Slack and GitHub discussions. The project originated at Uber, though that is not stated on the Apache pages themselves. Version 1.2.0 arrived in May 2026, with the 0.14 line still receiving maintenance releases.

Founded 2019 · Wilmington, Delaware · hudi.apache.org