Deequ

Overview

Deequ is a library for unit tests on data, built on Apache Spark and designed for datasets large enough that checking them any other way is impractical. AWS describes it as built for billions of rows.

It is Scala and JVM-first, which makes it the outlier here. A Python wrapper, PyDeequ, exists for callers who need it, and DQDL, a declarative rule language, is the second authoring surface.

Checks are a fluent Scala DSL chained off a check object with a severity level, asserting size, completeness, uniqueness, set membership and non-negativity, then run through a verification suite.

Three features set it apart from everything else in this category. Constraint suggestion profiles your data and proposes constraints, inverting the authoring problem entirely.

A metrics repository persists computed metrics so history is queryable rather than recomputed, and anomaly detection then runs over that history. No other library here keeps its own metric history.

Incremental computation is the architectural point. Metrics are computed with each new version of a dataset rather than by rescanning everything, using approximate sketches for expensive statistics such as quantiles.

It is Apache 2.0 and free, and it quietly underpins a paid service: AWS Glue Data Quality is built on Deequ, which AWS presents as giving portability without lock-in.

Maintenance is genuinely healthy, contrary to its reputation. Version 2.1.0 landed in September 2026 after six releases that year, with human commits throughout and multiple contributors.

Plan for one breaking change: from 2.1.0 Java 11 is required, where earlier 2.0 releases ran on Java 8, and the release also replaced deprecated Spark aggregators.

The honest gap is everything after detection. There are no dashboards, no alerting channels, no Slack or email integration, and no orchestrator or catalogue integrations; Deequ emits metrics and results for you to route.

Key features and capabilities

The same headings are used for every testing & validation entry, so two tools can be read side by side.

How checks are written

• A fluent Scala DSL chained off a check with an error or warning level, run through a verification suite
• Assertions include size, completeness, uniqueness, set membership and non-negativity
• DQDL is a declarative alternative, the same language AWS Glue Data Quality exposes
• PyDeequ provides a Python interface for callers who do not want Scala
• Constraint suggestion profiles the data and proposes constraints, inverting the authoring problem

What it can check

• Completeness, uniqueness and primary key checks
• Size and shape through row count, column count, column existence, length and schema match
• Ranges and set membership over numeric, string and date expressions
• Statistical checks including mean, standard deviation and correlation
• Freshness and cross-table row count matching, plus custom SQL, composable with and and or

Where checks run

• Spark only, distributed, with metrics computed through optimised Spark aggregation queries
• Published compatibility, Spark 3.1 to 3.5 for Deequ 2.x, with per-version artefacts
• Java 11 required from 2.1.0; earlier 2.0 releases run on Java 8
• Distributed by construction, reading from sources including Amazon S3
• Distributed through Maven Central with published Maven and sbt snippets

Results and alerting

• Verification produces a report of the derived metrics and constraint outcomes
• A metrics repository persists and lets you query metrics over time
• Anomaly detection runs over that persisted history
• No dashboards, no alerting channels and no Slack, email or Teams integration are published
• Severity is set per check, but no exit-code contract is published

Scale and performance

• Built for datasets in the billions of rows, living in a distributed filesystem or warehouse
• AWS describes the framework as managing petabyte-scale datasets
• Incremental computation, recomputing metrics with each new dataset version rather than rescanning
• Approximate sketch-based computation for expensive statistics such as quantiles
• Row-level sampling controls are not published

Integrations

• AWS Glue Data Quality is the headline integration and is built on Deequ
• Anything that runs Spark can host it, including EMR, Glue and Databricks
• PyDeequ for Python callers
• dbt, Airflow, Dagster, catalogues and continuous integration recipes are not published
• No REST API, since it is a JVM library

Where it runs and what it costs

• A library only, added as a JAR to a Spark job
• No self-hosted server, managed cloud, regions or pricing unit
• The managed path is to buy AWS Glue Data Quality instead
• Six releases in 2026, so maintenance is active rather than dormant

Pricing

Open sourceFree, Apache 2.0

Free and open source under Apache 2.0, distributed through Maven Central with no paid tier, no hosted offering, no licence keys and no trial. The commercial relative is AWS Glue Data Quality, which AWS states is built on Deequ and is billed pay as you go, though the unit price is not on the feature page. That relationship is the practical trade-off to weigh: self-host Deequ on your own Spark and pay only for compute, or buy the managed service and pay AWS for the convenience.

Vendor pricing page →

Demos and videos

• AWS Glue Data Quality overview, built on Deequ Product demo · YouTube
• Test data quality at scale with Deequ Tutorial · aws.amazon.com

About Amazon Web Services

Deequ is stewarded by Amazon Web Services through its awslabs GitHub organisation, and originated in Amazon research: the 2019 AWS blog introducing it, written by six Amazon authors, says it was already being used internally to verify the quality of many large production datasets. It is Apache 2.0 with around eighty contributors, and no foundation, charter or steering committee is published. Its commercial descendant is AWS Glue Data Quality, which AWS states uses Deequ to manage petabyte-scale datasets. Maintenance is now partly automated, with an AI-assisted review bot visible in recent commits.

Founded 2018 · github.com