Pandera

Overview

Pandera validates dataframes, and it does it by treating a schema as a type rather than as a test suite. That is what separates it from everything else in this category.

Schemas are written in Python two ways, an object-style dataframe schema or a class-based model with pydantic-style syntax, where each column is an annotated attribute with field constraints.

The typing integration is the real draw. A decorator validates function signatures typed as a dataframe of a given schema, and mypy integration is documented, so data contracts live in the same place as your type hints.

Data synthesis is its second distinctive feature. A schema can generate example data through hypothesis, so the same definition that validates your data also produces test fixtures for it.

Backend reach is unusually wide: define a schema once and apply it to pandas, polars, pyspark, dask, modin, ibis, pyarrow, geopandas or xarray, with narwhals as a unified opt-in layer.

Validation always runs in process, in the dataframe engine. Nothing is pushed to a warehouse as SQL, which is the clearest functional difference from Soda and Great Expectations.

Failure handling is well judged. By default it raises on the first problem; with lazy validation it collects everything and hands back failure cases as a dataframe you can inspect programmatically.

For production there are two environment switches worth knowing: validation depth can drop to schema-only in hot paths, and validation can be disabled wholesale.

Read one published performance hazard if you use Spark: each count command triggers a fresh Spark action, so a dataframe can be reprocessed repeatedly unless you enable the documented caching flags.

It is MIT licensed, the most permissive here, and free with no commercial edition. Note it is still pre-1.0 at 0.33.1 with over 450 open issues, though releases and commits are frequent.

Key features and capabilities

The same headings are used for every testing & validation entry, so two tools can be read side by side.

How checks are written

• Two APIs, an object-style dataframe schema and a class-based model with pydantic-style syntax
• Field constraints as keyword arguments, covering greater than, less than, membership and coercion
• Custom logic attaches as decorated column or whole-dataframe check methods
• Schema-wide behaviour goes in a nested config class, covering strictness, coercion and missing columns
• Schemas are ordinary Python classes, so they inherit and compose, and serialise to YAML or JSON

What it can check

• Comparison and membership built-ins, plus a full string family for matching, containment and length
• Type checking and coercion are first-class, since the dtype is the schema
• Uniqueness, including joint uniqueness across columns, and strictness rejecting undeclared columns
• Statistical and property-based checks through hypothesis objects with significance levels
• Freshness and cross-table referential integrity are not published, and SQL checks do not apply

Where checks run

• In-process only; nothing is pushed to a warehouse as SQL
• Backends include pandas, polars, pyspark, dask, modin, ibis, pyarrow, geopandas and xarray
• PySpark validation is native Spark rather than a conversion, using filters and counts
• Distributed execution through Dask partitions, Modin on Ray or Dask, and Fugue
• A new command-line interface validates files outside Python, covering CSV, Parquet, JSON, Feather and NetCDF

Results and alerting

• Fails fast by default, raising on the first problem found
• Lazy validation collects everything and raises once, carrying the failure cases and the data
• Failure cases come back as a dataframe with schema context, column, check, failing value and index
• PySpark collects errors on an attribute instead of raising, prioritising completing the load
• No dashboards, alerting integrations or result store are published

Scale and performance

• Validation depth is switchable between schema only, data only, or both, for hot paths
• Validation can be disabled wholesale through an environment variable
• Checks are vectorised by default rather than element-wise, for speed
• A published Spark hazard, each count triggering a new action, mitigated by documented caching flags
• Row-sampling arguments are not published, and the relevant API reference page is missing

Integrations

• Published integrations with FastAPI, Frictionless, Hypothesis, mypy and Pydantic
• Listed under Union.ai's orchestrator integrations alongside Flyte
• Type checking plus the decorator make it a natural citizen of unit tests and continuous integration
• dbt, Airflow, Dagster and data catalogues are not published as integrations
• No REST API, since it runs in process

Where it runs and what it costs

• A library, installed with pip, with extras for the command-line interface and data synthesis
• No self-hosted server, managed cloud, regions or pricing unit
• Synthetic data generation from a schema, in Python or from the command line
• Still pre-1.0 at 0.33.1, with over 450 open issues, though releases are frequent

Pricing

Open sourceFree, MIT licensed

Free and open source under the MIT licence, the most permissive in this category. There are no tiers, no free-tier limits, no trial, no quote-only plan and no licence keys. Union.ai sells a separate production AI runtime that integrates with Pandera, but Pandera itself is not gated by it and no paid edition exists. One point of confusion worth knowing: the Union.ai page footer links to a community licence, which governs Union.ai's own product rather than Pandera.

Vendor pricing page →

Demos and videos

• Intro to data validation with Pandera Tutorial · YouTube
• Introducing the Ibis backend for Pandera Webinar or talk · YouTube

About Union.ai

Pandera is a Union.ai open-source project, created and led by Niels Bantilan, who works at Union.ai, with the repository in the unionai-oss organisation. It is MIT licensed with around 215 contributors and an unusually distributed commit base for its size, and carries a pyOpenSci peer-review credential, the only formal quality signal published. Union.ai's paid product is a production AI runtime that integrates with Pandera and Flyte; there is no paid Pandera tier, no cloud edition and no enterprise version. Union.ai's own founding details are not published on the Pandera pages.

Founded 2018 · pandera.readthedocs.io