Polars

Overview

Polars is a dataframe library rather than a database, and it is the fastest common answer to a specific question: how do I transform a large dataset on one machine without reaching for Spark?

The engine is written in Rust, multi-threaded and columnar, and uses the Apache Arrow memory model, so data can move between Polars and other Arrow tools without copying.

There are two APIs and the choice matters. The lazy API defers work until you call collect, letting the planner apply predicate and projection pushdown; the eager API runs immediately and is really the lazy API underneath.

The project's own guidance is to prefer the lazy API unless you want intermediate results or are doing exploratory work, which is the opposite default from pandas.

Larger-than-memory data is handled by the streaming engine, enabled per collect call, which processes queries in batches and is claimed to outperform the in-memory engine.

One caveat to plan for: some operations are inherently non-streaming or not yet implemented that way, and Polars falls back to the in-memory engine for those, so a streaming query is not a memory guarantee.

Unlike most projects here it publishes its own benchmarks, using PDS-H derived from TPC-H on a 96-core AWS instance, where at scale factor 10 it beats DuckDB and at scale factor 100 DuckDB is ahead.

Read the disclaimer it publishes alongside those numbers: results from PDS are not comparable with published TPC-H benchmark results.

Interfaces are Python, Rust and JavaScript, with a DataFrame and expression API rather than SQL as the primary language, and GPU execution available through NVIDIA RAPIDS.

Scaling beyond one machine is the commercial path, Polars Cloud, which runs the same code on a single machine or a distributed cluster, with an on-premises option for air-gapped environments.

Key features and capabilities

The same headings are used for every query & processing engines entry, so two tools can be read side by side.

Execution model

• A single-node in-process library rather than a server or cluster
• Vectorised, columnar and multi-threaded, dividing work across available CPU cores
• Lazy and eager APIs, with a query plan you can inspect before running
• A streaming engine for batched, out-of-core execution

Languages and APIs

• Python, Rust and JavaScript
• A DataFrame and expression API rather than SQL as the primary interface
• In Rust, the lazy and streaming engines sit behind feature flags
• Arrow memory layout gives zero-copy interoperability with other Arrow tools

Data sources and formats

• Documented input and output for CSV, Excel, Parquet and JSON, including multiple files
• Hive-partitioned data, databases, cloud storage, BigQuery and Hugging Face
• Delta Lake and Iceberg are not listed in the input and output documentation
• IO plugins let third parties add their own sources

Performance and scale

• Multi-core parallelism with cache-coherent columnar algorithms
• The streaming engine handles datasets exceeding memory, falling back per operation where needed
• Published PDS-H results on a 96-core AWS instance, beating DuckDB at scale factor 10
• At scale factor 100 DuckDB is faster, and the project publishes the comparability disclaimer itself

What it suits

• Single-machine dataframe transformation and analytics, including data larger than memory
• Claimed gains of more than thirty times over pandas on tested workloads
• Eager API for exploratory and interactive work, lazy API for pipelines
• No published statement of workloads it is unsuitable for

Ecosystem

• Two extension points, expression plugins and IO plugins
• Community plugins for datetime handling, distance functions, bioinformatics and geospatial work
• GPU execution through NVIDIA RAPIDS, with NVIDIA holding copyright on parts of the code
• Community on Discord, with the code on GitHub under MIT

Where it runs and what it costs

• Installed as a library in Python, Rust or JavaScript, inside your own process
• Polars Cloud runs the same code remotely, on one machine or a distributed cluster
• AWS and Kubernetes supported now, with Google Cloud and Azure stated as later
• An on-premises enterprise option deploys entirely in your own infrastructure

Pricing

Open sourceFree, MIT licensed

The library is MIT-licensed and free, with no paid edition. The commercial product is Polars Cloud, published at $0.05 per vCPU hour, charged only for query execution with no minimum spend and no idle cost, on top of a free tier; cloud provider compute is additional. The on-premises enterprise tier includes 10,000 CPU hours a month at no cost, with the rest on enquiry. Trial length for Polars Cloud is not published on the pages read.

Vendor pricing page →

Demos and videos

• Polars channel Channel or playlist · YouTube
• The new streaming engine explained Webinar or talk · YouTube

About Polars

The project began in 2020 as open source, created by Ritchie Vink, who remains its lead maintainer, with a company established in 2023 to fund engineering and based in Amsterdam with a distributed team. It raised an €18m Series A in September 2025, led by Accel with BCV participating, on a stated plan to make the open-source engine fully streaming, build a distributed cloud engine behind one DataFrame API, and offer a managed data platform. The library is MIT-licensed. Its own pages name the legal entity inconsistently as Polars Inc. and Polars BV, which is worth noting on a procurement form.

Founded 2020 · Amsterdam, Netherlands · pola.rs