Polars 2.0 makes the streaming engine the default and spills to disk at about 80% of RAM
Serguey Asael Shinder1 min readPolars 2.0 shipped on 6 October, in a post by Ritchie Vink. The project says it did not set out to make a big feature release, but the version bump carries several defaults that change how existing code behaves.
Streaming by default. Calling collect on a LazyFrame now uses the streaming engine. The reason this needed a major version is row order: streaming does not guarantee it for some operations, including join, group_by and unpivot. Code that depends on observable order there has to opt in with maintain_order=True.
Out-of-core on by default. Polars now starts spilling to disk at about 80% of RAM, with a default disk budget of 64 GB, for operations that support it today (sort, window functions, many expressions). Joins and group-by are on the roadmap.
A Map type. Arrow's MapType is now read as a native Polars Map dtype with dictionary-like expressions, instead of List(Struct(key, value)).
Stricter types. The project says it wants errors to surface before a pipeline runs, and that collect_schema() lets humans and agents check a query's types without materialising data.
The benchmarks. Polars ran SQL derived from TPC-H and TPC-DS against DuckDB 1.5.6, a DuckDB 2.0 alpha and DataFusion 54.0.0 on two AWS machines (16 and 192 vCPUs), best of five hot runs per query. It reports default Polars fastest on all but one benchmark, says it has a constant overhead at 192 threads that hurts small queries, and notes that DataFusion timed out or ran out of memory on some queries, which were excluded for all engines. The benchmark code is published at github.com/pola-rs/polars-2.0-benchmark for anyone who wants to rerun it.