Python Polars 2.0.0 - Out of Core by Default and Streaming Engine Work

Python Polars 2.0.0, tagged py-2.0.0, was published on 2026 10 06. The headline change for anyone running large frames is that out of core execution is now on by default, with a memory threshold of 80 percent of available RAM and a default disk budget of 64 GB.

The full release notes and downloads are on the GitHub release page.

Two knobs changed. The stream now spills when memory reaches 80 percent of available RAM instead of waiting for a hard OOM, and the default disk budget for spilled data is 64 GB. There is also a new environment variable, POLARS_OOMKILL_THRESHOLD_MB, which gives operators an explicit ceiling to tune before the kernel OOM killer picks a victim for them.

For pipeline schedules that previously relied on the in memory engine and had generous node memory, this is a behavior change worth testing before rollout. Workloads that stayed under the old limits should be unaffected, but a job that ran near the edge may now start writing spill files. Watch disk pressure on your workers.

Most of the release is streaming engine work, and the pattern is consistent: reduce materialization, reorder work, and prefetch aggressively.

For scans, group record batch fetches on remote IPC streams, decode IPC and Parquet scans out of order when the caller does not observe ordering, use HTTP suffix range requests to read Parquet footers cheaply, and defer cached pread from prefetch to decode. For Iceberg, manifest files are now cached across scans and str.starts_with, is_nan, and != filters push down. scan_lance also gets filter pushdown.

For joins, there is a new runtime filter system: shared bloom filters between build threads, filtering of probe rows by the runtime join key range, and better build side selection via sampling for semi, anti, and outer joins. Hash table layouts for single key and multi key group-bys were reworked with block prefetching. Notably, a streaming out of core sort was added, and small cloud uploads now use a single PUT.

Also worth noting: BytecodeParser, which handles UDF execution, got several large optimizations, including reusing disassembly instead of repeating it per call.

A few behavior changes will surprise SQL heavy users:

  • SQL window functions now run over aggregated rows, and QUALIFY evaluates before the projection. Queries that relied on the old order need review.
  • Exact numeric literals in SQL are typed as Decimal, and % and DIV truncate. Float pipelines that depended on float literal semantics may shift behavior.
  • Parquet ENUM values are read as pl.String.
  • cut and qcut are deprecated.

The pipe_with_dtype expression is new and extends the plugin path, and deterministic expression plugins can now opt into common subexpression elimination if they want it.

The window function and QUALIFY ordering change is the one most likely to alter results, not just speed. If you run SQL through Polars in scheduled jobs, diff outputs against the 1.x series on a sample of your queries before upgrading. Decimal literal typing is the second one to check, especially for financial sums.