Python Polars 1.43.0 - Hive and Streaming Pipeline Changes

Python Polars py-1.43.0, published on July 21, 2026, makes Hive partition awareness the main operator change, with group operations and joins able to organize work around partition keys. The release also tightens streaming sinks, object storage access, and several data interchange paths. GitHub marks it as a regular release, not a prerelease.

The full release notes and downloads are on the GitHub release page.

Polars can now pre partition group operations on Hive keys. It can also pre partition left, right, and semi joins when the inputs use Hive partitioned data. For inner joins, the optimizer can rewrite the operation as a union of partition filtered joins. These changes matter when table layout already matches the group or join keys, since the engine has more information for separating the work before processing rows.

The new partition_hive optimization flag gives pipeline owners another planner control to track in regression tests. It is paired with predicate canonicalization, tighter removal of redundant comparisons, and propagation of filter constraints. Polars also avoids removing a cache when predicates could not be pushed to every input. That last fix is relevant to shared plans where one branch blocks pushdown but the cached result is still useful elsewhere.

Join planning receives two smaller controls. Redundant equality join keys can be optimized, and callers can set a build side preference. Neither change removes the need to inspect plans on skewed inputs. They do provide more control for workloads where partition shape and join memory are already measured.

The streaming IPC sink removes an excess memory copy, while single file IO sinks split morsels with awareness of chunk boundaries. A new POLARS_OOC_DISK_BUDGET_MB setting adds a concrete disk budget for out of core execution. Together, these changes give operators useful controls around memory pressure and spill capacity without changing the pipeline API.

Several fixes target failures rather than throughput. The release fixes a sink deadlock regression, an EntityTooSmall failure when writing IPC data to S3, and incorrect JSON output for null values inside Array columns. It also honors aws_checksum_algorithm, classifies S3 region request failures as IO errors, and restores correct sourcing of HF_TOKEN.

DNS handling changes as well. The object store DNS cache is now global and can serve stale records. There is also an environment variable for logging slow DNS lookups, although the release notes do not name that variable. This is operationally useful for distinguishing resolver delay from scan or sink work, but stale records remain a tradeoff that should be checked against endpoint rotation policy.

Native scan_iceberg now resolves metadata in parallel, understands the Iceberg field property initial-default, and no longer has a hard block for Iceberg V3 tables. Removal of the block is not a blanket compatibility claim. Pipelines using V3 features should still validate schema projection, defaults, and partition behavior against their own tables.

For Arrow integration, scan_arrow_c_stream adds a direct scan entry point. The release also fixes the offset used when exporting sliced struct arrays through Arrow FFI and hardens the IPC memory map API. Those correctness fixes deserve coverage in tests that exchange sliced or mapped data across process boundaries.

Database ingestion gets an Arrow fast path in read_database for python-oracledb. A separate ADBC fix stops requiring CREATE privileges for appends unless Polars confirms that the destination table does not exist. This narrows an unnecessary privilege requirement for append only jobs.

This release starts several deprecation paths. Numeric casts to categorical values, cat.get_categories(), cat.to_local(), and LazyFrame.profile() are deprecated. Casts from scalar types into lists and bitwise operations between integers and booleans are also deprecated. Calls to list.to_struct() or arr.to_struct() should now pass field names.

Configuration handling also renames missing_utf8_is_empty_string to empty_string_is_null. Audit configuration wrappers and warning filters before upgrading, especially if CI treats deprecation warnings as errors. The release also makes invalid temporal arithmetic and invalid Object implosion raise errors instead of producing ambiguous or invalid results.