Apache Spark had a busy week on master: 120 commits touched 457 files. The activity in apache/spark is not a release note dump; it is mostly data path cleanup around Python UDF transport, SQL correctness, security, and test determinism that affects teams running Spark as part of an ETL platform.
Python UDF Transport Got Real Fixes ¶
The most operator visible core change is in the Python runner path. The Unix Domain Socket follow up added support for Unix Domain Socket mode in pipelined Python UDF data transfer, then the blocking mode fix restored worker channel blocking mode after pipelined Python UDF execution. That matters because transport state leaks often surface as hangs, stale workers, or load only test failures.
The staged top changed files list shows PythonRunner.scala changed twice, and that is the right place to watch for downstream behavior. Another shorter temp base change keeps direct worker Unix Domain Socket paths from hitting AF_UNIX path length limits on macOS. That is a test fix on paper, but platform teams running mixed local and CI environments will recognize the failure mode.
Spark also removed unused ArrowStreamAggPandasUDFSerializer code and added an ASV microbenchmark for SQL_TRANSFORM_WITH_STATE_PANDAS_INIT_STATE_UDF. That keeps the stateful path visible before future changes land.
SQL Correctness And Benchmarks Moved Together ¶
The SQL side had performance work with explicit guardrails. The decimal division change improves decimal division by dividing at result scale, while the DecimalDivideBenchmark addition and result files such as DecimalDivideBenchmark results for JDK 21 give reviewers a place to compare behavior across JDKs. That is the pattern worth copying: change arithmetic, then publish enough benchmark surface to catch a regression.
Correctness fixes were not limited to decimals. Spark fixed an incorrect query result caused by inferAdditionalConstraints substituting attributes with collations that are not binary stable, rejected out of range variant_get array indices, and added nanosecond precision timestamp ordering tests. Together they reduce the class of data bugs where a query returns clean looking, wrong output.
There is also schema and connector churn. Incremental XML schema inference moved closer to CSV semantics in XmlInferSchema.scala, and JDBC value getters were refactored in JdbcUtils.scala. Those are internal changes, but data teams feel them with wide XML payloads, odd JDBC type mappings, or generated schemas that change between runs.
Security And Operator Defaults Tightened ¶
Several commits are directly relevant to locked down Spark clusters. The AuthV2 metadata authorization change enforces authorization on GetPrimaryKeys and GetCrossReference. That closes a quiet metadata gap: catalog metadata can expose table shape even when row access is controlled. In regulated data platforms, that distinction matters.
The transport authorization change makes StreamRequest authorization consistent with ChunkFetchRequest. Two request shapes that move data should not drift in access checks. Spark also switched temp file creation to Files.createTempFile with owner only permissions in the temp file permission change. That protects local spill and helper files from permissive defaults on shared hosts.
Kubernetes users get a smaller but practical cleanup. The entrypoint change avoids persisting Java options to disk in entrypoint.sh. That reduces the chance that command line settings with secrets or sensitive runtime detail remain in the container filesystem after process startup.
Test Flake Work Is Part Of The Data Path ¶
Much of this week was test reliability, which matters because Spark’s test suite is the contract that lets risky execution engine work land. MetricsFailureInjectionSuite was deflaked after the failure injection path sometimes did not trigger a recompute, leaving the metric at exactly 300. The staged diff notes an approximate 1/6 flake, more common on slower macOS arm64 runners. The file itself, MetricsFailureInjectionSuite.scala, was touched twice.
Other fixes target real resource hygiene. The ORC reader fix closes a reader in OrcSourceSuite to stop leaked file streams. RocksDB state store tests now isolate global memory manager state, Kafka source stress tests got deflaked, and SorterSuite gained retries around retryOnOOM. This reduces false signal in the same areas operators care about: shuffle recovery, streaming state, Kafka loss handling, and file stream cleanup.
The ML test fixes are in the same bucket. Gaussian mixture tests moved to better posed data, and single prediction checks now tolerate last ULP floating point differences on macOS. That tells you where Spark maintainers are spending time: making architecture and platform variance less likely to block core data changes.
What to watch ¶
The next risk area is Python worker transport. Unix Domain Socket mode plus pipelined Python UDF execution is valuable, but it sits close to task scheduling, worker lifecycle, and platform path limits. Watch future changes to PythonRunner.scala and any follow up commit around direct worker sockets.
SQL teams should track the benchmark files and error catalog churn. The top changed file is error conditions JSON, touched 13 times, and that usually means visible exceptions, error classes, or validation paths are moving. Stable errors matter for orchestration systems that classify failures.
For operators, the security commits are the ones to audit first when this lands in a release. Metadata auth, transport auth, owner only temp files, and less disk persistence of Java options are small changes with production value. They are also the kind of changes that can expose tests or local scripts that were relying on permissive defaults.