Apache Spark is a distributed engine for batch, streaming, and SQL workloads. Its master branch recorded 133 commits in the review window, changing 773 files with 32,314 insertions and 2,949 deletions. The useful data engineering story is tighter timestamp precision, a broader SQL surface, and fewer blind spots during reads and failures.
Nanosecond timestamps reach scan boundaries ¶
Nanosecond timestamp work landed as a sequence across input, storage, and expression paths. JDBC data sources gained support for nanosecond timestamp types. Matching changes added predicate pushdown for ORC and Parquet. That closes an important gap between representing a precise value and using it to prune data during a scan.
The SQL layer moved in the same direction. Spark now supports subtraction between nanosecond timestamp values, while current_timestamp and localtimestamp accept a precision argument. Pipelines can therefore keep one precision model across source reads, filters, generated timestamps, and interval calculations.
There are limits. The commits do not provide throughput numbers, and source systems may still truncate values before Spark sees them. Teams that depend on exact event ordering should test boundary values through JDBC, Parquet, and ORC rather than infer full path fidelity from the logical type alone.
Structured data functions move into SQL ¶
The newest change adds Unicode normalization through normalize. It supports NFC, NFD, NFKC, and NFKD, with NFC as the default. The implementation uses Spark’s bundled ICU4J data, so output does not depend on the Unicode tables supplied by a particular JVM vendor or version. That is useful before joins, deduplication, and key generation on text from mixed sources.
This is not limited to one language binding. The change reaches the Scala API, regular PySpark, and Spark Connect. The linked PySpark function implementation was one of the most active files in the window, with seven commits touching it.
Semi structured data received several additions. Spark gained the ANSI SQL JSON_VALUE scalar function, a variant_strip_nulls expression, and variant constructors for arrays and entries. The central FunctionRegistry was touched by eight commits. This is broad API work, not a single helper hidden behind one client.
Read paths gain narrower controls ¶
Archive ingestion now has an archivePathFilter option for selecting entries inside an archive. This matters when a landing object contains data, manifests, checksums, and unrelated payloads. Filtering entries at the reader boundary is cleaner than loading every entry and discarding rows later. The associated archive read test base also ranked among the frequently changed files, with four commits touching it.
Catalyst also gained a runtime filtering interface for DataSource V2 scans. The interface creates a route for filters discovered during execution to reach a scan. It does not prove that every connector can apply every filter. Connector support and metrics still decide whether the change reduces input bytes in a given pipeline.
Two documentation corrections are worth treating as contract clarification. One documents CSV reads when multiLine is disabled, and another corrects the claim that PERMISSIVE mode drops corrupt records. These are not advertised runtime changes. They are a reason to audit ingestion assumptions that may have been built from the old wording.
Failure paths expose more useful evidence ¶
Memory pressure should be easier to triage. An UNABLE_TO_ACQUIRE_MEMORY error now carries a bounded breakdown by consumer. Bounded output matters because a diagnostic must remain useful even when many consumers compete for memory. A separate fix stops UnifiedMemoryManager from reporting INVALID_DRIVER_MEMORY on an executor, removing a misleading subsystem label during incident response.
Kubernetes shuffle recovery now skips directories that cannot be listed. This lets recovery continue past one inaccessible path, but skipped data remains skipped. Operators should pair the resilience improvement with alerts for storage permissions and failed directory reads.
SQL correctness also received a focused repair: the single pass resolver no longer crashes on GROUPING SETS combined with HAVING or ORDER BY. The shared error condition catalog was the busiest listed file, touched by nine commits, which fits the wider effort to make new behavior fail with specific conditions.
What to watch ¶
These changes are on master. Confirm the release tag and client compatibility before changing production SQL.
Add precision fixtures around JDBC, ORC, and Parquet boundaries. Include filter plans and values that differ only below microsecond precision.
Measure archive and DataSource V2 filtering with connector metrics. An available filter path is useful only when fewer bytes reach the executor.