Apache Spark master moved 131 commits in the past week: 1117 files changed, 58013 insertions, 25851 deletions across the repository. The dominant theme is correctness in the SQL layer, mostly DSv2 CHECK constraints and generated columns, plus a planner fix that people running storage partitioned joins should read carefully. Java 17 support is gone.
DSv2 CHECK constraints get a long audit ¶
Dongjoon Hyun landed a run of fixes around ALTER TABLE ADD CONSTRAINT on DSv2 tables, and the churn concentrates in CheckAnalysis.scala, ResolveTableSpec.scala, and the CheckConstraintSuite, which grew nine commits of tests. The cases fixed:
- A constraint that always holds crashed with an internal error instead of validating cleanly (b7557f1b5).
- A condition subscripting a lambda variable made every subsequent write fail (7ffdda03a), and a followup repaired an Analyzer compilation regression from the first fix (b8168834c).
- CREATE and REPLACE TABLE accepted a CHECK constraint calling a generator function (d4b33cad3).
- ALTER TABLE targeting a view altered the underlying table (5eaa53ad7), and adding a constraint to a path threw a NoSuchElementException.
The first two are the ones worth an upgrade: a lambda variable inside a constraint expression previously broke writes at the whole table level, which is the kind of failure you discover in an ingest job, not in a DDL review. If your pipeline adds constraints programmatically, rerun that path against this week.
Storage partitioned join no longer rechecks a config at execution ¶
The most subtle correctness fix is f4eef9869 for SPARK-60260. Storage partitioned join relies on a keyed layout carried on the scan node, but reportedKeyedPartitioning rechecked spark.sql.sources.v2.bucketing.enabled every time the node was evaluated. Under AQE, PlanAdaptiveDynamicPruningFilters copies the planned scan at stage creation. If the config flipped between planning and execution, the copy dropped the keyed layout and the sort merge join aligned partitions against the wrong layout. Wrong rows, no error.
The fix moves the decision into DataSourceV2Strategy.scala: the config is read once at planning and the result is frozen into BatchScanExec. Related work in the same area skipped the pass through GroupPartitionsExec during alignment (1a524e44b) and made skewed join optimization per join instead of all or nothing (224287258). The larger lesson is a familiar one: do not read mutable session state twice for one plan decision.
Java 17 support is dropped ¶
21c599680 removes Java 17 support from the build, docs, and infra. Java 21 is the new baseline for benchmarks (2a90c7d21), and a documentation commit (152e072ad) now states that NodeManagers running the Spark Shuffle Service must be on a supported Java version. That last point is the operational trap: your driver and executors can be on Java 21 while a few older YARN nodes still run the shuffle service on a removed JDK, and failures will show up as shuffle fetch errors, not as an obvious version mismatch.
Check the migration guide, which took 19 commits of edits this week. Derby also moved to 10.17.1.0 (c17c63e5a), so any test harness pinning the embedded store should expect the bump.
Wrong result fixes worth scanning ¶
A handful of silent wrong result bugs landed, the worst category. ConstantFolding produced wrong output for size() of a map with duplicate keys (a475424a8) and dropped ANSI cast errors when ANSI mode was off (2c03f1681). Partition value handling also had two silent cases: missing datetime validation when ANSI mode is off (116f05ca3) and out of range TINYINT and SMALLINT partition values reading as the wrong numbers (7d0a478c1). If you consume partition columns as narrow integer types, the second fix can change query results on existing data.
Python and Connect clients ¶
The Python side saw the minimum grpcio version for Spark Connect raised to 1.84.1 (78246e106), a fix restoring offloaded ML model summaries during fetch (523596fdc), and empty Arrow buffers written uncompressed so PyArrow can read LZ4 batches (57f7ad61f). On the input side, text based file sources grew a decompression read option (73ca206be). Pin your pyspark environments before pulling the next build; grpcio floors have a habit of fighting with other pins in a locked environment. Separately, the streaming layer gained a DSv2 connector API for external lookup joins (957a72564).
What to watch ¶
- The CHECK constraint sequence is still settling; expect followups near
Analyzerpattern matching as more sessions exercise it. - Java 17 removal will propagate into downstream images. Audit NodeManager JDKs before your next shuffle service roll.
- Watch
SQLConf.scala, which took 12 commits this week; the ConcurrentHashMap registration change (0917659a4) may shift latency profiles for sessions that set many configs.