DuckDB Weekly: Checkpoint Index Bind Fixes and Optimizer Correctness Work


DuckDB is an embedded OLAP database, and the main branch of duckdb/duckdb absorbed 503 commits in the last week. Most of the operator visible work clusters around one theme: making the WAL checkpoint path safer when unbound indexes hold buffered operations. Around it sit a batch of optimizer correctness fixes and the usual CSV scanner hardening.

The most interesting change is a rework of SingleFileStorageManager::BindIndexesForCheckpoint, part of the pre checkpoint bind refinement in 7641a42cd0. Before this commit, any failed bind of an index could block the whole checkpoint, even when that index had no buffered WAL replays to lose. The new code tracks the failing table explicitly, rolls back the bind transaction before deciding, and only fails the checkpoint when buffered replays would actually be discarded.

The split matters for operators. On failure, an explicit CHECKPOINT still throws, but the error message now names the table and preserves the original exception type. An automatic checkpoint triggered by a commit logs a warning and keeps the WAL instead of aborting the write. The commit also moved OnCheckpointStart earlier in CreateCheckpoint, so storage extensions see the checkpoint begin before the WAL size check, and added a DebugCheckpointSleepMs setting to widen race windows in tests.

One FIXME in the diff is worth tracking: every commit that triggers an automatic checkpoint retries the bind until it succeeds. On a database with a permanently unbindable index, that could turn into repeated warning log lines.

Several commits protect older artifacts from newer logic. c48a147d5b keeps the type dependencies of a deserialized CREATE TABLE, which prevents a table loaded from storage from losing the catalog entries its column types depend on. 60a3522db5 keeps REPEATABLE samples written before v2.0.0 repeatable across the storage rewrite, and b3716e0dea from late October preserves the ASOF join predicate of plans serialized before v2.0.0.

The pattern is consistent: the v2.0.0 storage format transition is still shipping, and the team is closing the gap where a pre v2.0.0 database file or prepared plan would silently change semantics after reload. If you run DuckDB as a persistent catalog for ETL outputs, these commits are the reason old files keep producing identical results. The backward compatibility test runner also got maintenance, with d15b805d58 fixing version strings shorter than seven characters.

The single most edited file in the window was src/optimizer/duplicate_eliminated_domain_optimizer.cpp, touched by 18 commits. This is the domain optimizer work led by Laurens Kuiper and others: preserve fixed domains through grouped inputs in 0ebead4ee8, separate visible join domains from source provenance in f57a545dbc, cover derived join domains in 641f9db426, and preserve collation when proving domain key coverage in 5bb5ad7a26. These are correctness fixes, not speed features. A wrong domain proof lets the optimizer drop filters that actually matter, which shows up as quietly wrong query results.

Window function handling also got two small guards. 73f403bf38 keeps order sensitive aggregate inputs unchanged, and 65a7f2593e keeps the sign of a zero in a FLOAT or DOUBLE partition column of a top N window filter. Signed zero is rarely a correctness problem, but it is exactly the kind of value that breaks downstream comparison tests.

The CSV scanner keeps getting boundary fixes. 23980f85cd ensures a quoted value that ends at the end of the file is finished only once, and f828816e75 stops the scanner from overshooting a chunk when it finishes a boundary scan with an error row. Both files under src/include/duckdb/execution/operator/csv_scanner/ sit in the top changed list, so this is an active rewrite of the chunk boundary logic rather than isolated patches.

The shell side had 12 commits against tools/shell/shell.cpp plus 13 against its statement completeness tests, including a malloc check on multi line history input. Mostly test infrastructure, but the shell is the interface most people use first, so OOM shell tests failing in CI matter.

  • The automatic checkpoint bind retry noted in the FIXME in 7641a42cd0. If a bind fails permanently, watch for repeated warnings on every commit.
  • The domain optimizer churn. Expect more commits in duplicated_eliminated_domain_optimizer.cpp as the join domain proof work settles, and new tests under test/optimizer/ including the equality cast test file with 14 touches.
  • Parser leniency moving forward: b4e4f11e56 accepts a reserved keyword as a qualified table function name. Useful, but track it if your tooling counts on parse errors.