DuckDB Writes Parquet Encoding Stats And Guards Aggregates

DuckDB is an embedded analytical database used as a local query engine and as a Parquet writer in ETL pipelines. The DuckDB main branch took 213 commits in this window. The operator visible thread is a Parquet footer field that lake readers already expect, plus two aggregate correctness guards.

DuckDB already listed encodings on each column chunk. It did not write encoding_stats, the per page type counts in the footer. Downstream readers use that list to prove a chunk is fully dictionary encoded without decoding pages.

A commit that populates ColumnMetaData.encoding_stats closes that gap. The message cites a cold first touch of a 142 million row dictionary string column in Microsoft Fabric Direct Lake: 10.5 to 13.9 seconds on a file written by DuckDB, against about 0.5 seconds for the same data written by parquet-cpp or parquet-rs. Filling in only this field brought the DuckDB file down to 496.5 milliseconds across about 25 public runs.

Counting happens in the existing SetParquetStatistics page loop. Dictionary page entries are written first, matching physical page order and the order parquet-cpp and parquet-rs already produce. The field is also exposed on parquet_metadata() as encoding_stats LIST(STRUCT(page_type, encoding, count)), NULL when absent. A follow on test across writer column types checks one invariant over a table with 15 types: every chunk carries the field, dictionary chunks list DICTIONARY_PAGE first, and chunks without a dictionary open with DATA_PAGE.

SELECT path_in_schema,
       (encoding_stats[1]).page_type AS first_page_type
FROM parquet_metadata('file.parquet');

Lake jobs that rewrite Parquet with DuckDB and then hit Direct Lake or similar readers should pick up a nightly build, or wait for the next release. Files written before this change stay slow until they are rewritten.

Statistics propagation can precompute count, min, and max from partition stats at plan time. The partial path mixed those constants with an execution time scan that skipped partitions by index in the row group list.

That list is not stable. Concurrent appends and checkpoints can reorder it. A skipped partition then gets scanned again and its rows are counted twice.

The guard returns early whenever need_to_scan is true. Only the full precomputation path (no extra scan) stays enabled. Two EXPLAIN tests are skipped with a note about row group reordering, and the commit says they should come back once this skip is reverted. Result correctness for those aggregates still holds. The plan just does more work. The old partial branch is left in the file and is now dead.

This is a temporary correctness patch, not a new default that operators flip. Nightly builds will scan more row groups for filtered min/max/count until the skip is lifted.

The radix partitioned hash table used for grouped aggregates had a similar ordering bug. Local tables get abandoned when many threads are sinking so data can spill or resize. Other threads learn about that through any_abandoned.

Two commits landed a day apart. The first sets any_abandoned while calling ht.Abandon() in adaptivity and repartition paths that previously abandoned without flipping the flag. The second moves ht.Abandon() to after the flag write in the sink path.

The race is small. Calling Abandon() first can leave a window where the table is empty and the flag is still false. There is no new setting. High concurrency GROUP BY on nightly builds is the place to retest, especially queries that spill.

preserve_identifier_case used to be a boolean. It is now one enum with three modes: on (preserve as written, the default), lowercase, and uppercase. The change rejects a second capitalize_identifier boolean that would have done nothing unless the first flag was already false.

Aliases still work: true/false, preserve/lower/upper. Legacy BOOLEAN spellings such as 1, t, and yes still map onto preserve versus lowercase. The reported default for callers that never set the option is now on with input type VARCHAR, not true with BOOLEAN. Quoted identifiers stay exact in every mode. Folding to upper case no longer depends on the process locale.

SET preserve_identifier_case = 'uppercase';
CREATE TABLE MyTable(Id INTEGER);

Catalog dumps, codegen, and schema compare jobs that asserted true/false from duckdb_settings() need a one line update. Clients that only SET the old boolean keep working.

The skip of partial aggregate precomputation is meant to be temporary. EXPLAIN plans for filtered count/min/max will look heavier on nightly until the row group index problem is fixed for real.

The hottest engine files this window were the recursive CTE executor and its USING KEY UNION tests. That path is still moving (delta tracking, keyed preaggregation, state layout). Query text does not change. Jobs that use USING KEY recursive CTEs should rerun correctness tests on nightly.

Alpha Main.yml runs may let musl extension builds fail (linux_amd64_musl and linux_arm64_musl on workflow_dispatch alpha tags). A green alpha run is not musl coverage. FunctionSet is now immutable and held in shared pointers; extension authors who mutated overloads in place need ApplyToFunctions.