dbt-core is the transformation framework many ELT pipelines run on, and its main branch now looks like a Rust workspace with a lot of crates. This week saw 59 commits across 940 files. Most of it is internal, but a handful of fixes change behavior operators will notice.
The repository is at dbt-labs/dbt-core. The numbers are large because of generated and workspace churn, so this review sticks to the commits that touch runtime behavior.
Adapter and connection fixes ¶
The most operationally relevant change is closing a connection left mid transaction by a failed statement. A failed statement that leaves a connection inside an open transaction is a classic source of confusing follow on errors. Later statements on a pooled connection fail for reasons unrelated to their own SQL. Cleaning up at the adapter layer removes that class of noise.
Adapter code is also where most churn lands. adapter_impl.rs was touched by 7 commits and get_relation.rs by 5. Smaller warehouse specific fixes:
- BigQuery: external_volume and file_format are now optional for LRC catalogs.
- BigQuery: a table not found error handling fix.
- Spark: Microsoft Fabric Lakehouse support arrives as a platform flavor of the Spark adapter.
The commit subjects do not say how many users hit the BigQuery paths. Treat them as targeted fixes, not broad rewrites. If you run on those warehouses, read the diffs before upgrading and rerun your own failure cases, because adapter behavior on error paths is rarely covered by generic test suites.
Retry and run startup behavior ¶
The retry fix scopes parse failure entries in run_results to the selection, so retry respects --select. Before this, a parse failure could widen what a retry considered in scope. If you wrap dbt retry in orchestration for a subset of a large project, this is the one to read.
Two startup changes help with speed and diagnosis:
- Missing schemas are created concurrently before the run. Projects with many target schemas should see a shorter pre run phase. The commit does not cite numbers.
- The effective thread count is logged at startup. When a profile, a flag and an environment variable disagree, the log now tells you which value won.
dbt run --select tag:nightly --threads 8
# startup log now states the effective thread count
Unit tests and parsing ¶
Unit test work has three pieces. Unit test Jinja now renders on dbt-runtime workers, touching unit_test.rs. Separately, NUMBER precision and scale are kept in expected types, which matters when expected fixtures are compared against decimal columns.
On the parser side, a perf change skips linter only raw source spans in the parser path. That means less work during parse for data the parser never uses. An experiment to serialize Jinja node fields on demand is labeled as such, so do not expect it to be a settled design.
Crate boundaries and cleanup ¶
The latest commit is a refactor in which host crates reach the SQL frontend and DataFusion only through an interface, marked Phase 4. In dbt-datafusion-errors, the dependency on dbt-frontend-common is dropped from its Cargo.toml, and the frontend converts DataFusion errors itself. The practical effect is a smaller dependency graph and faster incremental builds for contributors.
Related moves: cancellation was extracted into a dbt-cancel crate, and the dbt-agent experiment was removed. Smaller operator facing items include dbt init offering DuckDB with a written profile and package hub TLS now trusting REQUESTS_CA_BUNDLE and CURL_CA_BUNDLE. The second helps anyone behind a corporate TLS proxy.
What to watch ¶
- Phase 4 implies more phases. Expect further crate splits and dependency changes in
Cargo.toml. - If you rely on
dbt retrywith--select, test the new scoping against a failed parse in staging. - Check startup logs for the thread count line after upgrading. It is a cheap way to catch a profile that silently sets fewer threads than you thought.