dlt 1.29.0 - Atomic Replace and Safer Staging Loads

dlt 1.29.0, published on July 13, 2026, adds atomic full replacement for ClickHouse through the staging-optimized strategy. BigQuery gets a separate opt in path for atomic replacement, while Airflow staging and Arrow schema handling receive fixes for production workloads.

The full release notes and downloads are on the GitHub release page.

ClickHouse can now run the staging-optimized replace strategy with EXCHANGE TABLES. dlt loads the replacement table in staging, then swaps it with the destination table as one atomic operation. Readers do not observe a partially loaded destination during a full replace.

The important detail is visibility. A large replacement can take time to prepare, but the final table change is instantaneous from the perspective of queries against the destination. This narrows the period where consumers could otherwise see an empty table or only part of the next snapshot.

BigQuery receives a related feature with different controls. The new enable_atomic_replace setting changes the truncate-and-insert strategy to issue one WRITE_TRUNCATE_DATA load job from GCS staging. The operation preserves table metadata and applies the full replacement transactionally.

This BigQuery path is opt in. Operators should check the destination configuration rather than assume every replace job uses it after upgrading. It also applies to the GCS staging flow described in the release notes, so pipelines using another loading path should not expect the same behavior from this flag.

Airflow runs using decompose="parallel" or decompose="parallel-isolated" no longer truncate shared staging data automatically. Before this fix, concurrent tasks could remove each other’s staging files. That failure mode is particularly awkward because each task can appear valid in isolation while another task removes the data it still needs.

dlt now disables staging truncation for those parallel modes. The change targets task interference, not Airflow scheduling itself. Existing DAG concurrency can remain in place while dlt avoids the unsafe cleanup step.

The ArrowToParquetWriter also handles safe type promotion across flush batches. A column that arrives as float32 in one batch and float64 in a later batch can now be promoted without crashing the writer. Previously, this could fail only after enough data had arrived to span multiple batches, which made small validation runs poor predictors of the production result.

This matters for sources where inferred types widen over time. The fix does not promise arbitrary schema coercion. It covers safe Arrow promotions and prevents batch boundaries from turning a compatible type change into a load failure.

dlt 1.29.0 adds an AWS Secret Manager configuration provider. Pipelines can resolve configuration and secret values directly from AWS instead of requiring application code to fetch them before dlt starts. The provider follows the model already used for Google Secret Manager.

The release also stops recreating the Google secrets client on every lookup. That is an internal lifecycle correction with an operator visible result: repeated configuration resolution no longer constructs a fresh client each time. Teams running short tasks or resolving several values per pipeline should see less avoidable client setup work.

Neither change alters the secret values themselves. Deployment roles and secret access policy still determine what a pipeline can read. The practical gain is a cleaner configuration path and more consistent provider behavior across AWS and Google environments.