dlt is a Python library for building ELT pipelines. The last week on the devel branch was almost all documentation, but the 1.32 release highlights contain breaking changes that matter to anyone running scd2 or refresh in production.
TL;DR ¶
Four commits landed in the window: 14 files changed, 1461 insertions, 44 deletions. No core runtime code changed. One commit adds the 1.32 release highlights, two rework verified source docs, and one tightens a test. Read the breaking changes first.
Breaking changes in the 1.32 highlights ¶
The release highlights commit adds 1.32.md. It lists three breaking changes.
- Naive scd2 timestamps are no longer read as UTC. A naive
boundary_timestamporactive_record_timestampis sent as written, so the database session timezone interprets it. On a server whose session timezone is not UTC, a naiveactive_record_timestampmay stop matching existing active records. refresh="drop_resources",pipeline.drop()anddlt pipeline dropnow drop every table of the dropped resources that exists in the destination, not only tables that saw data.- Arrow extract shares the resolved schema contract per table. Two resources with different
schema_contractwriting the same table now share the contract resolved for the first.
The second item has the largest blast radius. The notes say that when several pipelines share one dataset, a refresh can drop same named tables created by the others. If you share datasets between pipelines, audit table names before upgrading.
The fix for the first item is simple: pass timezone aware values.
Replace that keeps state, and placeholders ¶
The dict form of the write disposition now keeps resource state by default. An incremental cursor survives a full table replace, so each run loads only new records. The "replace" shorthand still resets state.
@dlt.resource(write_disposition={"disposition": "replace"})
def clients(updated_at=dlt.sources.incremental("updated_at")):
...
To get the old reset behavior from the dict form, set exclude_resource_state to False. This is a quiet default difference between two spellings of the same disposition. Pick one style per codebase and stay with it.
Merge also gains value placeholders. source_filter and destination_scope accept {load_id}, {curr_date}, date parts and your own extra_placeholders. The values live in the load package, not the schema, so a partition that changes every load does not create a new schema version. That is the useful part for daily partition pruning.
Operator tooling: trace metrics and dbt runner ¶
dlt pipeline <name> trace metrics prints files, rows and sizes of the last run. Extract is split by resource and table, normalize by table, and load shows files, sizes, job time and failed jobs. -v breaks it down per file and --steps picks steps.
dlt pipeline my_pipeline trace metrics -v --steps load
Normalize also balances files across workers by total size instead of count. That should help when a few large files used to leave one worker running alone.
The dbt runner accepts a load_id passed as a package variable, and an interval forwarded to microbatch models as --event-time-start and --event-time-end. Each DBTNodeResult now carries rows_affected, failures and adapter_response.
Verified sources docs rewrite ¶
Two commits, one on November 3 and one on November 6, add pages for ServiceNow and SharePoint and update Jira, Salesforce and MongoDB. Most of the diff is prose cleanup, but the signatures now document real options.
The MongoDB page now lists parallel, limit, filter_, projection, chunk_size, a backend of python or pyarrow, on_inferred_type_mismatch and partition_size_bytes (default 16 MiB). It also states that all collections load except system.* when you pass no names.
The Jira page has the most operator relevant change. The jira_search source now loads into a jira_search table, not issues, so it can share a dataset with jira. It also documents chunk_size, bulk_fetch_size, included_columns and excluded_columns. Rename the resource to change the table:
search = jira_search(jql_queries=["project = DEV AND issuetype = Epic"])
pipeline.run(search.jira_search.with_name("epics"))
These are docs commits, so check the source code before relying on any detail. The pages describe behavior, they do not change it.
A small test fix ¶
A test change in test_ai_command.py now checks that every toolkit resolves init as a transient dependency, instead of requiring it as a direct one. It also asserts that known toolkits exist in the workbench. This is minor.
What to watch ¶
- Test scd2 pipelines against a non UTC database session before moving to 1.32.
- Check which pipelines share a dataset before running refresh or drop.
- Review
replaceresources that use incremental cursors. The dict form and the shorthand now behave differently.