dlt 1.30.0 - Failed Load Recovery and Cross Destination Joins

dlt 1.30.0 was published on August 11, 2026. The main operator change is that failed load packages remain pending by default, preserving the evidence and jobs needed for inspection or recovery.

The full release notes and downloads are on the GitHub release page.

auto_abort_on_terminal_error now defaults to False. When raise_on_failed_jobs=True, a terminal job is retried, raises LoadClientJobTerminalRetry, and leaves its package pending. Previous releases raised LoadClientJobFailed and automatically aborted the package.

That is a better failure state for persistent pipeline storage because the package remains available for diagnosis. abort_packages records the abort instead of silently deleting the evidence. Each retry also writes its exception message under the package .exceptions directory.

Recovery controls are available through list_pending_retry_jobs_in_package(), fail_pending_job(), and retry_failed_job() on Pipeline. The CLI adds dlt pipeline <name> load-package <load-id> fail-job <job> for an explicit operator decision. Pipeline.drop_pending_packages() and drop-pending-packages are deprecated in favor of abort_packages and abort-packages.

Ephemeral storage still wipes the working directory and therefore still aborts automatically. Teams running short lived workers should not assume the new retention behavior applies there.

Datasets on separate destinations can now participate in one join. dlt attaches the foreign dataset to the query engine used by the dataset being read, then runs the join in that engine. Both eager and lazy materialization are supported for duckdb, motherduck, ducklake, lance, lancedb, and filesystem.

This removes an extra staging step for supported combinations, but it does not make every destination interchangeable. Pipeline owners should verify where materialization occurs and account for the data movement before using this path on large tables.

Traces now describe input and output lineage in relational form. Inputs are recorded per resource and data location on the extract step. dlt derives the corresponding outputs on the load step. The shape is close to OpenLineage and can be ingested by dlt itself, which gives operators a concrete mapping between an extraction source and its loaded tables.

The new retry_schema_update helper lets a pipeline place schema migration failures under a Tenacity policy. The release example uses five attempts, random exponential waiting, a maximum delay of 30 seconds, and reraise=True.

The useful boundary is narrow: only errors identified as schema update failures qualify. That avoids rerunning an entire load for an unrelated terminal condition. Error handling now also includes initialize_storage, so a failed CREATE SCHEMA can enter the same retry path.

Retries still need a bound. A permanent permission error will not improve with another attempt, while a transient catalog or warehouse failure may clear. The helper exposes that choice to pipeline code instead of imposing an unbounded loop.

The failed package change is breaking for automation that expects pending packages to disappear. Set auto_abort_on_terminal_error=True to retain the old behavior while runbooks and alerts are updated.

Filesystem layouts without {ext} now append the actual extension. A layout that previously produced an extension less object now writes a name such as mocked-table.jsonl; compressed jobs retain .jsonl.gz. Check downstream object matching, retention rules, and table discovery before upgrading an existing layout.

Table prefix generation also keeps its separator. With layout="{table_name}", the prefix becomes event. instead of event, preventing a replace of event from also matching events. warn_unsafe_layout_separators defaults to True and reports separators that can also appear inside table names.