Isolated DuckDB Connections and Doc Tooling in dlt

Recent updates to the dlt repository address runtime connection leaks during in memory dataset extraction and refine automated testing tooling. The changes eliminate state bleeding across concurrent DuckDB readers, standardize dependencies for educational execution scripts, and enforce uniform formatting rules across project documentation.

The _read_csv_duckdb function in the filesystem reader module previously relied on top level DuckDB module calls to parse incoming CSV files. Invoking duckdb.from_csv_auto directly uses the internal default connection of the Python process. When pipelines initialized multiple reader generators concurrently or processed batches in an interleaved stream, those generators shared a single active execution context.

DuckDB executes queries using vectors bounded by DUCKDB_VECTOR_SIZE, default to 2048 records. When extracting small files under 2048 rows in a single batch, interleaved readers might appear to operate without error because fetchmany pulls out of an already materialized data chunk. However, once dataset volume exceeds 2048 rows, DuckDB reenters the execution pipeline to compute the next vector. If an interleaved reader reentered the module level interface to read another chunk in the interim, it overwritten the pending result state of the active generator. As a result, pulling a subsequent batch from the original iterator past the 2048 row boundary caused result invalidation and query failures.

In the filesystem DuckDB fix, Jacob Curlin isolated each reader stream by instantiating an explicit DuckDB connection manager within _read_csv_duckdb. By wrapping file iteration inside with duckdb.connect() as conn:, every generator retains its own connection handle throughout execution:

with duckdb.connect() as conn:
    for item in items:
        with item.open() as f:
            file_data = conn.from_csv_auto(f, **duckdb_kwargs)

            for batch in helper(file_data, chunk_size):
                if add_filename:
                    for record in batch:
                        record["filename"] = item["file_name"]
                yield batch

The pull request adds test_read_csv_duckdb_interleaved_readers to verify that concurrent reader instances extract complete datasets across 4096 rows without invalidating each other. Pipelines using filesystem sources with PyArrow or DuckDB execution backends gain full safety during parallel or interleaved extraction tasks.

The educational training suite received structural updates to align sample scripts with current backend APIs. In the course update commit, Alena Astrakhantseva updated sample API extraction endpoints across course lessons from repository stargazers to repository issue comments.

The revised exercises call repos/dlt-hub/dlt/issues/comments with explicit pagination settings using params={"per_page": 100}. This change ensures that beginner pipelines process multi page REST payloads rather than smaller static endpoints.

In addition, the commit standardizes inline script metadata across lesson scripts. All tutorial files now explicitly list pyarrow under PEP 723 script inline dependencies:

# /// script
# dependencies = [
#     "dlt[duckdb]",
#     "numpy",
#     "pandas",
#     "pyarrow",
#     "sqlalchemy",
# ]
# ///

Installation commands inside interactive notebooks were updated from generic pip install dlt directives to pip install "dlt[duckdb]". The preprocessor script in docs/tools/preprocess_to_molab.py was also modified to ensure notebook transformation routines automatically inject pyarrow alongside numpy, pandas, and sqlalchemy into target execution environments.

To maintain documentation quality as the repository grows, Thierry Jean added automated Markdown linting rules in the mdsmith formatting commit. The repository now uses mdsmith to check formatting consistency across documentation source files.

Configuration settings in docs/.mdsmith.yml enable rule categories for accessibility, code block syntax, headings, link validity, table layouts, and whitespace integrity. Specific overrides disable duplicate heading checks on complex credential documentation while setting same-file-anchor: false to avoid conflicts between mdsmith and Docusaurus slug resolution on snake case anchor links.

The linting suite connects directly to automated developer workflows through docs/.pre-commit-config.yaml and docs/Makefile. The commit applies rule updates across more than 260 files, including core destination guides such as docs/website/docs/dlt-ecosystem/destinations/duckdb.md. Automated lint checks prevent broken internal anchors and improper code block syntax before changes land on the main branch.

  • Audit custom Python extractors that invoke DuckDB directly to ensure they instantiate dedicated connection handles rather than relying on top level module state.
  • Verify that standalone Python scripts running with uv or pipx specify pyarrow in inline metadata when working with DuckDB tabular streams.
  • Run mdsmith locally or through make target workflows before submitting documentation updates for new ecosystem destinations.