ingestr v1.1.46 - PostgreSQL CDC Barrier Checkpoints

ingestr v1.1.46 was published on 1 September 2026. The change that matters first is a PostgreSQL CDC barrier checkpoint fix. Aurora can return an LSN from pg_logical_emit_message() that cannot be compared with replication slot positions, and persisting that value made the next CDC run resume from an invalid location, take a full snapshot, and time out.

The full release notes and downloads are on the GitHub release page.

Pull request 1134 (commit a8a0413) changes how checkpoint LSNs are chosen at a CDC barrier. Both replicators now capture the matched logical decoding MESSAGE LSN from the replication stream. That decoded LSN is what gets written for CDC state, keepalives, and final slot confirmation.

The LSN returned by SQL stays available for diagnostics. The code warns when the two values diverge by a large amount. That split exists because Amazon Aurora initializes its own LSN sequence, independent of the source that seeded logical replication. Comparing the SQL result with slot positions is then invalid.

Operators running PostgreSQL CDC against Aurora should treat this as a correctness fix, not a new flag. If a pipeline was looping into full snapshots after a barrier, the resume position was the likely cause. Standard PostgreSQL uses the same path. Pull request 1134 adds regressions for Aurora, standard PostgreSQL, advanced position, and barrier matching.

DuckLake destinations now support table partitioning and sorting (pull request 1164). The operator surface is --partition-by and --cluster-by. Layout is applied and kept across schema evolution and table swaps, and redundant sort metadata is skipped.

Commit 282cde4 releases the staging layout after a committed swap. Cross schema swaps dropped the staging table and forgot its schema but left an entry in d.layouts. A long lived DuckLakeDestination then accumulated one stale layout per ingest. The hook is onStagingConsumed, fired after a successful commit, on both the rename path and the cross schema swap path.

Layout comment cleanup and lint fixes in the same pull request are internal. They do not change the CLI.

Commit b4fdff1 (pull request 1165) adds an iceberg+r2 destination shorthand for Cloudflare R2 Data Catalog. R2 Data Catalog is a managed Iceberg REST catalog. The shorthand is an alias over the existing REST catalog and vended credentials path, not a new catalog engine.

The URI shape is:

iceberg+r2://<account_id>/<bucket>?token=<r2_api_token>

The scheme assembles the catalog URI and warehouse name, forces HTTPS, and passes the R2 API token as token. R2 vends S3 storage credentials through the catalog, so AWS keys are not required. The scheme is registered in the destination registry, Schemes(), and the server connector list.

The same commit stops treating a REST warehouse identifier with no URI scheme as a local filesystem path. ensureLocalTableDirs no longer creates or chmods stray directories in the working directory for remote catalogs such as R2 or Glue. Catalogs that are not REST still honor relative warehouse and table_location paths.

Two BigQuery destination fixes land in this tag.

Commit 09e3323 (pull request 1109) stops retrying dataset location mismatches. A transient Not found: Dataset right after dataset creation remains retryable. The permanent BigQuery variant that includes was not found in location now fails immediately. The check lives in isRetryableLoadJobError and isDatasetNotFoundInLocation in pkg/destination/bigquery/load_job.go. Typed, wrapped, and untyped error forms are covered. The staging dataset name in the tests is _bruin_staging.

Commit 14e0826 (pull request 1163) caps copy job start retries. startCopyJobWithRetry retried an ambiguous copy job start (retryable error or NotFound) in an unbounded loop. A persistent 404, for example a missing dataset, could retry forever. The cap mirrors the load job start path at loadJobStartMaxAttempts. On exhaustion the code adopts an already submitted copy job of the same stable ID, or defers CDC reconciliation and fails.

Jobs that used to spin on a location mismatch will now fail on the first permanent error. That is the right outcome for a dataset that does not exist in the job location. The cost is that a misconfigured region no longer hides behind retries.

Agent skill work in this release is internal. A shared skills/ folder now feeds both .agents and .claude, and the add-source skill was tightened to cover interval options and OAuth. That does not change ingest runtime.

The notes do not list a migration step or a breaking API. Two behaviors change without a flag: Aurora CDC checkpoints now persist the stream LSN, and BigQuery location mismatch errors stop retrying. DuckLake --partition-by and --cluster-by are new flags. Inspect stored CDC positions on Aurora pipelines that already entered a full snapshot loop before this tag.