ingestr v1.1.51 - Postgres CDC Row Loss and MSSQL Stream

ingestr v1.1.51 was published on 10 September 2026. The patch contains two ingestion changes. The one that matters on a live pipeline is the PostgreSQL CDC fix: a schema rebuild keeps buffered rows for tables that stayed valid.

The full release notes and downloads are on the GitHub release page. The delta from v1.1.50 is the full changelog.

PostgreSQL CDC can omit rows after a schema change, and the missing table changes from run to run. Pull request 1182 from karakanb is the fix.

A primary key update after a column drop raises SchemaChangedError while buffered changes are still being materialized. The old flush stopped on the failing table and discarded remaining batches for every other table, including batches whose LSNs were already marked processed. Replay treated those LSNs as done and filtered the rows out. Go map iteration order chose the table, so one workload could lose different tables on different runs. Closing both writers still left the rows missing, before this fix landed.

The new flush writes unaffected tables before a rebuild that covers more than one table. Every schema failure is collected, and each failed table gets a replacement snapshot. Failed tables keep their buffered changes and low water marks through that flush. Their WAL stays unacknowledged until the replacement rows are delivered. Unaffected tables are written out in that earlier flush, and their batches acknowledge WAL only there.

TestPostgresCDC_StressComplexWorkload in tests/integration/postgres_cdc_stress_test.go is the reproduction. It waits for a durable WAL acknowledgement, shuts the writers down, and compares PostgreSQL with DuckDB for rows, schema, state, and tombstones. The check uses the WAL insert position because synchronous commit is off. Four full size runs passed, each with 180,000 operations, zero dropped operations, and zero errors. The rebuild path is in batch_accumulator.go.

Destination writers and dependencies match v1.1.50. Rows the old flush already discarded stay missing. A destination that copied through a schema change on the previous build needs a full refresh of the affected tables. A later revert leaves that hole in place. The pull request calls the risk medium. The changed path is PostgreSQL CDC rebuilds that span several tables.

mssql+ct:// gains --stream in pull request 1187 from turtleDev. The source used to run once and return. Change Tracking is poll based. Each read asks SQL Server for the current version and loads changes up to that version. MSSQLChangeTrackingSource implements StreamingSource, and the tail of changeTrackingTable.Read in pkg/source/mssql/change_tracking.go polls until the context is cancelled.

Repeated updates to one row inside a single read collapse to the current row. A shorter interval narrows that window, so a resident stream keeps more intermediate net changes than a scheduled one shot job. The payload is still the current row. Per change history stays on the CDC source.

poll_interval is a URI parameter, default 1s. ingestr strips it before building the driver connection string, so SQL Server never sees it as a connection property. Omit --stream and the source still exits after one read.

The resume cursor is the destination column _cdc_lsn. On a quiet table an idle read restamps a row so the stored version stays inside CHANGE_RETENTION. A stamp on every poll would rewrite a destination row every cycle. ctHeartbeatGate spaces the stamp out. The interval is one quarter of database retention, capped at five minutes and floored at five seconds. ensureDatabaseChangeTracking already reads sys.change_tracking_databases, and the heartbeat takes the retention period and the cleanup flag from that row. A one shot run keeps a zero interval, so a scheduled job matches the behavior from before --stream.

With the database version unchanged, the heartbeat stays idle. The cursor is still inside retention. A stamp at or below the stored version would drag MAX(_cdc_lsn) backwards and rewind the resume point.

ReplicationLag reports the version gap as source and destination positions, plus a flag for whether the stream is caught up. A Change Tracking version has no byte offset and no commit time. Lag follows the durable position. StreamCommitter emits a commit token at the end of each window, and the position advances only after the streaming executor confirms the flush. Buffered rows, or a failed destination write, stay behind. The starting version is already durable. It is the resume cursor, or the snapshot rows just emitted.

A one shot run still drains every batch. A stream cancelled during a flush used to block the producer on a send and leak the goroutine plus its connection. Batch sends select on ctx.Done() and release the batch when the context ends.

A mssql+ct:// stream reads one table through StreamingSource. Several Change Tracking tables means one stream per table.