ingestr v1.1.35 - CDC Table Subset Selection

ingestr v1.1.35 was published on 18 August 2026. The tag is not a prerelease. --source-table can now name a subset of CDC tables, and the run resolves that list against the discovered inventory.

The full release notes and downloads are on the GitHub release page.

--source-table accepts a CDC table list ¶

CDC table selection used to be binary. Pass --source-table and the run took exactly one table. Omit it and the run replicated every table in the source capture set. Naming several tables meant reshaping the source: hand managing a Postgres publication, or enabling CDC on only some SQL Server tables. For MySQL and Vitess that option did not exist.

The flag now accepts a comma separated list on CDC sources. One name keeps the old single table path byte for byte. Two or more names restrict capture to that set. Omit the flag and the run still takes all tables. The split happens only for CDC sources, and only when the value contains a comma. Custom queries such as query:SELECT a, b FROM t and SaaS specs such as campaigns:123,456 keep their commas and stay on the unchanged code path. The same list applies to PostgreSQL, MySQL, MariaDB, SQL Server, MongoDB, Vitess, and PlanetScale.

ingestr ingest --source-table "public.users,public.orders"

Selection lives on the source through a TableSelector capability, set after Connect and before discovery. A pipeline filter would not be enough. postgres_cdc.ReadAll lists tables itself. Under --stream it treats anything outside opts.KnownTables as newly observed, which would announce every excluded table as new. Postgres also lists again from the discover_interval timer and from rebuildStream. Storing the selection on the source covers those paths, and GetTables returns the narrowed set.

TableSelection generalizes mssql_cdc.selectTables. It keys on names rather than SourceTableInfo so each source can filter before per table schema queries. Postgres leaves public schema tables unqualified, so public.users and users resolve to the same table.

Commit c04103f changed how a requested name is matched. The first cut used case insensitive matching as a per name predicate. That silently selected two tables when a source held both users and Users. The collision is real for MongoDB collections, a quoted PostgreSQL identifier, MySQL on a case sensitive filesystem, or SQL Server under a case sensitive collation.

Includes, FilterNames, and FilterTables gave way to Resolve(inventory). An exact spelling wins over a case insensitive one. A request that matches several discovered names with none exact is ambiguous and fails, rather than guessing or ingesting all of them. Every source collects its name list and resolves once, before per table schema work.

A requested name that matches nothing fails the run. Ingesting nothing would hide a typo, and in streaming mode the job would poll forever. A table that exists but cannot be replicated reports the reason instead of “not found”. MySQL, MongoDB, Vitess, and PlanetScale previously skipped unmatched names silently. They now fail the same way. Filtering before the per table checks also unblocks Vitess and PlanetScale keyspaces that contain a keyless table, which used to fail the whole run.

The run summary shown before --yes lists the selected tables. Telemetry gains table_selection with values single, subset, or all. Table names are not sent.

A subset participates in the connector identity. It gets its own replication slot, run lease, and CDC state. Sharing those with the all tables run is unsafe in both directions. The slot would advance past changes to tables the run filters out, so a later wider run would resume from a position that had already skipped them. --full-refresh resets completion markers wholesale, which would discard snapshot state for every table outside the subset.

The selection is appended to the identity only when there is one. Existing connectors keep the ID they have and do not snapshot again on upgrade. That invariant is asserted by a test in pull request 1101, merged as 2f95b29.

The Postgres managed publication is deliberately left covering everything. ingestr_publication is a shared, database scoped object. Every managed run reconciles it with ALTER PUBLICATION ... SET TABLE. Narrowing it per run would make concurrent pipelines overwrite each other’s capture set. Operators who want the WAL itself narrowed keep supplying ?publication=, which ingestr never rewrites. Extra WAL traffic on unused tables is the cost. Two subset pipelines cannot clobber each other’s publication.

MultiTableReadOptions.Tables is an internal cleanup. Nothing set it once selection moved onto the source, so it was removed. That also drops a MySQL inventory branch whose “no longer available, run with --full-refresh” message was misleading for a typo.

This tag is a patch, not a prerelease. Single table --source-table and omitted --source-table keep their previous behavior. Existing CDC connectors keep their identity.

Do not share a subset run with an all tables run on the same slot or state. A wider capture later is a new connector. To narrow WAL on Postgres, pass ?publication= rather than expecting ingestr_publication to shrink. Unmatched names now fail. Check the --yes list before the first subset run.