ingestr v1.1.54 - CDC Overlap Warning And Archive Hash

ingestr v1.1.54 was published on 11 September 2026. The pipeline change is in pull request 1191: a CDC run that cannot prove it is the only live run of its connector now warns and continues, where v1.1.53 rejected it. That admits mssql+ct, mssql+cdc, mongodb+cdc, vitess+cdc, and ps_mysql+cdc into BigQuery and Snowflake when the schedule stays serial. The same tag adds an optional SHA-256 check on the installer archive in pull request 1184.

The full release notes and downloads are on the GitHub release page. The patch range against the previous tag is the v1.1.53 to v1.1.54 changelog.

validateCDCRunSerialization required proof that only one run of a connector was live before a merge into a destination that cannot enforce primary key uniqueness. On BigQuery, PRIMARY KEY is NOT ENFORCED, a query optimizer hint. Snowflake has the same gap. Where the key is enforced, two overlapping inserts collapse on the unique index. Without it both inserts land, later merges update both copies, and nothing repairs the duplicate.

PostgreSQL CDC already proved exclusivity with its replication slot. MySQL CDC proved it with a destination side ManagedCDCRunLeaser (pg_try_advisory_lock or GET_LOCK). BigQuery and Snowflake have no equivalent, so every other change source was rejected. mssql+ct into bigquery could not run.

Pull request 1191 renames the check to warnUnserializedCDCRuns. Detection is unchanged. The result is output.Warnf where the old path returned fmt.Errorf. The warning names both schemes, points at tracking issue 1190, and tells the operator to run the connector serially. The hazard needs two overlapping runs of one connector. A serial schedule avoids it. The old rejection also blocked a single scheduled run.

Newly admitted sources against BigQuery and Snowflake are mssql+ct, mssql+cdc, mongodb+cdc, vitess+cdc, and ps_mysql+cdc. postgres+cdc still relies on the exclusive slot. Destinations with an enforced primary key are unchanged. --full-refresh does not merge into existing rows, so it skips the warning.

docs/getting-started/cdc.md adds Overlapping runs on BigQuery and Snowflake: the duplicate key path, why PostgreSQL CDC is immune, and three mitigations (runs that do not overlap, a scheduler level lock, or one long lived --stream process). The SQL Server pages point there. skills/bigquery-destination/SKILL.md now matches the warning. Restoring the error later is a one line change in the same function.

mysql+cdc and mariadb+cdc skip the warning. Pipeline.Run still requires a ManagedCDCRunLeaser, and validateMySQLCDCMutationFencing still requires the CDCConditional* interfaces. BigQuery and Snowflake implement neither, so warnUnserializedCDCRuns returns early. The CDC guide states that restriction. Stock mysql+cdc and mariadb+cdc still fail on those warehouses on any schedule. mssql+ct, mssql+cdc, and ps_mysql+cdc stay in the admitted set.

Issue 1190 still tracks a durable fix. The pull request lists an ETag compare and swap lease on BigQuery table metadata, incarnation fencing, a measurement of concurrent DML, and a duplicate repair pass. None of those are in v1.1.54. Admission changed. Overlap can still write two rows for one key.

Pull request 1184 adds an optional archive hash to install.sh and tests/test_install.py. Attestation, cosign, jq, and a signing manifest sit outside this pull request.

The caller passes a digest of the archive bytes, the .tar.gz or the Windows .zip, through -s. The digest covers the archive, not the extracted binary and not the installer script. Trust in install.sh itself is separate. The value is 64 ASCII hex characters, either case, with no prefix and no whitespace.

With -s set, the script accepts exactly one stable tag matching ^v[0-9]+\.[0-9]+\.[0-9]+$. A missing tag, latest, a prerelease, and extra positional arguments are rejected. The download uses that tag directly. Omitting -s keeps the old flags and the old version selection.

The check runs before extraction, execution, installation, and any overwrite of the destination path. A download failure, a mismatch, missing hash tools, or a bad digest fails closed and deletes the temp file. The script uses an already installed gsha256sum, sha256sum, shasum, or openssl, fixes the OpenSSL helper, and applies strict curl failure handling only when -s is set.

On HUP, INT, or TERM the interactive path reaps the child tree, so install cannot keep writing after the parent exits. A repeated signal does not leave cleanup blocked on curl, tar, or install.

Pipelines that already cleared the serialization check keep that behavior. The set is postgres+cdc, MySQL CDC on a destination that implements ManagedCDCRunLeaser, and any destination with an enforced primary key.

Pairs that v1.1.53 rejected can run on v1.1.54 and emit a warning: mssql+ct, mssql+cdc, mongodb+cdc, vitess+cdc, and ps_mysql+cdc into BigQuery or Snowflake. A second run that starts before the first merge finishes can insert the same key twice. Use one run at a time, a scheduler lock, or a single --stream process. The warning means the run is allowed. Overlap is still unsafe.

mysql+cdc and mariadb+cdc into those warehouses still fail on the missing lease and fencing interfaces.

Installer calls that omit -s keep the old download path. Pass -s with a trusted digest of the archive for that exact stable tag. A prerelease tag, a floating latest, or a digest of the extracted binary fails the new checks.

Binaries and the note text are on the GitHub release page.