ingestr v1.1.58 - Keyed CDC Resume After a Killed Run

ingestr v1.1.58 was published on 29 September 2026. The change that matters on a killed job is keyed CDC resume: a merge that continued a verified snapshot can replay from the last complete generation instead of taking another full snapshot. The same tag also stops SQLite, MySQL, and Athena from dropping Arrow lists, small integers, and millisecond dates on write.

The full release notes and downloads are on the GitHub release page. Commits since v1.1.57 are listed on the compare page.

Pull request 1200 changes resume after an incomplete CDC generation. A run killed between BeginRun and Persist leaves the newest generation unfinished. ResumePosition refused it, and the connector took a full snapshot.

A primary key merge can replay when it continued a verified snapshot and never started replacing the target. The killed run only staged its work. ResumePositionForKeyedMerge walks back to the newest generation that certified the table, and prune keeps that certificate until the current generation certifies it.

BeginRun writes a resuming marker only after a keyed merge accepts a valid certificate. Historical resume requires the marker on every interrupted generation. The check rejects an invalidated snapshot, a changed source or destination identity, a conflicting run, and a destination cursor that is missing or newer than the certificate. An emptied table, or one holding changes the old generation cannot explain, still snapshots.

A cursor at or below the previous checkpoint is not proof. Batch and streaming replacement invalidate resume eligibility before they truncate, because a partial reload can keep the same physical identity. Batch invalidation keeps the original destination binding and the accepted snapshot metadata, so a finished WAL truncate can still be certified by a checkpoint only commit.

Only merges with primary keys use the fallback, on single table and multi table pipelines. Legacy runs with no resuming marker still snapshot. The empty state gate for unproven legacy targets is unchanged.

Pull request 1244 fixes SQLite lists. The writer stored the whole Arrow List debug string on every row that was not null, so [[7,null,-2],null,[]] landed on each present row. After the parent null guard it now calls List.ValueStr. Child nulls, SQL NULL, an empty list, and a nested list stay distinct.

Pull request 1243 is the MySQL case for typed Arrow List and LargeList. The debug string is invalid in a MySQL JSON column, and every present row got the same text, shaped like [[7 (null) -2] (null) [] [42]] instead of [7,null,-2]. INSERT arguments and the LOAD DATA TSV serializer both use the per row JSON form. Null handling, type mapping, and deduplication stay as they were.

Pull request 1222 fixes Athena INSERT literals. arrowLiteral had no arm for Int8, Uint8, Uint16, Uint32, Uint64, or Date64, so a value that was not null became NULL. Unsigned integers now print without signed overflow, including a Uint64 past MaxInt64. Date64 uses Date64.ToTime on both sides of the epoch. Real nulls still render as null. No live Athena write was run. Athena still enforces column type and range, and rows already stored as NULL are not rewritten.

Pull request 1253 closes a false success. CleverTap ignored the strategy, so delete and replace uploaded rows and reported success. Profiles now accept only merge. Events accept only append. Anything else fails before the client sends. --full-refresh, and an --incremental-strategy that does not match the table, fail before the read. The table strategy wins over the one the source selected. If --incremental-strategy is unset, each table keeps its own strategy, so runs that never passed the flag keep the old default. PrepareTable does the check once.

The other six notes do not change ingestion. Pull request 1249 fixes a Postgres CDC fixture. The old test expected a snapshot only batch to include a later insert and failed before FinalizeBatch. Keepalive must not advance confirmed_flush_lsn before the destination is durable. Pull request 1228 covers DynamoDB BatchWriteItem retries and CDC merges where delete wins an LSN tie. Pull request 1235 maps CLI flags into runIngest and found no defect. Pull request 1250 replaces Frankfurter fixtures that used one timestamp for start and end with a nonempty midnight to noon interval. Pull request 1231 and Pull request 1225 exercise pipeline incrementality and Google Sheets writes over HTTP.

CleverTap jobs that passed delete, replace, --full-refresh, or a mismatched --incremental-strategy used to upload and exit zero. On v1.1.58 the run fails before the read. Profiles need merge. Events need append. Leave --incremental-strategy unset when the table default was enough.

Generations written before this tag have no resuming marker, so the first run after upgrade still snapshots. A partial reload still forces a snapshot.

SQLite and MySQL list columns change stored text from the Arrow debug string to a JSON array. SQL NULL, a child null, and an empty list stay different. Athena NULL values already written for Int8, unsigned integers, or Date64 stay null until a later load. Backfill if those stored nulls are wrong.