ingestr v1.1.43 - Vertica Destination And Swap Safety

ingestr v1.1.43 was published on 28 August 2026. The main operator visible change is a Vertica destination that bulk loads with COPY FROM STDIN and publishes replace jobs through an atomic table rename. The same tag also adds Sklik and 2Checkout sources.

The full release notes and downloads are on the GitHub release page.

Pull request 1133 adds Vertica as a destination, using the vertica-sql-go driver. Rows stream through an io.Pipe into COPY FROM STDIN with a control byte encoding, so delimiters, newlines, and nulls survive. Supported strategies are replace, append, merge, and delete+insert. scd2 returns an unsupported error.

replace loads into a staging table, then publishes with a multi table RENAME swap. The first cut reused a fixed _old backup name for the displaced target. A leftover table from a crashed prior swap, or a real user table already using that name, made the rename collide and left the loaded staging table unpublished.

The swap now probes for a free backup name derived from the staging table, which is unique per run, so two concurrent replace jobs do not pick the same identifier. The destination never drops an already existing table to free a name. If the chosen backup name is occupied, the atomic rename fails with staging and target still intact.

Cross schema replace loads into a temp table in the target schema first, then swaps, so a failed load never drops the live target. That temp table is claimed with a strict CREATE, not CREATE ... IF NOT EXISTS. A table that appears between probe and create is not reused. A later failure would otherwise CASCADE drop it. JSON, array, and interval values are stored as text. Vertica has no native type for those.

Sklik is now a registered source in pull request 1128. It is the paid search product of Seznam.cz and talks Sklik JSON-RPC, including a rolling session from client.loginByToken. The connector exposes five entity snapshots (campaigns, groups, ads, keywords, conversions) and two report tables (campaign_stats_daily, search_queries).

Sklik rate limits per account per minute. Over the limit a report returns HTTP 200 with an empty report array, which is byte identical to a genuinely empty window. The source treats createReport totalCount as the only reliable signal and retries with linear backoff only while that count promises rows the read has not produced.

search_queries keys on query, keyword_id, and date. Sklik reports the same search term once per matching keyword. Keying on (query, date) alone would keep one keyword row and drop the rest of the spend. queries.createReport also returns empty unless the request is scoped to campaigns, groups, or keywords. An unrestricted call can look like an account with no search terms.

Pull request 1129 adds 2Checkout through REST API 6.0. The URI scheme is twocheckout, not 2checkout, because RFC 3986 requires a scheme to start with a letter. Tables are orders, subscriptions, products, and promotions. There is no customers table. REST 6.0 only exposes /customers/search/, which requires an Email parameter, so the customer base cannot be enumerated.

On /orders/, GrossPrice and GrossDiscountedPrice arrive as a running cumulative total across the page. NetPrice, NetDiscountedPrice, and VAT are correct, so the source recomputes gross rather than storing the raw field. Orders are keyed on (RefNo, Status). A refund row negates the money fields, and keying on RefNo alone lets merge overwrite the original charge.

/subscriptions/ uses only ModifiedAfter. It refuses to page past 20000 results and silently ignores StartDate, EndDate, and ProductCode.

The cloudflare-radar source used to emit one Arrow record batch per API page, ignoring opts.MaxBatchBytes. Pull request 1132 chunks emitted batches by that cap so this source matches the others.

Pull request 1135 changes Adjust daily and hourly reports to one request per day so period aggregates stay correct. Reports that contain day or hour are split, and extract parallelism still applies. Week, month, quarter, and year custom reports stay a single request.

Pull request 1136 is a CI fix, not an ingest path change. test_release_platform_uses_goreleaser_asset_names called the real _linux_uses_musl(), which globs /lib and /usr/lib for ld-musl-*.so.1. On a runner with a musl loader present, the Linux branch hit the glibc guard and failed. The test now mocks that probe.

The notes do not list a breaking change. Vertica, Sklik, and 2Checkout are additive.

If you point a new job at Vertica, scd2 is not supported. User tables that already occupy a generated backup name will block the swap rather than be dropped. That is the intended failure mode.