ingestr v1.1.47 - Streaming HTTP File Source

ingestr v1.1.47 shipped on 2 September 2026. The tag adds a streaming HTTP file source so CSV, JSON, JSONL, gzip, and Parquet files can be read over HTTP(S) through the existing decoders with bounded heap use. The same release also fixes ClickHouse destination schema readback for decimal columns, which was failing every merge or append sync after the first run.

The full release notes and downloads are on the GitHub release page.

Pull request 1156 lands the new source in commit 465bbbb. Remote files stream through the current CSV, JSON, JSONL, gzip, and Parquet decoders instead of buffering the whole object. Pipelines that already decode those formats from local disk or object storage get the same decode path over HTTP(S).

The URI contract covers static authentication, custom headers, redirects, retries, cancellation, conditional requests, checksums, and metadata. Resume is in process only. It needs a strong ETag or a valid Last-Modified validator plus a consistent 206 Content-Range response. Conditional validators are caller supplied and are not written to disk. A job that dies and restarts does not pick up a Range resume from a previous process.

Commit 97b3dfc validates HTTP resume identity so a resumed 206 cannot concatenate a different object when the ETag is missing or Last-Modified changed. Concatenating two representations of a file is silent row corruption, not a successful retry.

doRequest used to overwrite User-Agent with the ingestr default after cloning caller headers. Commit e109f49 applies the default only when the caller did not set header.User-Agent on the source URI. Commit 6d6b2aa trims encoding hints so a value such as csv, encoding=windows-1252 no longer slices to =windows-1252. Commit debf105 is an internal refactor. detectFormat now parses the URL once and returns whether the resource is gzip encoded, dropping a second isGzipped pass that parsed the same URL again.

The streaming client only set ResponseHeaderTimeout. A server that sent headers and then stalled mid body without closing the socket hung forever. The previous buffered source was bounded by a 120s overall Client.Timeout. That hang is a regression operators would feel on flaky origin servers and on objects that start and then stop sending bytes.

Commit b5f60a4 adds a rolling per read deadline. The default is 2 minutes. Override it with read_timeout. Set 0 to disable. A stall fails. A slow but progressing stream is not killed. Commit 18b18ca is lint cleanup on that timeout path.

Pull request 1167 is the destination side of schema evolution. ClickHouseDestination.GetTableSchema left Precision and Scale unset. Schema evolution then compared a real Decimal(38, 9) source column against a phantom Decimal(38, 0). Every merge or append sync after the first failed with:

decimal widening requires precision 47 (integer digits 38 + scale 9),
maximum supported precision is 38

Commit d97e3e0 stops hand parsing DESCRIBE output. The destination now reads system.columns and reuses the source mapper MapClickHouseToDataType. That fills Precision, Scale, and ArrayType. DateTime64(..., 'UTC') stays TypeTimestampTZ so timezone aware timestamps round trip. GetTableSchema is used only for schema evolution comparison and existence detection. It does not change writes, table creation, or row data.

Width suffixed ClickHouse types still broke the mapper. Decimal32, Decimal64, Decimal128, and Decimal256 encode scale as the single argument and imply precision from the width. The regex treated that argument as precision, so Decimal128(9) came back as (9, 0) instead of (38, 9). Commit 9bd2db2 captures the width and derives precision from it.

system.columns returns zero rows, not an error, for a missing table. An empty schema that is not nil made first run preparation treat the target as existing. Commit 9f7d97f asserts GetTableSchema returns nil in that case so the first sync creates the table.

No migration of existing ClickHouse tables is described. Pipelines that already failed on the phantom decimal widening should succeed on a later merge or append once this tag is running. Watch the HTTP idle timeout if you ingest large files from slow origins. The default 2 minute read_timeout fails a stalled body, and 0 disables the bound. Resume still does not survive process restart.