ingestr v1.1.26 - Okta Source And Decimal Cast Fixes

ingestr v1.1.26 was published on 12 August 2026. The release adds an Okta Management API source for identity and access records, and it fixes --columns decimal overrides so a precision overflow fails the run instead of writing a bad value.

The full release notes and downloads are on the GitHub release page.

Commit a060752 adds the Okta source (pull request 1088). Authentication is an SSWS API token on a URI of the form okta://<org>.okta.com?api_key=....

The source exposes 11 tables: users, groups, group_members, applications, application_users, application_groups, system_log_events, devices, policies, policy_rules, and roles. That set covers directory objects, app assignments, devices, policy, and the system log.

system_log_events always sends bounded since and until query parameters. Okta polling query mode can truncate a full load without a hard failure. The bounded window keeps the extract complete.

Most Okta list endpoints paginate through a Link header cursor. /iam/roles is the outlier: it paginates via _links.next.href in the body. The client uses two rate limit buckets, a shared core bucket for most endpoints and a tighter one for /logs. Retries are capped at 5 attempts with backoff up to 65 seconds, which matches the Okta rate limit reset window of about 60 seconds rather than a generic HTTP client default.

Load strategy is per table, not global. users, groups, applications, devices, policies, and policy_rules merge on id and lastUpdated. system_log_events merges on uuid and published. group_members, application_users, application_groups, and roles use replace. The replace tables have no incremental key; they emit the current set only.

Merge and replace have different failure modes in the warehouse. Merge upserts what the API returns. A user or group deleted in Okta may remain in the destination until a separate cleanup. Replace rewrites the table each run, so a removed group member or app assignment disappears. That is the correct choice for assignment tables. It is the wrong choice if the goal was cheap incremental loads on those four tables.

Commit 101a029 is the Greptile review fix. The first version of application_users and application_groups used merge. Those readers list only current assignments, so an unassignment would never be emitted and the old row would stay forever. Both tables now use replace, matching group_members. The page cap is also a hard error. If a listing exceeds maxPages, the source fails instead of reporting success with missing rows.

--columns decimal overrides ¶

Commit bc44d14, merged as pull request 1089, fixes Arrow casts that run in memory before the bulk load when --columns overrides a column to a sized decimal such as decimal(p,s). Destinations bulk load already typed Arrow, Parquet, or COPY binary. There is no SQL CAST seam on the write path, so the conversion lives in pkg/databuffer/file.go.

Two Arrow casts were wrong. A decimal to decimal safe cast returned a malformed error (invalid: %!s(<nil>)) when the target scale dropped digits. Any --columns rescale of a decimal source column failed the run. An int to decimal cast refused unless the target precision could hold the entire integer type range, so int64 to decimal(10,2) failed even when the actual values fit.

The fix rescales decimal to decimal with an unsafe cast. A smaller scale truncates toward zero, matching the existing truncating decimal and float to int conversions. Int to decimal widens through decimal(38,0) first, then rescales. After the unsafe rescale, ensureDecimalFits checks that every value’s magnitude fits the target precision. Overflow is a hard error (value overflows decimal(p,s)). Scale reduction stays silent because that is the requested type. float to decimal and string to decimal already worked and are unchanged.

Concrete cases from the change: decimal(8,5) to decimal(10,2) turns 3.14159 into 3.14. int64 12345 to decimal(10,2) becomes 12345.00. 12345.678 as decimal(10,3) to decimal(5,2) errors because 12345.67 does not fit decimal(5,2). An int64 of 99999999999 to decimal(10,2) also errors.

Pipelines that never pass --columns decimal overrides should see no behavior change. Pipelines that do override to a tighter decimal(p,s) can now finish rescales that used to abort on Arrow’s safe cast. They can also fail later if a value does not fit the precision. Treat that error as a schema problem. Do not retry it as a transient ingest fault.

The Okta source is additive. Other sources do not need a migration. Assignment tables on Okta use replace, so the destination table is a snapshot of current assignments, not a history of every grant.

CI commit 8b78485 (pull request 1086) sets INGESTR_QUIET_PROGRESS=1 in test-integration-nodocker. That discards the per pipeline “Initiated the pipeline” banner and the metrics table (Total Rows, Duration, Peak Memory) so go test output stays readable. It is test plumbing, not a change to default CLI logging.