CloudQuery Headless Sync and Pipeline Maintenance

CloudQuery had a busy maintenance window: 47 recent commits touched 295 files, with 1663 insertions and 1398 deletions. The useful part for data engineers is not the raw churn. It is the CLI work around headless platform sync, followed by broad dependency and CI refresh across plugin build paths.

CloudQuery CLI v6.38.0 is the meaningful user facing change in this batch. The feature landed in the headless platform sync commit, then was cut in the CLI v6.38.0 release commit. The practical effect is simple: cloudquery sync and cloudquery migrate can run against a CloudQuery Platform tenant with a pre minted cqpd_ token and no cloudquery login.

That matters for scheduled data jobs. A sync process can now carry CQ_PLATFORM_TOKEN, or use CLOUDQUERY_API_KEY when the value starts with cqpd_. The CLI reads tenant URL and team name from token claims, uses the token for plugin download, injects the platform destination, and passes the same credential to spawned plugins when no user supplied API key is already set.

The version pinning path is also worth noting. In the headless path, the CLI asks the tenant whoami endpoint for the recommended platform plugin version. The precedence is explicit: CQ_PLATFORM_PLUGIN_VERSION, then the platform recommendation, then the CLI default. That keeps automation from drifting onto a platform destination build that the tenant did not expect.

The release did not stop at cli/CHANGELOG.md. A follow up CLI dependency update moved workflow references from v6.37.0 to v6.38.0 in the manual command workflow and the plugin publish workflow. That is the right surface to watch if your internal fork depends on CloudQuery workflows for plugin release automation.

The root module also picked up the new CLI module version, and cli go.mod was one of the most touched files in the window. Seven commits touched it, while cli go.sum saw six. That pattern says this was less about new source logic and more about keeping generated, module, and workflow state aligned after the CLI release.

For operators, the risk is not a new flag breaking existing specs. The old login path remains. The sharper edge is older automation that pins a CLI version outside the repo workflows, then expects platform destination injection behavior from v6.38.0.

The dependency wave was broad across destination code. PostgreSQL go.mod, Elasticsearch go.mod, and DuckDB go.mod all appeared in the top changed files. The commits included pgx to v5.10.0, MongoDB Go driver to v2.7.0, Elastic transport to v8.11.0, and Kafka Sarama to v1.50.3.

This is not a schema migration by itself. The blast radius is lower, but it is not zero. Database drivers can change retry timing, TLS defaults, error wrapping, bulk write handling, and edge behavior under stale connections. Destination plugin owners should run smoke syncs against real service versions, not just unit tests with fakes.

The same point applies to observability and runtime libraries. The activity included OpenTelemetry Go and collector updates, plus Go support packages such as golang.org/x/net and golang.org/x/sync. If you watch sync latency or table error rates, keep the next few runs comparable to prior baselines before treating this as harmless dependency noise.

The most touched file was the Airtable pnpm lock, hit by eight commits. The latest commit in the input updated eslint plugin unicorn to v68, while nearby commits moved pnpm, eslint, prettier, got, dayjs, and typescript eslint. This is lint and package maintenance, not an Airtable extraction change.

CI also saw a large mechanical refresh. The GitHub Actions update touched 55 files, including destination workflows, source workflows, release workflows, and publish workflows. Separate commits then moved actions/cache to v6 and actions/checkout to v7. The files most worth tracking are the publish paths such as publish plugin to hub DuckDB, publish plugin to hub Snowflake, and source test paths such as source test.

This kind of update often looks boring until a runner image, cache key, or action input shifts under release jobs. The good news is that the change is visible and mostly isolated to workflow references. The next useful signal is whether publish jobs stay stable after the new CLI version and Actions pins are both in play.

  • For scheduled sync jobs, decide whether CQ_PLATFORM_TOKEN should replace interactive login. Validate that existing CLOUDQUERY_API_KEY values are not masking the platform token.
  • For destination plugin tests, rerun smoke syncs after the pgx, DuckDB, Elasticsearch, Snowflake, Kafka, SQLite, and MongoDB bumps. The change set is dependency heavy, so regressions would probably show as driver behavior.
  • For CI owners, watch the new Actions pins and cli v6.38.0 workflow references. A failed publish path is more likely than a source schema change in this window.