Argo Workflows Large Workflow Operator Notes

Argo Workflows is a Kubernetes native workflow engine that shows up often in batch data pipelines, model jobs, and internal ETL platforms. The recent activity is worth reading because it changes how large workflow status is stored, how persistence failures time out, and how the project tests a new pod layout.

The most direct data pipeline change is the node status compression change. Argo already compresses large /status/nodes payloads before it falls back to SQL offload. This commit makes the algorithm configurable through WORKFLOW_COMPRESSION_ALGORITHM, with gzip, zstd, and brotli as the documented choices. WORKFLOW_COMPRESSION_LEVEL tunes the selected algorithm, and docs environment variables now names both knobs.

The docs in docs offloading large workflows give numbers on synthetic node status JSON. The useful part is not just smaller payloads. It is the trade between controller CPU and the point where status has to move out of etcd. The doc says zstd level 3 and brotli level 9 can raise the effective node count ceiling by roughly 20 percent and 50 percent. It also calls out that brotli level 11 compresses best but is about 80 times slower than level 9.

There is a compatibility trap. Decompression now detects the algorithm, but older Argo versions and simple kubectl tooling that assumes gzip cannot read statuses written with a new algorithm. The implementation in util file fileutil.go keeps gzip as the default. For operators, that means the rollout order matters. Upgrade all readers first, then change the controller setting.

The largest commit in the window is the beta initless pod work. It adds a controller wide initlessPod setting in config config.go and updates docs workflow controller configmap. The feature removes the old argoexec init container and uses a supervisor container plus Kubernetes image volumes to place the executor binary where the main container can run it.

That is not a cosmetic pod spec change. Init containers run in sequence, so removing them can reduce startup latency for workflows with artifact handling. The plugin model also changes: input and output plugin work can go through one long running sidecar per plugin, driven by supervisor, instead of splitting input load through init containers and output save through sidecars.

The risk is also concrete. The feature depends on Kubernetes image volumes. The docs say image volumes are beta in Kubernetes 1.33 through 1.35 behind a feature gate, and GA from 1.36. The controller does not probe the cluster for this. On clusters below that line, or with the gate off, pod creation fails at the Kubernetes API layer.

The more subtle workflow author change is input artifact staging. Legacy mode used bind mounts. Initless mode can use symlinks for input artifact paths after supervisor prepares the files. Most programs that open and read the path will not care. Code that calls lstat, readlink, removes the path, or runs with readOnlyRootFilesystem can see different behavior. This is why the beta label matters.

The database timeout fix adds connectionTimeoutSeconds to the shared database config used by persistence and synchronization. The default is 5 seconds. The field appears in both PersistConfig and SyncConfig docs in docs workflow controller configmap.

This matters for archive and node status offload users because a broken database path should fail in bounded time. The tests model a server that accepts TCP and never completes the startup handshake. Before this kind of fix, a controller could hang during session creation even though the network path looked open.

The implementation treats Postgres and MySQL differently because the drivers expose different controls. Postgres gets connect_timeout in the DSN. MySQL gets a connector wrapper that bounds connect and handshake work without applying a read timeout to every query. That distinction is good operational hygiene. A query timeout policy is not the same thing as a connection establishment timeout.

The build activity is not just developer convenience. The Tilt migration moves local and CI execution closer to what the controller now needs to prove: real images, a k3d cluster, port forwards, and initless matrix coverage. The changed ci build workflow adds initless entries for several end to end suites.

The Make side also got cleanup in the vendor dependency change. The updated Makefile gives vendor/modules.txt real dependency meaning for builds, code generation, lint, and end to end tests. That is mundane, but it closes a common CI gap where a touched marker file can make later Go commands see an empty or stale vendor tree.

There is also a smaller controller signal fix: parallelism backpressure now logs at info, with the change in workflow controller operator.go. Hitting max parallelism is normal scheduling pressure, not a workflow error. For noisy clusters, that reduces false error volume without hiding a real failed pod or template.

  • Treat compression algorithm changes as a full component rollout. Do not write zstd or brotli statuses until every reader can decode them.
  • Test initless pods on a non critical namespace first. Focus on image volume support, artifact paths, plugins, and templates that use readOnlyRootFilesystem.
  • Set connectionTimeoutSeconds explicitly if the workflow archive or synchronization database is across a fragile network path. The default is sane, but production failure budgets are rarely default shaped.