Argo Workflows Fixes Memoized Resubmit and Artifact GC

Argo Workflows is the Kubernetes workflow controller that runs DAG and step graphs as pods. In the seven days ending 23 September 2026, main took 53 commits and changed 230 files (8619 insertions, 5019 deletions), most of it test fixtures and lockfiles. The changes that affect a running cluster are a memoized resubmit rewrite, artifact GC that retries, and a sync queue cleanup that keeps other controllers’ waiters.

argo resubmit --memoized could sit Running forever when the failure sat inside a withItems or withParam fan out over a nested DAG or Steps template. The TaskGroup stayed Pending after the failed pod had succeeded on the rerun. The rewrite drops the 2018 loop in FormulateResubmitWorkflow and uses the same reset planner as argo retry.

The old loop deleted Skipped and Omitted nodes and left their IDs in children and outboundNodes. assessDAGPhase treats a missing child as still running, so the inner DAG never finished. A template resolution error or an expired suspend node used to produce an empty plan, and memoized resubmit then completed Failed without running a step.

Nested fan outs complete. A failed Retry node drops old attempts, succeeded nodes stay Succeeded, and a continueOn survivor stays Failed with original pod: recorded. A when off the failed path is left as it was. The shared planner also changes argo retry, so keep the change off release branches.

Node IDs are still <workflow>-<fnv32a(name)>. custom-job-thbh7[0] and a longer descendant both hashed to 1277661780, the graph cycled, and the controller overflowed the stack. On synthetic fan out names, about 2 percent of 10,000 node workflows and about 39 percent of 50,000 node workflows collide, usually with the second sibling never running. The 17 September fix stores the losing name as a 64 bit FNV-64a id, and a second collision fails initializeNode. The follow up test changed because memoized resubmit now deletes a failed colliding leaf. See workflow/util/node_id_collision_test.go. v4.1.4 has the collision fix.

Artifact GC marked a strategy processed inside a defer, so one failed task or pod create was stored as success and never retried. The workflows.argoproj.io/artifact-gc finalizer stayed, and an empty podsRecouped map blocked forceFinalizerRemoval. In the reported namespace, namespaceParallelism filled with completed workflows and every new submit stayed Pending.

Processing is recorded only on success. Failure sets ArtifactGCError and requeues after one minute. After that failure, once the workflow has been complete (or deleted, for OnWorkflowDeletion) longer than ARGO_ARTIFACT_GC_RETRY_WINDOW (default 60 minutes), the strategy is abandoned and the finalizer can drop. Create returns the stored task so the GC pod owner UID is set.

Sync queue GC fixes issue 16737. Controllers sharing a sync database ran CheckWorkflowExistence once a minute, listed sync_state by lock name, and asked the local informer whether the workflow existed. Another controller’s workflow always looked deleted. ReleaseHeld filtered on controller. RemoveFromQueue did not, so it deleted the foreign pending row. Controller A could then release the lock, notifyWaiters saw an empty queue, and the lock sat free until B reconciled, about 20 minutes at a typical informer resync. RemoveFromQueue now requires controllerName on the DELETE. The mock diff is util/sync/db/mocks/SyncQueries.go. controllerName still defaults to empty, and nothing checks uniqueness. Two controllers on that default can delete each other’s held rows, and a database mutex can be held twice.

The changelog update on 18 September records four private fork merges. The same patches are on main. v4.1.4 ships them with the node ID collision fix.

Directory artifact download on S3, GCS, Azure, and OSS joined the key remainder onto the local path with no containment check. A key with .., or one that follows a symlink out of the destination, could write anywhere the executor can write in the pod. LocalPathForObject rejects those keys. Archived get returned not found before the RBAC check and permission denied only when the object existed, which let a caller enumerate archived names. Delete, resubmit, and retry share that path. The check now runs first.

ListArchivedWorkflows authorized list on the namespace in metadata.namespace!= and then ran the negated query, so list permission in one namespace read every other namespace. The live list already uses a cluster wide check there. Client mode is the fourth fix. --auth-mode=client built clients from the bearer token without checking it, so GetInfo, GetVersion, GetUserInfo, LintWorkflow, and dry run CreateWorkflow accepted any token shaped string. Each request now does a SelfSubjectReview. Successes cache for ARGO_SERVER_TOKEN_REVIEW_CACHE_TTL (default 1 minute) under a SHA256 of the Authorization header.

Span and trace IDs now come from the workflow, the node, the phase, and the creation timestamp, so a restart keeps them and a delete plus recreate does not collide. A retry of the same node still reuses the previous span ID. markNodePhase also called EndNode after ChangeNodePhase had already closed the span, which logged no existing trace for a running node on every finished node. New IDs are built in util/telemetry/builder/tracing.go.

Controller logs were the volume. In a 6 hour sample, 73,394 of 104,380 lines were was unable to obtain the node from the DAG lookup for tasks that had not started. That line and the pod event logs are removed. Per node misses move to debug, and Error updating workflow moves to info. The call sites are workflow/controller/dag.go and workflow/controller/operator.go.

HTTP1 headers split on every colon, so Example-URI:http://example.com/a.html was rejected. parseHeaders now uses strings.Cut and keeps the remainder of the value. ARGO_WORKFLOW_NAME was set twice on init, wait, and the supervisor. workflow/controller/workflowpod.go is now the only writer.

The executor series is an internal refactor. Process state splits from task state, commands run a Prepare, Run, Collect plan, and legacy init is that Prepare phase. Init errors gain a stage prefix (write-template). workflow/executor/plan.go picks the plan. go.mod moved to Go 1.27.1 and OTel SDK 1.46.0.

v4.1.4 has the four auth and path merges and the node ID collision fix. The memoized resubmit planner, the artifact GC retry, and the controllerName filter are on main after that tag.

Give every controller on a shared sync database its own non empty controllerName. The default is empty, and two empties still share one GC scope, including held rows.

ARGO_ARTIFACT_GC_RETRY_WINDOW is 60 minutes. After a recorded failure ages past it, the strategy is abandoned and force finalizer removal can free the parallelism slot while the objects remain. Alert on ArtifactGCError when that matters. The new trace IDs also repeat when the same node is retried.