Argo Workflows v4.0.11 - Empty OTLP Endpoints and Lock Status

Argo Workflows v4.0.11 was published on 11 September 2026. The release commit, dc33efd, stops the controller from starting an OTLP metrics exporter when OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_METRICS_ENDPOINT is set to an empty string. On the 4.0 line that empty value used to dial localhost:4317 for the life of the process.

The full release notes and downloads are on the GitHub release page. The page body is the quick start: a curl install for the CLI and a kubectl apply of install.yaml. Selected changes for this tag are in the changelog under v4.0.11, as the diff from v4.0.10.

os.LookupEnv reports a variable set to "" as present. The 4.0 gate used that check on OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, so an empty string opened the exporter. With no usable endpoint the SDK fell back to localhost:4317 and the controller kept pushing metrics there. docs/metrics.md already said a blank value would leave export disabled. #16937 makes the code follow that sentence.

v4.0.11 calls resolveOTLPEndpoint in util/telemetry/otlp.go and requires a real address. Empty values stay off. Whitespace only values stay off. A configured endpoint such as http://localhost:4317 still exports. The log line “Starting OTLP metrics exporter” now includes the endpoint. 4.0 exports metrics through this path.

An empty variable set to reach the SDK default loses the localhost dial. An empty variable set because the docs said blank means off now gets that behavior. A populated endpoint is unchanged.

4.0 built the meter resource with resource.NewWithAttributes and hardcoded semconv.ServiceName to workflows-controller. metricsdk.WithResource merged resource.Environment() first and that object second. On a shared key the second value wins, so the hardcoded name overwrote OTEL_SERVICE_NAME and service.name in OTEL_RESOURCE_ATTRIBUTES. With both variables set to argo-workflows, a reader still collected service.name=workflows-controller next to k8s.namespace.name.

#16934 (e943741) is the 4.0 fix for the bug filed against 4.1.2 as #16927. Static attributes go on first inside workflowsResource. Environment attributes follow through WithFromEnv. WithSchemaURL keeps the schema URL 4.0 already exported. util/telemetry/resource_test.go covers the default name, both environment sources, their precedence, and the schema URL.

Unset variables keep the name workflows-controller. Set them and a dashboard keyed on the old service.name splits. The new name is attached to metrics.

#16913 (ba4ab7e) closes #16397. After downstream steps finish, Argo kills argoexec in the daemon pod. SIGTERM is exit 143. SIGKILL is exit 137. v3 forced NodeSucceeded when a daemon pod hit PodFailed. v4 dropped that guard while adding daemon retries, and inferFailedReason treated the main container like any other pod. v4.0.11 keeps NodeSucceeded for those two codes on a daemon template. Any other non zero exit still becomes NodeFailed. Alerts on a clean daemon teardown go quiet.

#16873 (6926264) covers containers whose names start with artifact-plugin-. The wait container, or the supervisor container when the pod has no init container, stops them only after outputs and logs are saved. terminateContainers also SIGTERMs them through kubectl exec of argoexec kill. If that exec races PID 1, the runtime can store a phantom status. CI saw exit code 2 on artifact-passing-explicit-plugin.yaml with a clean sidecar log. inferFailedReason now ignores those sidecar exit codes. The main container and the wait or supervisor container still decide the node. A plugin that dies during a save still fails the node, because the aux container save fails. A sidecar exit code by itself is no longer a failure signal.

A Pending workflow on a database lock printed the opposite fraction from a lock kept in memory. workflow/sync/semaphore.go reports Lock status: as available slots over the limit, so a held mutex shows Lock status: 0/1. workflow/sync/database_semaphore.go printed holders over the limit, so the same mutex showed Lock status: 1/1. #16911 (44233ee) reports available slots, limit minus holders, over limit. A held database mutex now reads Lock status: 0/1 on Postgres and on MySQL.

Alerts that treated Lock status: 1/1 as a held database mutex will miss that state. 0/1 means held on both backends. The denominator is still the limit.

#16834 (68314fd) fixes a nil interface on the same path. SynchronizationStatus.GetStatus wrapped a nil *SemaphoreStatus or *MutexStatus in the SynchronizationAction interface. A nil pointer stored in an interface compares as set, so s != nil in Manager.Release passed and LockReleased dereferenced nil for a workflow released with unset lock status. GetStatus now returns an untyped nil. staticcheck SA4023, from golangci-lint 2.13.2, flagged the typed nil.

The v4.0.11 selected changes carry no breaking marker. The release page points at the upgrading guide and at open issues for jumps past this patch.

The changelog marks security updates in the same tag. google.golang.org/grpc moves to v1.83.2, with an intermediate line at v1.83.1. The qs dependency moves to v6.16.0.

Install the controller and server from the GitHub release page:

kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v4.0.11/install.yaml

The CLI snippet on that page downloads argo-linux-amd64.gz or argo-darwin-amd64.gz and installs the binary as /usr/local/bin/argo. The snippet targets amd64. Run argo version after the move.