OpenShift Origin Stabilizes TNF Recovery Tests And CI Allowlists

The openshift/origin repo is the OpenShift end to end test suite, not a cluster runtime. Four commits landed this week across 7 files (385 insertions, 46 deletions). The bulk of the diff is DualReplica Two Node Fencing recovery flake hunting; two smaller changes stop CI from treating expected OKD launch job names and EgressIP events as failures.

Recovery tests for DualReplica TNF were passing at 57% to 95%. The disruption sibling suite sat at 100%. The gap was test hygiene, not a new etcd bug.

Failed cases left the cluster in maintenance mode, with etcd-clone disabled and stale CRM attributes still set. The next case inherited that state and failed for reasons unrelated to the scenario under test. tnf_recovery.go now runs an AfterEach that mirrors the disruption pattern: drop maintenance, unstandby nodes, reenable etcd-clone, clear CRM attributes, run pcs resource cleanup, then wait for cluster and etcd health.

Cleanup itself had a footgun. An earlier draft used Items[0] as the debug node. After a failed recovery that node can be unreachable. The merged code waits up to 2 minutes for a Ready node and issues pcs commands from there.

The second root cause was Pacemaker’s default retry budget. Node failure tests exhausted it, then Pacemaker stopped restarting etcd and the suite reported a false failure. Five cases now set migration-threshold=INFINITY under DeferCleanup:

  • double graceful shutdown
  • sequential graceful shutdowns
  • graceful plus ungraceful failure
  • kernel panic recovery
  • simultaneous graceful shutdown

AfterEach also checks that the override did not leak past DeferCleanup.

Double reboot tests had a separate timeout problem. longRecoveryTimeout is 10 minutes and was sized for single node container kill. After both nodes reboot, bare metal still has BIOS POST, OS boot, kubelet start, and operator recovery. CI showed AllNodesReady passing while MonitorClusterOperators timed out on 503. The suite now gates etcd membership checks on waitForClusterHealthyWithPeriodicCleanup with clusterReachableAfterDoubleReboot at 25 minutes. Callers that passed ThirtySecondPollInterval into validateEtcdRecoveryState were ignored before; the helpers hardcoded a 5 second poll. That is fixed.

tnf_etcd_disruption.go had a quieter bug. pacemakerDegradedObserver dropped API errors, so an apiserver hiccup during the majority loss window looked identical to the condition never firing. Errors are counted and logged. The observation is informational; dedicated latency coverage stays in tnf_pacemaker_healthcheck.go.

The same recovery commit also reworked connectivity after a node is destroyed and reprovisioned. tnf_node_replacement.go and tnf_node_replacement_ovn_vm.go wait on PodNetworkConnectivityCheck for pod to pod traffic.

Two independent failures showed up in CI. OVN Kubernetes often does not resync the dataplane for a new chassis until pods restart, so Reachable=True never arrived. The restart of ovnkube-node and control plane pods lived in AfterEach, which runs after the test has already failed. That recovery is now in the test flow: if the first 12 minute check fails, restart those pods, wait 60 seconds, then retry.

The second failure was a stale check object. After replacement the old PNCC kept an empty Reachable status for 24 minutes or more because the target endpoint changed. resetStalePNCC deletes the stale object and restarts network-check-source so the cluster network operator creates a fresh check against the new pod IP.

Pod restart can also move the source onto the replacement node, which changes the PNCC name. resolveEastWestNodes discovers the actual source. waitForNetworkCheckSourcePodReady skips pods with a DeletionTimestamp, because the old code treated a terminating pod as Ready.

This is still a test workaround. If OVN Kubernetes needs a pod restart to learn a new chassis in production, that is a product question, not something this suite fixes.

validateStandaloneNodeOS compares cluster version to the Prow job name. If the cluster is OKD SCOS (okd-scos in clusterVersion.Status.Desired.Version) the raw job name must also contain okd-scos. That is a reasonable guard for payload jobs whose names encode variant and platform.

cluster-bot launch jobs are variant agnostic by design. Names look like release-openshift-origin-installer-launch-aws-modern. The [sig-ci] [Early] job name test hard failed on OKD SCOS launches even when the upgrade and the rest of the suite passed.

The suite now skips the OKD name check for those jobs when rawJobName contains installer-launch. OS stream, OSImageStream CR, and node OS checks still run:

isOKD := strings.Contains(clusterVersion.Status.Desired.Version, "okd-scos")
isLaunchJob := strings.Contains(rawJobName, "installer-launch")
if isOKD && !strings.Contains(rawJobName, "okd-scos") && !isLaunchJob {
    e2e.Failf("cluster is OKD but job name %q does not contain 'okd-scos'", jobName)
}

A follow up comment records why the exemption exists. The filter is coarse. Any other variant agnostic job whose name lacks installer-launch will still fail the same assertion.

EgressIP e2e creates EgressIP objects before nodes are labeled k8s.ovn.org/egress-assignable. OVN Kubernetes then emits NoMatchingNodeFound (“no assignable nodes for EgressIP”) on a loop until the label appears. That is the test sequence, not a production incident.

The pathological event library still treated the repeats as a failure. duplicated_event_patterns.go now registers EgressIPNoMatchingNodeFound in NewUniversalPathologicalEventMatchers. Repeats of that reason and message are allowed.

The matcher is global for jobs that load the universal set. A real assignable node outage that produces the same event will also be ignored. That is the tradeoff: quieter EgressIP e2e, less signal if the same reason fires outside those tests.

Payload shards still stall on a stuck cluster etcd operator update-setup job after node replacement. The TNF recovery change tried raising that wait from 5 minutes to 10. Runs still timed out at the new cap, so the bump was reverted. A stuck job, not a slow one. That path still needs a product side look.

Topology aware monitor softening (operator log scraper, kubelet log collector, node invariants) was split out of this merge. DualReplica and SNO disruptive tests will keep producing 503s, lease errors, and container restart storms. HA monitors stay strict. Until that follow on lands, reduced topology jobs can still fail on expected recovery noise.

None of these commits change cluster APIs or operator defaults. They change what origin CI will fail on. If you run payload jobs, cluster-bot OKD launches, DualReplica recovery, or EgressIP e2e, expect fewer false reds and a 25 minute wait after simultaneous node reboot. If you operate production DualReplica, treat the OVN chassis resync and PNCC staleness notes as test observations, not as shipped fixes.