https://wintermutecore.com/tags/site-reliability/Recent content in Site Reliability on Project WintermuteHugoen-usMon, 09 Feb 2026 12:00:00 +0200https://wintermutecore.com/posts/kubernetes-ingress-outage-postmortem/Mon, 09 Feb 2026 12:00:00 +0200https://wintermutecore.com/posts/kubernetes-ingress-outage-postmortem/<p><strong>TL;DR.</strong> A backend deployment lost all healthy pods. nginx active health checks marked the upstream pool empty. That pool was the <code>default_server</code> for ports 80 and 443, so every unmatched hostname returned 502 for 6 hours and 38 minutes. The trigger was Kubernetes-side. The blast radius was a configuration choice we made years ago. The post-incident fixes were almost all on the nginx side.</p> <p>We had a P0 outage on a public ingress tier. Two redundant nginx instances, both showing the same symptoms, both serving production traffic to dozens of hostnames. This is the writeup, sanitised and reduced to the parts that generalise.</p>