It was 3:14 AM on a Tuesday when the first 503 errors started hitting the ingress-nginx logs, and by 3:20, the entire kubernetes cluster was a graveyard of CrashLoopBackOffs.
I’m writing this because if I don’t document this nightmare now, I’ll probably hallucinate that it was all a bad dream induced by too much caffeine and not enough REM sleep. I’ve been staring at a terminal for 72 hours. My eyes feel like they’ve been scrubbed with steel wool. The “Architects” are all tucked away in their beds, dreaming of microservices and “agility,” while I’m sitting here wondering why we ever thought running a stateful database on a distributed scheduler with a flaky CNI was a good idea.
This isn’t a “success story.” This is a post-mortem of a cluster that decided to commit ritual suicide because of a single misconfigured YAML indentation and a kernel parameter that everyone ignored.
Table of Contents
Incident Timeline: The 72-Hour Descent
03:14 UTC: PagerDuty triggers. IngressHighErrorRate. I check the logs. ingress-nginx is throwing 503s like it’s going out of style. I figure it’s just a pod that got OOMKilled. I was wrong.
03:22 UTC: kubectl get nodes returns nothing. The API server is timing out. I try to SSH into the master node. Connection refused.
03:45 UTC: I finally get into the control plane. The load average is 450 on a 16-core machine. etcd is screaming. The logs are filled with apply entries took too long.
# etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \
endpoint health
127.0.0.1:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Error: unhealthy cluster
04:10 UTC: The entire cluster is now in a NotReady state. The kubelets have lost contact with the API server.
# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
prod-worker-01 NotReady <none> 142d v1.29.2 10.0.12.4 Ubuntu 22.04.3 LTS 5.15.0-101-generic containerd://1.7.1
prod-worker-02 NotReady <none> 142d v1.29.2 10.0.12.5 Ubuntu 22.04.3 LTS 5.15.0-101-generic containerd://1.7.1
prod-master-01 NotReady control-plane 142d v1.29.2 10.0.12.1 Ubuntu 22.04.3 LTS 5.15.0-101-generic containerd://1.7.1
06:00 UTC: I realize the etcd database has hit the 8GB quota limit. Why? Because some “Senior Developer” decided to enable debug logging for a service that stores its entire state in a ConfigMap, which was being updated every 500ms. The raft log was bloated beyond recovery.
Root Cause Analysis: The Etcd Quorum Collapse
The core of the failure was the etcd cluster. In Kubernetes v1.29.2, etcd is the brain. If the brain has a stroke, the body starts twitching. We were seeing disk fdatasync latencies of over 500ms. For those of you who don’t live in the real world, etcd needs sub-10ms latency to maintain quorum.
The disk I/O was saturated because we were running the control plane on the same SSD array as the Prometheus WAL (Write-Ahead Log). When the metrics spike happened, Prometheus started churning IOPS, starving etcd. etcd lost quorum, the API server stopped responding, and the Kubelet on every node decided that since it couldn’t talk to the master, it should just stop working.
But it gets worse. Because the nodes went NotReady, the controller-manager started trying to reschedule 4,000 pods simultaneously. This created a thundering herd problem that crushed the API server the moment I tried to bring it back up.
Networking Hell: CNI Encapsulation and MTU Mismatches
While I was fighting etcd, the CNI (Container Network Interface) decided to join the party. We use Calico with VXLAN encapsulation. Somewhere in the chaos, the MTU (Maximum Transmission Unit) settings on the virtual interfaces got out of sync with the underlying physical network.
The physical network was set to 1500, but with VXLAN overhead, the packets were hitting 1550 and getting dropped. This meant that even when a pod was “Running,” it couldn’t talk to the database. The iptables rules were a mess. kube-proxy had crashed, leaving behind a legacy of 15,000 orphaned rules that were being evaluated for every single packet.
# iptables -nvL | grep KUBE-SERVICES | wc -l
15422
The latency for a simple curl between two pods was hitting 2000ms. I had to manually go into each node and flush the conntrack tables because they were full. The kernel parameter net.netfilter.nf_conntrack_max was set to the default 262144, which is a joke for a cluster this size. We were dropping packets at the edge because the connection tracking table was overflowing.
I also found that net.ipv4.ip_forward had been disabled on two of the worker nodes by a “security hardening” script that someone ran via Ansible without testing it. You can’t run a Kubernetes node without IP forwarding. It’s like trying to run a car without a fuel pump.
The PVC Deadlock: Finalizers and the CSI Ghost in the Machine
By hour 30, I had etcd back up after a manual defrag and a quota increase. But then I hit the Persistent Volume Claim (PVC) wall.
When the nodes went down, the pods were stuck in Terminating. Because the nodes were NotReady, the CSI (Container Storage Interface) couldn’t unmount the volumes. We had 200 volumes stuck in a “Multi-Attach” error state. The AWS EBS API was refusing to attach the volumes to new nodes because it thought they were still attached to the dead ones.
# kubectl describe pod api-service-7f5d6b8d-9z2x4
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedAttachVolume 14m attachdetach-controller Multi-Attach error for volume "pvc-0a1b2c3d" Volume is already used by pod...
Warning FailedMount 2m kubelet, prod-worker-03 MountVolume.SetUp failed for volume "pvc-0a1b2c3d" : rpc error: code = Internal desc = volume is in use
The only way to fix this was a manual, surgical strike on the finalizers in the API. I had to write a bash loop to patch the JSON of every stuck PVC and Pod to remove the kubernetes.io/pvc-protection and foregroundDeletion finalizers. This is dangerous. If you do this wrong, you end up with orphaned resources that you have to clean up in the cloud provider’s console like a janitor.
# Forcing deletion of stuck pods
for pod in $(kubectl get pods --all-namespaces | grep Terminating | awk '{print $2}'); do
kubectl patch pod $pod -p '{"metadata":{"finalizers":null}}' --type=merge
done
The “Fix”: Patching the Control Plane with Duct Tape
The “fix” wasn’t some elegant architectural shift. It was a series of desperate hacks to keep the cluster from collapsing again.
First, I moved the etcd data directory to a dedicated NVMe drive. I had to manually edit the /etc/kubernetes/manifests/etcd.yaml file on all three master nodes. One tiny typo—a single extra space before the hostPath—and the kubelet would refuse to start the static pod. I spent forty minutes debugging a YAML indentation error while the CTO was breathing down my neck on Zoom.
Second, I had to fix the OOMKiller behavior. The developers hadn’t set any limits on their sidecars. A logging agent was consuming 4GB of RAM because it was trying to buffer all the errors from the crash. I had to force-apply a LimitRange to the namespace to stop the bleeding.
# kubectl describe node prod-worker-01
Non-terminated Pods: (12 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
production heavy-service-54d9f7-x89 2 (25%) 4 (50%) 8Gi (50%) 16Gi (100%) 12m
monitoring prometheus-node-exporter-2z4 100m (1%) 200m (2%) 256Mi (1%) 512Mi (3%) 142d
The heavy-service pod was set to a memory limit of 100% of the node’s capacity. When the kernel hit a low-memory condition, it didn’t just kill the pod; it killed the containerd process because it had a lower oom_score_adj. I had to manually tune the kube-reserved and system-reserved settings in the Kubelet configuration to ensure the OS had enough breathing room to actually function.
Post-Mortem Architecture Critique: The Cost of Complexity
Why did this happen? It happened because we’ve built a Rube Goldberg machine and called it “infrastructure.”
We are using Kubernetes v1.29.2, which is great, but we are treating it like a magic black box. The initial design was flawed because it assumed the network was reliable and that disk I/O was infinite. We put our control plane on a shared storage network. We didn’t set PriorityClasses for critical components, so when the cluster got squeezed, the API server was evicted to make room for a dev-environment Jenkins runner.
The CNI was configured with default settings that work in a lab but fail in a high-throughput production environment. We didn’t account for the iptables overhead. With 1,500 services, every packet has to traverse a chain of rules that grows linearly. We should have moved to IPVS mode months ago, but “it wasn’t a priority.”
And don’t get me started on the “Managed” Kubernetes service we use. They manage the control plane, sure, but they don’t manage the underlying VPC routing or the disk throughput limits. When etcd started failing, their “health check” just kept rebooting the nodes, which made the raft election cycle even more unstable. It was a feedback loop of failure.
Stabilization: The Long Road Back
To stabilize the cluster, I had to perform a rolling restart of every single node, but not before I manually cleared the containerd shim leaks.
# Cleaning up orphaned shims
ps aux | grep containerd-shim | awk '{print $2}' | xargs kill -9
I then had to re-apply the CoreDNS ConfigMap. During the outage, CoreDNS had cached the NXDOMAIN responses for our internal services. Even when the pods came back up, the services couldn’t find each other because the DNS cache was poisoned with failure. I had to scale the CoreDNS deployment to zero and back up to three to clear the internal cache.
The final step was fixing the kube-proxy mess. I switched the mode from iptables to ipvs. This required loading the ip_vs, ip_vs_rr, ip_vs_wrr, and ip_vs_sh kernel modules on every single worker node.
# Verifying IPVS modules
lsmod | grep -i ip_vs
ip_vs_sh 16384 0
ip_vs_wrr 16384 0
ip_vs_rr 16384 0
ip_vs 176128 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack 172032 1 ip_vs
Once IPVS was active, the CPU load on the worker nodes dropped by 20% instantly. The KUBE-SERVICES chain was no longer being evaluated for every packet.
The Warning
If you’re reading this and you think your cluster is “safe” because you’re using a cloud provider and a fancy CNI, you’re delusional. You are one bad YAML commit away from a 72-hour shift.
Stop ignoring your etcd metrics. If your wal_fsync_duration_seconds is spiking, you are on borrowed time. Stop letting developers deploy pods without resource requests and limits. An OOMKiller is not a cleanup strategy; it’s a failure of governance.
Check your kernel parameters. If net.ipv4.ip_forward isn’t 1, your cluster is a paperweight. If your nf_conntrack_max is still at the default, you’re going to drop traffic the moment you get a marketing spike.
And for the love of everything holy, stop putting your Prometheus data on the same physical disks as your etcd state. You’re building a suicide pact, not a platform.
I’m going to sleep now. If the pager goes off again, I’m throwing it into the ocean.
Technical Appendix: Sub-System Analysis
The CNI (Calico) Deep-Dive
The failure of the CNI was exacerbated by the BGP peering flapping. When the nodes went NotReady, the BGP sessions timed out. Calico tried to withdraw the routes, but because the API server was down, the calico-node pods couldn’t update the IPAM (IP Address Management) state. This led to “IP exhaustion” errors even though we had thousands of available IPs. The fix required a manual reset of the BGP speaker nodes and a forced flush of the IPAM block affinities.
The CSI (EBS) Deep-Dive
The EBS CSI driver (v1.25.0) has a known issue where it can get stuck in a loop if the VolumeAttachment object isn’t cleaned up by the external-attacher. In our case, the external-attacher was itself stuck in a CrashLoopBackOff because it couldn’t reach the API server. This is a circular dependency that Kubernetes documentation rarely mentions. If the thing that fixes the storage needs the storage to run, you’re stuck in a deadlock.
Kube-Proxy and Iptables
The transition to IPVS is not just a performance boost; it’s a necessity for clusters exceeding 500 services. In iptables mode, the kernel uses a sequential search through the rules. In ipvs mode, it uses a hash table. The difference in O(n) vs O(1) complexity is the difference between a functional cluster and a pile of smoking silicon. We were at 1,500 services. We were already dead; we just hadn’t fallen over yet.
Resource Limits and the OOMKiller
When a pod exceeds its memory limit, the kernel sends a SIGKILL. However, if the node itself is out of memory, the kernel’s OOMKiller logic takes over. It looks at the oom_score. By default, the kubelet and containerd have a very low oom_score_adj to protect them. But if you don’t configure eviction-hard thresholds in your kubelet config, the node will lock up before the OOMKiller can even decide what to kill. We had no eviction-hard settings, so the kernel just panicked.
Final Stats:
– Total Downtime: 14 hours (partial), 4 hours (total).
– Coffee Consumed: 18 cups.
– YAML Files Edited: 42.
– Will to Live: 0.2%.
Related Articles
Explore more insights and best practices: