Timestamp: 03:14 UTC. The pager went off because the API server decided it didn’t feel like responding to heartbeats anymore.
$ kubectl get nodes
Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get nodes)
$ kubectl get pods -A
Unable to connect to the server: dial tcp 10.0.0.1:6443: i/o timeout
I stared at the terminal, the blue light of the monitor searing my retinas while the rest of the house slept. This is the reality of the modern “kubernetes cluster”. We were told it would be self-healing. We were promised that the “reconciliation loop” would be our mechanical savior, a tireless deity that would ensure our desired state always matched our actual state.
Instead, I’m looking at a dead control plane and a Slack channel full of automated alerts from Prometheus that are essentially screaming “Everything is on fire and I don’t know why.”
The culprit, as it turns out, wasn’t a massive traffic spike or a sophisticated DDoS attack. It was a single, misconfigured PodDisruptionBudget (PDB) combined with a minor version upgrade on a managed node group in a v1.28.x environment. We tried to drain a node. The scheduler tried to move a pod. The PDB said “No.” The automation tried again. And again. And again. Eventually, the kube-controller-manager got stuck in a logic loop that bloated the etcd transaction log until the disk I/O on the master nodes hit 100% utilization.
Welcome to the future of infrastructure. It’s just a thousand shell scripts in a trench coat, pretending to be an operating system.
Table of Contents
The Etcd Ghost in the Machine
When people talk about a “kubernetes cluster”, they usually focus on the shiny parts—the YAML, the containers, the “serverless” abstractions. They rarely talk about the brittle, temperamental heart of the beast: etcd.
In this specific incident, our etcd quorum collapsed because of a latency spike that would have been a rounding error in any other system. But in a “kubernetes cluster”, etcd is the only source of truth. If the disk takes 15ms instead of 5ms to commit a proposal, the whole house of cards starts to wobble.
# journalctl -u etcd -f
Mar 14 03:16:22 ip-10-0-42-12 etcd[2104]: store.index: error: failed to get key "/registry/events/default/log-processor-7f8d9b.17c2a..."
Mar 14 03:16:23 ip-10-0-42-12 etcd[2104]: raft: 8e9234c21a... is starting a new election at term 42
Mar 14 03:16:24 ip-10-0-42-12 etcd[2104]: raft: 8e9234c21a... failed to send message to 9f21... (exceeded max message size)
The “max message size” error is the kiss of death. It means your cluster state has become so bloated with event metadata—mostly garbage generated by failing health checks—that the nodes can no longer synchronize. We’ve built a system where the diagnostic data about the failure is the very thing that prevents the system from recovering.
The “kubernetes cluster” architecture relies on a consensus algorithm (Raft) that is notoriously sensitive to network jitter. We run these things on virtualized hardware, with virtualized networking, on top of “bursty” SSDs, and then we act surprised when the consensus breaks. I spent two hours manually compacting the etcd keyspace using etcdctl just to get the API server to stop timing out. This isn’t “cloud-native” engineering; it’s digital archaeology, digging through layers of compacted JSON to find the one record that’s stalling the entire pipeline.
The Networking Layer of Lies (CNI and the Iptables Nightmare)
Once I got the control plane back, I realized the “kubernetes cluster” was still a ghost town. The pods were “Running,” but they couldn’t talk to each other.
This brings us to the CNI (Container Network Interface). In our case, we’re running a standard VPC-CNI on AWS, which is supposed to be “robust.” What they don’t tell you is that every time a node joins or leaves the “kubernetes cluster”, the CNI has to perform a frantic dance of attaching ENIs (Elastic Network Interfaces), assigning secondary IP addresses, and updating the local iptables or ipvs rules.
If you’ve never looked at the iptables output on a production node with 200+ services, don’t. It’s a horror story. Thousands of lines of rules, chains, and jumps that the kube-proxy has to manage.
# iptables -L -t nat | wc -l
14282
Fourteen thousand rules. Every packet that enters that node has to traverse a significant portion of that list just to find out which local socket it belongs to. We’ve replaced a simple router with a massive, distributed, eventually-consistent lookup table that breaks if the kube-proxy pod restarts too quickly.
In this incident, the CNI had entered a “race condition” where it thought it had assigned an IP address to a new pod, but the underlying VPC hadn’t finished plumbing the route. The pod started, the readinessProbe failed because the network wasn’t there, the kubelet killed the pod, and the cycle started over. Each cycle left a “zombie” veth pair on the host. By 04:00 AM, the node had run out of available network interfaces, not because we were out of capacity, but because the “kubernetes cluster” was too fast for its own underlying infrastructure.
The Scheduler: An Over-Confident Matchmaker
The “kubernetes cluster” scheduler is a piece of software that thinks it’s much smarter than it actually is. It looks at CPU and Memory “requests” as if they are hard facts, rather than the wild guesses made by developers who haven’t looked at a resource graph in six months.
During the recovery, I watched the scheduler try to bin-pack our heaviest Java microservices onto a single worker node because that node happened to have the most “available” memory. It didn’t account for the fact that these services are all I/O intensive and would immediately starve the kubelet of disk cycles.
Here is the manifest that nearly killed us. A developer, in their infinite wisdom, had applied this PodDisruptionBudget:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: critical-app-pdb
spec:
minAvailable: 1
selector:
matchLabels:
app: critical-app
On the surface, it looks fine. “Keep at least one pod alive.” But the critical-app only had one replica. When the node it was on needed to go down for a security patch, the “kubernetes cluster” entered a deadlock. The NodeDrainer couldn’t remove the pod because it would violate the PDB. The AutoScaler couldn’t kill the node because it wasn’t empty. The UpgradeController just kept retrying the drain every 10 seconds, flooding the API server with requests.
This is the “magic” of automation. It’s a system that will happily spend $5,000 in compute time trying to move a single 100MB container because a YAML file told it to, without ever stopping to ask a human, “Hey, this isn’t working, should I stop?”
The Kubelet and the Cgroup v2 Migration Trap
We recently moved the “kubernetes cluster” to nodes running v1.28 on an Amazon Linux 2023 base, which defaults to cgroup v2. You’d think this would be a “seamless” transition (to use a word I despise). It wasn’t.
The way kubelet interacts with the Linux kernel’s resource accounting changed. Suddenly, our “OOMKilled” (Out of Memory) events weren’t just killing the container; they were occasionally causing the entire containerd runtime to hang because of how the memory pressure was being reported up the chain.
I spent thirty minutes at 04:30 AM looking at dmesg logs on a worker node, trying to figure out why a Python script was able to lock up the entire node’s container runtime.
[ 1420.4821] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=kubepods-besteffort-pod...
[ 1420.4822] Memory cgroup out of memory: Killed process 12904 (python3) total-vm:1048576kB, anon-rss:512000kB, file-rss:0kB, shmem-rss:0kB
[ 1420.5100] containerd-shim[12800]: segfault at 0 ip 000055...
The “kubernetes cluster” is supposed to isolate these failures. That’s the whole point of containers. But the abstraction is leaky. The kernel doesn’t care about your “Namespace” or your “ResourceQuota” when it’s starving for pages. It will kill whatever it needs to kill to stay alive, and often, the first victim is the very agent (kubelet) that is supposed to be managing the mess.
The Hidden Cost of Managed “Kubernetes Cluster” Services
We use a managed service because we thought it would mean fewer 3:00 AM pages. That was a lie. Managed services just move the “black box” further away from you.
When the API server went down, I couldn’t check the logs. I had to wait for the cloud provider’s “Control Plane Logging” to eventually ingest the data into a separate log-aggregation service, which has a 5-minute ingestion delay. When you’re in the middle of a cascading failure, 5 minutes is an eternity.
The “kubernetes cluster” managed by a provider often hides the etcd metrics. You don’t see the disk latency until it’s already triggered a leader election. You don’t see the CPU throttling on the API server until the kubectl commands start failing. We are paying a premium for the privilege of being blindfolded during an emergency.
And let’s talk about the “Add-ons.” The VPC-CNI, the CoreDNS autoscaler, the storage drivers. These are all just more pods running in your “kubernetes cluster”, often with “PriorityClass: system-cluster-critical”. If one of these “managed” components has a bug—like the time a CoreDNS update caused a circular dependency in DNS resolution—it can take down your entire application stack, and you have no way to roll it back because the managed service “owns” that configuration.
The Reconciliation Loop as a DoS Vector
The core philosophy of a “kubernetes cluster” is the reconciliation loop.
1. Observe state.
2. Compare to desired state.
3. Act to fix the difference.
This sounds great in a textbook. In practice, it’s a Distributed Denial of Service attack waiting to happen. When our cluster started failing, every single controller in the system—the DeploymentController, the ReplicaSetController, the EndpointSliceController—all noticed the failure at the same time.
They all started hammering the API server with “Update” and “Patch” requests. The API server, already struggling with etcd latency, started queueing these requests. The kube-apiserver has a “Priority and Fairness” (APF) configuration, but in v1.29, if you haven’t tuned those “FlowSchemas” perfectly, the system will prioritize “system” traffic so heavily that you, the administrator, can’t even get a get nodes request through to see what’s happening.
We’ve built a system that prioritizes its own internal bureaucracy over the actual work it’s supposed to be doing. The “kubernetes cluster” was so busy telling itself that it was broken that it didn’t have any resources left to actually fix itself.
Hard-Won Lessons from the Trenches
It’s 05:45 AM now. The “kubernetes cluster” is stable, mostly because I manually deleted half the failing pods and scaled the deployments to zero to give the API server room to breathe. My coffee is cold, and I have a meeting in three hours to explain why “the cloud” failed.
If you’re going to run a “kubernetes cluster” and you want to sleep, here is the survival guide.
1. Your PDBs are a Suicide Note.
Never, ever set minAvailable to equal your total replica count. If you have 2 replicas, minAvailable should be 1. If you have 1 replica, don’t use a PDB at all. The “kubernetes cluster” will respect your PDB until it kills the entire node group. It is a suicide pact, not a safety net.
2. Etcd is Not a Database; It’s a Bomb.
Monitor your etcd_disk_wal_fsync_duration_seconds and etcd_network_peer_round_trip_time_seconds like your life depends on it, because it does. If those numbers trend upward, your “kubernetes cluster” is about to have a heart attack. Use the fastest NVMe drives you can find for etcd storage. Do not use “General Purpose” cloud storage.
3. Resource Limits are Lies.
A limits: memory: 1Gi doesn’t mean your app gets 1Gi. It means the kernel will kill your app the millisecond it touches 1.00001Gi. Always set your requests equal to your limits to avoid the “Burstable” QoS class. “Burstable” is just another word for “The scheduler will over-provision this node and then the OOMKiller will pick a random victim when things get tight.”
4. The CNI is the Weakest Link.
The networking in a “kubernetes cluster” is a fragile overlay. If you are using AWS, increase the WARM_IP_TARGET in your VPC-CNI settings so you always have a buffer of IPs. If you’re using Calico or Flannel, watch your MTU settings. A 20-byte mismatch in MTU will result in “random” connection resets that will haunt your dreams and are nearly impossible to debug with standard tools.
5. Trust No Automation.
The “Operator Pattern” is just a way to move manual mistakes into code. Every “Operator” you add to your “kubernetes cluster”—whether it’s for Prometheus, Postgres, or Certificates—is another reconciliation loop that can go rogue. We had an “Auto-healing” operator that once deleted 40 nodes because it misinterpreted a temporary network partition as a total hardware failure.
6. Keep the Control Plane Lean.
Stop putting everything in the “kubernetes cluster”. If you can run it on a managed database service or a simple VM, do it. The more you cram into the cluster, the more complex the dependency graph becomes. When the “kubernetes cluster” fails, you want your database and your logging to be outside the blast radius.
7. Learn the Low-Level Tools.
When the API server is down, kubectl is useless. You need to know how to use crictl to inspect containers directly on the node. You need to know how to read journalctl logs. You need to understand iptables-save. If you only know the “kubernetes cluster” through the lens of the API, you are not an SRE; you are a passenger.
The “kubernetes cluster” is a marvel of engineering, but it is also a monument to our own hubris. We took the simple problem of “running a program on a computer” and added ten layers of abstraction, three layers of virtualized networking, and a distributed consensus algorithm. We got “scalability,” sure. But we traded our sleep for it.
I’m going to bed. If the pager goes off again, I’m quitting and becoming a carpenter. Wood doesn’t have reconciliation loops. Wood doesn’t have a CNI. Wood just sits there. And right now, that sounds like heaven.
Related Articles
Explore more insights and best practices: