TO: Engineering All
FROM: Senior SRE (On-Call Rotation 4)
DATE: 04:12 AM – Day 3 of Incident #8842
SUBJECT: The Corpse of the Production Cluster and the Myth of Automation
It’s 4:12 AM. I haven’t slept since Tuesday. My vision is tunneling, and the blue light from my third monitor is starting to feel like a physical weight on my retinas. If you’re reading this, the prod-us-east-1 cluster is technically “up,” but don’t mistake uptime for health. It’s a shambling zombie held together by duct tape, shell scripts, and my own dwindling sanity.
We were told that kubernetes orchestration would solve our scaling problems. We were told it was “self-healing.” Well, I’ve spent the last 72 hours watching it “self-heal” us into a recursive death spiral that nearly wiped our entire stateful set.
This is the reality of our stack. It’s not a “solution.” It’s a high-maintenance engine that demands blood.
$ kubectl get pods -n prod-checkout
NAME READY STATUS RESTARTS AGE
checkout-api-7f8d9465b-2v9sq 0/1 CrashLoopBackOff 42 (5m ago) 3h12m
checkout-api-7f8d9465b-8n2px 0/1 Terminating 0 14m
checkout-api-7f8d9465b-9lkkj 0/1 Pending 0 2m
checkout-api-7f8d9465b-m4rtz 0/1 ImagePullBackOff 0 8m
checkout-db-0 0/1 Pending 0 48h
Look at that. That’s the heartbeat of our company. A CrashLoopBackOff followed by a Pending state that’s been sitting there for two days because the scheduler has decided that our nodes—despite having 64GB of RAM free—don’t meet the “affinity requirements.”
Welcome to the machine.
Table of Contents
1. The Control Plane is a Fragile Dictator
We upgraded to Kubernetes v1.28 last month. The changelog mentioned “refined control plane stability.” What they didn’t mention is that if your etcd quorum loses more than 10ms of latency on the disk I/O, the entire API server decides to stop talking to the nodes.
At 2:00 AM on Wednesday, the etcd leader started flapping. Why? Because a “self-healing” cronjob decided to trigger a massive backup at the same time the logging agent was dumping 40GB of garbage into /var/log. The disk choked. The quorum broke.
When the control plane loses its mind, kubernetes orchestration becomes a weapon against you. The API server started sending “delete” signals to pods because it couldn’t verify their status. It wasn’t “healing”; it was purging. I had to manually intercept the kube-apiserver process and force-feed it a config change just to get it to stop killing the database pods.
$ journalctl -u kubelet -f
-- Logs begin at Mon 2024-05-13 08:00:01 UTC. --
May 15 02:14:10 node-01 kubelet[1042]: E0515 02:14:10.123456 1042 remote_runtime.go:116] "StopPodSandbox from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" podSandboxID="a1b2c3d4..."
May 15 02:14:12 node-01 kubelet[1042]: E0515 02:14:12.987654 1042 pod_workers.go:965] "Error syncing pod, skipping" err="failed to "StartContainer" for "checkout-api" with CrashLoopBackOff: back-off 5m0s restarting failed container=checkout-api pod=checkout-api-7f8d9465b-2v9sq_prod-checkout" pod="prod-checkout/checkout-api-7f8d9465b-2v9sq"
May 15 02:14:15 node-01 kubelet[1042]: I0515 02:14:15.000123 1042 status_manager.go:567] "Failed to get status for pod" pod="prod-checkout/checkout-db-0" err="etcdserver: request timed out"
The etcdserver: request timed out error is the sound of the abyss staring back at you. If the state store is gone, the cluster is just a collection of expensive heaters. We spent four hours just trying to restore the quorum because the v1.28 kube-proxy had a race condition with the CNI that prevented the master nodes from seeing each other on the internal overlay network.
2. The Scheduler is Not Your Friend
The scheduler is supposed to be the “intelligence” of the system. In reality, it’s a blind accountant with a grudge.
We use topologySpreadConstraints to ensure high availability. Great in theory. In practice, when one availability zone (AZ) goes down—which it did—the scheduler refuses to place pods in the remaining two zones because it violates the “maxSkew” parameter.
Think about that. The system would rather have zero pods running than have an “unbalanced” distribution of pods. I watched 200 replicas of the checkout-api sit in Pending for an hour because the scheduler was waiting for a zone that didn’t exist anymore.
This is the dark side of kubernetes orchestration. It follows the rules you give it, even if those rules lead to a total service outage. I had to manually patch the Deployment to remove the constraints while the site was throwing 503s for every customer.
And don’t get me started on “bin-packing.” The scheduler tried to cram fifteen memory-intensive sidecars onto a single worker node because it only looks at “requests,” not “actual usage.” The node hit a kernel panic within ten minutes. The “self-healing” then tried to move those same fifteen pods to another node, killing that one too. It’s a digital plague. A cascading failure of “intelligence.”
3. CNI Plugins: The 3:00 AM Networking Hell
If you haven’t had to debug an encapsulated VXLAN packet at 3:00 AM, you haven’t lived. Or rather, you’ve lived a much better life than me.
Our CNI (Container Network Interface) decided that IP exhaustion was a fun game to play. We’re running a /24 per node. We had a spike in traffic. The HPA (Horizontal Pod Autoscaler) did its job—it spun up 300 pods. But the CNI didn’t release the IPs of the old, terminated pods fast enough.
The result? FailedCreatePodSandBox.
$ kubectl describe pod checkout-api-7f8d9465b-m4rtz
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox: add cmd: failed to assign an IP address to container
I’m staring at the logs, and I see the pods are stuck because the CNI’s local cache is out of sync with the API server. I had to manually ssh into forty different worker nodes to flush the iptables and restart the CNI daemonset.
This is the “seamless” experience we were promised.
In Kubernetes v1.29, they introduced some changes to how SidecarContainers are handled. We thought this would help with our service mesh. Instead, it created a dependency loop where the application container would start before the mesh proxy was ready to route traffic. So the app would try to connect to the database, fail, and crash. The mesh proxy would then see the app crashed and restart itself.
Round and round we go. The kubernetes orchestration engine just keeps spinning the wheel, oblivious to the fact that the car has no tires.
4. Resource Quotas as a Self-Inflicted DOS
We implemented ResourceQuotas to prevent one team from hogging the entire cluster. A sensible move for a “comprehensive” management strategy, right? Wrong.
During the peak of the failure, the checkout-api was trying to scale up to handle the backlog of requests. But the ResourceQuota for the prod-checkout namespace was calculated based on “requests,” not “limits.”
One of the junior devs had set the memory request to 2Gi but the limit to 4Gi. When the HPA tried to scale, the cluster-wide quota hit its ceiling. But here’s the kicker: the scheduler had already killed the old pods to make room for the new ones.
So we were stuck. We had 0 pods running, and we couldn’t start any new ones because the “quota” was being held by the “terminating” pods that couldn’t die because the CNI was hung.
It’s a deadlock. A perfect, beautiful, technical deadlock.
I had to use kubectl patch pod ... -p '{"metadata":{"finalizers":null}}' --type=merge on over a hundred pods just to force the API server to forget they existed. Do you know how dangerous that is? If those pods were actually still writing to a volume, I’d have corrupted the entire database. But at 4:00 AM, with the CTO breathing down my neck via a Slack thread, you stop caring about data integrity and start caring about the “Running” status.
5. YAML is a Programming Language for Masochists
I have spent the last three days staring at white space.
Our Helm charts have grown into 5,000-line monsters. We have templates calling templates. We have values.yaml files that are nested ten levels deep. And if you get the indentation wrong by two spaces in a ConfigMap, the entire deployment fails with an error message that is about as helpful as a “Check Engine” light in a spaceship.
# The "Simple" Deployment that broke everything
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout-api
spec:
replicas: 200 # This was a mistake
selector:
matchLabels:
app: checkout-api
template:
metadata:
labels:
app: checkout-api
spec:
containers:
- name: checkout-api
image: our-registry.io/checkout:v2.4.1
resources:
requests:
cpu: "500m"
memory: "2Gi"
limits:
cpu: "2"
memory: "4Gi"
readinessProbe: # The probe that killed us
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 30 # Why is this so high?
The failureThreshold was set to 30. That means the pod could be “unhealthy” for five minutes before the kubernetes orchestration logic decided to restart it. During those five minutes, the service was still sending traffic to a dead pod.
Why was it set to 30? Because someone, somewhere, three months ago, had a “slow startup” issue and decided to “fix” it by making the health check effectively useless.
We aren’t writing code anymore. We’re writing configuration for a system that we barely understand, hoping that the abstractions don’t leak. But they always leak. They leak through the CNI, through the CSI (Container Storage Interface), and through the Kubelet’s eviction manager.
6. The Illusion of “Self-Healing”
The most cynical part of this whole ordeal is the “self-healing” marketing.
Kubernetes doesn’t “heal” anything. It just restarts things. If your app has a memory leak, Kubernetes will restart it. If your app has a race condition, Kubernetes will restart it. If your database is corrupted, Kubernetes will restart the pod, which will then try to mount the corrupted volume, fail, and… you guessed it, restart.
It’s a loop. It’s an infinite loop of failure that masks the underlying problems until they explode into a 72-hour on-call nightmare.
We’ve built a system so complex that no single human can hold the entire state of it in their head. We’ve traded the simplicity of a “boring” VM for the “orchestration” of a thousand moving parts, each with its own failure mode, its own versioning quirks, and its own cryptic logging format.
I’m looking at the kubectl get events output one last time before I pass out on my desk.
$ kubectl get events -n prod-checkout --sort-by='.lastTimestamp' | tail -n 10
2m Warning FailedScheduling pod/checkout-api-7f8d9465b-9lkkj 0/45 nodes are available: 15 node(s) had untolerated taint {node.kubernetes.io/unreachable: }, 30 node(s) had volume node affinity conflict. preemption: 0/45 nodes are available: 45 Preemption is not helpful for scheduling.
1m Normal Scheduled pod/checkout-api-7f8d9465b-2v9sq Successfully assigned prod-checkout/checkout-api-7f8d9465b-2v9sq to node-22
1m Normal Pulling pod/checkout-api-7f8d9465b-2v9sq Pulling image "our-registry.io/checkout:v2.4.1"
45s Normal Pulled pod/checkout-api-7f8d9465b-2v9sq Successfully pulled image "our-registry.io/checkout:v2.4.1" in 15.2s (15.2s total)
30s Normal Created pod/checkout-api-7f8d9465b-2v9sq Created container checkout-api
25s Normal Started pod/checkout-api-7f8d9465b-2v9sq Started container checkout-api
10s Warning Unhealthy pod/checkout-api-7f8d9465b-2v9sq Readiness probe failed: HTTP probe failed with statuscode: 500
“Preemption is not helpful for scheduling.”
That’s the most honest thing this cluster has said to me all week. Preemption isn’t helpful. My intervention isn’t helpful. We are just observers in a chaotic system that we’ve tricked ourselves into thinking we control.
The kubernetes orchestration is finally stable, for now. The pods are green. The traffic is flowing. But the cost isn’t just the cloud bill. It’s the grey hair, the caffeine-induced tremors, and the knowledge that at any moment, a single etcd packet could drop, and we’ll be right back here in the dark.
Go home. Don’t push any code. Don’t even look at a YAML file. If you touch the cluster, I will personally revoke your kubeconfig access and move you to the documentation team.
I’m going to sleep. If the pager goes off, throw it in the river.
— Senior SRE (Out)
Related Articles
Explore more insights and best practices: