It is 4:42 AM. The sun isn’t up, but my blood pressure is. I’ve spent the last 48 hours staring at a Grafana dashboard that looked like a heart monitor of a patient in active cardiac arrest. My eyes feel like they’ve been rubbed with sandpaper, and the smell of stale, burnt coffee—the kind that’s been sitting in the pot since Tuesday—is the only thing keeping me tethered to this mortal plane.
You “clever” developers finally did it. You didn’t just break the app; you managed to turn a high-availability, multi-zone Kubernetes cluster into a very expensive heater for a data center in Northern Virginia.
Here is the post-mortem. Read it. Print it out. Eat it for all I care. Just stop doing this to me.
Table of Contents
The Incident: Death by a Thousand API Calls
At 03:14 AM on Saturday, the kube-apiserver on our production control plane decided it had seen enough of this world. It didn’t just crash; it OOM-killed itself with such violence that the underlying etcd nodes started desyncing.
Why? Because one of you geniuses decided to deploy a “lightweight” microservice that didn’t have resource limits and, for reasons known only to God and your poorly written Go code, decided to list every single Secret in the namespace every 500 milliseconds.
When the API server died, the Kubelets panicked. When the Kubelets panicked, they started killing pods. When the pods died, the ReplicaSets tried to recreate them. But since the API server was in a recursive loop of death, the scheduler couldn’t bind the pods to nodes.
This is what my terminal looked like for six hours:
$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default awesome-new-app-6f9d5f7d57-abc12 0/1 CrashLoopBackOff 142 (4m ago) 2h
default awesome-new-app-6f9d5f7d57-def34 0/1 Terminating 0 2h
kube-system kube-apiserver-ip-10-0-1-10.ec2.internal 0/1 Error 15 48h
kube-system kube-controller-manager-ip-10-0-1-10 0/1 CrashLoopBackOff 12 48h
monitoring prometheus-k8s-0 0/2 ImagePullBackOff 0 5h
The ImagePullBackOff was a nice touch—turns out that when the control plane is melting, the NAT gateway also decided to hit its connection limit because of the millions of retries.
This isn’t v1.18 anymore. We are on v1.29. If you are still using policy/v1beta1 for your PodDisruptionBudgets or ignoring the fact that selfLink is gone, you aren’t just behind the times; you are a liability.
1. The Resource Limit Lie: Why ‘Best Effort’ is a Suicide Pact
You love to talk about “elasticity.” You think Kubernetes is a magical infinite bucket of RAM. It isn’t. When you omit requests and limits from your YAML, you are telling the scheduler, “I don’t know what I’m doing, just figure it out.”
Kubernetes assigns a Quality of Service (QoS) class to every pod. If you don’t define limits, you get BestEffort. Do you know what happens to BestEffort pods when a node gets tight on memory? They are the first ones lined up against the wall and shot by the OOM Killer.
The “Clever” Developer YAML (The Suicide Pact):
apiVersion: apps/v1
kind: Deployment
metadata:
name: memory-hog
spec:
template:
spec:
containers:
- name: app
image: our-registry.io/app:latest # FIREABLE OFFENSE
# No resources defined. "It'll be fine," you said.
The SRE-Approved YAML (The “I Want to Sleep” Version):
apiVersion: apps/v1
kind: Deployment
metadata:
name: stable-app
spec:
template:
spec:
containers:
- name: app
image: our-registry.io/app:v1.4.2 # Specific tag, you cowards
resources:
requests:
memory: "256Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "500m"
If your limit is significantly higher than your request, you are creating “burstable” pods. That’s fine for a dev environment, but in prod, it leads to bin-packing nightmares. If the node is overcommitted and your pod tries to hit its limit, the kernel’s OOM score for your process skyrockets. I spent three hours debugging why the order-processor kept vanishing, only to find out it was trying to burst to 2GB on a node that only had 200MB of slack.
2. Probes that Kill: How Your Liveness Check Created a Cascading Failure
I am begging you: stop using livenessProbes as a crutch for bad code. A liveness probe is meant to catch a deadlocked process, not to restart an app because the database is slow.
During the outage, the payment-gateway service started lagging because the API server was slow. What did your liveness probe do? It saw a 2-second delay, decided the pod was “dead,” and killed it.
This happened across all 50 replicas simultaneously.
Now, instead of a slow service, we had no service. And when the new pods came up, they had to perform their “startup routine”—which involves hitting the database to cache schema. Fifty pods hitting the DB at once killed the DB.
The “Death Spiral” Probe:
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 3
periodSeconds: 5
failureThreshold: 1 # Why would you do this?
If your failureThreshold is 1, a single network hiccup or a long Garbage Collection (GC) pause kills your container. Congratulations, you’ve built a self-destruct mechanism. Use startupProbes for heavy lifting and give your readinessProbes some breathing room. A pod that isn’t “Ready” just stops receiving traffic; a pod that isn’t “Live” gets executed. Learn the difference.
3. RBAC is Not Optional: Stop Giving ‘Cluster-Admin’ to Your CI/CD Bot
I found a ServiceAccount in the dev-tools namespace yesterday named jenkins-deployer. It had a ClusterRoleBinding to cluster-admin.
Do you know what that means? It means if anyone compromises that Jenkins instance—which, let’s be honest, is running a version of Java from the Mesozoic era—they own the entire cluster. They can delete the kube-system namespace. They can steal the TLS certs. They can spin up 5,000 crypto-miners on our Spot instances.
RBAC (Role-Based Access Control) is painful because it requires you to actually know what your application does. I know that’s a lot to ask. But “I need to list pods” does not mean you need verbs: ["*"] on resources: ["*"].
The “I’m Lazy” RBAC:
rules:
- apiGroups: [""]
resources: ["*"]
verbs: ["*"]
The “I Actually Care About Security” RBAC:
rules:
- apiGroups: [""]
resources: ["pods", "services"]
verbs: ["get", "list", "watch"]
Stop using the default ServiceAccount for everything. It has no permissions by design. Don’t “fix” it by adding permissions to it. Create a scoped ServiceAccount for your deployment. It takes ten lines of YAML. Just do it.
4. The Hidden Cost of “kubernetes best” Intentions
Most of you treat kubernetes best practices like the “Terms and Conditions” of a software update—you scroll past the reality of how distributed systems actually fail just to click ‘Accept’ on a broken deployment. You read a medium article about “Service Meshes” and suddenly you’re injecting Istio sidecars into every namespace without looking at the overhead.
We are paying a “Sidecar Tax” of 1.5 vCPUs per node just to handle the Envoy proxies for services that literally only talk to one other service over plain HTTP. You want “observability,” but you won’t even instrument your code with Prometheus metrics. You expect the infrastructure to magically tell you why your Python script is leaking memory.
And don’t get me started on Helm. Helm is a great way to install software you don’t understand. I’ve seen charts where the values.yaml is 4,000 lines long, and you’re just changing replicaCount and hoping the other 3,999 lines don’t blow up the cluster. When the outage hit, I tried to use helm rollback, but the release was in a PENDING_UPGRADE state because the last five deploys failed their readiness checks. I had to manually delete the Secret holding the Helm release state just to get the cluster to stop trying to deploy a broken image.
5. Networking Sprawl: When Your CNI Decides to Stop Routing Traffic
Kubernetes networking is a lie built on top of iptables and BGP. Our CNI (Container Network Interface) plugin, Cilium, is fantastic—until you overwhelm the conntrack table on the underlying Linux nodes.
During the “Great API Meltdown,” the number of orphaned connections reached 65,536. At that point, the kernel started dropping packets. Not just for the broken app, but for everything. CoreDNS couldn’t resolve internal names. The Kubelet couldn’t heartbeat to the control plane.
$ kubectl describe pod coredns-78fcdf6894-q4v7b -n kube-system
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 2m (x24 over 10m) kubelet Liveness probe failed: Get "http://10.0.1.15:8080/health": dial tcp 10.0.1.15:8080: connect: connection refused
The reason? You guys are using ndots:5 in your dnsConfig. Every time your app tries to talk to database.internal, it first tries to resolve:
1. database.internal.my-namespace.svc.cluster.local
2. database.internal.svc.cluster.local
3. database.internal.cluster.local
4. database.internal.us-east-1.compute.internal
That’s four DNS queries for every single connection. Multiply that by 1,000 requests per second, and you’re DDoS-ing our own CoreDNS. If you’re talking to a service in the same namespace, just use the service name. If you’re talking to an external URL, use a FQDN with a trailing dot to skip the search path.
6. Storage Classes and the Myth of Statelessness
“We’re cloud-native,” you said. “Our apps are stateless,” you said. Then why do I see 400 PersistentVolumeClaims (PVCs) in the production namespace?
The biggest headache during the recovery was the ReadWriteOnce (RWO) access mode. We have a multi-AZ cluster. When Node A in us-east-1a died, the scheduler tried to move the pod to Node B in us-east-1b. But the EBS volume is locked to us-east-1a.
The pod stayed in ContainerCreating for 45 minutes because the volume was “already attached to another node.”
The Error Log from Hell:
Warning FailedAttachVolume 5m attachdetach-controller Multi-Attach error for volume "pvc-12345" Volume is already used by pod "old-app-pod"
If you need state, use a managed service like RDS or S3. If you insist on running a database inside K8s, you better have a StatefulSet and a very good reason. And for the love of all that is holy, stop using strategy: Recreate for your deployments just because your app can’t handle two versions of the schema running at once. That’s not a deployment strategy; that’s an admission of failure.
A RollingUpdate is the standard. If your app can’t handle a RollingUpdate, your app isn’t ready for Kubernetes.
The “image: latest” Fireable Offense
I am making this its own section because I am tired of explaining it. If I see image: latest in a production manifest again, I am revoking your git push rights.
When you use latest, you have no idea what code is actually running. If a node reboots and pulls the image again, it might get a different version than the other nodes in the same deployment. Now you have a distributed system running two different versions of the code with the same version string. Debugging that is like trying to find a black cat in a coal cellar at midnight.
Use immutable tags. Use the Git SHA. Use a semantic version. Just don’t use latest.
Why I’m Still Here
I spent 48 hours fixing this because I actually care about the uptime. I care about the fact that our customers couldn’t process payments for four hours. I care that the junior SRE on call was crying because he thought he’d deleted the production database when, in reality, the networking was just so hosed he couldn’t see it.
Kubernetes is a tool, not a personality trait. It is incredibly powerful, but it is also a loaded gun pointed directly at your foot. Every line of YAML you write is a decision. If you make those decisions based on “it worked in my local Minikube,” you are going to keep waking me up.
I’m going home. I’m going to sleep for 14 hours. When I come back, I expect to see every one of your deployments updated with resource requests, proper probes, and pinned image tags.
If I see another OOMKilled event caused by a BestEffort pod, I’m not fixing it. I’m just going to let the cluster burn and send you the bill for the AWS bill.
Tools you should actually learn to use before your next PR:
– Stern: To tail multiple pod logs without losing your mind.
– Kube-capacity: To see how much of the cluster you’re actually wasting.
– Kustomize: To manage your YAML without the Helm-chart-hell.
– Prometheus/Grafana: Look at the container_memory_working_set_bytes metric. It’s the only one that matters for OOMs.
Fix your YAML. Leave me alone.
— Your exhausted SRE.
Related Articles
Explore more insights and best practices: