INCIDENT REPORT: POST-MORTEM #8842-B
TIMESTAMP: 2024-05-14T03:14:15Z
STATUS: CRITICAL / DEGRADED
INCIDENT LEAD: Senior SRE (Grizzled)
SUBJECT: The “Great Cloud Meltdown” of the Production Cluster
Table of Contents
1. THE INCIDENT SUMMARY
At 03:14:15Z, my pager didn’t just beep; it screamed. I was three hours into a four-hour sleep cycle, dreaming about a world where people actually read documentation. Instead, I woke up to a Slack channel that looked like a digital war zone. The monitoring dashboard, usually a boring sea of green, was a pulsating, angry neon red. Our primary production cluster—running Kubernetes v1.29.2—wasn’t just failing; it was undergoing a systematic, cascading collapse.
The first log I pulled from the jump box told the whole story. It wasn’t a subtle bug. It was a suicide pact between the microservices.
$ kubectl get pods -n production -o wide
NAME READY STATUS RESTARTS AGE IP NODE
api-gateway-7f8d9b6c5-2x9wl 0/1 CrashLoopBackOff 42 (3m ago) 14h 10.2.4.15 node-01
payment-processor-5d4f3e2-m8zqp 0/1 OOMKilled 12 (1m ago) 2h 10.2.4.88 node-01
inventory-service-9a1b2c3-k4j5h 1/1 Running 0 14h 10.2.4.92 node-01
auth-service-1a2b3c4-p9o8i 0/1 Error 5 10m 10.2.4.101 node-01
I checked the Kubelet logs on node-01. The kernel was screaming for mercy.
May 14 03:16:22 node-01 kernel: [12445.678] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4f3e2.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4f3e2.slice,task=java,pid=12345,uid=1000
May 14 03:16:22 node-01 kernel: [12445.679] Memory cgroup out of memory: Killed process 12345 (java) total-vm:8542304kB, anon-rss:4194304kB, file-rss:0kB, shmem-rss:0kB, spid:12345, oom_score_adj:985
May 14 03:16:23 node-01 kubelet[1022]: E0514 03:16:23.123456 1022 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"payment-processor\" with CrashLoopBackOff: back-off 5m0s restarting failed container=payment-processor pod=payment-processor-5d4f3e2-m8zqp_production" pod="production/payment-processor-5d4f3e2-m8zqp"
The “kubernetes best” practices we’ve been preaching for three years were ignored like a “No Smoking” sign in a 1970s bowling alley. We didn’t have a cluster; we had a high-speed collision of YAML files written by people who think “memory limits” are just suggestions.
2. THE ROOT CAUSE ANALYSIS (RCA)
I. The OOMKiller’s Scythe and the Myth of Unlimited Memory
The primary failure point was the payment-processor service. Looking at the Helm v3.14 charts used for deployment, I found that the resources block was either missing or set to values that a toddler would find optimistic. In Kubernetes v1.29.2, the scheduler relies on these numbers to make adult decisions. When you leave them blank, you’re telling the Kubelet, “Hey, feel free to let this Java app eat the entire node’s RAM.”
The Linux kernel uses cgroups v2 to manage these boundaries. When the payment-processor hit its limit—or rather, when it exhausted the node’s physical memory because no limit was set—the kernel’s OOM (Out Of Memory) Killer stepped in. It doesn’t ask questions. It doesn’t send a polite email. It looks for the process with the highest oom_score_adj and puts a bullet in its head. Because these pods were in the BestEffort QoS class (the lowest tier), they were the first to be executed.
To follow “kubernetes best” practices, every single container must have requests and limits explicitly defined. Requests are what the scheduler uses to find a home for the pod; limits are the hard ceiling enforced by the kernel. Without these, you aren’t running a distributed system; you’re running a game of Russian Roulette where every chamber is loaded. We saw the oom_score_adj at 985—that’s basically a neon sign saying “Kill Me First.”
II. Liveness Probes: The Thundering Herd and the Death Spiral
Once the payment-processor died, the api-gateway started failing. Why? Because the liveness probes were configured by someone who apparently hates our infrastructure. Here is the kubectl describe output for the failing gateway:
Liveness: http-get http://:8080/healthz delay=0s timeout=1s period=2s #success=1 #failure=3
Readiness: http-get http://:8080/ready delay=0s timeout=1s period=2s #success=1 #failure=3
A delay=0s on a liveness probe is a death sentence. The moment the container starts, the Kubelet starts hammering the /healthz endpoint. If the application takes more than two seconds to initialize its internal connection pools—which it does—the Kubelet decides the pod is dead and kills it. This is called a “death spiral.”
The “kubernetes best” approach here is to use startupProbes for heavy lifting and give the livenessProbe enough breathing room to account for temporary network jitters. By setting the periodSeconds to 2 and the failureThreshold to 3, we gave the app exactly 6 seconds to be perfect. In the real world, things are never perfect. The gateway was being killed while it was still trying to connect to the database, leading to the CrashLoopBackOff we saw in the logs. It’s like shooting a marathon runner because they didn’t finish the first mile in ten seconds.
III. Istio 1.20 and the Sidecar Memory Tax
We are running Istio 1.20 for our service mesh. It’s a powerful tool, but it’s also a hungry one. Every pod has an istio-proxy (Envoy) sidecar. In this incident, we discovered that the sidecars were consuming more memory than the actual applications. This happens because, by default, Envoy is sent the configuration for every single service in the entire cluster.
As our cluster grew to 500+ services, the Envoy memory footprint ballooned. We hadn’t implemented Sidecar resources to limit the scope of configuration discovery. Each proxy was holding a massive, redundant map of the entire network in its memory. When the node started feeling memory pressure from the OOM-happy Java apps, the sidecars pushed the node over the edge.
The “kubernetes best” practice for service meshes is to use the Sidecar Custom Resource Definition (CRD) to restrict the namespace visibility. You don’t need the payment service to know the IP address of the internal Jenkins runner. By failing to prune these configurations, we turned our networking layer into a memory-hogging anchor that dragged the whole ship down.
IV. The Scheduler’s Nightmare: Lack of Pod Anti-Affinity
When I looked at the node distribution, I realized that four of our most critical services were all scheduled on node-01. This is a classic “all your eggs in one rusted bucket” scenario. When node-01 started thrashing due to the OOM issues, it didn’t just take down one service; it took down the entire checkout flow.
Kubernetes is supposed to be a distributed system, but the scheduler isn’t a psychic. If you don’t tell it to spread pods across different physical nodes or availability zones, it will bin-pack them onto the first node it finds with available “paper” capacity.
Following “kubernetes best” practices means using podAntiAffinity with topologyKey: kubernetes.io/hostname. This forces the scheduler to act like a sensible person and put replicas on different hardware. We had three replicas of the auth-service, and all three were sitting on the same dying node. When that node’s Kubelet stopped responding because the kernel was too busy killing processes, the entire auth layer vanished. It’s the equivalent of building a three-story house and putting all the support beams in the same corner.
V. Network Policies: The Wild West of Internal Traffic
During the meltdown, we noticed a spike in traffic to the internal database that didn’t originate from the payment-processor. It turns out a compromised dev-tool pod in the same namespace was able to reach out and touch the production DB. Why? Because we had zero NetworkPolicies in place.
In a default Kubernetes environment, the network is flat. Every pod can talk to every other pod. It’s a plumber’s nightmare where every pipe is connected to every other pipe, regardless of whether it’s carrying fresh water or raw sewage.
The “kubernetes best” way to handle this is a “default-deny” ingress and egress policy. You should have to explicitly permit every single connection. If the api-gateway needs to talk to the payment-processor, you write a policy for it. You don’t just leave the door open and hope for the best. Our lack of isolation meant that the “noise” from the failing services was able to saturate the network interfaces of unrelated pods, turning a localized fire into a forest fire.
VI. Helm Template Hell and Hardcoded Secrets
Finally, we have to talk about the configuration. The Helm v3.14 charts used for this deployment were a mess of hardcoded values and missing abstractions. When we tried to scale up the inventory-service to handle the load, the new pods failed because they couldn’t pull their configuration. The ConfigMap was updated, but the pods didn’t restart because there was no checksum annotation in the deployment template.
In Kubernetes, a ConfigMap update doesn’t automatically trigger a rolling update of a Deployment. This is a known behavior, yet we keep falling for it. The “kubernetes best” practice is to include a hash of the configuration in the pod’s annotations. When the config changes, the hash changes, and Kubernetes realizes it needs to roll out new pods.
Instead, we had half the pods running on “Config A” and the other half failing on “Config B.” It was a split-brain scenario that made debugging nearly impossible. We were trying to fix a leak in a pipe while the pipe was constantly changing its diameter.
3. FIXING THE MESS
We aren’t just going to patch this; we’re going to rebuild it correctly. Here is the corrected manifest for the payment-processor. If I see another deployment without resource limits, I’m revoking your kubectl access and sending you back to manual VM provisioning.
Corrected Deployment Manifest (payment-processor-v2.yaml):
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-processor
namespace: production
labels:
app: payment-processor
version: "2.1.0"
spec:
replicas: 3
selector:
matchLabels:
app: payment-processor
template:
metadata:
annotations:
# This ensures a rollout when the config changes
checksum/config: ${CONFIG_HASH}
sidecar.istio.io/inject: "true"
labels:
app: payment-processor
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- payment-processor
topologyKey: "kubernetes.io/hostname"
containers:
- name: payment-processor
image: our-registry.io/payment-processor:v2.1.0
ports:
- containerPort: 8080
resources:
requests:
memory: "1Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 30 # Give the JVM time to wake up
periodSeconds: 15
failureThreshold: 3
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
securityContext:
runAsNonRoot: true
runAsUser: 1000
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
Enforcing Network Isolation:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-gateway-to-payment
namespace: production
spec:
podSelector:
matchLabels:
app: payment-processor
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: api-gateway
ports:
- protocol: TCP
port: 8080
Terminal Commands for Recovery:
To get the cluster back into a sane state, I had to manually prune the dead weight and force a clean rollout.
# 1. Delete the pods that are stuck in the death spiral
kubectl delete pods -n production -l app=payment-processor --force --grace-period=0
# 2. Apply the new resource-constrained manifests
kubectl apply -f payment-processor-v2.yaml
# 3. Verify the QoS Class is 'Guaranteed' or 'Burstable' (not 'BestEffort')
kubectl get pod -n production -l app=payment-processor -o jsonpath='{.items[0].status.qosClass}'
# 4. Check the Istio sidecar logs for configuration sync issues
istioctl proxy-status
4. LETTER TO THE JUNIOR DEVS
Listen closely, because I’m only going to say this once before I go back to my cave and wait for the next pager alert.
I don’t care that “it worked on your machine.” Your machine is a controlled environment with 64GB of RAM and a single user. Production is a chaotic, hostile environment where the network is unreliable, the hardware is shared, and the kernel is looking for any excuse to kill your process.
When you omit resource limits, you aren’t being “flexible.” You’re being a bad neighbor. You’re saying that your service is more important than every other service on that node. When you write a liveness probe with a zero-second delay, you’re essentially building a self-destruct button and asking the Kubelet to press it every time the wind blows.
Kubernetes is not a magic wand that fixes bad code. It is an orchestrator. If you give it a bad score, it will play a bad symphony. The “Great Cloud Meltdown” wasn’t a failure of the cloud; it was a failure of engineering discipline. We treat YAML like it’s just configuration, but in this world, YAML is the infrastructure. A single misplaced space or a missing resources block is the equivalent of a plumber using duct tape on a high-pressure steam line.
Next time you open a Pull Request, don’t tell me about the new features. Show me the resources block. Show me the securityContext. Show me the podAntiAffinity. If you can’t tell me how your service will behave when it’s under 90% load and the network is dropping 5% of packets, then you aren’t ready to ship to my cluster.
Now, if you’ll excuse me, I have a date with a bottle of ibuprofen and a very long nap. Don’t touch anything.
— The SRE
Related Articles
Explore more insights and best practices: