The Graveyard of Our Infrastructure: A Handover Memo for the Unfortunate
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-42-12 NotReady worker 89d v1.29.2
ip-10-0-42-13 NotReady worker 89d v1.29.2
ip-10-0-42-14 Ready worker 89d v1.29.2
ip-10-0-42-15 NotReady worker 89d v1.29.2
$ kubectl describe pod checkout-api-7f8d9b6c5-x4z2l
Name: checkout-api-7f8d9b6c5-x4z2l
Namespace: prod-main
Status: Running
IP: 10.2.14.155
Containers:
checkout-svc:
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Mon, 12 Aug 2024 03:14:22 -0400
Finished: Mon, 12 Aug 2024 03:15:01 -0400
You’re reading this because I’m gone. By the time you’ve decoded my Git history, I’ll be somewhere without cell service, trying to forget the sound of a PagerDuty alert screaming at 3:14 AM. You’ve inherited the “Phoenix” cluster. It was named ironically. It doesn’t rise from the ashes; it just stays on fire.
The terminal output above is your new reality. Three out of four nodes are NotReady because the Kubelet has effectively given up on life. The one “Ready” node is currently being hammered by every single pod in the prod-main namespace, which is why the checkout-api is getting OOMKilled every forty seconds.
We ignored every kubernetes best practice because “the business needed to ship.” Now the business is losing $40k an hour, and you’re the one holding the shovel.
Table of Contents
The Resource Limit Lie and the OOM Killer’s Wrath
We treated resource requests and limits like they were optional suggestions. They aren’t. In Kubernetes 1.29, the interaction between cgroups v2 and the Linux Out-Of-Memory (OOM) Killer is precise and unforgiving. We didn’t set limits on the checkout-api. We let it “burst.”
When a container doesn’t have a memory limit, it thinks it owns the entire node. The Java heap, being the greedy pig it is, expanded until it hit the node’s physical capacity. At that point, the Linux kernel stepped in. The OOM Killer doesn’t care about your microservices architecture. It looks at the oom_score. Since we didn’t follow the kubernetes best practice of defining Guaranteed Quality of Service (QoS) classes, our critical pods had the same priority as a cronjob that calculates employee of the month.
The kernel calculates oom_score_adj based on the ratio of memory requested to memory used. Because we set requests to 256Mi but let the app use 8Gi, the score was astronomical. The kernel killed the process, the container exited with code 137, and the Kubelet—already struggling with disk pressure—just stopped responding.
The Wrong Way (What we did):
# This is a suicide note in YAML format
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout-api
spec:
template:
spec:
containers:
- name: checkout-svc
image: checkout:latest
resources:
requests:
memory: "256Mi"
cpu: "100m"
# No limits. "Let it scale," they said.
The Fixed Way:
# This is how you stop the bleeding
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout-api
spec:
template:
spec:
containers:
- name: checkout-svc
image: checkout:v1.29.4 # Use specific tags, for the love of god
resources:
requests:
memory: "2Gi"
cpu: "1000m"
limits:
memory: "2Gi" # Limits = Requests for Guaranteed QoS
cpu: "1000m"
By setting limits equal to requests, you ensure the pod is in the Guaranteed class. The OOM Killer will target almost everything else before it touches this pod. Also, notice the CPU. We didn’t use limits there either, which led to “CPU Throttling.” When the CFS (Completely Fair Scheduler) quota kicks in because you didn’t define a limit, your 10ms latency turns into 500ms.
The Readiness Probe Suicide Pact
Last Tuesday, we had a “partial” outage that turned into a total blackout. Why? Because we misconfigured readiness probes. A readiness probe tells Kubernetes when a pod is ready to accept traffic. If it fails, the pod is removed from the Service’s endpoints.
Our genius move was making the /health endpoint of the checkout-api dependent on the database connection. When the database got slightly slow, the readiness probe failed. Kubernetes, doing exactly what we told it to do, pulled the pod out of rotation. This increased the load on the remaining pods, which then slowed down, failed their probes, and were also pulled. Within three minutes, we had 50 healthy pods running, but 0 endpoints in the LoadBalancer. The cluster was a ghost town.
In a RollingUpdate, this is fatal. Kubernetes starts a new pod, waits for it to be “Ready,” then kills an old one. If the new pod never becomes “Ready” because the database is under load, the update hangs, or worse, you end up with no pods at all if maxUnavailable is set poorly.
The Wrong Way:
# A recipe for cascading failure
readinessProbe:
httpGet:
path: /health # This endpoint checks DB, Redis, and an external API
port: 8080
initialDelaySeconds: 0
periodSeconds: 1
The Fixed Way:
# Decouple your health from your dependencies
readinessProbe:
httpGet:
path: /ready # Only checks if the app server is up
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 3
startupProbe: # Use this for slow-starting legacy junk
httpGet:
path: /health
port: 8080
failureThreshold: 30
periodSeconds: 10
Use startupProbes for your slow-ass Java apps. It prevents the Kubelet from killing the container before it has even finished loading the Spring Boot context. This is a basic kubernetes best practice we ignored because we thought initialDelaySeconds was enough. It wasn’t.
CoreDNS and the Five-Second Search Penalty
If you look at the logs for the payment-gateway service, you’ll see thousands of EAI_AGAIN errors. You might think the network was down. It wasn’t. We were just DDoSing our own CoreDNS.
By default, Linux’s resolver uses a setting called ndots:5. This means if your app tries to resolve google.com, it first tries:
1. google.com.prod-main.svc.cluster.local
2. google.com.svc.cluster.local
3. google.com.cluster.local
4. google.com.us-east-1.compute.internal
5. And finally, google.com.
Each one of those is a DNS query. Our apps make hundreds of external API calls. CoreDNS was processing 50,000 queries per second, most of them for junk domains that didn’t exist. The latency spiked, the UDP packets dropped, and the apps timed out.
We should have used NodeLocal DNSCache. It runs a thin agent on every node that intercepts these queries so they don’t have to hop across the network to the CoreDNS pods. But we didn’t. We just scaled the CoreDNS deployment to 20 replicas and hoped for the best. It wasn’t enough.
RBAC: The Key to the Kingdom Under the Mat
Security was an afterthought. We gave the default service account in the prod-main namespace cluster-admin privileges because one developer couldn’t figure out why their Prometheus sidecar wasn’t scraping metrics.
In Kubernetes 1.29, the attack surface is huge. By leaving that RoleBinding in place, any pod that gets compromised—like that unpatched WordPress site the marketing team insisted on running in the same cluster—has full control over the API server. They can delete namespaces, steal secrets, or spin up Monero miners on our GPU nodes.
The Wrong Way:
# How to get fired by an auditor
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: allow-everything-to-everyone
subjects:
- kind: ServiceAccount
name: default
namespace: prod-main
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
The Fixed Way:
# Least privilege is not a suggestion
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: prod-main
name: pod-reader
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-pods
namespace: prod-main
subjects:
- kind: ServiceAccount
name: my-app-sa
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
Stop using the default service account. Create a specific ServiceAccount for every deployment. It takes ten extra lines of YAML, but it prevents a single compromised pod from becoming a cluster-wide catastrophe.
Sidecars and the 1.29 Lifecycle Revolution
We had a massive problem with our logging sidecars. The main app would finish its job and exit, but the logging sidecar would keep running, keeping the pod in a Running state forever and preventing Job completion. Or, the sidecar would take too long to start, and the main app would crash because it couldn’t find the local log socket.
Kubernetes 1.29 finally moved SidecarContainers to GA. You can now define a container in the initContainers list with a restartPolicy: Always. This ensures the sidecar starts before your main app and shuts down after it. We didn’t implement this. We were still using the old, broken pattern of just stuffing two containers in a pod and praying to the scheduler.
The Fixed Way (1.29 Native Sidecars):
apiVersion: v1
kind: Pod
metadata:
name: telemetry-pod
spec:
initContainers:
- name: network-proxy
image: proxy:v1
restartPolicy: Always # This makes it a formal sidecar
containers:
- name: main-app
image: app:v1
This ensures the network-proxy is up before main-app starts. If the proxy dies, it’s restarted. If the main app exits, the proxy is sent a SIGTERM. This is the kubernetes best practice for any auxiliary process. Use it, or enjoy your zombie pods.
The Ephemeral Storage Death Spiral
We didn’t set ephemeral-storage limits. One of the microservices had a bug where it would dump its entire debug log to a local file instead of stdout. It filled up the node’s root partition in twenty minutes.
When a node runs out of disk space, the Kubelet starts evicting pods. But it doesn’t do it gracefully. It panics. It starts killing pods to save itself. Because we didn’t have Taints or Tolerations set up to protect the system critical services, the Kubelet ended up evicting the aws-node CNI plugin and the kube-proxy.
Suddenly, the node wasn’t just out of disk; it was off the network. The API server marked it as NotReady. The scheduler saw 200 pods that needed a home and tried to cram them onto the remaining three nodes. Those nodes then ran out of memory, and the “Great Collapse of 2024” began.
You need to set requests and limits for ephemeral-storage just like you do for memory. And for the love of all that is holy, use a logrotate sidecar or stream everything to a centralized collector. Do not trust the developers to manage their own file handles.
The Silence of the Alerts
If you look at Alertmanager, you’ll find a silence titled “Temporary – Fix later.” It was created eight months ago. It silences all KubeNodeNotReady alerts for the prod-main cluster.
The previous lead (my boss, before he “retired” to a goat farm) got tired of the alerts firing every time we did a node rotation. He silenced it for “two hours” and forgot. We’ve been flying blind for the better part of a year.
The Prometheus instance is also currently OOMing because we’re trying to ingest 2 million cardinality metrics from a defunct A/B testing framework that someone forgot to turn off. The prometheus-k8s pods are stuck in a loop because the Write-Ahead Log (WAL) is corrupted from the last three hard reboots of the node they were on.
To fix this, you’ll need to:
1. Delete the corrupted WAL volume (yes, you’ll lose data, no one cares).
2. Increase the memory limit to at least 16Gi.
3. Remove the silence in Alertmanager.
4. Prepare for the 500 emails you’re about to get.
The Final Descent
The cluster isn’t broken because Kubernetes is bad. It’s broken because we treated it like a giant VPS instead of a distributed orchestrator. We ignored the kubernetes best practices because they felt like “overhead.”
We didn’t use PodDisruptionBudgets, so when the cloud provider performed maintenance on the underlying instances, Kubernetes killed all replicas of the auth-service at the same time.
We didn’t use TopologySpreadConstraints, so all our API pods ended up on the same physical rack in the same Availability Zone. When that AZ had a power flicker, the entire stack went dark.
We didn’t use NetworkPolicies, so when a dev’s laptop was compromised via a phishing link and they had kubectl access, the attacker could scan the entire internal network from a pod in the dev namespace.
The documentation is all there. The 1.29 release notes are clear. The tools exist. But you won’t have time to read them because the checkout-api just crashed again.
I left a bottle of high-proof bourbon in the bottom drawer of the desk. You’re going to need it when you realize that the etcd backup script has been failing since February because the S3 bucket it was writing to was deleted to “save costs.”
Good luck. You’re going to need more than luck, actually. You’re going to need a miracle and a lot of YAML.
Signed,
The SRE who saw too much.
Related Articles
Explore more insights and best practices: