INCIDENT LOG: 2024-05-20T03:04:12Z
LOCATION: us-east-1 (Production Cluster ‘Ares-01’)
STATUS: CRITICAL FAILURE
03:04:12Z – PagerDuty alert: KubePodNotReady (Namespace: checkout-prod).
03:05:45Z – Node ip-10-244-12-84.ec2.internal enters NotReady state.
03:06:10Z – Cascading failure detected. kube-dns latency spikes to 4500ms.
03:07:30Z – API Server unresponsive. kubectl commands timing out.
03:10:00Z – On-call engineer attempts manual recovery.
$ kubectl get pods -n checkout-prod -o wide
NAME READY STATUS RESTARTS AGE IP NODE
checkout-api-7f8d9b6c5d-2v8xl 0/1 OOMKilled 4 (12s ago) 14m 10.244.2.45 ip-10-244-12-84
checkout-api-7f8d9b6c5d-9z2pq 0/1 ImagePullBackOff 0 2m 10.244.3.11 ip-10-244-13-19
payment-gw-5d4f3a2b1c-m4n5b 1/1 Running 0 12h 10.244.2.46 ip-10-244-12-84
redis-master-0 0/1 Pending 0 1m <none> <none>
03:12:45Z – Root cause identified: A “Cloud Native” developer pushed a deployment with no memory limits and a broken sidecar configuration, triggering a node-level OOM event that took down the CNI plugin.
Table of Contents
Your Resource Limits are a Mathematical Lie
If you think setting resources.limits.cpu: "1" means your pod gets exactly one core, you shouldn’t be touching a production cluster. You are confusing abstraction with reality. Kubernetes v1.29 and v1.30 rely heavily on Linux cgroups v2, and if you haven’t bothered to understand the unified hierarchy, you are just guessing.
In cgroups v1, memory and CPU were separate silos. In cgroups v2, they are integrated. When your pod hits a memory limit, the kernel doesn’t just kill the process; it looks at the memory.high and memory.max settings. If you haven’t configured your kubelet with MemoryQoS=true, you are leaving your node’s stability to the whims of the OOM killer, which is about as reliable as a coin toss in a hurricane.
The CPU limit is even worse. It’s implemented via the Completely Fair Scheduler (CFS) quota. If you set a limit of 100ms per 100ms period, and your application is multi-threaded, you will hit that quota in 10ms and be throttled for the remaining 90ms. Your “low latency” Java app is now stuttering because you don’t understand how cpu.cfs_quota_us works.
$ kubectl describe node ip-10-244-12-84.ec2.internal
Conditions:
Type Status LastHeartbeatTime Reason
---- ------ ----------------- ------
MemoryPressure True Mon, 20 May 2024 03:15:22 +0000 KubeletHasInsufficientMemory
DiskPressure False Mon, 20 May 2024 03:15:22 +0000 KubeletHasNoDiskPressure
PIDPressure False Mon, 20 May 2024 03:15:22 +0000 KubeletHasSufficientPID
Ready False Mon, 20 May 2024 03:15:22 +0000 KubeletNotReady
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning SystemOOM 2m14s kubelet System OOM encountered, victim process: checkout-api
Normal NodeNotReady 1m58s kubelet Node ip-10-244-12-84.ec2.internal status is now. NotReady
To fix this, you need to stop using limits as a “safety net” and start using them as a hard constraint based on actual profiling. Use GOMEMLIMIT for Go apps or MaxRAMPercentage for JVM. If you don’t align your application’s internal runtime awareness with the cgroup limits, you are just begging for a 3:00 AM wake-up call. This is “kubernetes best” practice: alignment, not guesswork.
Stop Blindly Trusting Your Cloud Provider’s Default CNI
Most of you are running the default AWS VPC CNI or Azure CNI and wondering why your pod networking feels like it’s running over a 56k modem. These plugins are designed for the lowest common denominator. They eat IP addresses like candy and introduce massive overhead in the kernel’s routing table.
When you have 200 pods on a node, and each pod has its own ENI or secondary IP, the iptables rules grow exponentially. Every single packet has to traverse a chain of thousands of rules. If you haven’t switched to eBPF-based networking like Cilium, you are living in 2015. eBPF bypasses the iptables bottleneck by hooking directly into the kernel’s socket layer.
$ istioctl proxy-status
NAME CLUSTER CDS LDS EDS RDS ECDS ISTIOD VERSION
checkout-api-7f8d9b6c5d-2v8xl.checkout-prod Ares-01 STALE (14m) STALE (14m) STALE (14m) NOT SENT NOT SENT istiod-789456-abcde 1.21.0
payment-gw-5d4f3a2b1c-m4n5b.checkout-prod Ares-01 SYNCED SYNCED SYNCED SYNCED NOT SENT istiod-789456-abcde 1.21.0
Look at that STALE status. That’s what happens when your CNI can’t handle the churn. The control plane is trying to push XDS updates to Envoy, but the network is so congested with ARP requests and iptables lookups that the sidecar just gives up. You need to tune your conntrack tables. If net.netfilter.nf_conntrack_max isn’t set to at least 1,048,576 on a high-traffic node, you’re an amateur.
Your Sidecars are Parasites, Not Features
The “Service Mesh” hype train has convinced you that every pod needs a sidecar. Congratulations, you’ve just increased your memory footprint by 50% and added 10ms of latency to every hop. In the incident above, the istio-proxy was consuming 150MB of RAM just to route traffic to a local Redis instance.
If you are using Istio, you must use Sidecar resources to limit the configuration sent to each proxy. By default, every sidecar knows about every other service in the cluster. That is thousands of endpoints being shoved into a tiny Envoy process.
# A production-grade Deployment that doesn't suck
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout-api
namespace: checkout-prod
labels:
app: checkout-api
tier: backend
spec:
replicas: 3
selector:
matchLabels:
app: checkout-api
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 1
template:
metadata:
labels:
app: checkout-api
annotations:
sidecar.istio.io/inject: "true"
sidecar.istio.io/cpuLimit: "200m"
sidecar.istio.io/memoryLimit: "256Mi"
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
containers:
- name: checkout-api
image: internal-reg.io/checkout-api:v1.4.2
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
cpu: "1000m"
memory: "1Gi"
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 15
periodSeconds: 20
failureThreshold: 3
readinessProbe:
httpGet:
path: /readyz
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
Notice the securityContext. If you’re running as root, you’ve already lost. Notice the initialDelaySeconds. If you set this to 0, the kubelet will kill your app before it even finishes loading its config, leading to a CrashLoopBackOff. This is “kubernetes best” practice: give your application room to breathe before you start poking it with sticks.
DNS is Always the Culprit Because You’re Lazy
When the API server went down at 03:07Z, it wasn’t because of a bug in Kubernetes. It was because kube-dns (CoreDNS) was being hammered by 50,000 requests per second from pods trying to resolve redis-master.checkout-prod.svc.cluster.local.
By default, the ndots setting in /etc/resolv.conf is set to 5. This means for every DNS lookup of google.com, the resolver tries:
1. google.com.checkout-prod.svc.cluster.local
2. google.com.svc.cluster.local
3. google.com.cluster.local
…and so on.
You are effectively quintupling your DNS traffic for no reason. Use NodeLocal DNSCache. It runs a DNS caching agent on every node as a DaemonSet, intercepting these requests before they ever hit the network.
$ kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 180d
node-local-dns ClusterIP 10.96.0.11 <none> 53/UDP,53/TCP,9153/TCP 180d
$ kubectl top pods -n kube-system | grep coredns
coredns-64444d474-8vdfv 15m 42Mi
coredns-64444d474-9zpxl 12m 38Mi
If your CoreDNS pods are using more than 100MB of RAM, you have a leak or a massive misconfiguration in your Corefile. Stop using the autopath plugin if you don’t know exactly what it does; it’s a notorious source of race conditions in high-concurrency environments.
NetworkPolicies are Mandatory, Not Optional
Running a cluster without NetworkPolicies is like leaving your front door open and putting a sign out front that says “Free Money.” By default, every pod in Kubernetes can talk to every other pod. If your frontend web server gets compromised, the attacker has a direct line to your internal database and your metadata service (169.254.169.254).
You need a “Default Deny” policy in every namespace. Period. No exceptions.
# Production-grade NetworkPolicy: Deny All except what is explicitly allowed
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: checkout-prod
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-checkout-to-redis
namespace: checkout-prod
spec:
podSelector:
matchLabels:
app: checkout-api
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
app: redis
ports:
- protocol: TCP
port: 6379
This isn’t just about security; it’s about blast radius. When a service goes rogue and starts scanning the network, a NetworkPolicy will drop those packets at the kernel level (via iptables or eBPF), preventing the “thundering herd” from taking down your entire infrastructure.
Admission Controllers: Your Last Line of Defense Against Junior Devs
You cannot rely on “documentation” or “best practices” wikis to keep your cluster stable. People will ignore them. You must use Admission Controllers to enforce your standards. Whether it’s OPA Gatekeeper or Kyverno, you need to block any deployment that doesn’t have resource requests and limits, or that tries to run as root.
$ kubectl get events -n checkout-prod --field-selector involvedObject.name=checkout-api
LAST SEEN TYPE REASON OBJECT MESSAGE
2m Warning FailedCreate ReplicaSet Error creating: admission webhook "validation.gatekeeper.sh" denied the request: Container <checkout-api> must have memory limits defined.
If your CI/CD pipeline is the only thing checking your YAML, you are vulnerable. Someone will eventually use kubectl edit or a manual helm install and bypass your checks. The Admission Controller is the only way to ensure that what is running in the cluster actually matches your “kubernetes best” standards.
Storage is Not a Magic Bucket
If you are still using In-Tree storage drivers, you are living in a museum. Kubernetes v1.29+ has fully migrated to the Container Storage Interface (CSI). If your kube-controller-manager is still trying to talk to the AWS API to attach an EBS volume, you are asking for deadlocks.
The most common failure mode in storage is the Multi-Attach error. A node dies, the controller tries to move the volume to a new node, but the old node still has a lock on it.
$ kubectl describe pod checkout-api-7f8d9b6c5d-9z2pq
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 4m kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data]: timed out waiting for the condition
Warning FailedAttachVolume 4m attachdetach-controller Multi-Attach error for volume "pvc-12345" Volume is already used by pod checkout-api-7f8d9b6c5d-2v8xl on node ip-10-244-12-84
To mitigate this, you must use ReadWriteOncePod access mode (introduced in v1.22 and stable in v1.27) if you are on a supported CSI driver. Also, ensure your StorageClass has volumeBindingMode: WaitForFirstConsumer. If you let Kubernetes pick a zone for your volume before it knows where the pod will be scheduled, you will end up with a pod in us-east-1a trying to mount a volume in us-east-1b. It will never work.
PodDisruptionBudgets and the Illusion of Availability
You claim your app is “highly available” because you have 3 replicas. Then, the cloud provider performs a scheduled maintenance on the underlying nodes, or your cluster autoscaler decides to bin-pack the nodes, and it terminates all 3 replicas at the same time.
A PodDisruptionBudget (PDB) is the only way to tell the Kubernetes scheduler: “I don’t care what you’re doing, you are not allowed to have fewer than 2 replicas of this service online.”
# Production-grade PodDisruptionBudget
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: checkout-api-pdb
namespace: checkout-prod
spec:
minAvailable: 2
selector:
matchLabels:
app: checkout-api
Without a PDB, your “availability” is just a suggestion. During the 03:00 AM incident, the cluster autoscaler tried to drain node ip-10-244-12-84 because it was under pressure. Because there was no PDB, it killed the only healthy payment-gw pod, turning a minor slowdown into a total outage.
$ kubectl get pdb -n checkout-prod
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
checkout-api-pdb 2 N/A 1 14d
payment-gw-pdb 1 N/A 0 12h
If ALLOWED DISRUPTIONS is 0, the kubectl drain command will hang until you manually intervene or until the timeout is reached. This is exactly what you want. It forces the system to wait until a new replica is Ready (passing its readinessProbe) before killing the old one.
The Kernel is the Source of Truth
Finally, stop treating the node as a black box. You need to monitor kernel metrics. If your node-exporter isn’t scraping node_vmstat_oom_kill or node_sched_stat_waiting_seconds_total, you are flying blind.
In Kubernetes v1.30, the interaction between the kubelet and the kernel’s oom_score_adj has been refined. The kubelet assigns a score to each process based on its QoS class (Guaranteed, Burstable, or BestEffort). If you don’t set both requests and limits to the same value, your pod is Burstable. In a memory crunch, the kernel will kill BestEffort pods first, then Burstable. If you want your database to stay alive, it must be in the Guaranteed class.
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-10-244-12-84.ec2.internal Ready <none> 180d v1.30.1 10.244.12.84 Ubuntu 22.04.4 LTS 5.15.0-1058-aws containerd://1.7.13
ip-10-244-13-19.ec2.internal Ready <none> 180d v1.30.1 10.244.13.19 Ubuntu 22.04.4 LTS 5.15.0-1058-aws containerd://1.7.13
Check your kernel version. If you are running anything older than 5.10, you are missing out on critical cgroups v2 improvements and eBPF features that make Kubernetes actually manageable at scale.
Your cluster is not a “platform.” It is a complex, fragile orchestration of Linux processes, network namespaces, and filesystem mounts. If you continue to treat it like a magic cloud bucket where you just “deploy code,” it will continue to burn. Stop reading marketing blogs and start reading the kernel documentation.
Fix your YAML. Tune your sysctls. Or get used to the sound of PagerDuty.
Related Articles
Explore more insights and best practices: