2024-05-14T03:14:22.891Z [ERROR] checkout-service-v2-7f89db4c5b-x9z2p: Liveness probe failed: Get “http://10.42.12.84:8080/healthz”: dial tcp 10.42.12.84:8080: connect: connection refused
2024-05-14T03:14:25.102Z [INFO] kubelet, ip-10-0-45-122.ec2.internal: Container checkout-service failed liveness probe, will be restarted
2024-05-14T03:14:28.443Z [WARN] kubelet, ip-10-0-45-122.ec2.internal: Pod checkout-service-v2-7f89db4c5b-x9z2p failed to terminate gracefully: terminated by SIGKILL
2024-05-14T03:14:30.001Z [ERROR] checkout-service-v2-7f89db4c5b-x9z2p: CrashLoopBackOff: Back-off 10s restarting failed container=checkout-service pod=checkout-service-v2-7f89db4c5b-x9z2p_production

My pager didn't just beep; it screamed. It’s that specific high-pitched frequency that cuts through a REM cycle like a serrated knife. I’ve been awake for 72 hours. My eyes feel like someone rubbed them with fiberglass insulation, and the lukewarm dregs of a twelve-hour-old espresso are the only thing keeping my heart beating at a semi-regular rhythm. 

We’re running Kubernetes v1.28.4 on a fleet of AWS m5.2xlarge instances. It was supposed to be stable. We did the upgrades. We ran the benchmarks. But production doesn't care about your benchmarks. Production is a chaotic god that demands sacrifices, and today, it wanted our entire checkout pipeline.

## H2: The 3 AM PagerDuty Screech and the Thundering Herd

The first alert was a standard "High Error Rate" on the checkout service. Simple, right? Probably a bad deploy. I checked the logs, saw the `CrashLoopBackOff`, and figured I’d just roll it back. But when I tried to run `kubectl get pods`, the terminal just sat there. Hanging. 

```bash
$ kubectl get pods -n production
# ... 30 seconds of silence ...
Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get pods)

That’s when the cold sweat started. If the API server isn’t responding, you aren’t just looking at a service failure; you’re looking at a cluster-wide cardiac arrest. I jumped into the AWS console. The m5.2xlarge nodes were pinned at 100% CPU. Not just one node. All of them.

The checkout-service had a memory leak, sure. But the real killer was the liveness probe. When the service lagged, the liveness probe failed. Kubernetes, being the dutiful executioner it is, killed the pod. Because we had imagePullPolicy: Always and a massive container image, the simultaneous restart of 200 pods triggered a thundering herd. Every node started pulling a 2GB image at the same time, saturating the NAT gateway and the internal container registry.

Table of Contents

H2: Blaming the Network (The Classic Mistake)

For the first four hours, we were convinced it was a Cilium issue. We’re running Cilium v1.14.2 with eBPF acceleration. It’s powerful, but when it breaks, it breaks in ways that make you want to go back to managing physical switches in a basement.

I looked at the node logs. The cilium-agent was throwing errors about identity allocation. We thought the CNI was failing to assign IPs to the new pods, causing the network stack to collapse.

$ kubectl -n kube-system logs -l k8s-app=cilium | grep "Error"
level=error msg="Unable to update policy" error="update identity: context deadline exceeded" subsys=daemon
level=warning msg="Failed to release IP allocation" error="context deadline exceeded" ip=10.42.12.84
level=error msg="Error while updating bpf map" error="key not found" mapName=cilium_ipcache

We spent two hours tuning bpf-map-dynamic-size-ratio and checking the conntrack tables. I was convinced the m5.2xlarge ENI limits were being hit. Each m5.2xlarge can handle a certain number of IP addresses per interface, and I thought we’d leaked so many pods that the AWS VPC CNI was choking.

But the network wasn’t the problem. The network was a symptom. The real horror was happening deeper down, in the brain of the kubernetes cluster.

H2: When Etcd Decides to Die

By 7 AM, the API server was completely unresponsive. I managed to SSH into one of the control plane nodes. I ran top. The etcd process was consuming 14GB of RAM and thrashing the disk.

When you’re running a kubernetes cluster, Etcd is the source of truth. If Etcd is slow, everything is slow. If Etcd stops, the world stops. I checked the journalctl logs for the Etcd service, and what I saw was a nightmare of raft election failures.

# journalctl -u etcd -f
May 14 07:12:10 ip-10-0-1-10.ec2.internal etcd[2102]: store.index.rebuild took 5.2s
May 14 07:12:15 ip-10-0-1-10.ec2.internal etcd[2102]: failed to send out heartbeat on time (deadline exceeded for 1.2s)
May 14 07:12:15 ip-10-0-1-10.ec2.internal etcd[2102]: server is likely overloaded
May 14 07:12:18 ip-10-0-1-10.ec2.internal etcd[2102]: apply entries took too long [4.8s for 1 entries]
May 14 07:12:18 ip-10-0-1-10.ec2.internal etcd[2102]: avoid high disk i/o latency or CPU utilization

The “apply entries took too long” message is the SRE equivalent of a flatline on a heart monitor. Our EBS volumes (gp3) were hitting their IOPS limit. Why? Because the checkout-service was crashing so fast that it was generating thousands of events per second. Kubernetes was trying to write every single “Pod Failed,” “Back-off restarting,” and “Liveness probe failed” event into Etcd.

The database was bloated with millions of event objects that hadn’t been compacted yet. The disk couldn’t keep up with the write pressure. Etcd lost consensus. The control plane went dark. We were flying a plane where the cockpit instruments had just been replaced with static.

H2: The YAML Sin That Broke the World

We finally found it around noon on day two. We weren’t looking for a “thought leadership” solution; we were looking for the idiot who forgot a decimal point. It wasn’t an idiot, though. It was a “standardized” Helm chart update that had been pushed by the platform team to “optimize” resource usage.

I finally got a describe pod to return after killing the kube-scheduler to stop the restart loop and give the API server some breathing room.

$ kubectl describe pod checkout-service-v2-7f89db4c5b-x9z2p
Name:         checkout-service-v2-7f89db4c5b-x9z2p
Namespace:    production
Containers:
  checkout-service:
    Image:      our-registry.io/checkout:v2.4.1
    Limits:
      cpu:     200m
      memory:  512Mi
    Requests:
      cpu:     100m
      memory:  256Mi
    Liveness:  http-get http://:8080/healthz delay=0s timeout=1s period=2s # <--- THE KILLER
  logging-sidecar:
    Image:      fluent-bit:2.1.0
    Resources:  {} # <--- THE OTHER KILLER

There it was. Two fatal flaws hidden in plain sight.

First, the Liveness probe had a initialDelaySeconds of 0. As soon as the container started, Kubernetes started hitting the /healthz endpoint. But the checkout-service takes 15 seconds to initialize its database connections. The probe would fail immediately, the container would be killed, and the cycle would repeat.

Second, the logging-sidecar had no resource limits. In Kubernetes, if you don’t specify limits, the container can consume as much as the node allows. The sidecar was buffering logs because the network was saturated, and it started eating RAM like a starving dog. It was the sidecar that was actually causing the OOMKills, but the checkout-service was taking the blame in the logs.

The “Aha!” moment wasn’t a moment of triumph. It was a moment of pure, unadulterated rage. We had built a system so complex that a missing initialDelaySeconds: 15 and an empty resources: {} block could bring down a multi-million dollar revenue stream.

H2: CPR on a Dying Control Plane

Recovery wasn’t a “seamless” process. It was a brutal, manual slog. We had to stop the bleeding before we could heal the patient.

Step one: We scaled the checkout-service deployment to zero. We couldn’t do it via kubectl because the API server was still choking on Etcd latency. I had to go into the Etcd member directly using etcdctl and manually delete the deployment object keys. This is the SRE version of performing open-heart surgery with a rusty spoon.

# etcdctl del /registry/deployments/production/checkout-service
# etcdctl del /registry/events/production/checkout-service... (thousands of these)

Step two: We had to clear the container runtime on the worker nodes. The m5.2xlarge nodes were cluttered with thousands of “dead” containers that containerd hadn’t had the CPU cycles to clean up.

# On each worker node:
$ sudo crictl ps -a | grep "Exited" | awk '{print $1}' | xargs sudo crictl rm
$ sudo systemctl restart containerd
$ sudo systemctl restart kubelet

Step three: We patched the YAML. We added the initialDelaySeconds, set sane resource limits for the sidecar, and changed the imagePullPolicy to IfNotPresent. We also bumped the Etcd storage limit and triggered a manual defragmentation to reclaim space.

$ etcdctl defrag --cluster
Finished defragmenting etcd member[https://10.0.1.10:2379]
Finished defragmenting etcd member[https://10.0.1.11:2379]
Finished defragmenting etcd member[https://10.0.1.12:2379]

Finally, slowly, the cluster started to breathe again. The API server response times dropped from 30 seconds to 20ms. The CPU usage on the m5.2xlarge nodes returned to a beautiful, boring 20%.

H2: The Scar Tissue (Hard-Learned Lessons)

It’s now 4 AM on day four. The cluster is stable. The developers are back to pushing code, oblivious to the fact that their “minor optimization” almost burned the house down. I’m sitting here, staring at the Grafana dashboard, waiting for the next spike that will inevitably come.

What did we learn?

Kubernetes is a feedback loop. If you don’t configure your probes correctly, the orchestrator becomes an aggressor. A liveness probe without a delay is just a suicide switch.
Sidecars are first-class citizens. If you don’t give them limits, they will take everything. There is no such thing as a “lightweight” sidecar when you’re running at scale.
Etcd is the bottleneck. You can have the fastest network and the beefiest nodes, but if your Etcd disk latency spikes, your kubernetes cluster is a paperweight. We’re moving Etcd to dedicated i3en instances with NVMe drives. No more gp3 EBS volumes for the source of truth.
Events are noise. We’re implementing an event-exporter to offload Kubernetes events to an external database. Keeping millions of “Pod Restarted” events in Etcd is like storing your trash in your brain.
Default settings are dangerous. imagePullPolicy: Always is great for dev, but in production, it’s a distributed denial of service attack against your own registry during a failure event.

I’m going home now. I’m going to sleep for fourteen hours. I’m going to dream of raw terminal output and the smell of ozone. And when I come back, I’ll start building the next layer of duct tape and baling wire to keep this kubernetes cluster from killing us all again. Because that’s the job. No fluff, no “thought leadership,” just the endless war against the next 3 AM page.

The system isn’t “evolving.” It’s just getting harder to break. And that’s the best we can hope for.

“`bash
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-45-122.ec2.internal Ready worker 14d v1.28.4
ip-10-0-45-123.ec2.internal Ready worker 14d v1.28.4
ip-10-0-45-124.ec2.internal Ready worker 14d v1.28.4

Explore more insights and best practices:

Everything is green. For now.

Mastering the Kubernetes Cluster: A Comprehensive Guide