2024-10-24T03:14:07.821Z [ERROR] controller-runtime.manager.controller.pod-lifecycle-controller: Reconciler error {"error": "failed to allocate IP from CIDR block 10.128.0.0/24: address already in use by peer-vpc-02-us-east-1", "stacktrace": "github.com/kubernetes-sigs/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/github.com/kubernetes-sigs/[email protected]/pkg/internal/controller/controller.go:329"}
2024-10-24T03:14:08.001Z [WARN] kubelet: Readiness probe failed: HTTP probe failed with statuscode: 503 for container "api-gateway" (pod "api-gateway-7f8d9b4c5-xk2m9_prod")
2024-10-24T03:14:08.112Z [FATAL] kernel: [192837.441] Out of memory: Kill process 29384 (envoy) score 942 or sacrifice child
Table of Contents
INCIDENT-882: The Ghost of Overlapping Subnets
It’s 3:15 AM. I’m staring at a monitor that’s too bright for my aging retinas, smelling the phantom scent of ozone and dust that used to permeate the data centers where I started my career. Back then, if a Sun Fire V240 went down, you knew why. You could hear the fans screaming or see the “Service Required” LED glowing like a malevolent eye. Today, I’m chasing ghosts in a virtualized, abstracted, containerized hellscape where the “hardware” is just a line item on a bill I’m not allowed to see.
The log above is the result of what happens when “marketing-led engineering” meets the reality of the networking stack. Someone in Product decided we needed a “multi-region global footprint” by the end of Q3. So, a junior developer—bless their heart—fired up a Terraform module they found on a medium post and applied it to our production environment. They didn’t check the existing VPC peering routes. They didn’t look at the routing tables. They just assumed the cloud would “handle it.”
The result? A CIDR block collision that effectively black-holed half of our internal API traffic. The CNI plugin, trying to be helpful, kept attempting to assign IPs from a range that was already being advertised by a legacy VPC we use for database replication.
# Checking the routing table on the node... or what's left of it.
$ ip route show table main
default via 10.128.0.1 dev eth0 proto dhcp src 10.128.0.45 metric 100
10.128.0.0/24 dev eth0 proto kernel scope link src 10.128.0.45
10.128.0.0/24 via 10.200.1.1 dev tun0 # <--- There's the culprit. Overlapping route from the VPN tunnel.
169.254.169.254 dev eth0 proto php scope link
We followed every devops best practice listed in the glossy brochures. We had CI/CD. We had automated testing. We had “Infrastructure as Code.” But Terraform 1.5+, for all its fancy import blocks and check assertions, doesn’t know that your legacy VPN gateway is going to squat on a subnet you just tried to provision for a new EKS node group.
The terraform plan looked clean because the VPN gateway isn’t managed by Terraform. It’s a “manual hotfix” from 2019 that became permanent infrastructure. This is the reality of the “modern” stack: a thin veneer of YAML covering a mountain of technical debt and undocumented manual changes.
The Readiness Probe Death Spiral
While the networking team (which is just me and a guy named Dave who is currently on a flight to Tokyo) tries to unfurl the routing mess, the Kubernetes scheduler is making things worse. Because the CIDR collision is dropping packets, the readiness probes for the api-gateway pods are failing.
In K8s 1.30, the kubelet is more aggressive than ever. It sees the 503, marks the pod as Unready, and pulls it from the Service endpoint list. Now, the remaining pods—which are also struggling with intermittent packet loss—are being slammed with 100% of the traffic.
$ kubectl describe pod api-gateway-7f8d9b4c5-xk2m9
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 2m14s (x24 over 12m) kubelet Readiness probe failed: Get "http://10.128.0.88:8080/healthz": dial tcp 10.128.0.88:8080: connect: connection refused
Normal Killing 2m14s kubelet Container api-gateway failed liveness probe, will be restarted
The “self-healing” nature of Kubernetes is currently a circular firing squad. The liveness probe fails because the application is too busy handling a massive spike in retries from the frontend, which is also failing because the backend is unreachable. We’ve built a system that is so “resilient” it will happily kill itself trying to stay alive.
I’ve seen this before. In 2005, we called it a “thundering herd.” In 2024, we call it “cloud-native scaling.” It’s the same physics, just with more layers of indirection. We’ve replaced simple socket errors with complex, multi-layered failure modes that require a PhD in distributed systems to debug at 3 AM.
Why Your CI/CD Pipeline is a Glorified Bash Script
The “devops best” crowd loves to talk about “shifting left.” They want developers to own the infrastructure. But when you give a developer a Terraform manifest, they don’t see a representation of physical hardware, subnets, and BGP sessions. They see a configuration file. They treat it like a JSON object.
We’ve moved from “Infrastructure as Code” to “Infrastructure as Manual Hotfixes” disguised as code. Look at our current pipeline. It’s 4,000 lines of YAML spread across twenty different repositories. It uses “reusable workflows” that are so abstracted no one knows what they actually do.
When the deployment failed tonight, the CI/CD pipeline reported a “Success.” Why? Because the helm upgrade command returned an exit code of 0. Helm doesn’t care if the pods actually start; it only cares that the Tiller-less API server accepted the new manifest.
# The "Successful" deployment log
$ helm upgrade --install api-gateway ./charts/api-gateway --namespace prod --wait --timeout 5m
Release "api-gateway" has been upgraded. Happy Helming!
NAME: api-gateway
LAST DEPLOYED: Thu Oct 24 03:10:02 2024
NAMESPACE: prod
STATUS: deployed
REVISIONS: 142
“Happy Helming!” The sheer audacity of that message while the production environment is actively melting down. The --wait flag timed out, but because of a bug in the wrapper script someone wrote three years ago, the error was swallowed and the pipeline moved on to the next stage: “Send Slack Notification.”
So, while I’m digging through tcpdump outputs, the rest of the company is getting a message saying the new feature was successfully deployed. This is the “synergy” of modern engineering.
The Silent Failure of Silent Retries
One of the most insidious things we’ve introduced in the last five years is the service mesh. We’re running Istio because someone read a whitepaper about “zero trust networking” and “observability.” Now, every single pod has an Envoy sidecar.
When the CIDR collision started dropping packets, Envoy did what it was programmed to do: it started retrying. Exponential backoff, jitter, the whole nine yards. But it did it silently. The application code didn’t see a “Connection Refused” or a “Timeout.” It just saw increased latency.
By the time the latency hit the 99th percentile and triggered an alert, the Envoy sidecars had already opened ten thousand connections to a non-existent endpoint, exhausting the conntrack table on the underlying worker nodes.
# Checking conntrack on the node
$ sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_count = 262144
net.netfilter.nf_conntrack_max = 262144
# We are at the limit. No new connections can be tracked. Everything drops.
This is the “observability” we were promised. We have dashboards that show “Gold Signals,” but they don’t show the kernel-level exhaustion that’s actually killing the system. We’ve built so many layers of abstraction that we’ve lost sight of the machine. We’re debugging YAML when we should be debugging the kernel.
Back in the Solaris days, I could run dtrace and see exactly what the kernel was doing with a thread. Now, I have to jump through three different jump-hosts, get temporary IAM credentials, exec into a “debug container” that has the right tools, and hope that the containerd runtime hasn’t already recycled the namespace I’m trying to inspect.
The Containerd Transition and the Loss of Tooling
Speaking of containerd, the transition away from the Docker shim was supposed to be “seamless” (a word I loathe). In reality, it broke every single one of our legacy monitoring scripts that relied on the Docker socket.
We’re running K8s 1.30 now. The dockershim is a distant memory, and we’re stuck with crictl. It’s technically superior, sure. It’s more “standard-compliant.” But it’s also another layer of friction when you’re trying to figure out why a process is stuck in D state (uninterruptible sleep).
I tried to strace the hanging process in the api-gateway pod. In the old days, I’d just find the PID on the host and attach. Now, with cgroups v2 and the way containerd handles namespaces, it’s a chore.
# Trying to find the actual PID of the process inside the container
$ crictl inspectp 7f8d9b4c5 | grep pid
"pid": 29384
$ strace -p 29384
strace: attach: ptrace(PTRACE_SEIZE, 29384): Operation not permitted
# Ah, right. SecurityContext. AllowPrivilegeEscalation: false.
# I have to edit the deployment YAML and restart the pod to debug it.
# Which I can't do because the CIDR block is full.
This is the “devops best” practice of “Security by Default.” It’s great for preventing hackers, but it’s even better at preventing SREs from doing their jobs. We’ve locked the doors so tightly that we can’t get in to put out the fire.
Persistent Volume Claims and the Lie of Statelessness
The “marketing-led” push for “cloud-native” everything has led us to try and run stateful workloads in Kubernetes. We have a Kafka cluster running on top of EBS volumes managed by the AWS EBS CSI driver.
When the nodes started failing due to the conntrack exhaustion, the scheduler tried to move the Kafka pods to “healthy” nodes. But the EBS volumes were still “attached” to the dead nodes.
$ kubectl get events -n kafka
Warning FailedAttachVolume 4m attachdetach-controller Multi-Attach error for volume "pvc-88234-..." Volume is already used by pod kafka-0 on node ip-10-128-0-45.ec2.internal
This is the “Multi-Attach” error that has haunted my dreams for the last three years. The cloud provider’s API is slow, the CSI driver is buggy, and the Kubernetes controller is impatient. The result is a “zombie” volume that is stuck in “Attaching” state forever.
The only way to fix it is to manually go into the AWS Console—the ultimate admission of failure for an SRE—and force-detach the volume. So much for “Infrastructure as Code.” It’s “Infrastructure as a Series of Desperate Clicks in a Web UI.”
We’re told that containers are ephemeral. That we should treat our servers like cattle, not pets. But a 2TB Kafka partition isn’t a cow. It’s a building. You can’t just “reschedule” it and expect it to be fine. We’ve taken the most difficult part of systems engineering—state management—and tried to hide it under a layer of YAML. It didn’t work. It just made the failures more opaque.
The Service Mesh Tax: Istio, Envoy, and the 50ms Latency Floor
We were promised that Istio would give us “insights.” What it actually gave us was a 50ms latency floor on every internal request and a memory footprint that rivals the actual application.
Every time a packet travels from Service A to Service B, it has to go through:
1. The application’s network stack.
2. The Envoy sidecar’s network stack.
3. The host’s iptables (which are a mess of ISTIO_INBOUND and ISTIO_OUTPUT chains).
4. The virtual ethernet pair.
5. The bridge.
6. The physical (virtual) NIC.
7. And then the whole thing in reverse on the other side.
We’ve added six layers of indirection to a simple TCP handshake. And for what? So we can have a pretty graph in Kiali that shows us the traffic is failing? I could have told you the traffic was failing by looking at the stderr logs, which are free.
The “devops best” approach would be to simplify. To use standard Linux tools. To trust the kernel. But there’s no money in simplicity. There are no “Certified Simplicity Architect” badges you can put on your LinkedIn profile. So we build these Rube Goldberg machines and act surprised when they break in ways we didn’t anticipate.
The Silent Failure of “Self-Healing” Systems
The most dangerous part of the modern stack is the belief that the system can fix itself. The “Horizontal Pod Autoscaler” (HPA) is a perfect example.
Tonight, as the latency increased, the HPA saw the CPU usage spike (because Envoy was burning cycles on retries). It decided we needed more pods. So it started spinning up new api-gateway instances.
But each new pod needs an IP address. And where do those IP addresses come from? The CIDR block that is already exhausted.
So the HPA is trying to scale up, which is triggering more CNI errors, which is putting more pressure on the Kube-API server, which is already struggling because the etcd cluster is on the same overloaded network. It’s a feedback loop of failure.
# HPA trying to be helpful
$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
api-gateway Deployment/api-gateway 240%/50% 5 50 32 142d
32 replicas. All of them “Pending” or “CrashLoopBackOff.” All of them consuming IP addresses that don’t exist. All of them contributing to the noise in the logs.
In the old days, if a server was overloaded, it just got slow. You could see the load average climb. You could see the swap space filling up. Now, the system tries to “help” by adding more fuel to the fire. We’ve automated our own destruction.
Technical Debt as a Physical Force
People talk about technical debt like it’s an abstract concept, like a credit card balance you’ll eventually pay off. It’s not. It’s a physical force. It’s the friction that makes every change harder. It’s the “manual hotfix” from five years ago that is now a “critical dependency.”
Tonight’s incident wasn’t caused by a single bug. It was caused by the accumulation of “good enough” decisions.
– The “good enough” VPC design from 2019.
– The “good enough” CI/CD pipeline that swallows errors.
– The “good enough” monitoring that doesn’t look at the kernel.
– The “good enough” Terraform module that doesn’t check for existing routes.
When you follow “devops best” practices without understanding the underlying systems, you’re just building a faster way to fail. You’re automating the chaos instead of eliminating it.
I’m looking at the strace output now. I finally got into a debug container.
$ strace -f -p 29384
[pid 29384] epoll_wait(6, [], 1024, 0) = 0
[pid 29384] epoll_wait(6, [], 1024, 0) = 0
[pid 29384] epoll_wait(6, [], 1024, 0) = 0
[pid 29384] epoll_wait(6, [{events=EPOLLIN, data={u32=11, u64=11}}], 1024, 100) = 1
[pid 29384] read(11, 0xc000456000, 4096) = -1 EAGAIN (Resource temporarily unavailable)
EAGAIN. The universal sign that the system is screaming “I can’t keep up!” The application is fine. The code is perfect. The “business logic” is flawless. But the environment it’s running in is a toxic wasteland of overlapping subnets and exhausted tables.
The Fallacy of the “Single Pane of Glass”
Marketing loves to sell the “single pane of glass.” The one dashboard that tells you everything. Tonight, that pane of glass is a lie.
The Datadog dashboard says the “Service Health” is 99.9%. Why? Because it’s measuring the success rate of the load balancer, which is successfully returning 503s. To the load balancer, a 503 is a “successful” response—it successfully communicated that the backend is dead.
The “Error Rate” widget is green because we’re filtering out 5xx errors that are “expected” during a deployment.
We’ve spent millions of dollars on “observability” tools, and I’m still sitting here using ip route and tcpdump like it’s 1998. Because at the end of the day, the only thing that matters is the packet. The packet doesn’t care about your “holistic” view or your “synergy.” It only cares if there’s a route to the destination.
A Note to the Junior Who Triggered This
You’re probably sleeping right now. You’ll wake up tomorrow, see the Slack thread with 400 messages, and feel a pit in your stomach. You’ll think you failed because you didn’t follow the “devops best” practices.
You didn’t fail. The system failed you. It was designed to be too complex to understand, too fragile to change, and too opaque to debug. You were told that YAML was “code” and that the cloud was “abstracted.” You were lied to.
The cloud is just someone else’s computer, and it’s running the same buggy Linux kernel we’ve been using for decades. The abstractions are just masks.
If you want to survive the next twenty years in this industry, stop reading “Thought Leadership” blogs and start reading the man pages. Learn how iptables actually works. Learn the difference between L4 and L7. Learn why a /24 subnet is different from a /23.
And for the love of all that is holy, never trust a tool that tells you “Happy Helming!”
Now, I have to go manually delete thirty-two “Pending” pods and force-detach an EBS volume because the “self-healing” automation is currently stuck in a loop.
Go back to sleep. The “modern” stack will still be broken when you wake up.
Related Articles
Explore more insights and best practices: