I’m staring at a git blame from three years ago and I want to scream.
The coffee is cold. My eyes feel like they’ve been scrubbed with industrial-grade sandpaper. The sun is coming up, and for the third day in a row, I’m watching the kubectl get pods -A output scroll past like a digital obituary. Seventy-two hours. That’s how long it took to untangle the “elegant” solution some mid-level engineer—who is now safely ensconced at a FAANG company with “Platform Evangelist” in his LinkedIn bio—decided to drop into our core routing logic.
He wanted to use a “bleeding edge” service mesh configuration because he read a Medium article. He wanted to prove he could handle “complex traffic shifting.” What he actually did was build a digital suicide pact.
Here is the “masterpiece” I found at 3:00 AM on Tuesday, buried in a repo titled infrastructure-live-v2-final-REALLY-FINAL:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: checkout-api-gw
namespace: prod-checkout
spec:
hosts:
- checkout.api.internal
http:
- route:
- destination:
host: checkout-v1
subset: v1
weight: 90
- destination:
host: checkout-v2-canary
subset: v2
weight: 10
retries:
attempts: 5
perTryTimeout: 200ms
retryOn: "5xx,connect-failure,refused-stream,gateway-error"
timeout: 100ms # <--- THE SMOKING GUN
Look at that. Look at the timeout vs. the perTryTimeout. This absolute genius set a total request timeout that is half the duration of a single retry attempt. When the checkout-v2-canary started throwing latent 503s because the RDS instance hit a connection limit, Istio did exactly what it was told: it entered a recursive death spiral. It tried to retry, realized the total timeout had already expired, killed the connection, and then spawned a new one, effectively DDOSing our own control plane.
Table of Contents
The Anatomy of a Self-Inflicted Wound
I spent six hours just trying to get a shell into the nodes. The kubelet was choking so hard it couldn’t even report status. Here’s what journalctl -u kubelet looked like on node-04-prod-us-east-1:
May 14 04:12:21 node-04 kubelet[1204]: E0514 04:12:21.442103 1204 pod_workers.go:965] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=istio-proxy pod=checkout-api-v2-7f8d9b-x2z_prod-checkout\""
May 14 04:12:25 node-04 kubelet[1204]: I0514 04:12:25.112998 1204 cpu_manager.go:412] "Reconcile finished"
May 14 04:12:30 node-04 dmesg[882]: [259201.12] oom-kill: constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/kubepods.slice/kubepods-burstable.slice/pod-istio-proxy,task=envoy,pid=31222,uid=1337
May 14 04:12:30 node-04 dmesg[882]: [259201.15] Out of memory: Killed process 31222 (envoy) total-vm:4.2GB, anon-rss:1.8GB, file-rss:0B, shmem-rss:0B
The Envoy sidecars were eating 2GB of RAM each just trying to hold the state of the retry budget. We were running K8s v1.27.2, and the scheduler just gave up. It started bin-packing pods onto nodes that were already in an I/O wait hellscape.
I had to manually kill the istiod deployment just to stop the bleeding, which of course meant that for twenty minutes, no new pods could get an IP address because the CNI was waiting on a sidecar injection that would never come. This is what “devops best” looks like in the real world: a circular dependency that requires a Senior Architect to manually edit iptables rules at 4 AM while crying into a lukewarm Red Bull.
YAML is Not a Programming Language (And You’re Not a Programmer)
We have reached a point where we spend more time debugging indentation than we do debugging logic. The “devops best” crowd told us that Infrastructure as Code would save us. They lied. They just traded a shell script that works for a 5,000-line Helm chart that no one understands.
I looked at the values.yaml for the checkout service. It’s 800 lines long. It has nested conditionals that would make a Lisp programmer blush. Why? Because someone wanted to make it “reusable.”
“Reusable” is a code word for “I don’t know what I’m doing, so I’ll make it everything’s problem.” We have {{- if .Values.global.legacy.enabled }} blocks inside {{- range }} loops that reference secrets that don’t exist in the staging environment. When the deployment failed, Helm didn’t tell us why. It just sat there for ten minutes and then spat out: Error: timed out waiting for the condition.
Thanks, Helm. Very helpful. I’ll just go check the 45 different places where a typo could have happened.
The “devops best” approach to YAML is to add more YAML. Need a secret? Use an Operator. Need a cert? Use an Operator. Pretty soon, you have twenty different controllers all fighting over the same CustomResourceDefinition, and your etcd latency looks like a heart attack victim’s EKG. We had etcd heartbeats hitting 500ms because the “resume-driven” architect decided every single pod needed its own unique ConfigMap generated at runtime. Bit rot doesn’t just happen in binaries; it happens in the logic of your orchestration.
Your Microservices are a Distributed Monolith
We split the “Order” service into “Order-Header,” “Order-Items,” “Order-Validation,” and “Order-Tax.” You know what that gave us? Four times the chance of a network failure and a $12,000 monthly bill for cross-AZ data transfer.
During the outage, I watched the trace in Honeycomb. A single request to /checkout was triggering 42 downstream RPC calls. One of those calls—order-tax-v2—was failing. Because we “followed devops best” and implemented aggressive circuit breaking, the failure of the tax service (which isn’t even critical for a cart preview!) caused the entire checkout flow to hard-fail with a 500 Internal Server Error.
We didn’t build microservices. We built a distributed monolith where every function call now requires a TLS handshake and a JSON serialization overhead.
I found a relabel_config in our Prometheus setup that was supposed to “simplify” the metrics coming off these services. Instead, it was creating a high-cardinality explosion. Every time a pod restarted (which was often, thanks to the CrashLoopBackOff), it generated a new set of metrics with a unique pod_name label. Prometheus was swapping to disk. The “devops best” monitoring solution was blind exactly when we needed it most because someone wanted to see a “vibrant” dashboard with per-pod granularity that no one ever looks at.
The Monitoring Lie: Dashboards Won’t Save You
If I see one more Grafana dashboard with 50 spinning dials and no clear “is it broken?” indicator, I’m going to throw my MacBook into the parking lot.
We have “Observability,” but we don’t have “Visibility.” We have millions of traces, but when the site went down, the only thing that told us was a PagerDuty alert from a synthetic ping that’s been failing for three years and everyone just ignores.
The “devops best” practice is to “measure everything.” Okay, we measured everything. We have 4TB of logs in CloudWatch that cost more than our database. But when I needed to see why the envoy proxy was dropping packets, I had to kubectl exec into a container and run tcpdump like it was 1998.
The logs were useless. They were filled with “Info” level garbage about heartbeat checks and “Successful” health probes. The actual error—the ALPN mismatch—was buried in a buffer that got rotated every 30 seconds because the log volume was too high. We are drowning in data and starving for information. We’ve built a “tapestry” of metrics that is actually just a shroud.
CI/CD is a Rube Goldberg Machine
Our deployment pipeline has 14 stages. It takes 45 minutes to deploy a one-line CSS change.
- Linting (which fails if you use double quotes instead of single quotes).
- Unit tests (which are all mocked and pass even if the DB is on fire).
- Security scanning (which flags
lodashfor the 400th time). - Container build.
- Push to ECR.
- Terraform plan.
- Terraform apply (manual approval).
- Integration tests in a “preview” environment that doesn’t mirror prod.
- …and so on.
During the outage, I tried to push a hotfix. The pipeline blocked me because the “security scan” couldn’t reach its upstream server. I had to manually patch the deployment using kubectl edit, which now means our “Source of Truth” in Git is out of sync with reality.
This is the “devops best” trap. We’ve automated the easy stuff and made the hard stuff (emergency fixes) impossible. We’ve built a “seamless” process that is actually a series of brittle gates maintained by people who don’t understand the application code. The “devops best” engineers love the pipeline because the pipeline is their product. They don’t care if the application actually works, as long as the little circles in GitHub Actions turn green.
I saw a Makefile in the repo that calls a Python script that generates a Bash script that runs a Terraform command. Why? Because the guy who wrote it wanted to “abstract the complexity.” He didn’t abstract it; he just hid it behind a curtain of “oopsie-ops” logic that broke the moment we had to scale.
State is the Enemy (And You’re Losing)
“Go stateless!” they screamed. “Use S3 for everything!”
Then we realized we needed a database. So we put Postgres in Kubernetes because someone wanted to “simplify the stack.”
Let me tell you about the joy of a PersistentVolumeClaim that gets stuck in Terminating status while your production database is offline. I spent four hours yesterday fighting with the AWS EBS CSI driver because it couldn’t detach a volume from a node that had been terminated by the Auto Scaling Group.
The “devops best” advice for running stateful workloads on K8s is basically “don’t do it unless you’re an expert,” but the “resume-driven” crowd does it anyway because “Cloud Native” looks better on a CV than “RDS.”
We had a split-brain scenario because the etcd cluster for our “highly available” database operator lost quorum during the network storm. The operator started killing “unhealthy” nodes, which were actually the only nodes that had the correct data. It was a suicide bot. I had to manually intervene to stop the operator from deleting our entire production dataset.
“Devops best” says we should have “self-healing” infrastructure. In reality, we have “self-mutilating” infrastructure. If the system is too stupid to know the difference between a network glitch and a fatal disk error, it shouldn’t have the power to delete things.
The Resume-Driven Development Plague
This is the root of it all. We don’t build systems to solve business problems anymore; we build them to satisfy the “Skills” section of our next job application.
Why did we use a service mesh for a three-service app? Resume.
Why did we use a graph database for a flat list of users? Resume.
Why did we implement a custom Kubernetes controller in Go when a cron job would have worked? Resume.
The engineer who built this mess is gone. He’s probably at a conference right now, giving a talk about “Scaling Microservices with Istio” and showing “comprehensive” slides of the architecture he left behind—the architecture I just spent 72 hours trying to keep from exploding.
He got the “pivotal” experience he wanted. I got the gray hair.
The “devops best” culture rewards complexity. It rewards the person who introduces the new tool, not the person who maintains the old one. It rewards the “journey” to a new platform, not the “destination” of a stable, boring system.
I’m looking at the relabel_configs again.
- source_labels: [__meta_kubernetes_pod_label_app]
target_label: application
regex: (.*)
replacement: $1
action: replace
This looks innocent. But in our environment, it was part of a 200-line Prometheus configuration that was being reloaded every time a new service was discovered. The reload was taking 10 seconds. We have 500 services. Do the math. The Prometheus pod was spending 80% of its CPU time just parsing its own config file.
“Devops best.”
I’m done. I’m going to sleep for fourteen hours. When I come back, I’m deleting the service mesh. I’m deleting the “elegant” traffic shifting. I’m going back to a load balancer and a simple health check.
If you want to build a “multifaceted” “tapestry” of “vibrant” “cloud-native” technologies to “unlock” your “potential,” do it on your own time. On my time, we’re building things that don’t break when a single packet gets dropped in US-EAST-1.
The next person who mentions “thought leadership” or “the devops journey” in a post-mortem is getting their sudo access revoked. We don’t need a journey. We need a website that stays up.
I’m staring at the git blame again. It’s my own name on the timeout: 100ms line from three years ago. I was the one who added it during a “quick fix” session when I was trying to be “clever.”
I want to scream. But I’m too tired.
The “devops best” practice I should have followed? Go to bed. Don’t touch the YAML when you’re tired. And for the love of God, stop trying to make your infrastructure “robust.” Just make it simple enough that a sleep-deprived architect can fix it at 4 AM without needing a PhD in Envoy filter syntax.
The outage is over. The “burn book” is closed. For now. Until the next “visionary” decides we need to migrate to a serverless-wasm-blockchain-mesh.
I’m going home. Don’t page me. If the site goes down, just turn it off and turn it back on again. It’s the only “devops best” practice that actually works.
Related Articles
Explore more insights and best practices: