POST-MORTEM AND MANIFESTO: THE 72-HOUR DEATH MARCH
Document ID: PM-2024-05-12-PROD-COLLAPSE
Status: Bitter.
Author: Senior SRE (On-call since Tuesday, currently vibrating from caffeine toxicity)
I’m writing this because if I don’t, the “DevOps Transformation Committee” will spin this as a “learning opportunity” in their next slide deck. It wasn’t a learning opportunity. It was a preventable disaster caused by layers of abstraction we don’t understand, managed by people who think “reliability” is a Jira ticket status.
I haven’t slept. My eyes feel like they’ve been scrubbed with steel wool. The following is the raw reality of what happened while the “thought leaders” were dreaming of “digital acceleration.”
Table of Contents
Timeline of Failure
- 03:14:00 – PagerDuty triggers.
ServiceUnavailableon the Checkout API. - 03:16:22 – Attempted to scale the deployment.
kubectl scalehangs. - 03:18:45 – Realized
metrics-serverisOOMKilled. HPA (Horizontal Pod Autoscaler) is dead. - 03:25:10 – Discovered Nginx Ingress Controller is throwing 504s. Upstream nodes are unresponsive.
- 03:40:00 – First attempt to roll back the “Standardized Infrastructure Update” from the previous afternoon.
- 04:10:15 – Terraform plan indicates it wants to destroy 412 resources, including the production RDS instance. Rollback aborted.
- 05:30:00 – Discovered a race condition in the CI/CD pipeline that pushed a “latest” tag to the container registry, overwriting the stable build.
- 09:00:00 – Management asks for an “ETA on the fix” while I’m deep in an
stracedump. - 14:00:00 (Day 2) – DNS propagation issues begin because someone “cleaned up” the Route53 records to “reduce clutter.”
- 02:00:00 (Day 3) – Finally stabilized the control plane. Total data loss: 4 hours of non-critical telemetry, 100% of my remaining sanity.
1. The Incident: 03:14 AM and the Cascading Failure of Logic
It started with a whisper and ended with a scream. The Checkout API didn’t just die; it underwent a total existential collapse. The “DevOps Transformation” promised us that by moving to Kubernetes v1.29.1, we would have a self-healing infrastructure.
Self-healing is a lie. It’s a marketing term for “it will try to restart the dumpster fire until the entire data center smells like smoke.”
The root cause was a “performance optimization” pushed by the Platform Team. They decided to “leverage” (I apologize for using a consultant word) the default kernel parameters on our worker nodes without testing the impact on high-concurrency workloads. Specifically, they ignored net.ipv4.tcp_tw_reuse.
Under the load of our 3:00 AM batch processing, the nodes hit the ephemeral port limit. Thousands of sockets stuck in TIME_WAIT. The Nginx ingress controller, running on the same nodes, couldn’t open new connections to the backends.
# Raw output from node-04-prod during the peak of the outage
$ netstat -atn | awk '{print $6}' | sort | uniq -c
14 LISTEN
42 ESTABLISHED
65122 TIME_WAIT
3 CLOSE_WAIT
$ dmesg | tail -n 5
[259201.412] TCP: request_sock_TCP: Possible SYN flooding on port 80. Sending cookies. Check SNMP counters.
[259205.110] TCP: TCP: Hash table overflow
[259210.001] nf_conntrack: table full, dropping packet
The “DevOps” approach here was to let the HPA handle it. But the HPA relies on metrics. The metrics-server was running on a node that was already saturated. It couldn’t report the CPU spike because it couldn’t even talk to the API server. We were flying blind in a cockpit where every gauge was painted on.
2. The YAML Graveyard: Why Our Abstractions are Leaking
We have built a cathedral of YAML. It is a fragile, sprawling mess of Helm charts that reference sub-charts that reference “standardized” templates maintained by a team that hasn’t touched a production server in three years.
During the outage, I tried to manually override the ingress configuration. I found myself digging through six layers of Kustomize overlays. Every time I changed a value, something else broke because of a “helpful” validation script in the CI pipeline that enforces “best practices.”
These abstractions don’t simplify; they hide. They hide the fact that at the end of the day, a process is running on a Linux kernel, and that process needs file descriptors and memory. We’ve replaced understanding the system with understanding the configuration of the system.
Look at this snippet from the “Standard Ingress Template” that caused the 504s:
# The "Standardized" Ingress that killed us
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/proxy-connect-timeout: "15" # Too low for our legacy DB
nginx.ingress.kubernetes.io/proxy-read-timeout: "15"
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "X-DevOps-Transformation: Success"; # This is literally in the template
spec:
ingressClassName: nginx-internal
rules:
- host: api.prod.internal
http:
paths:
- path: /v1/checkout
pathType: Prefix
backend:
service:
name: checkout-service
port:
number: 80
The proxy-connect-timeout was hardcoded to 15 seconds. Our legacy database, which the Checkout API talks to, occasionally takes 16 seconds to respond under load. Because the Platform Team wanted “consistency,” they locked the timeout. When the load hit, the ingress started dropping connections, which triggered retries, which increased the load, which filled the conntrack table.
This is “Day 2” operations. “Day 1” is getting the YAML to apply. “Day 2” is realizing your YAML created a feedback loop that destroys your business.
3. Version Pinning and the Lie of “Latest”
We are currently running a mix of Terraform v1.6.4 and various providers that were “auto-updated” by a Dependabot-style tool. This is a recipe for statefile corruption.
Someone in their infinite wisdom decided that we should always use the “latest” version of our internal base images. “It ensures we have the latest security patches,” they said.
What it actually ensures is that your environment is non-deterministic. At 05:30 AM, while I was trying to redeploy the Checkout API, the CI pipeline pulled a new “latest” base image that included a breaking change in OpenSSL 3.0.7. Suddenly, our service-to-service mTLS was broken.
# Logs from checkout-api-v2-7f9db9c84b-l2x9z
[ERROR] 2024-05-12T05:32:11Z: Failed to initialize SSL context.
[ERROR] 2024-05-12T05:32:11Z: OpenSSL error: error:03000086:digital envelope routines::initialization error
[INFO] 2024-05-12T05:32:12Z: Service exiting...
The “DevOps” way is to automate everything. But if you automate the deployment of unpinned versions, you are just automating your own demise. We spent two hours bisecting base images to find out which one introduced the regression.
We talk about “immutable infrastructure,” but our build process is as mutable as a Google Doc. If I can’t rebuild the exact same binary from a commit hash from three months ago, I don’t have a pipeline; I have a random number generator.
4. The “DevOps Engineer” Title is a Trap
I am tired of being a “DevOps Engineer.” It’s a nonsense title. DevOps was supposed to be a culture of shared responsibility. Instead, it became a way to fire the SysAdmins and tell the Developers they’re now responsible for the entire stack, while hiring a few “SREs” to clean up the blood when the “transformation” hits the fan.
The organization treats us like janitors with better salaries. We are expected to know the intricacies of the Linux kernel, the nuances of BGP routing, the specific syntax of three different cloud providers, and the “agile” way to write a post-mortem.
The “DevOps Transformation” meant we stopped talking about “servers” and started talking about “resources.” But the cloud is just someone else’s server, and it’s usually a server that’s over-provisioned and under-monitored.
We’ve created a culture of “toil.” We spend 80% of our time fixing the tools that were supposed to save us time. We spend hours “bikeshedding” over whether to use Pulumi or Terraform, while our actual application logs are being sent to a /dev/null equivalent because the Prometheus query is too expensive to run.
Speaking of Prometheus, here is the query that was supposed to alert us to the connection exhaustion:
# The alert that never fired
sum(rate(container_network_transmit_errors_total{namespace="prod"}[5m])) by (pod) > 0
Why didn’t it fire? Because container_network_transmit_errors_total doesn’t track TIME_WAIT saturation at the node level. It tracks interface errors. The packets weren’t being dropped by the interface; they were being dropped by the kernel before they even got to the interface. We were looking at the wrong layer of the OSI model because our “standardized dashboards” were built by someone who thinks the network is a “seamless” (ugh) abstraction.
5. Statefile Corruption and the Prayer to the S3 Backend
When the rollback failed, it wasn’t because the code was wrong. It was because the Terraform statefile was locked by a “ghost” process in a failed GitHub Action.
We use S3 for our backend with DynamoDB for locking. During the chaos, the lock didn’t release. I had to manually intervene in the AWS Console—the ultimate admission of failure for an SRE—to delete the lock entry.
But it got worse. Because of the “drift” between our “snowflake servers” (yes, we still have them, they’re just called “long-running EC2 instances” now) and the Terraform code, the plan output was a nightmare.
$ terraform plan
...
# module.vpc.aws_route_table.public will be updated in-place
~ resource "aws_route_table" "public" {
- tags = {
"ManagedBy" = "Terraform"
} -> null
# (4 unchanged attributes hidden)
}
# module.db.aws_db_instance.primary will be destroyed and recreated
-/+ resource "aws_db_instance" "primary" {
~ engine_version = "14.7" -> "15.3" # Force replacement
~ instance_class = "db.r6g.xlarge" -> "db.t4g.medium" # Wait, what?
...
}
Plan: 12 to add, 5 to change, 412 to destroy.
The “DevOps” team had “leveraged” (sorry) a community module for the VPC that had a breaking change in its latest minor version. Because we didn’t pin the module version, a simple terraform init -upgrade during the emergency changed the entire plan.
If I had hit apply, I would have deleted the production database. In the middle of an outage. This is the “safety” our modern tooling provides. It’s a chainsaw without a guard, and we’re using it in the dark.
6. Hard Truths for the Next On-Call Rotation
We need to stop lying to ourselves.
First, DNS is not a “set and forget” service. It is the foundation of everything, and ours is held together by hope and legacy records. The ndots: 5 default in Kubernetes is killing our latency. Every internal lookup is doing a five-step search through the DNS suffix list. We are DDOSing our own CoreDNS pods.
Second, our IAM roles are a disaster. We claim to follow “least privilege,” but every time a developer hits a “403 Forbidden,” we just add AdministratorAccess until it works, promising to “clean it up in the next sprint.” That sprint never comes. We have service accounts with more power than the CTO.
Third, “Day 2” operations are the only operations that matter. Anyone can spin up a cluster. Very few people can keep it running when the disk fills up on a Friday night because the log rotation script failed. We are optimizing for “Day 1” (speed of delivery) and ignoring the “toil” of “Day 2” (maintenance).
Fourth, we need to kill the “DevOps Engineer” title. Call us “Infrastructure Janitors” or “Systems Sufferers.” At least then the expectations would be honest.
Letter to the Junior
To the new hire who just finished a 6-week bootcamp and thinks “DevOps” is about writing YAML and “leveraging” (I am so sorry) the cloud:
Turn back.
You think you’re an architect, but you’re a plumber in a building where the pipes are made of glass and the water is pressurized acid. You’ve been told that the cloud is “serverless” and “elastic.” It’s not. It’s just a very expensive way to run someone else’s Linux kernel.
You will spend your days fighting with tools that were designed to solve problems you don’t have, creating new problems you can’t solve. You will be told that “automation is the key,” but you will find that you are just automating the speed at which you can break things.
You will learn to hate the word “transformation.” You will learn that “best practices” are just things people say when they don’t want to think about the specific constraints of their own system. You will learn that the most important tool in your kit isn’t kubectl or terraform; it’s strace, tcpdump, and the ability to stay calm when the world is burning.
Being an SRE isn’t about “innovation.” It’s about holding the line. It’s about being the person who knows that net.ipv4.tcp_tw_reuse actually does something. It’s about being the one who stays awake for 72 hours so that everyone else can pretend the “DevOps Transformation” is working.
If you still want the job, fine. But don’t come to me asking why the “seamless” (god, I hate that word) deployment failed. I’ll be in the server room, listening to the fans and waiting for the next PagerDuty alert.
Now, if you’ll excuse me, I have to go delete a manual IAM policy before it becomes a permanent part of our infrastructure. Or maybe I’ll just go to sleep.
Probably not. The statefile is still drifting.
EOF
Post-Mortem Summary for Management (The Lie):
The incident was a result of unforeseen micro-service interactions during a period of high growth. We are implementing enhanced observability and refining our automated scaling protocols to ensure a more robust user experience. We continue to prioritize our DevOps journey.
Post-Mortem Summary for SREs (The Truth):
The kernel ran out of ports because we’re idiots who don’t test our “standardized” templates. Terraform tried to kill us. The junior pushed to prod on a Thursday. Everything is still broken, but it’s broken in a way that looks like it’s working. Don’t touch anything. I’m going to bed.
Related Articles
Explore more insights and best practices: