POST-MORTEM REPORT: INCIDENT #4092 – GATEWAY OOM COLLAPSE DATE: 2024-10-24 PREPARED BY: Senior Systems Engineer (Embedded/Firmware Dept.) SUBJECT: THE CATASTROPHIC FAILURE OF ABSTRACTION: WHY THE PYTHON LIST IS NOT AN ARRAY 0. THE SMOKING GUN: TERMINAL LOGS I was woken up at 03:15 by the pager because the production gateway—a machine with 16GB of ECC … Read more
I’m writing this from the floor of the cold aisle in Data Center 4. The 3 AM cron job just triggered a cascade failure because someone decided to ‘innovate’ with a non-deterministic API call. [ 136829.402183] out_of_memory: Kill process 14022 (python3) score 942 or sacrifice child [ 136829.402195] Killed process 14022 (python3) total-vm:128402176kB, anon-rss:124021760kB, file-rss:0kB, … Read more
text stderr: 2024-05-14T03:14:02.882Z ERROR [service-api-gateway] [go-routine-442] connection reset by peer: context deadline exceeded (client_timeout=5000ms) stderr: 2024-05-14T03:14:02.883Z FATAL [service-api-gateway] [main] failed to initialize upstream circuit breaker: redis connection refused at 10.96.0.15:6379 stderr: 2024-05-14T03:14:03.101Z INFO [kernel] [oom-killer] invoked: oom_score_adj=999, total_vm=18446744, anon_rss=16223412, file_rss=0, shmem_rss=0 $ kubectl get pods -n production -l app=order-processor NAME READY STATUS RESTARTS AGE order-processor-7f8d9b6c5d-2v8xl … Read more
The smell of burnt coffee is the only thing keeping my eyes open. It’s 04:15 AM. I’ve been staring at a Grafana dashboard that looks like a heart monitor for a patient in active cardiac arrest for the last three days. My PagerDuty alert didn’t just beep; it screamed. It screamed because some “Full Stack … Read more
INTERNAL DOCUMENT: POST-MORTEM REPORT – PROJECT “SILVER LINING” (MIGRATION FAILURE) FROM: Senior Systems Architect (Infrastructure & Physical Security) TO: The C-Suite and the “Cloud Native” Evangelists who broke the bank. DATE: 2024-05-22 SUBJECT: Why we are broke and why my pager didn’t stop buzzing for 72 hours. { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowDevsToBreakEverything", … Read more
INTERNAL POST-MORTEM: PROJECT “ICARUS” / INCIDENT REPORT #8842-B TO: Engineering Leadership, DevOps, and anyone else who thinks they can “just run a script” FROM: Silas Thorne, Principal Systems Architect (Infrastructure & Recovery) SUBJECT: The Smoldering Remains of our “Machine Learning” Pipeline It is 4:42 AM. I have been awake for thirty-eight hours. The air in … Read more
$ kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-7689d884b-l2v98 0/1 CrashLoopBackOff 42 (3m ago) 72h kube-system kube-proxy-z4m2n 0/1 Error 15 72h production api-gateway-v2-7f5d9c8d4b-9w2k1 0/2 ImagePullBackOff 0 14m production order-processor-5566778899-abc12 0/1 CreateContainerConfigError 0 12m production payment-service-8899aabbcc-xyz34 0/1 Terminating 0 72h production auth-service-66778899aa-def56 0/1 Pending 0 5m monitoring prometheus-server-0 0/1 CrashLoopBackOff 112 72h … Read more
text [root@prod-node-04 ~]# docker pull registry.internal.corp/analytics/data-cruncher:latest latest: Pulling from analytics/data-cruncher d5a1f291072d: Already exists f23a467d5e21: Pull complete 4f4fb5514a3d: Pull complete 7d23456789ab: Extracting [==================================================>] 2.1GB/2.1GB 8e34567890cd: Extracting [========================> ] 1.2GB/2.4GB failed to register layer: Error processing tar file(exit status 1): write /usr/lib/x86_64-linux-gnu/libLLVM-15.so.1: no space left on device [root@prod-node-04 ~]# df -h /var/lib/docker Filesystem Size Used Avail Use% … Read more
It is 4:12 AM. My eyes feel like someone rubbed them with industrial-grade sandpaper. The third pot of coffee tastes like battery acid and failed dreams. I’ve been staring at a Grafana dashboard that looks like a heart monitor of a patient in mid-arrest for the last three days. Why? Because a “senior” developer decided … Read more
Incident ID: #8829-OMEGA. Status: Resolved (Barely). Subject: The day the load balancer decided to become a random number generator. Incident Summary * Duration: 02:04 UTC to 06:12 UTC (4 hours, 8 minutes). * Impact: Total loss of ingress traffic for the api.production.internal and checkout.production.internal zones. Estimated revenue loss: $2.1M. * Root Cause: A “minor” update … Read more