Kubernetes: The Distributed System You Probably Don’t Need (But Are Stuck With Anyway) It was 3:14 AM on a Tuesday in 2021. I was staring at a Grafana dashboard that looked like a heart monitor for someone having a massive coronary. We were running a fleet of microservices on Kubernetes v1.20 in us-east-1. I had … Read more
text [root@ops-bastion-01 ~]# aws ce get-cost-and-usage \ –time-period Start=2024-05-18,End=2024-05-21 \ –granularity DAILY \ –metrics “UnblendedCost” \ –group-by Type=DIMENSION,Key=SERVICE { “ResultsByTime”: [ { “TimePeriod”: {“Start”: “2024-05-18”, “End”: “2024-05-19”}, “Groups”: [ {“Keys”: [“Amazon Elastic Compute Cloud – Compute”], “Metrics”: {“UnblendedCost”: {“Amount”: “412.45”, “Unit”: “USD”}}}, {“Keys”: [“Amazon Simple Storage Service”], “Metrics”: {“UnblendedCost”: {“Amount”: “89.12”, “Unit”: “USD”}}}, {“Keys”: [“Amazon … Read more
$ du -sh ./node_modules 1.2G ./node_modules $ du -sh ../rtos_project/src 420K ../rtos_project/src $ npm install npm ERR! code ERESOLVE npm ERR! ERESOLVE could not resolve dependency npm ERR! Conflicting peer dependency: [email protected] … $ node -v v21.7.1 I’ve spent the last two decades optimizing interrupt service routines to fit within 50-microsecond windows. I’ve fought for … Read more
Timestamp: 03:14 UTC. The pager went off because the API server decided it didn’t feel like responding to heartbeats anymore. $ kubectl get nodes Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get nodes) $ kubectl get pods -A Unable … Read more
It is 3:14 AM. The air in the server room is a dry, static-charged hum, and the coffee in my mug is a bitter, lukewarm sludge that tastes like burnt rubber and regret. I’ve been staring at a debugger for six hours because some “Senior Full-Stack Evangelist” decided that “developer velocity” was more important than … Read more
text <— 2024-05-22 03:14:12.441 —> <— Last few GCs —> [28492:0x64c0000] 14201 ms: Mark-sweep 2038.4 (2082.1) -> 2038.1 (2082.1) MB, 1142.1 / 0.0 ms (average mu = 0.142, current mu = 0.002) allocation failure; GC in old space requested [28492:0x64c0000] 15403 ms: Mark-sweep 2038.1 (2082.1) -> 2038.1 (2050.1) MB, 1201.8 / 0.0 ms (average mu … Read more
The Graveyard of Our Infrastructure: A Handover Memo for the Unfortunate $ kubectl get nodes NAME STATUS ROLES AGE VERSION ip-10-0-42-12 NotReady worker 89d v1.29.2 ip-10-0-42-13 NotReady worker 89d v1.29.2 ip-10-0-42-14 Ready worker 89d v1.29.2 ip-10-0-42-15 NotReady worker 89d v1.29.2 $ kubectl describe pod checkout-api-7f8d9b6c5-x4z2l Name: checkout-api-7f8d9b6c5-x4z2l Namespace: prod-main Status: Running IP: 10.2.14.155 Containers: checkout-svc: … Read more
It is 4:42 AM. The sun isn’t up, but my blood pressure is. I’ve spent the last 48 hours staring at a Grafana dashboard that looked like a heart monitor of a patient in active cardiac arrest. My eyes feel like they’ve been rubbed with sandpaper, and the smell of stale, burnt coffee—the kind that’s … Read more
json { “error”: { “code”: “Forbidden”, “message”: “Access denied. Caller was not found in any access policy. \nCaller: appid=––;oid=––;iss=https://sts.windows.net/***/\nVault: secret-store-prod-001; Location: eastus”, “innererror”: { “code”: “AccessDeniedByVNet” } }, “activityId”: “f47ac10b-58cc-4372-a567-0e02b2c3d479”, “clientRequestId”: “b94d27b9-934c-4512-8058-7a3e77033cbc”, “timestamp”: “2024-05-20T14:22:01.442Z” } The marketing team promised us a fortress, but they gave us a glass house. They told us that migrating to … Read more
Stop Treating HTML Like a Solved Problem It was 2014. I was working at a high-frequency trading firm that, for some reason, decided to build their internal monitoring dashboard as a “modern” single-page app. We were streaming live order-book data over a primitive WebSocket implementation. I was the lead SRE, and I made a classic … Read more