text [ 402.119283] NVRM: Xid (PCI:0000:41:00): 79, GPU has fallen off the bus. [ 402.119288] NVRM: GPU 0000:41:00.0: High Temperature (104C) detected! Throttling. [ 402.119291] NVRM: GPU 0000:41:00.0: Fan speed at 100% (12500 RPM). [ 402.119295] NVRM: GPU 0000:41:00.0: Thermal violation. Clock dropped to 210 MHz. [ 402.119302] nvidia-smi: Critical: PCIe Bus Error on Device … Read more
Docker is Not a Virtual Machine, and Your Images are Too Fat It was 3:14 AM on a Tuesday in 2017. I was the “on-call hero” for a fintech startup that shall remain nameless. We had a monolithic Python API that we’d recently “containerized” to be modern. I pushed a change to the master branch, … Read more
POST-MORTEM REPORT: INCIDENT #8842-OMEGA DATE: 2024-05-14 TIME: 03:14:22 EST AUTHOR: Murphy, Senior Systems Engineer (Level IV) SUBJECT: Total Production Collapse / Kernel Panic on Node-04 1. The Incident Log I was woken up at 03:14 AM by the sound of a pager that has been in service longer than the junior developer who caused this … Read more
02:41:15 AM – PRIMARY NODE FAILURE [129384.492031] Out of memory: Killed process 4921 (python3.11) total-vm:64210432kB, anon-rss:61023412kB, file-rss:0kB, shmem-rss:0kB [129384.492045] oom_reaper: reaped process 4921 (python3.11), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [129384.492102] pcieport 0000:00:01.0: AER: Uncorrected (Fatal) error received: 0000:00:01.0 [129384.492105] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Receiver ID) [129384.492108] pcieport 0000:00:01.0: device [8086:1901] error … Read more
INCIDENT REPORT: DOM_CORRUPTION_0x042 TIMESTAMP: 03:14:22 AM STATUS: CRITICAL / MIGRATION IN PROGRESS ENGINEER: Senior Systems Lead (Systems Purist) The following script was found at the heart of the failure. It is a standard imperative approach to updating a dashboard. It failed because the network latency on the GET /api/v1/telemetry call caused a race condition where … Read more
text Traceback (most recent call last): File “/opt/deploy/services/analytics/processor.py”, line 442, in main() File “/opt/deploy/services/analytics/processor.py”, line 118, in main results = [process_record(r) for r in large_dataset_query] File “/opt/deploy/services/analytics/processor.py”, line 118, in results = [process_record(r) for r in large_dataset_query] File “/opt/deploy/services/analytics/processor.py”, line 89, in process_record buffer.append(deep_copy_transform(r)) MemoryError [ 172834.123456] Out of memory: Kill process 12834 (python3.11) score … Read more
Cybersecurity is Just a Fancy Word for “Not Getting Fired for Someone Else’s Bad Code” I once took down a regional payment gateway because I trusted a “secure” internal API. It was 2:14 AM on a Tuesday. I was trying to debug a latency spike in our order-processor service. I noticed that the service was … Read more
CODE REVIEW: overhyped_model_v1.py Reviewer: Senior Embedded Systems Engineer (Staff Level, 32 Years Experience) Date: October 24, 2023 Subject: Stop wasting my cycles on this statistical alchemy. EXECUTIVE SUMMARY: A Monument to Inefficiency I was asked to review this repository, and frankly, I’d rather be debugging a race condition in a 1992 interrupt handler written in … Read more
AWS is Not a Cloud; It’s a Billable API for Someone Else’s Technical Debt I once cost a former employer $14,200 in forty-eight hours because I misunderstood a single checkbox in the VPC console. I was setting up a “highly available” log processing cluster in us-east-1. I provisioned a NAT Gateway in each Availability Zone … Read more
2024-05-14T03:14:22.891Z [ERROR] checkout-service-v2-7f89db4c5b-x9z2p: Liveness probe failed: Get “http://10.42.12.84:8080/healthz”: dial tcp 10.42.12.84:8080: connect: connection refused 2024-05-14T03:14:25.102Z [INFO] kubelet, ip-10-0-45-122.ec2.internal: Container checkout-service failed liveness probe, will be restarted 2024-05-14T03:14:28.443Z [WARN] kubelet, ip-10-0-45-122.ec2.internal: Pod checkout-service-v2-7f89db4c5b-x9z2p failed to terminate gracefully: terminated by SIGKILL 2024-05-14T03:14:30.001Z [ERROR] checkout-service-v2-7f89db4c5b-x9z2p: CrashLoopBackOff: Back-off 10s restarting failed container=checkout-service pod=checkout-service-v2-7f89db4c5b-x9z2p_production My pager didn’t just beep; it … Read more