Uncategorized

[72439.123456] out_of_memory: Kill process 14201 (python3.11) score 942 or sacrifice child [72439.123501] Killed process 14201 (python3.11) total-vm:16777216kB, anon-rss:14567424kB, file-rss:0kB, shmem-rss:0kB [72439.123550] oom_reaper: reaped process 14201 (python3.11), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [72439.123600] traps: python3.11[14202] general protection fault ip:7f8a2c3d4e5f sp:7f8a1b2c3d4e error:0 in libc.so.6[7f8a2c300000+1a0000] Segmentation fault (core dumped) $ gdb python3.11 core.14201 (gdb) bt 0 0x00007f8a2c3d4e5f in _int_malloc … Read more

Stop Chasing Certifications: The Reality of Cybersecurity Jobs in a Post-Cloud World I once left a Jenkins instance exposed to the public internet because I thought the VPC security group was “good enough” for a temporary test. It wasn’t. Within four hours, a botnet had found the `/script` console, executed a Groovy script, and dropped … Read more

React Native: The Bridge is a Bottleneck and Your Build Pipeline is a Lie It was 3:14 AM on a Tuesday in 2021. I was staring at a Grafana dashboard that looked like a heart attack. Our flagship app, built on react native 0.63 at the time, was crashing for 40% of users on Android. … Read more

Log Entry: 03:42 AM. The build failed again, and if I see one more ‘Invariant Violation,’ I’m throwing this MacBook Pro into the Hudson River. — FATAL EXCEPTION: mqt_js — Process: com.enterprise.monolith, PID: 28492 java.lang.RuntimeException: Error: Exception in HostFunction: Failed to create JS object from C++ pointer. at com.facebook.react.bridge.CatalystInstanceImpl.jniCallJSFunction(Native Method) at com.facebook.react.bridge.CatalystInstanceImpl.callFunction(CatalystInstanceImpl.java:272) at com.facebook.react.bridge.AppRegistry.runApplication(AppRegistry.java:35) at … Read more

The Ghost in the Machine: A 72-Hour Descent into the Azure Portal The cursor is mocking me. It’s a steady, rhythmic blink in a PuTTY window that hasn’t seen a packet in four minutes. I’m sitting in a room that is too quiet, too warm, and smells like nothing but stale coffee and dry wall. … Read more

text [ 402.119283] NVRM: Xid (PCI:0000:41:00): 79, GPU has fallen off the bus. [ 402.119288] NVRM: GPU 0000:41:00.0: High Temperature (104C) detected! Throttling. [ 402.119291] NVRM: GPU 0000:41:00.0: Fan speed at 100% (12500 RPM). [ 402.119295] NVRM: GPU 0000:41:00.0: Thermal violation. Clock dropped to 210 MHz. [ 402.119302] nvidia-smi: Critical: PCIe Bus Error on Device … Read more

Docker is Not a Virtual Machine, and Your Images are Too Fat It was 3:14 AM on a Tuesday in 2017. I was the “on-call hero” for a fintech startup that shall remain nameless. We had a monolithic Python API that we’d recently “containerized” to be modern. I pushed a change to the master branch, … Read more

POST-MORTEM REPORT: INCIDENT #8842-OMEGA DATE: 2024-05-14 TIME: 03:14:22 EST AUTHOR: Murphy, Senior Systems Engineer (Level IV) SUBJECT: Total Production Collapse / Kernel Panic on Node-04 1. The Incident Log I was woken up at 03:14 AM by the sound of a pager that has been in service longer than the junior developer who caused this … Read more

02:41:15 AM – PRIMARY NODE FAILURE [129384.492031] Out of memory: Killed process 4921 (python3.11) total-vm:64210432kB, anon-rss:61023412kB, file-rss:0kB, shmem-rss:0kB [129384.492045] oom_reaper: reaped process 4921 (python3.11), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [129384.492102] pcieport 0000:00:01.0: AER: Uncorrected (Fatal) error received: 0000:00:01.0 [129384.492105] pcieport 0000:00:01.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Receiver ID) [129384.492108] pcieport 0000:00:01.0: device [8086:1901] error … Read more

INCIDENT REPORT: DOM_CORRUPTION_0x042 TIMESTAMP: 03:14:22 AM STATUS: CRITICAL / MIGRATION IN PROGRESS ENGINEER: Senior Systems Lead (Systems Purist) The following script was found at the heart of the failure. It is a standard imperative approach to updating a dashboard. It failed because the network latency on the GET /api/v1/telemetry call caused a race condition where … Read more