text
\n[ 259200.412834] Out of memory: Killed process 124092 (python3) total-vm:154234824kB, anon-rss:82341204kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:302144kB oom_score_adj:0
\n[ 259200.413102] oom_reaper: reaped process 124092 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
\n[ 259200.415001] Kernel panic – not syncing: Fatal exception in interrupt
\n[ 259200.415050] Kernel Offset: 0x24000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
\n[ 259200.415110] —[ end Kernel panic – not syncing: Fatal exception in interrupt ]—<\/p>\n
## The PagerDuty Alert That Ruined My Life\n\nIt was 3:14 AM on a Tuesday. I know it was 3:14 because I remember thinking about pi, and how much more sense circles make than the jagged, broken architecture of our production cluster. The alert didn't just chirp; it screamed. A persistent, rhythmic assault on my eardrums that signaled the total collapse of the inference tier. I haven't slept more than four hours a night since we decided to pivot to "machine learning" as our primary product driver. I use quotes because, from where I sit, it\u2019s not learning. It\u2019s just a very expensive way to turn electricity into heat and technical debt.\n\nI crawled to my desk, the blue light of my monitor searing my retinas. Ubuntu 22.04 LTS was staring back at me, or rather, the lack of it was. The SSH session to the head node was dead. The serial console showed the kernel panic I pasted above. Total memory exhaustion. The OOM killer had tried to do its job, but the process was so bloated, so engorged with unoptimized tensors and leaked CUDA contexts, that the kernel just gave up the ghost.\n\nWe are running Python 3.10.12. Why? Because the "machine learning" team insisted that 3.11 broke their specific flavor of broken dependencies. So here we are, stuck on a version that\u2019s fine, I guess, but it\u2019s wrapped in a container image that\u2019s 14GB. Fourteen gigabytes for a REST API. It\u2019s a crime against engineering. Every time we pull that image, the network fabric groans.\n\nI tried to reboot the node. Nothing. The IPMI was unresponsive. I had to remote-cycle the PDU. While waiting for the BIOS to post, I looked at the Grafana dashboards. The memory usage graph for the last six hours looked like the north face of the Eiger. A steady, relentless climb until the cliff edge. This wasn't a spike; it was a slow-motion train wreck.\n\n## Tracing the Memory Leak into the Abyss\n\nOnce the node came back up, I started digging through the wreckage. I needed to see what the GPUs were doing right before the crash. I managed to pull a log fragment from the telemetry sidecar that had been scraping `nvidia-smi` every five seconds.\n\n```text\n+---------------------------------------------------------------------------------------+\n| NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.1 |\n|-----------------------------------------+----------------------+----------------------+\n| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |\n| Fan Temp Perf Pwr:Usage\/Cap | Memory-Usage | GPU-Util Compute M. |\n| | | MIG M. |\n|=========================================+======================+======================|\n| 0 NVIDIA A100-SXM4-80GB On | 00000000:00:04.0 Off | 0 |\n| N\/A 64C P0 312W \/ 400W | 79422MiB \/ 81920MiB | 98% Default |\n| | | Disabled |\n+-----------------------------------------+----------------------+----------------------+\n| 1 NVIDIA A100-SXM4-80GB On | 00000000:00:05.0 Off | 0 |\n| N\/A 67C P0 345W \/ 400W | 79110MiB \/ 81920MiB | 99% Default |\n| | | Disabled |\n+-----------------------------------------+----------------------+----------------------+\n| Processes: |\n| GPU GI CI PID Type Process name GPU Memory |\n| ID ID Usage |\n|=======================================================================================|\n| 0 N\/A N\/A 124092 C python3 79410MiB |\n| 1 N\/A N\/A 124092 C python3 79100MiB |\n+---------------------------------------------------------------------------------------+\n<\/code><\/pre>\nLook at that. 79GB utilized on an 80GB card. That\u2019s not “efficient resource utilization.” That\u2019s a suicide note. The “machine learning” engineers told me that PyTorch 2.1.0 would handle memory pooling better. They lied. Or rather, they read the marketing blog<\/a> post and didn’t bother to check how torch.cuda.empty_cache()<\/code> actually interacts with the Linux kernel’s memory management.<\/p>\nThe problem is that the model\u2014some bloated transformer variant with more parameters than the company has revenue\u2014wasn’t just sitting in VRAM. It was thrashing the swap. When the GPU memory filled up, the driver started trying to pin host memory. But the host was already squeezed because we\u2019re running sixteen of these containers on a single box to “maximize ROI.” <\/p>\n
I checked the dmesg<\/code> logs again. The nvidia-uvm<\/code> (Unified Memory Management) driver was screaming. It was trying to migrate pages between the CPU and GPU over an NVLink bus that was already saturated. The latency spiked, the heartbeat checks failed, and Kubernetes, in its infinite, automated wisdom, decided the pod was unhealthy and tried to restart it. <\/p>\nDo you know what happens when you try to restart a 14GB container that requires 80GB of VRAM while the previous instance is still stuck in a Terminating<\/code> state because it\u2019s deadlocked in a kernel wait-state? You get a cascade. The new pod tries to schedule, fails to find resources, triggers the cluster autoscaler, which then tries to spin up more $30-an-hour GPU nodes that will also fail because the underlying container image is a bloated mess of technical debt.<\/p>\n\nTable of Contents<\/p>\n