{"id":4886,"date":"2026-09-18T23:46:50","date_gmt":"2026-09-18T18:16:50","guid":{"rendered":"https:\/\/itsupportwale.com\/blog\/understanding-machine-learning-models-a-complete-guide-3\/"},"modified":"2026-09-18T23:46:50","modified_gmt":"2026-09-18T18:16:50","slug":"understanding-machine-learning-models-a-complete-guide-3","status":"publish","type":"post","link":"https:\/\/itsupportwale.com\/blog\/understanding-machine-learning-models-a-complete-guide-3\/","title":{"rendered":"Understanding Machine Learning Models: A Complete Guide"},"content":{"rendered":"

02:14 – [CRITICAL] KERNEL PANIC – NODE_04<\/strong><\/p>\n

The smell of ozone is the only thing keeping me awake. Node 04 just went dark. I didn\u2019t even get a graceful shutdown; the watchdog timer just bit the dust and took the whole stack with it. I\u2019ve been staring at this terminal for fourteen hours, and the silicon is finally screaming back. <\/p>\n

[ 420.691234] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010\n[ 420.691288] PGD 0 P4D 0 \n[ 420.691312] Oops: 0000 [#1] PREEMPT SMP PTI\n[ 420.691345] CPU: 3 PID: 12402 Comm: python3 Tainted: G           OE     5.15.0-84-generic #93-Ubuntu\n[ 420.691398] Hardware name: Supermicro AS -1014S-WTRT\/H12SSW-NT, BIOS 2.1 06\/02\/2021\n[ 420.691445] RIP: 0010:uv_nmi_setup+0x1a\/0x40 [uv_nmi]\n[ 420.691487] Code: 48 89 e5 48 83 ec 10 48 8b 05 ...\n[ 420.691621] RSP: 0018:ffffb1234c567d88 EFLAGS: 00010246\n[ 420.691665] RAX: 0000000000000000 RBX: ffff987654321000 RCX: 0000000000000000\n<\/code><\/pre>\n
Python 3.10.12. That\u2019s the culprit. Or rather, the bloated, inefficient abstraction layer that some “data scientist” decided was acceptable for an edge deployment. They handed me a container image that\u2019s 14GB. 14 gigabytes for a model that\u2019s supposed to run on a cluster of aging T4s in a closet that hasn’t seen a functional HVAC unit since the Obama administration. <\/p>\n
I\u2019m looking at the dmesg<\/code> logs on the serial console. The memory address 0x0000000040001000<\/code> is throwing parity errors. It\u2019s not the software\u2014well, it is<\/em> the software\u2014but the software is pushing the hardware into a thermal grave. These T4s are rated for 70W, but the way this “transformer” architecture hammers the tensor cores, they\u2019re hitting 88\u00b0C within three minutes of cold boot. <\/p>\n
The fans are spinning at 15,000 RPM. It sounds like a jet engine is trying to take off inside a coffin. I\u2019ve got Ubuntu 22.04.3 LTS running on these nodes, and the kernel is doing its best to manage the interrupts, but the PCIe bus is saturated. We\u2019re moving weights back and forth because the VRAM is too small for the model. 16GB of GDDR6 is a joke in 2024, yet here we are, trying to cram a 30B parameter model into a space meant for a glorified calculator.<\/p>\n
05:45 – THE PYTHON BLOAT AND THE L3 CACHE BLEED<\/strong><\/p>\n
I\u2019ve spent the last three hours trying to strip the environment down. Why does a “machine learning” framework need pandas<\/code>, scipy<\/code>, matplotlib<\/code>, and three different versions of numpy<\/code> just to run an inference loop? It\u2019s a cancer. Every layer of abstraction is a layer of latency. <\/p>\n
I ran top<\/code> to see where the cycles are going. <\/p>\n
top - 05:46:12 up 14:22,  1 user,  load average: 18.42, 15.21, 12.10\nTasks: 342 total,   2 running, 340 sleeping,   0 stopped,   0 zombie\n%Cpu(s): 88.2 us,  9.1 sy,  0.0 ni,  1.2 id,  0.0 wa,  0.0 hi,  1.5 si,  0.0 st\nMiB Mem :  32041.2 total,    412.4 free,  28411.8 used,   3217.0 buff\/cache\nMiB Swap:   2048.0 total,      0.0 free,   2048.0 used.   2104.5 avail Mem \n\n    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND\n  12402 root      20   0   42.1g  26.4g   1.2g R  398.4  82.4  421:14.22 python3\n<\/code><\/pre>\n
Look at that. 398% CPU usage. The Python interpreter is fighting the GIL (Global Interpreter Lock) while the CUDA driver is trying to manage the DMA (Direct Memory Access) transfers. The memory pressure is so high that the kernel is swapping to a SATA SSD. A SATA SSD! We might as well be writing the weights on stone tablets with a chisel. <\/p>\n
The L3 cache is thrashing. I can see it in the performance counters. We\u2019re missing the cache 40% of the time because the model\u2019s attention mechanism is jumping across memory addresses like a caffeinated squirrel. You can\u2019t optimize for hardware you don\u2019t understand. These kids write code in Jupyter notebooks and think the “cloud” is some magical ether where resources are infinite. It\u2019s not. It\u2019s a rack of silicon that gets hot and breaks.<\/p>\n
I\u2019ve got the side panel off Node 04. I touched the heatsink and left a layer of skin behind. 92\u00b0C. The thermal threshold is 95\u00b0C. We\u2019re three degrees away from a hard throttle that will drop the clock speed to 300MHz, and then the whole pipeline stalls.<\/p>\n
13:20 – THE QUANTIZATION LIE<\/strong><\/p>\n
They told me, “Just use INT8 quantization, it’ll be fine.” <\/p>\n
Liars. <\/p>\n
Quantization isn’t a free lunch. It\u2019s a desperate attempt to hide the fact that your math is inefficient. I\u2019m looking at the output of nvidia-smi<\/code> right now, and the power draw is fluctuating wildly. <\/p>\n
+-----------------------------------------------------------------------------+\n| NVIDIA-SMI 525.125.06   Driver Version: 525.125.06   CUDA Version: 11.8     |\n|-------------------------------+----------------------+----------------------+\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n| Fan  Temp  Perf  Pwr:Usage\/Cap|         Memory-Usage | GPU-Util  Compute M. |\n|                               |                      |               MIG M. |\n|===============================+======================+======================|\n|   0  Tesla T4            Off  | 00000000:01:00.0 Off |                    0 |\n| N\/A   89C    P0    68W \/  70W |  15842MiB \/ 16384MiB |     99%      Default |\n|                               |                      |                  N\/A |\n+-------------------------------+----------------------+----------------------+\n<\/code><\/pre>\n
The accuracy loss on INT8 is making the model hallucinate garbage. It\u2019s supposed to be detecting anomalies in industrial vibration data. Instead, it\u2019s flagging the cooling fan as a “high-probability cat.” Why? Because the weights were clipped so aggressively that the manifold collapsed. <\/p>\n
I\u2019m trying to implement a custom CUDA kernel to handle the 4-bit dequantization on the fly, but CUDA 11.8 is fighting me on the memory alignment. If I don’t align the pointers to 128-bit boundaries, the performance drops by 30%. But the “framework” handles memory allocation like a toddler with a bucket of Legos. It\u2019s scattering tensors across the heap with no regard for spatial locality.<\/p>\n
I\u2019ve been awake for 36 hours. My eyes feel like they\u2019ve been rubbed with sandpaper. I\u2019m drinking coffee that\u2019s more sludge than liquid. The air in this server room is 35\u00b0C because the building\u2019s chiller is “optimizing for energy efficiency.” I\u2019ll show them energy efficiency when I bypass the thermal sensors just to get this job to finish.<\/p>\n
22:10 – DAY 2: THE PCI-E BOTTLENECK AND THE OOM KILLER<\/strong><\/p>\n
I thought I had it stable. I was wrong. <\/p>\n
The OOM (Out of Memory) killer just woke up and decided to execute the main inference process. <\/p>\n
[54123.987654] Out of memory: Killed process 12402 (python3) total-vm:44123456kB, anon-rss:27123456kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:88246kB oom_score_adj:0\n<\/code><\/pre>\n
Why did it die? Because the “smart” batching logic decided to queue up 500 requests at once. The PCIe Gen3 x16 bus can only move 15.75 GB\/s. That sounds like a lot until you realize you\u2019re trying to move a 12GB model every time the context window shifts. We\u2019re spending 80% of our time waiting for the bus. It\u2019s a traffic jam on a highway where everyone is driving a semi-truck full of zeros.<\/p>\n
I\u2019m looking at the latency numbers. 1,200ms per token. That\u2019s not “real-time.” That\u2019s “glacial.” <\/p>\n
I\u2019ve started writing a raw C++ wrapper using libtorch<\/code> to bypass the Python overhead. If I can get rid of the Python runtime, I might save 2GB of RSS (Resident Set Size). But the model weights are in a proprietary .safetensors<\/code> format that requires a specific version of a specific library that only works with\u2014you guessed it\u2014Python. <\/p>\n
It\u2019s a hostage situation. The software is holding the hardware hostage. <\/p>\n
I\u2019ve replaced the thermal paste on Node 04 with some high-end diamond-carbon junk I found in my bag. It dropped the temp by 4 degrees. I\u2019m now at 85\u00b0C. I\u2019ve also rigged a floor fan to blow directly into the intake. The noise is deafening. I have to wear my shooting range ear protection just to sit at the console.<\/p>\n
04:30 – DAY 3: THE VOLTAGE SAG AND THE DESPAIR<\/strong><\/p>\n
The power supply on Node 02 just popped. A literal “pop.” A capacitor gave up the ghost. <\/p>\n
I\u2019m down to three nodes. I have to redistribute the load, which means the remaining T4s are going to run even hotter. I\u2019m looking at the power distribution unit (PDU) logs. We\u2019re pulling 14.2 amps on a 15-amp circuit. One more spike and the breaker trips, and I\u2019m in total darkness.<\/p>\n
The math doesn’t add up. They want 99.9% uptime on hardware that was end-of-lifed three years ago, running software that was written last week by someone who thinks “bare metal” is a genre of music. <\/p>\n
Let\u2019s talk about the packet loss. The edge cluster is connected via a 10G SFP+ link that\u2019s dropping frames. Why? Because the CPU is so busy handling the interrupt storms from the GPU that it\u2019s neglecting the NIC (Network Interface Card). <\/p>\n
# ethtool -S eth0 | grep errors\n     rx_errors: 14202\n     tx_errors: 0\n     rx_over_errors: 1201\n     rx_fifo_errors: 882\n<\/code><\/pre>\n
The FIFO buffers are overflowing. The data is arriving faster than the CPU can move it from the NIC to the system RAM. I tried to enable Jumbo Frames, but the switch\u2014a piece of junk from a surplus auction\u2014doesn’t support them. So we\u2019re stuck with 1500-byte MTUs, chopping our data into tiny pieces and wondering why the overhead is killing us.<\/p>\n
I\u2019m currently rewriting the data ingestion pipeline in Rust. It\u2019s the only way to get the memory safety without the garbage collector (GC) pauses. Every time the Python GC kicks in, the inference stalls for 200ms. In a high-speed industrial environment, 200ms is the difference between a successful read and a catastrophic machine failure.<\/p>\n
11:15 – THE GHOST IN THE SILICON<\/strong><\/p>\n
I\u2019m seeing bit-flips. <\/p>\n
I ran a checksum on the model weights in VRAM, and they don’t match the source on disk. It\u2019s the heat. At 88\u00b0C, the electrons start getting jumpy. Cosmic rays? No, just pure, unadulterated thermal agitation. <\/p>\n
I\u2019m having to implement a redundant check-summing routine that runs every ten minutes. If the weights are corrupted, I have to reload them from the NVMe drive. That\u2019s another 12 seconds of downtime. <\/p>\n
The “solution” from the software team? “Just use a larger model with more redundancy.” <\/p>\n
I nearly threw my terminal through the window. More redundancy means more parameters, which means more memory, which means more heat, which means more bit-flips. It\u2019s a feedback loop of stupidity. <\/p>\n
I\u2019ve started undervolting the GPUs. I\u2019m dropping the voltage by 50mV to see if I can stabilize the thermals. It\u2019s a gamble. Too much and the chip crashes. Too little and it melts. <\/p>\n
# nvidia-smi -i 0 -lgc 900,1200\n# nvidia-smi -i 0 -pl 65\n<\/code><\/pre>\n
I\u2019ve locked the clocks at 1200MHz and capped the power at 65W. The latency just went up by 15%, but the temperature dropped to 78\u00b0C. This is the “optimization” I\u2019m reduced to: crippling the hardware so the software doesn’t kill it.<\/p>\n
19:45 – THE FINAL STRETCH OR THE FINAL BREAKDOWN<\/strong><\/p>\n
70 hours. I haven’t slept. I\u2019ve eaten three protein bars and a bag of jerky that tasted like cardboard. <\/p>\n
The cluster is “stable,” if you define stability as “barely functioning and emitting a high-pitched whine.” I\u2019ve managed to get the inference time down to 450ms by bypassing the entire high-level API and talking directly to the CUDA driver via a custom C++ shim. <\/p>\n
I had to manually manage the memory pools. I\u2019ve allocated a contiguous block of 14GB at boot and I\u2019m manually carving it up for the tensors. No malloc<\/code>, no free<\/code>, just raw pointer arithmetic. It\u2019s dangerous. One wrong offset and I\u2019m overwriting the kernel\u2019s page tables. But it\u2019s the only way to avoid the fragmentation that was killing the performance.<\/p>\n
The “data scientists” called. They want to know if we can “easily” swap the model for a new version they just trained. <\/p>\n
I told them the next person who mentions “easy” is getting locked in the server room with the broken PDU. <\/p>\n
The new model is 40B parameters. It won’t fit. It physically won’t fit. You can\u2019t download more VRAM. You can\u2019t “optimize” your way out of the laws of physics. <\/p>\n
I\u2019m looking at the logs one last time before I collapse.<\/p>\n
[252143.123456] nvidia-nvlink: Nvlink Core is being initialized, major 470, minor 129\n[252143.123789] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  525.125.06\n[252145.443221] pcieport 0000:00:01.1: AER: Corrected error received: 0000:01:00.0\n[252145.443235] nvidia 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)\n[252145.443240] nvidia 0000:01:00.0:   device [10de:1eb8] error status\/mask=00000001\/0000e000\n[252145.443244] nvidia 0000:01:00.0:    [ 0] RxErr\n<\/code><\/pre>\n
Still getting receiver errors on the PCIe bus. The signal integrity is degrading. Probably the cheap cables or the fact that the motherboard is warping from the heat. I don’t care anymore. The model is running. The data is flowing. The fans are screaming.<\/p>\n
I\u2019m going to my car. If the building catches fire, the smoke detectors will handle it. I\u2019ve done more with this silicon than it was ever meant to do. I\u2019ve fought the bloat, the heat, and the sheer incompetence of the modern software stack, and I\u2019ve won a pyrrhic victory. <\/p>\n
The shift is over. The hardware is dying, but the “machine learning” is happening. God help us all. <\/p>\n
23:59 – LOG ENDS.<\/strong><\/p>\n
Related Articles<\/h2>\n
Explore more insights and best practices:<\/p>\n