[2023-10-24 03:14:22.891] ERROR: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.50 GiB (GPU 0; 80.00 GiB total capacity; 64.22 GiB already allocated; 1.12 GiB free; 72.45 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2023-10-24 03:14:25.102] WARNING: GPU 0: Thermal Throttling Active. Temp: 89C. Clock: 420MHz (Target: 1410MHz). Power Draw: 685W / 700W.
[2023-10-24 03:14:25.105] FATAL: Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 48
I don’t care about your ‘neural’ metaphors; if your weights don’t fit in the L3 cache, you’re just heating the room for nothing.
You software types come in here with your “artificial intelligence” dreams and your Python scripts, thinking the hardware is some infinite, ethereal plane where logic just happens. It isn’t. It’s a collection of silicon gates screaming under the pressure of your inefficient abstractions. You treat a GPU like a magic box, but it’s just a very fast, very stupid array of arithmetic logic units (ALUs) that are currently choking on your garbage memory management. You’ve spent the last decade wrapping everything in layers of “convenience”—Docker on top of a VM on top of a hypervisor on top of a kernel that’s trying to manage a memory bus it barely understands. Every layer is a tax. Every abstraction is a latency penalty. And now you’re trying to run “artificial intelligence” models that require more bandwidth than the laws of physics want to give you, and you’re wondering why the rack is melting.
I’ve been in these windowless rooms since we were hand-soldering serial ports, and I’m telling you: the party is over. The “free lunch” of Moore’s Law is dead, and your software is the bloated corpse. If you want to deploy “artificial intelligence” without burning down the data center or going bankrupt on H100 compute credits, you need to stop thinking like a mathematician and start thinking like an electron.
Table of Contents
DIRECTIVE 1: Respect the VRAM or Get Out of My Data Center
You see an NVIDIA H100 with 80GB of HBM3 and you think, “Great, I can fit a 70B parameter model in there.” You’re wrong. You’re catastrophically wrong. You haven’t accounted for the KV cache, the activation buffers, or the overhead of the CUDA context itself. When you’re running “artificial intelligence” at scale, the memory wall isn’t a suggestion; it’s a physical barrier.
An H100 SXM5 has a memory bandwidth of about 3.35 TB/s. That sounds fast until you realize your model is doing trillions of operations per second. If your data isn’t sitting exactly where the Tensor Cores can grab it, those cores sit idle. An idle Tensor Core is a $30,000 paperweight. You’re wasting cycles waiting for the bus.
Stop using FP32. If I see a single FP32 weight in a production inference pipeline, I will personally pull the power cord on your rack. We are in the era of FP8 and INT4. If your “artificial intelligence” can’t handle a little quantization noise, your architecture is brittle and your training data is garbage. Use the Transformer Engine in the H100. It’s there for a reason. It handles the dynamic scaling between FP8 and FP16 so you don’t have to, but you’re too busy writing “clean code” to actually read the hardware manual.
DIRECTIVE 2: Purge the Dependency Hell of Python 3.11.4
Your requirements.txt is a suicide note. You’re pulling in 400 dependencies to do a simple matrix multiplication. Look at this mess I found on the dev server yesterday:
# BROKEN DEPENDENCY HELL - DO NOT REPLICATE
torch==2.1.0+cu121
nvidia-cuda-runtime-cu12==12.2.140
numpy==1.24.3
pandas==2.0.3
transformers==4.31.0
accelerate==0.21.0
bitsandbytes==0.41.1
# Conflict: bitsandbytes requires a specific CUDA version
# that conflicts with the torch-bundled runtime.
# Result: Segmentation fault (core dumped)
You’re running Python 3.11.4. Fine. It’s faster than 3.10, but it’s still Python. The Global Interpreter Lock (GIL) is still there, lurking like a ghost in the machine, making sure your multi-core CPU spends 90% of its time waiting for a mutex. When you’re deploying “artificial intelligence,” you shouldn’t even be in Python for the hot path. Python is for the configuration; C++ and Triton are for the execution.
And stop using pip install. If you aren’t pinning your versions to the exact hash, you aren’t an engineer; you’re a gambler. I’ve seen production clusters go down because a sub-dependency of a sub-dependency updated from 1.0.2 to 1.0.3 and changed the way it handles memory pinning. In CUDA Toolkit 12.2, the way memory is mapped has changed. If your library is expecting the 11.8 behavior, you’re going to get a silent corruption that will make your “artificial intelligence” output gibberish for three weeks before you even notice.
DIRECTIVE 3: Quantize or Suffer the Thermal Throttling
Let’s talk about the 700W TDP of an H100. That is not a peak; that is a sustained reality when you’re pushing “artificial intelligence” workloads. If you have eight of those in a 4U chassis, you are trying to dissipate 5.6 kilowatts of heat from a box the size of a microwave.
If you run your models at full precision, you are moving twice as many bits as necessary across the memory bus. Moving bits costs energy. Energy creates heat. Heat triggers the thermal controllers. When that GPU hits 83C, the clock speed drops. When it hits 89C, it throttles to the floor. Your “high-performance” model is now running slower than a 2015 GTX 980 because you were too lazy to implement 4-bit quantization.
Use NF4 (NormalFloat 4). It’s designed for the distribution of weights in “artificial intelligence” models. It’s not perfect, but it’s better than the alternative: a rack that sounds like a jet engine and performs like a calculator. And don’t give me that “accuracy loss” excuse. If a 4-bit quantization ruins your model, your model was overfitted to begin with. Prune the dead weights. If a weight is less than 0.001, it’s noise. Kill it. Your L3 cache will thank you.
DIRECTIVE 4: The PCIe Gen5 Bus is a Bottleneck, Stop Pretending Otherwise
You think because you have a PCIe Gen5 slot, you have infinite bandwidth. You don’t. You have 64 GB/s. That is a trickle compared to the 900 GB/s of NVLink. If you are building “artificial intelligence” systems that rely on moving large tensors back and forth between the CPU and GPU, you have already failed.
The “artificial intelligence” should live on the GPU. It should stay on the GPU. The CPU is just a glorified traffic cop. I see people using tensor.to('cpu') in the middle of a loop and I want to weep. Do you know what that does? It stalls the entire pipeline. It flushes the command queue. It forces a synchronization point. It’s the equivalent of stopping a freight train to check if a single passenger has their ticket.
If you must move data, use DMA (Direct Memory Access). Use pinned memory. Stop letting the Linux kernel’s virtual memory manager decide when to swap your tensors to disk. Set hugepages. If you aren’t using mmap for your model weights, you’re letting the OS bloat your latency with page faults.
DIRECTIVE 5: The Lie of Infinite Scalability
Marketing people love to talk about “seamlessly” scaling “artificial intelligence” in the cloud. There is nothing “seamless” about it. Scaling means more cables, more switches, and more points of failure. When you scale to 1,000 GPUs, you aren’t 1,000 times faster. You’re lucky if you’re 600 times faster, because the rest of that compute is being eaten by the collective overhead of NCCL (NVIDIA Collective Communications Library) trying to keep all those weights in sync.
The “artificial intelligence” hype ignores the physical reality of the speed of light. Electrons can only move so fast through a copper trace. When you’re doing all-reduce operations across a cluster, the latency of your InfiniBand switch becomes the heartbeat of your system. If one cable is slightly crimped, if one transceiver is overheating, your entire $100 million cluster slows down to the speed of that one failing component.
You want scalability? Optimize your kernels. Use Triton to write custom fused kernels that combine the activation function with the matrix multiplication. This reduces the number of times you have to write to VRAM. Every VRAM write is a power-hungry, slow operation. Fusing kernels is the only way to beat the memory wall. But that requires actual engineering, not just importing a library from GitHub.
DIRECTIVE 6: Stop Treating “Artificial Intelligence” Like Magic
It’s just math. It’s just a massive amount of dot products. There is no “soul” in the machine, just a bunch of floating-point numbers that are currently being handled with the grace of a sledgehammer.
The current trend of “artificial intelligence” development is to throw more hardware at the problem. “Just add more H100s,” they say. That is the philosophy of the incompetent. A real architect makes the model smaller, faster, and more efficient. They look at the memory access patterns. They look at the bank conflicts in the shared memory of the GPU. They understand that a warp of 32 threads should never diverge, or you’re wasting 31 threads of execution.
We are building a “Field Manual for the Resistance” because the current path is unsustainable. We are reaching the limits of what air cooling can handle. We are reaching the limits of what the power grid can provide to a single city block. If we don’t start respecting the silicon, the silicon is going to stop respecting us.
THE TECHNICAL MINUTIAE OF SURVIVAL
Let’s get into the weeds, because that’s where the failures happen. You’re using CUDA 12.2. Do you even know what the new memory allocator does? It tries to be “smart” by caching allocations. But in “artificial intelligence” workloads with variable sequence lengths, this leads to massive fragmentation. You’ll see 72GB “reserved” but only 40GB “allocated,” and your script will crash with an OOM error because it can’t find a contiguous 4GB block for the next attention head.
You need to set PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512. Or better yet, write your own memory pool. Stop relying on PyTorch to hold your hand. It’s a general-purpose tool, and “artificial intelligence” at scale is not a general-purpose problem.
And what about the “garbage” that is the Python interpreter? Every time you create a new tensor object, Python has to allocate a small object on the heap, increment a reference count, and eventually garbage collect it. When you’re doing this millions of times a second, the overhead is non-trivial. I’ve seen “artificial intelligence” benchmarks where 15% of the wall-clock time was spent in Python’s gc.collect(). Turn off the garbage collector during the inference loop. Manually manage your memory. It’s what we did in the 90s, and it’s what you need to do now if you want to survive the “artificial intelligence” gold rush without going broke.
THE REALITY OF THE H100 TRANSFORMER ENGINE
The H100 isn’t just a faster A100. It has a dedicated hardware block for handling the scaling factors of FP8. If you aren’t using the transformer_engine library from NVIDIA, you are leaving 3x performance on the table. But of course, that library requires a specific version of ninja and a specific version of g++, and your environment is a mess of conflicting versions.
Here is what your environment should look like, if you had any discipline:
– OS: Ubuntu 22.04.3 LTS (Kernel 5.15.0-84-generic)
– Compiler: GCC 11.4.0
– CUDA: 12.2 Update 2
– Driver: 535.104.05
– Python: 3.11.4 (compiled from source with --enable-optimizations)
If you’re running on a “Data Science” AMI you found in the AWS marketplace, you’ve already lost. Those images are packed with bloatware that steals CPU cycles and pollutes your LD_LIBRARY_PATH. Build your own stack. Know every library that is linked to your binary.
PRUNING, SPARSITY, AND THE BITTER LESSON
Rich Sutton wrote about “The Bitter Lesson”—that general-purpose methods that leverage compute are the most effective. He was right, but he forgot to mention that compute isn’t free. The next stage of “artificial intelligence” isn’t “bigger models”; it’s “smarter hardware utilization.”
NVIDIA’s Ampere and Hopper architectures support 2:4 structured sparsity. This means for every four weights, two must be zero. If you do this, the Tensor Cores can double their throughput. Are you using this? No. You’re too busy chasing the next leaderboard spot to bother with a pruning schedule during training. You’d rather pay for 2x the GPUs than do 1x the engineering.
This is why the “Resistance” exists. We are the ones who have to keep these machines running when the “artificial intelligence” hype cycle hits the wall of physical reality. We are the ones who have to explain to the CFO why the electricity bill for the data center is higher than the revenue from the product.
FINAL DIRECTIVE: MONITOR THE BARE METAL
Stop looking at your “Weights & Biases” dashboard and start looking at nvidia-smi dmon. Watch the pwr and temp columns. Watch the mclk (memory clock) and pclk (processor clock). If you see the pclk dropping while the util is at 100%, you are thermal throttling. You are failing.
“Artificial intelligence” is a resource management game. The weights are your inventory, the VRAM is your warehouse, and the PCIe bus is your delivery truck. If you try to cram a million-ton shipment into a pickup truck, it doesn’t matter how “intelligent” your routing algorithm is; the truck is going to break an axle.
Go back to your code. Strip out the abstractions. Look at the memory layouts. Check your strides. Ensure your tensors are contiguous in memory before you pass them to a CUDA kernel. If you don’t know what a “stride” is in the context of a multi-dimensional array, you shouldn’t be allowed to call yourself an “artificial intelligence” engineer. You’re just a script kiddie playing with expensive toys.
The silicon doesn’t care about your “neural” metaphors. It only cares about the electrons. Respect the electrons, or they will turn into heat and destroy everything you’ve built.
END OF DEBRIEF.
STATUS: THERMAL CRITICAL.
ACTION: SHUTTING DOWN NON-ESSENTIAL ABSTRACTIONS.
Related Articles
Explore more insights and best practices: