The smell of ozone and scorched FR4 is something you never forget. It’s the smell of failure, usually the kind that costs six figures and a week of sleep. At 03:14 AM, the secondary cooling loop on the thermal vacuum chamber didn’t just fail; it committed suicide. I was staring at a console window where a “modern” automation suite—written in what some mid-level manager called “highly maintainable python code”—had decided that now was a great time to trigger a generational garbage collection cycle.
While the Python interpreter was busy traversing a massive graph of useless objects to see if they “sparked joy,” the pressure transducer was screaming over the I2C bus. The script, frozen in a stop-the-world GC event, missed the interrupt. The valve stayed closed. The pressure spiked. The seal blew.
I looked at the top output on the industrial controller before the kernel panicked. It was a graveyard of efficiency:
top - 03:14:22 up 12 days, 4:21, 1 user, load average: 4.52, 3.10, 2.15
Tasks: 142 total, 2 running, 140 sleeping, 0 stopped, 0 zombie
%Cpu(s): 94.2 us, 5.8 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 512.0 total, 12.4 free, 482.1 used, 17.5 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 8.2 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1242 root 20 0 412.4m 380.2m 4.2m R 98.2 74.2 14:22.11 python3.11
Look at that. 380MB of resident memory for a script that’s supposed to read four sensors and toggle a GPIO pin. In C, I could have done this in 16KB of SRAM on an STM32 and still had enough room left over to write a flight controller. But no, we needed “rapid development.” We needed “python code.” Well, we rapidly developed a localized explosion.
Table of Contents
The Sin of Abstraction: Why Your Integer is a Fat Liar
In the world I spent thirty years in, an integer is a piece of hardware. It’s four bytes in a register. You know where it is, you know what it’s doing, and you know exactly what happens when you add one to it. In Python, an integer is a PyObject. It’s a heap-allocated monstrosity that carries around a reference count, a pointer to its type definition, and a variable-sized array of “digits.”
When you write x = 42 in your “clean” python code, you aren’t just setting a memory location. You’re invoking a constructor, allocating memory on the heap, and incrementing a reference counter. If you’re doing this in a tight loop—say, processing a 100kHz signal from a logic analyzer—you are effectively sandblasting your CPU’s L1 cache with garbage.
Compare this to a simple C loop:
uint32_t i;
for (i = 0; i < 1000000; i++) {
process_data(i);
}
The compiler puts i in a register. The CPU increments it in one clock cycle. The branch predictor loves it.
Now look at the equivalent python code:
for i in range(1000000):
process_data(i)
Every iteration, the interpreter has to fetch the next object from the range iterator, check its type, increment its reference count, and then—when the loop finishes that iteration—decrement the count and potentially trigger a deallocation. It’s a miracle anything runs at all. You’re not writing logic; you’re managing a bureaucracy.
If you’re forced into this hellscape, you have to use struct. It’s the only way to keep the data from bloating into a balloon animal. You have to treat Python like a wrapper for memory buffers, not a language.
import struct
import mmap
# This is how you stop the bleeding. You don't use Python lists.
# You use raw memory. You treat Python like the glorified
# macro language it is.
def handle_sensor_stream(file_desc):
# Map the hardware buffer directly into memory
# No copying, no PyObject overhead.
mem = mmap.mmap(file_desc.fileno(), 4096, access=mmap.ACCESS_READ)
# We use struct.unpack_from to read the raw bytes.
# We avoid creating thousands of 'int' objects.
offset = 0
while offset < 4096:
# Read a 32-bit unsigned int (big-endian)
# This is still slow, but it's better than the alternative.
val = struct.unpack_from('>I', mem, offset)[0]
if val > 0xDEADBEEF:
trigger_emergency_stop(val)
offset += 4
The Global Interpreter Lock: A Single-Threaded Straitjacket
I’ve had people tell me, with a straight face, that Python is “great for concurrency.” These people have clearly never had to synchronize a high-speed data acquisition system. The Global Interpreter Lock (GIL) is the ultimate insult to modern multi-core processors. It’s a mutex that prevents multiple native threads from executing Python bytecodes at once.
You have a 16-core Xeon? Python doesn’t care. It’s going to use one core to do the work and the other fifteen to heat your office. In C++, I’d use std::atomic or a lock-free queue to pass data between threads with nanosecond latency. In Python, if you use the threading module, you’re just playing a shell game. The threads spend more time fighting for the GIL than they do processing data.
If you try to bypass this with multiprocessing, you’ve just traded one problem for another. Now you’re serializing and deserializing data across IPC pipes using pickle. Do you have any idea how much CPU time is wasted turning a dictionary into a byte stream just to send it to another process? It’s offensive.
I was debugging a system last month where the “python code” was supposed to handle a 10Gbps network stream. The developers were shocked—shocked!—that they were seeing 90% packet loss. I ran gdb on the running process and looked at the stack:
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7ffff7fc4740 (LWP 1234) "python3" 0x00007ffff7b123d4 in __GI_***_select (...)
2 Thread 0x7ffff6f9d700 (LWP 1235) "python3" 0x00007ffff7bc1a11 in PyEval_RestoreThread (tstate=0x5555557a1230)
3 Thread 0x7ffff679c700 (LWP 1236) "python3" 0x00007ffff7bc1a11 in PyEval_RestoreThread (tstate=0x5555557a4560)
Every thread was stuck in PyEval_RestoreThread. They were all waiting for the GIL. The system was basically a very expensive heater that occasionally processed a packet. To fix it, I had to rip out the “clean” logic and drop into ctypes to call a shared library I wrote in C that actually handles the buffer management outside the GIL’s jurisdiction.
The Garbage Collector: The Silent Killer of Real-Time Systems
In an embedded system, determinism is everything. If I say a routine needs to finish in 500 microseconds, it needs to finish in 500 microseconds every single time. Python’s garbage collector is the enemy of determinism. It’s a non-deterministic beast that decides to wake up whenever it feels like it.
Python 3.12.1 made some “improvements” here, but it’s still a mess of reference counting and cyclic garbage collection. When the gc module decides that the heap is too fragmented, it pauses your execution. If that happens while you’re bit-banging a custom protocol to a legacy FPGA, you’re dead. The timing is gone. The FPGA times out. The system halts.
In C, I manage the heap. Or better yet, I don’t use the heap at all. I allocate everything statically at compile time. I know exactly where every byte lives. In your python code, you’re at the mercy of the gc.collect() threshold.
If you must use Python for anything that resembles real-time work, you have to manually disable the GC and trigger it yourself when you know the system is idle. It’s ugly, it’s hacky, and it’s the only way to survive.
import gc
import time
def mission_critical_loop():
# Disable the automatic garbage collector.
# We are taking the wheel now. God help us.
gc.disable()
try:
while True:
start_time = time.perf_counter()
# Do the actual work
perform_hardware_io()
# Manually check if we have a window to clean up the mess
# Python's objects have left behind.
if time.perf_counter() - start_time < 0.001: # We have 1ms of slack
# Only collect the youngest generation to save time
gc.collect(0)
finally:
gc.enable()
This is what “high-level” programming looks like in the real world: you spend half your time fighting the language features that were supposed to make your life easier.
The Bloat of PyObject: A Memory Post-Mortem
Let’s talk about memory density. I had a project where we needed to store 10 million sensor readings in memory for a quick statistical analysis. In C, that’s an array of double. 10,000,000 * 8 bytes = 80MB. Easy.
In Python, if you put those in a list, you’re looking at over 300MB. Why? Because each float in Python is an object.
(gdb) p *(((PyFloatObject *)0x7ffff6f9d700))
$1 = {
ob_refcnt = 1,
ob_type = 0x5555558a2340 <PyFloat_Type>,
ob_fval = 3.141592653589793
}
The ob_refcnt is 8 bytes. The ob_type pointer is 8 bytes. The ob_fval is 8 bytes. That’s 24 bytes for an 8-byte value. And that’s not counting the overhead of the list itself, which is just an array of pointers to these objects. So you’re adding another 8 bytes per element for the pointer. 32 bytes to store 8 bytes of data. A 4x overhead just for the privilege of using a “friendly” language.
When you’re working on an edge gateway with 512MB of RAM, this isn’t just “inefficient”—it’s a fatal flaw. You start hitting the OOM (Out of Memory) killer before you’ve even started the actual data processing.
The solution? array.array or numpy. But even then, you’re just using Python as a thin, shaky bridge to C code. If you’re using numpy, you’re not really writing Python; you’re calling C functions that happen to have Python bindings. And the moment you try to iterate over that numpy array in a standard Python loop, the performance collapses because you’re back to boxing and unboxing those values into PyObject wrappers.
Interfacing with the Real World: ctypes and the Nightmare of Pointers
Eventually, your python code has to talk to something that isn’t a string or a dictionary. It has to talk to a C library, a kernel driver, or a memory-mapped register. This is where ctypes comes in. It’s a bridge, but it’s a bridge made of wet cardboard.
The number of times I’ve seen a Python script segfault because someone passed a Python string to a C function expecting a char* is staggering. Python strings are not null-terminated arrays of bytes. They are complex structures (especially in Python 3 with PEP 393’s flexible string representation).
To talk to the hardware, you have to get your hands dirty with ctypes. You have to manually define structures that match the C alignment. If you get one padding byte wrong because of a compiler difference, you’re writing data into the wrong register. In C, I’d just include the header file. In Python, I have to play “guess the struct alignment.”
from ctypes import Structure, c_uint32, c_uint16, POINTER, cast
class HardwareRegisterMap(Structure):
# You have to be surgical here. One mistake and you've
# bricked the controller.
_fields_ = [
("control_reg", c_uint32),
("status_reg", c_uint32),
("data_buffer", c_uint16 * 1024),
("interrupt_mask", c_uint32),
]
def initialize_hardware(base_address):
# We are literally casting an integer to a pointer.
# This is the kind of thing Python was supposed to prevent.
# Welcome to the basement.
reg_ptr = cast(base_address, POINTER(HardwareRegisterMap))
# Direct register access. No safety nets.
# If base_address is wrong, the OS will kill us with a SIGSEGV.
reg_ptr.contents.control_reg = 0x01
while not (reg_ptr.contents.status_reg & 0x80):
# Busy-waiting in Python. My ancestors are crying.
pass
This is “python code” that looks like C but runs at 1/100th the speed. It’s the worst of both worlds. You have the danger of pointer manipulation with the overhead of an interpreted runtime.
The Bytecode Abyss: Why “Optimization” is a Myth
People talk about “optimizing” Python. They talk about Cython, Numba, or PyPy. But at the end of the day, if you’re running on the standard CPython interpreter (which 99% of production environments are), you’re running bytecode.
You can use dis to see the horror for yourself. Every simple operation is a dozen bytecode instructions.
import dis
def add_values(a, b):
return a + b
dis.dis(add_values)
Output:
2 0 LOAD_FAST 0 (a)
2 LOAD_FAST 1 (b)
4 BINARY_OP 0 (+)
8 RETURN_VALUE
BINARY_OP isn’t a simple ADD instruction. It’s a call to PyNumber_Add, which triggers a massive search through the type hierarchy of the objects to find the correct __add__ method, checks for operator overloading, handles potential type conversions, and finally—maybe—adds the numbers.
In C, a + b is one instruction. add eax, ebx.
When you’re writing automation for a production line, these microseconds add up. If you’re polling a sensor at 1kHz, you have 1ms to do everything. If your “python code” takes 800us just to navigate the bytecode for a few additions and a list lookup, you have no margin for error. A single background process spikes the CPU, and your 1ms window is gone. You’ve missed a sample. Your PID loop goes unstable. The robotic arm starts oscillating. People start running for the exits.
I’ve spent thirty years making sure the hardware does exactly what I tell it to do. Python feels like trying to perform surgery while wearing oven mitts. You can do it, but why would you want to? The only reason we’re in this mess is that it’s easier to hire people who know Python than people who know how a CPU actually works. We’ve traded reliability and efficiency for a lower barrier to entry.
The Final Reckoning: Survival Strategies
If you find yourself trapped in a project where “python code” is mandatory for a mission-critical task, you have to stop thinking like a Python developer. You have to start thinking like an embedded engineer who is using a very bloated, very slow macro language.
- Pre-allocate everything. Never append to a list in a critical loop. Use
bytearrayorarray.arrayand pre-size them. - Avoid the heap. Use
structto pack data into flat buffers. - Bypass the GIL. If you need performance, write the heavy lifting in C and call it via
ctypesor a custom C extension. Don’t try to do math in Python. - Control the GC. Disable it during critical sections and run it manually when the system is idle.
- Use
mmap. If you’re talking to hardware or sharing data between processes, use memory-mapped files. It’s the only way to avoid the catastrophic overhead of Python’s copy-on-write behavior and object serialization.
I’m looking at the charred remains of that vacuum chamber controller on my desk. The root cause wasn’t a hardware flaw. The hardware did exactly what it was told. The problem was that it was told to wait for a garbage collector that didn’t care about the laws of thermodynamics.
Next time, I’m writing the safety interlock in Assembly. I don’t care if it takes longer to write. At least I’ll be able to sleep at night knowing that a PyObject_Head isn’t the only thing standing between a successful test and a fireball.
Anyway, the sun is coming up. I need more coffee and a fresh copy of the CPython ceval.c source code so I can figure out why this new “optimized” dictionary lookup is causing a cache miss every three microseconds. It never ends. You think you’ve escaped the low-level grind, but Python just finds new and creative ways to make you miss the simplicity of a bare-metal pointer.
Don’t talk to me about “clean code.” Show me your memory map. Show me your interrupt latency. If you can’t tell me where your bytes are, you aren’t a programmer; you’re a poet. And poets shouldn’t be in charge of high-pressure valves.
Related Articles
Explore more insights and best practices: