text
Traceback (most recent call last):
File “/opt/deploy/services/analytics/processor.py”, line 442, in
main()
File “/opt/deploy/services/analytics/processor.py”, line 118, in main
results = [process_record(r) for r in large_dataset_query]
File “/opt/deploy/services/analytics/processor.py”, line 118, in
File “/opt/deploy/services/analytics/processor.py”, line 89, in process_record
buffer.append(deep_copy_transform(r))
MemoryError
[ 172834.123456] Out of memory: Kill process 12834 (python3.11) score 942 or sacrifice child
[ 172834.123489] Killed process 12834 (python3.11) total-vm:64213424kB, anon-rss:60123412kB, file-rss:0kB, shmem-rss:0kB
[ 172834.123510] oom_reaper: reaped process 12834 (python3.11), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
I’ve been staring at this screen for forty-eight hours. My eyes feel like they’ve been scrubbed with industrial-grade sandpaper. The fluorescent lights in this data center annex are humming at a frequency that is currently vibrating my last remaining nerve. All of this—the downtime, the lost revenue, the frantic Slack pings from "Stakeholders"—because a junior developer decided that their **python code** was too "elegant" to follow basic memory management principles.
They wanted "readability." They wanted "idiomatic" structures. What they got was a 64GB heap exhaustion that triggered a cascading failure across the entire analytics cluster. When the OOM killer starts hunting your processes like a wolf in a sheep pen, you don't talk about "clean code." You talk about survival.
## INCIDENT #4092-B: THE OOM KILLER’S MIDNIGHT SNACK
The call came in at 3:14 AM on a Sunday. The primary ingestion node for our telemetry pipeline went dark. By 3:20 AM, the failover nodes followed suit. This wasn't a network partition. This wasn't a hardware failure. This was a slow, agonizing death by a thousand allocations.
I pulled the logs and saw the `MemoryError` screaming from the stderr of our main processing service. We’re running Python 3.11.5. It’s supposed to be faster, right? The "Specialized Adaptive Interpreter" is supposed to optimize our bytecode. But no amount of interpreter magic can save you from **python code** that treats RAM like an infinite resource.
The culprit was a single script designed to "summarize" user activity logs. The junior dev—let’s call him Kevin, because it’s always a Kevin—decided that instead of streaming the data from our PostgreSQL instance (running SQLAlchemy 2.0.21, by the way), he would just load the entire result set into a list comprehension.
```bash
# ps aux output during the spike
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 12834 98.2 94.1 64213424 60123412 ? Rs 03:00 12:45 python3.11 processor.py
Look at that RSS. 60GB. On a machine with 64GB of physical RAM. The kernel was gasping for air. I ran strace -p 12834 and watched a literal wall of brk() and mmap() calls. The python code was begging the kernel for more pages, and the kernel, in its infinite mercy, finally decided to put the process out of its misery.
Table of Contents
TICKET #991: THE “ELEGANT” GENERATOR THAT WASN’T
Kevin thought he was being smart. He used a generator expression at first, but then he realized he needed to “sort the results for the UI.” So, he cast the whole thing to a list.
# The offending python code snippet
def get_data():
query = session.query(ActivityLog).filter(ActivityLog.timestamp > last_week)
return [r for r in query] # Kevin's "fix" for sorting later
In Python 3.11.5, every object has overhead. A simple integer isn’t just 4 or 8 bytes like in C. It’s a PyObject structure. It’s 28 bytes. Now multiply that by the 150 million rows Kevin was trying to pull into memory. Then add the overhead of the list itself, which needs to over-allocate to maintain O(1) append time. Then add the SQLAlchemy model overhead, where each instance tracks its own state, its own identity map, and its own internal dictionary.
The python code was effectively creating a massive directed acyclic graph of objects that the Garbage Collector (GC) couldn’t even begin to reason about. I looked at the gc.get_stats() output before the crash. Generation 0 was firing every few milliseconds. Generation 2—the place where objects go to die—was ballooning.
The problem with python code in a high-throughput environment is that the GC is a stop-the-world event. When you have 60GB of objects, the GC has to traverse those object references to see what can be deleted. It’s walking a minefield. Every time the GC ran, the process would freeze for seconds, causing our heartbeat checks to fail, which caused Kubernetes to think the pod was dead, which caused a restart loop. A “death spiral” in the most literal sense.
DEBUG LOG: THE DICTIONARY LOOKUP DEATH SPIRAL
I spent the next six hours digging into why the memory wasn’t being reclaimed even when we limited the batch size. It turns out Kevin’s python code was using a dictionary to “cache” lookups for user metadata.
# More "clever" python code
user_cache = {u.id: u for u in session.query(User).all()}
In Python, dictionaries are hash tables. They are fast, sure. But they are also memory-hungry. Even with the PEP 468 optimizations that made dicts more compact, you’re still looking at a significant footprint. Kevin had 5 million users in that cache.
I ran a quick script to check the size: sys.getsizeof(user_cache). It was massive. But the real killer wasn’t the dict itself; it was the fragmentation. Python’s memory allocator (obmalloc) handles small objects (less than 512 bytes) by grouping them into “arenas” of 256KB. When Kevin’s python code deleted an object, the memory wasn’t necessarily returned to the OS. If one tiny object in that 256KB arena was still “alive” (perhaps referenced by a stray pointer in a traceback or a global variable), the entire arena stayed resident in RAM.
I had to use tracemalloc to find the leak.
import tracemalloc
tracemalloc.start()
# ... run Kevin's garbage ...
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')
for stat in top_stats[:10]:
print(stat)
The output confirmed my fears. The SQLAlchemy identity map was holding onto every single ActivityLog object ever instantiated. Kevin hadn’t used session.expunge_all() or session.close(). He just kept piling more data into the same session. The python code was essentially a memory leak by design.
ANALYSIS: THE GIL AS A CHOKEPOINT FOR CONCURRENCY
Once I patched the memory leak by forcing yield_per(1000) on the SQLAlchemy queries and manually clearing the session, I hit the next wall: the Global Interpreter Lock (GIL).
Kevin’s python code was trying to be “performant” by using threading.Thread to process the logs in parallel.
threads = []
for i in range(cpu_count()):
t = threading.Thread(target=process_logs, args=(chunks[i],))
threads.append(t)
t.start()
I wanted to scream. This is a CPU-bound task. In Python, due to the GIL, only one thread can execute bytecode at a time. By spawning ten threads, Kevin wasn’t making the python code faster; he was making it slower. The overhead of context switching between threads, combined with the contention for the GIL, was driving the CPU load to 100% while the actual throughput dropped by 40%.
I watched the mpstat output. One core was pinned at 100%, while the others were sitting at 10-15%, mostly handling kernel interrupts and I/O wait. The python code was effectively serialized, but with the added penalty of thread management.
I checked the strace again. I saw a constant stream of futex(..., FUTEX_WAIT_BITSET_PRIVATE, ...) calls. That’s the sound of threads begging for the GIL. It’s the sound of a system dying. Python 3.11.5 has some improvements in how it handles internal locks, but it can’t bypass the fundamental law of the GIL: one thread to rule them all, and in the darkness, bind them.
To fix this, I had to rip out the threading module and replace it with multiprocessing. But multiprocessing comes with its own baggage. You have to serialize data (pickle) to send it between processes. Pickle is slow. Pickle is heavy. If you’re sending large objects, you’re back to square one with memory exhaustion because each child process gets its own copy of the memory space (until a write triggers Copy-on-Write).
DATABASE TRACE: SQLALCHEMY 2.0.21 AND THE N+1 APOCALYPSE
While the python code was struggling with its own internal memory, it was also hammering our database into submission. I looked at the PostgreSQL slow query logs.
SELECT users.id, users.name FROM users WHERE users.id = 123;
SELECT users.id, users.name FROM users WHERE users.id = 124;
SELECT users.id, users.name FROM users WHERE users.id = 125;
Kevin had implemented a classic N+1 query pattern. For every ActivityLog he processed, his python code would reach out to the database to fetch the associated User object.
# The N+1 disaster
for log in logs:
print(f"User {log.user.name} did something") # This log.user.name triggers a new query
In SQLAlchemy 2.0.21, you have to be explicit about your joins. Kevin wasn’t. He was relying on “lazy loading,” which is a fancy way of saying “I want my database to die a slow death.”
I had to rewrite the query to use joinedload or subqueryload.
query = session.query(ActivityLog).options(joinedload(ActivityLog.user)).filter(...)
But even then, the python code was still too slow. The overhead of converting SQL rows into Python objects is non-trivial. When you’re dealing with millions of rows, the “Object-Relational” part of ORM becomes a massive tax. I ended up bypassing the ORM entirely for the heavy lifting, using raw SQL and fetching results as simple tuples.
# Stripping away the "magic"
results = session.execute(text("SELECT l.id, u.name FROM activity_logs l JOIN users u ON l.user_id = u.id")).fetchall()
The memory usage dropped instantly. A tuple of primitives is significantly smaller than a full-blown SQLAlchemy model instance. The python code started to behave. But I wasn’t done.
RESOLUTION: STRIPPING THE ABSTRACTIONS TO THE BONE
By hour thirty-six, I had the memory under control and the GIL contention minimized. But the script was still taking too long to finish. The “clever” logic Kevin had written involved a lot of string manipulation and datetime parsing.
In Python, strings are immutable. Every time you do s += "new bit", you’re creating a new string object and copying the old one. Kevin’s python code was doing this inside a loop that ran millions of times.
I replaced his string concatenations with a list of strings and a "".join(list_of_strings) at the end. It’s a basic optimization, but it’s one that people who care more about “elegance” than “execution” always seem to forget.
Then there was the datetime parsing. datetime.strptime is notoriously slow because it has to handle all the complexities of locales and formats. I replaced it with a custom parser that just sliced the string, since our logs are in a fixed ISO format.
# Fast ISO parsing
def fast_parse(ts_str):
# ts_str is "2023-10-27T12:00:00"
return datetime(int(ts_str[:4]), int(ts_str[5:7]), int(ts_str[8:10]), ...)
This shaved another 20% off the execution time.
But the real win came from using mmap. I realized that instead of loading the log files into memory at all, I could just map them into the process’s address space. This lets the OS handle the paging. If the python code needs a specific byte, the kernel fetches that page from disk. If memory gets tight, the kernel can just drop the page because it’s backed by a file.
import mmap
with open("huge_log.txt", "r+b") as f:
mm = mmap.mmap(f.fileno(), 0)
# Now we can treat 'mm' like a giant bytearray without loading it all
I also looked into the __slots__ optimization for our internal data structures. By default, Python objects use a dictionary to store their attributes. This allows for the flexibility of adding new attributes at runtime, but it’s a memory disaster. By defining __slots__, we tell Python exactly what attributes an object will have, allowing it to use a fixed-size array instead of a dictionary.
class ProcessedRecord:
__slots__ = ['id', 'user_id', 'timestamp', 'action_code']
def __init__(self, id, user_id, timestamp, action_code):
self.id = id
self.user_id = user_id
self.timestamp = timestamp
self.action_code = action_code
This change alone reduced the memory footprint of our record objects by nearly 60%. When you’re dealing with millions of these things, that’s the difference between a stable system and a 3 AM wake-up call.
THE AFTERMATH: LESSONS IN STABILITY
It’s now 5:00 AM on Tuesday. The script is running. It’s processing the entire backlog at a rate of 50,000 records per second. The memory usage is a flat line at 4GB. The CPU usage is evenly distributed across all cores thanks to a properly implemented multiprocessing pool.
I’ve spent the last two days fixing python code that should have never passed a code review. But we don’t have time for thorough reviews when we’re “moving fast and breaking things.” Well, things broke. They broke hard.
The problem with modern software development is that we’ve become too comfortable with abstractions. We think that because we’re writing in a high-level language, we don’t need to understand how the kernel manages memory or how the CPU schedules tasks. We treat the interpreter like a black box that will magically handle our inefficiencies.
But the black box has limits. The GIL is real. The GC overhead is real. The cost of a PyObject is real.
I’m looking at the final pip freeze of the patched environment:
SQLAlchemy==2.0.21
psycopg2-binary==2.9.9
numpy==1.26.1
python-dateutil==2.8.2
I added numpy because, frankly, if you’re doing heavy numerical processing or large-scale data manipulation, you shouldn’t be using pure python code anyway. You should be using something that drops down into C and manages its own memory buffers.
I’m going home now. I’m going to sleep for fourteen hours. When I come back, I’m going to have a very long, very unpleasant conversation with Kevin about the difference between “clever” code and “production” code.
If I see one more list comprehension being used to load a multi-gigabyte dataset, I’m revoking everyone’s sudo access and moving the entire stack to assembly. At least then, people will have to think about where their bytes are going.
The system is stable. The alerts are silent. The “magic” is gone, replaced by boring, predictable, and efficient logic. That’s how it should be. Stability isn’t a feature; it’s a prerequisite. And if your python code can’t respect the hardware it runs on, it doesn’t belong in my data center.
I’m out. Don’t page me. Unless the building is literally on fire, and even then, check the temperature sensors first. I don’t trust “clever” fire alarms either. They probably run on a micro-service architecture with a 500ms latency.
End of report.
# Final status check
$ uptime
05:12:01 up 2 days, 14:22, 1 user, load average: 0.05, 0.12, 0.15
$ free -h
total used free shared buff/cache available
Mem: 62Gi 4.2Gi 48Gi 1.0Mi 10Gi 57Gi
Swap: 2.0Gi 0B 2.0Gi
The numbers don’t lie. The “elegant” code is in the trash, and the “ugly” code is actually working. I’ll take ugly and working over elegant and broken any day of the week. Especially on a Sunday at 3 AM.
Now, where is my coffee? Actually, forget the coffee. I need a drink that’s stronger than my hatred for unoptimized bytecode.
The outage is over. The autopsy is complete. The cause of death was “Cleverness.” The cure was “Reality.”
Goodnight. Or good morning. Whatever. Just stay off my servers.
Related Articles
Explore more insights and best practices: