It was 03:14 AM when the Prometheus alerts bypassed my ‘Do Not Disturb’ settings, and I knew exactly which piece of ‘python code’ was responsible.
I didn’t even have to look at the Grafana dashboard to know that the worker-api-v2 pods were in a death spiral. My eyes were already burning from seventy-two hours of staring at the same flickering terminal, a cocktail of cheap espresso and pure spite keeping my heart beating. This wasn’t a standard “the database is slow” alert. This was the “everything is melting and the fire department is on strike” alert. The latency on the /v2/process-batch endpoint hadn’t just spiked; it had flatlined at a hard timeout because the pods were too busy choking on their own heap to respond to a liveness probe.
This is the third time this week. Every time we think we’ve patched the leak, another “clever” abstraction in the “python code” finds a new way to bypass the garbage collector and squat on the RAM like a landlord during a housing crisis.
Table of Contents
INCIDENT-8821: THE OOMKILLER COMETH
The timeline of the failure is a masterclass in how not to build a distributed system. We’ve been running on a skeleton crew since the “restructuring,” and it shows. The “python code” that handles our batch processing was supposed to be a temporary bridge. That was eighteen months ago.
03:14:02 – PAGERDUTY: High Error Rate on worker-api-v2. 95th percentile latency exceeds 30s.
03:15:45 – KUBE-EVENT: OOMKill detected on worker-api-v2-7f9b8c-4x2z. Node ip-10-0-42-11 under heavy memory pressure.
03:17:10 – Auto-scaling group attempts to spin up three new nodes to handle the “load.”
03:19:30 – New nodes are immediately saturated. The python code starts consuming 4GB of RSS within 45 seconds of startup.
03:22:00 – Redis connection pool exhausted. The workers are failing so fast they aren’t closing their connections properly.
03:25:12 – I’m logged in, staring at a terminal that feels like it’s mocking me.
# top - 03:26:44 up 12 days, 4:12, 1 user, load average: 42.12, 38.05, 22.10
Tasks: 142 total, 4 running, 138 sleeping, 0 stopped, 0 zombie
%Cpu(s): 88.2 us, 10.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 1.3 si, 0.0 st
MiB Mem : 32142.4 total, 142.1 free, 31850.2 used, 150.1 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 120.4 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4102 root 20 0 12.421g 8.124g 4212 R 99.8 25.3 4:12.44 python3.12
4103 root 20 0 11.812g 7.912g 4110 R 98.2 24.6 4:08.12 python3.12
4104 root 20 0 10.912g 7.512g 4092 R 97.5 23.4 4:05.11 python3.12
4105 root 20 0 9.122g 6.812g 4011 R 96.1 21.2 3:55.09 python3.12
Look at those numbers. 8GB of Resident Set Size for a worker that’s supposed to be doing simple JSON transformation and database upserts. That’s not “load.” That’s a crime scene.
INCIDENT-8822: TRIAGE AND THE RED HERRINGS
The first instinct of the junior dev on call—who I had to wake up because I refuse to suffer alone—was that we were under a DDoS. “Silas, the request volume is up 20%!” he shouted over Zoom, his voice cracking with the kind of optimism only the uninitiated possess.
I told him to shut up and look at the strace output. We weren’t being DDoSed. We were being strangled by our own middleware. I initially suspected uvloop==0.19.0. We’d recently upgraded to gain some performance on the event loop, and I thought maybe there was a bug in how it was handling the file descriptors under high pressure. I spent forty minutes digging through lsof outputs, looking for leaked sockets.
Nothing. The sockets were being cleaned up. The file descriptors were stable.
Then I looked at the database. We’re using psycopg2-binary==2.9.9. I’ve told the team a thousand times to compile from source to ensure we’re linking against the right libpq version, but “it takes too long in the CI pipeline.” So we use the binary. I thought maybe the binary wheel had a memory alignment issue with the new kernel we pushed last Tuesday. I wasted another hour swapping out the binary for a source-compiled version in a test container.
The leak persisted. It wasn’t the library. It wasn’t the loop. It was the “python code” inside the business logic layer.
INCIDENT-8823: THE DEEP DIVE INTO THE HEAP
I pulled a heap dump using memray. If you haven’t used memray on a production pod while it’s actively dying, you haven’t lived. It’s like trying to perform heart surgery on a marathon runner while they’re sprinting.
The memray profile showed something disgusting. We weren’t just leaking strings or small objects. We were leaking entire execution contexts. Every time a request hit the /v2/process-batch endpoint, the “python code” was creating a massive dictionary to “track state” and then failing to release it because of a circular reference that the garbage collector couldn’t see through.
# memray run -m app.worker
# memray report summary memray-worker.bin
Allocation metadata:
-------------------
Command line: ['app/worker.py']
Total memory allocated: 14.2GB
Total allocations: 85,122,904
Peak memory footprint: 8.1GB
Top 5 locations by memory usage:
--------------------------------
1. app/utils/context.py:142 (create_tracking_context) -> 4.2GB
2. psycopg2/_json.py:88 (load_json) -> 1.8GB
3. app/models/base.py:54 (__init__) -> 1.2GB
4. <frozen importlib._bootstrap>:241 -> 0.4GB
5. app/worker.py:212 (process_payload) -> 0.3GB
The culprit was app/utils/context.py:142. I opened the file and felt a physical wave of nausea.
INCIDENT-8824: THE “CLEVER” DEVELOPER’S SIN
The “python code” in question was written by a “Senior Software Architect” who spent more time on LinkedIn talking about “clean code” and “design patterns” than actually reading the Python documentation. He had implemented a custom “Context Manager” that was supposed to “simplify” how we pass database connections and logging metadata through the stack.
Here is the “clever” part that killed us:
class GlobalStateTracker:
def __init__(self):
self.history = []
self._cache = {}
def add_event(self, event, metadata):
# "Clever" developer thought this was a good way to debug
# but forgot that this list grows indefinitely per worker process
self.history.append({
'event': event,
'metadata': metadata,
'timestamp': time.time(),
'callback': self._cleanup_callback # CIRCULAR REFERENCE ALERT
})
def _cleanup_callback(self):
# This never gets called because the object is pinned in history
pass
tracker = GlobalStateTracker()
def create_tracking_context(request_id):
ctx = tracker
ctx.add_event("request_start", {"id": request_id})
return ctx
The developer had created a singleton GlobalStateTracker. Every single request that came through the system appended a dictionary to self.history. But it gets worse. The dictionary included a reference to a method of the tracker itself (self._cleanup_callback).
In Python, specifically before the recent improvements in 3.12, circular references involving objects with custom __del__ methods or complex closure captures could occasionally confuse the generational garbage collector, especially when the objects are being allocated in a tight loop driven by uvloop.
But the real sin wasn’t just the circular reference. It was the fact that this “python code” was storing the entire raw JSON payload from psycopg2-binary in the metadata field of the history list. We process 10,000 batches an hour. Each batch is roughly 500KB of JSON.
10,000 * 500KB = 5GB of memory per hour.
The garbage collector wasn’t “napping.” It was being buried alive. It was trying to traverse a list of millions of dictionaries, each containing nested objects and circular references. The CPU spike we saw in top wasn’t the application doing work; it was the GC thread frantically trying to mark-and-sweep a heap that was growing faster than it could scan.
INCIDENT-8825: THE 4 AM HOTFIX
By 4:00 AM, I was done being surgical. I didn’t have the energy to explain to the “Architect” why his design was a pile of hot garbage. I needed the servers to stop exploding so I could sleep for three hours before the Monday morning stand-up.
I opened the production container via kubectl exec and edited the “python code” directly in the site-packages like a goddamn animal. This is the kind of code that makes “clean code” influencers weep. I don’t care. It worked.
# THE 4 AM HACK - DO NOT MERGE (BUT I WILL ANYWAY)
import gc
import os
import psutil
def brutal_memory_manager():
process = psutil.Process(os.getpid())
if process.memory_info().rss > 2 * 1024 * 1024 * 1024: # 2GB Hard Limit
print("CRITICAL: Memory threshold exceeded. Purging history.")
from app.utils.context import tracker
tracker.history.clear() # The nuclear option
tracker._cache.clear()
gc.collect() # Force the collector to actually do its job
I hooked this function into the uvloop as a background task that ran every thirty seconds. It’s ugly. It’s a hack. It bypasses every single “safety” mechanism we have. But it stopped the OOMKills. The pods stabilized at 2.1GB of RSS. The latency dropped from 30 seconds to 200ms.
I watched the logs for twenty minutes, drinking a lukewarm cup of what I think was once coffee, and waited for the screaming to stop.
INCIDENT-8826: THE PERMANENT REFACTOR (PYTHON 3.12.2)
Once I had six hours of sleep and a shower that almost washed away the shame of that hotfix, I sat down to do it right. We are running Python 3.12.2, and it’s time we actually used the features we’re paying for in overhead.
The fix wasn’t just clearing a list. It was about moving away from that bloated, singleton-based “python code” and using PEP 684 (Per-Interpreter GIL) and the new sys.monitoring API to handle our tracking without killing the heap.
Here is the refactored, production-grade “python code” that I’ve now forced into the main branch:
import gc
import sys
import typing
from dataclasses import dataclass, field
from datetime import datetime
from collections import deque
# Using slots to minimize memory footprint per object
@dataclass(slots=True, frozen=True)
class EventMetadata:
event_type: str
request_id: str
timestamp: float = field(default_factory=lambda: datetime.utcnow().timestamp())
class OptimizedTracker:
"""
A memory-efficient tracker using Python 3.12 features.
We use a deque with a maxlen to ensure we never exceed memory limits.
"""
def __init__(self, max_history: int = 1000):
# Deque automatically drops old items, preventing the leak
self._history: typing.Deque[EventMetadata] = deque(maxlen=max_history)
def add_event(self, event: str, request_id: str):
# We no longer store the entire raw JSON payload.
# If you want the payload, look at the logs.
meta = EventMetadata(event_type=event, request_id=request_id)
self._history.append(meta)
def get_stats(self):
return len(self._history)
# Using Python 3.12's new monitoring features to keep an eye on things
def monitor_callback(event: str, arg: any):
if event == "gc_start":
# Log GC pressure if needed
pass
# sys.monitoring is way more efficient than old-school profiling
# sys.monitoring.register_callback(sys.monitoring.DEBUGGER_ID, monitor_callback)
By using slots=True in the dataclass, we tell Python not to create a __dict__ for every instance of EventMetadata. This reduces the memory overhead of each object by about 60%. By using a collections.deque with a maxlen, we provide a hard cap on how much “history” the “python code” can retain. No more infinite lists. No more 8GB heaps.
Furthermore, I updated the psycopg2-binary usage. We stopped fetching the entire JSONB column into a Python dictionary if we only needed two fields. We now use extras.RealDictCursor more judiciously and ensure that large result sets are handled via server-side cursors to keep the memory footprint in the application layer minimal.
# Refactored DB Access
def fetch_batch_data(cursor_name: str, batch_size: int = 100):
with conn.cursor(name=cursor_name) as cur: # Server-side cursor
cur.itersize = batch_size
cur.execute("SELECT id, payload->'required_field' FROM massive_table")
for row in cur:
yield row
This ensures that even if we have a million rows to process, the “python code” only ever holds batch_size rows in memory at any given time.
THE SRE’S ULTIMATUM
Listen closely, because I’m only going to say this once before I start rejecting every PR that contains the word “Manager” or “Wrapper.”
The next person who commits “python code” that uses a global list as a “cache” without a TTL or a size limit will have their production access revoked. I don’t care if it’s for debugging. I don’t care if it’s “temporary.” Temporary code is the most permanent thing in this building.
We are not building a playground. We are building a high-throughput data pipeline. If you want to play with design patterns, go write a blog post. If you want to write code for this platform, you will respect the GIL, you will respect the garbage collector, and you will learn how to use strace.
I have spent seventy-two hours cleaning up after your “abstractions.” I have seen the depths of the PyObject struct, and it is full of your mistakes.
The next time the pager goes off at 3 AM because of a memory leak, I’m not fixing it. I’m just going to delete the service and go back to sleep.
Fix your “python code.” Or I will fix it for you. And you won’t like my solution.