INCIDENT REPORT: #882-BRAVO-TANGO
TIMESTAMP: 2023-10-14 03:14:22 UTC
STATUS: RESOLVED (After 72 hours of manual state recovery)
SYSTEM: Real-time Analytics Ingestion Engine (The “Data-Sucker-9000”)
PRIMARY ON-CALL: Senior SRE (Current Mood: Resignation Letter is drafted)
Table of Contents
1. Incident Summary
At 03:14 UTC, the primary ingestion cluster for our “Real-time Analytics Dashboard” didn’t just fail; it committed ritualistic suicide. Within forty seconds, the Resident Set Size (RSS) across all 48 nodes in the production-A pool spiked from a healthy 4.2GB to the hard limit of 64GB. The Linux Kernel 6.2 OOM (Out of Memory) killer did exactly what it was programmed to do: it started hunting. It didn’t just kill the Python processes; it nuked the SSH daemon, the monitoring agents, and eventually caused a kernel panic on three nodes because the system was so starved for pages it couldn’t even handle an interrupt.
The cause? A “clean code” refactor pushed by the “Platform Optimization Team” (a group of people who have clearly never seen a production workload larger than a TodoMVC app). They replaced a battle-tested, albeit “ugly,” generator-based pipeline with what they called “modern, idiomatic Python.” They used Pydantic v2.5 for everything, heavy decorators, and list comprehensions that would make a functional programming enthusiast weep with joy—until they saw the bill from AWS. This report is being published internally and externally because I am tired of explaining why “pretty” code that breaks at scale is just expensive garbage.
2. The “Crime Scene” (Broken Code)
Below is the “modern” Python 3.11.4 script that brought a multi-million dollar infrastructure to its knees. It looks “clean.” It has type hints. It uses the “latest” patterns. It is also a technical debt bomb.
# The "Clean" Version that killed the cluster
import asyncio
from pydantic import BaseModel, Field
from typing import List, Optional
import time
class TelemetryData(BaseModel):
sensor_id: int
value: float
timestamp: int
metadata: Optional[dict] = Field(default_factory=dict)
class DataProcessor:
def __init__(self, raw_payloads: List[dict]):
# The first sin: Loading everything into memory at once
self.data = [TelemetryData(**item) for item in raw_payloads]
async def process_all(self):
# The second sin: Creating a massive list of coroutines
tasks = [self.transform(item) for item in self.data]
return await asyncio.gather(*tasks)
async def transform(self, item: TelemetryData):
# The third sin: Blocking the event loop with "clean" logic
item.value = item.value * 1.0000001 # Simulated "complex" math
await asyncio.sleep(0.0001) # Artificial "io"
return item
async def main():
# Simulated 10 million rows from a "clean" API call
raw_data = [{"sensor_id": i, "value": i * 1.5, "timestamp": int(time.time())} for i in range(10_000_000)]
processor = DataProcessor(raw_data)
results = await processor.process_all()
print(f"Processed {len(results)} items")
if __name__ == "__main__":
asyncio.run(main())
3. The Investigation
H2: The 3:00 AM Wake-up Call and the OOM Killer.
When my pager went off, the first thing I saw in the logs wasn’t a Python traceback. It was the silence of a dead node. I logged into the jump box and ran dmesg -T. The output was a graveyard of memory addresses:
[Oct 14 03:15:22] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/docker.service,task=python3,pid=14202,uid=1000
[Oct 14 03:15:22] Out of memory: Killed process 14202 (python3) total-vm:72142016kB, anon-rss:62142016kB, file-rss:0kB, shmem-rss:0kB
[Oct 14 03:15:22] oom_reaper: reaped process 14202 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Python 3.11.4 is fast, but it isn’t magic. The garbage collector (GC) in Python works on reference counting supplemented by a generational collector for detecting cycles. However, when you allocate 10 million Pydantic objects in a single list comprehension, you aren’t just creating data; you are creating a massive, contiguous block of pointers in the heap.
The TelemetryData class, despite being “modern,” is a memory hog. Each instance of a Pydantic model carries with it a __dict__ (unless __slots__ is used, which it wasn’t) and a significant amount of metadata for validation. In our case, each TelemetryData object was taking up approximately 160 bytes. Multiply that by 10 million, and you’re looking at 1.6GB just for the objects. But wait, there’s more. The list itself—the container—needs to store the pointers. That’s another 80MB. Then there’s the raw_payloads list which was also in memory. We had three copies of the data in various states of “transformation” before the first line of actual processing even started. The OOM killer didn’t stand a chance. It saw a process asking for 72GB of virtual memory on a 64GB machine and did its job.
H2: Why Your ‘Elegant’ List Comprehensions Ate 16GB of RAM.
The junior developers love list comprehensions. They think it’s “python best” practice because it’s concise. It’s not. It’s a memory allocation trap.
When you write [TelemetryData(**item) for item in raw_payloads], Python must evaluate the entire expression and build the full list in memory before it can move to the next line of code. This is “eager evaluation.” For a small script, it’s fine. For a production pipeline handling millions of telemetry points, it’s catastrophic.
Compare this to a generator expression: (TelemetryData(**item) for item in raw_payloads). The difference is a single character—a bracket vs a parenthesis—but the architectural difference is the gap between a functioning system and a 72-hour outage. A generator is “lazy.” It yields one item at a time, keeping only the current item in memory.
In the “Crime Scene” code, the DataProcessor.__init__ method forced the entire dataset into a list. Then, the process_all method created another list of coroutines using another list comprehension: tasks = [self.transform(item) for item in self.data]. At this point, we have the raw data, the Pydantic objects, and the coroutine objects all co-existing in the heap.
I ran a tracemalloc on a subset of this logic. The results were sickening:
python3.11/site-packages/pydantic/main.py:314: size=4200 MiB, count=1000000, average=4404 B
infrastructure/ingestor.py:15: size=1200 MiB, count=1000000, average=1258 B
The overhead of the “pretty” abstraction was 4x the size of the actual data.
H2: The ‘Python Best’ Fallacy: Type Hinting Isn’t a Safety Net.
There is a pervasive myth that following “python best” practices like heavy type hinting and Pydantic validation makes your code “production-ready.” Let me be clear: mypy is a static analysis tool. It does nothing for you at 3:00 AM when your logic is flawed.
In the incident code, the developers used Pydantic for “safety.” But Pydantic’s strength—runtime validation—is its greatest weakness in a hot loop. Every time TelemetryData(**item) was called, Pydantic performed a suite of checks: Is sensor_id an int? Can value be coerced to a float? This is fine for an API endpoint receiving one JSON object. It is a performance nightmare when done 10 million times in a row.
Furthermore, the type hints gave the developers a false sense of security. They thought that because the code passed flake8 and mypy, it was “correct.” They ignored the fundamental physics of the machine. They forgot that Python is an interpreted language where every abstraction has a cost. They used Optional[dict] = Field(default_factory=dict), which sounds great until you realize that for every one of those 10 million objects, a new dictionary is being instantiated and tracked by the GC.
The “python best” way to handle this would have been to use __slots__ to prevent the creation of __dict__ and __weakref__ for every instance, saving roughly 40-50 bytes per object. But that isn’t “pretty,” is it? It doesn’t look like the examples in the Pydantic documentation.
H2: Threading vs. Multiprocessing: A Comedy of Errors.
The “Crime Scene” code used asyncio.gather(*tasks). This is a classic mistake. asyncio is designed for I/O-bound tasks—waiting for a database, waiting for a network socket, waiting for a file. It is not a tool for parallelizing CPU-bound data transformation.
Because Python has a Global Interpreter Lock (GIL), only one thread can execute Python bytecode at a time. When the developers ran await asyncio.gather(*tasks), they weren’t running 10 million transformations in parallel. They were telling the single-threaded event loop to manage 10 million coroutine objects. The overhead of the event loop trying to schedule 10 million tiny tasks actually took longer than if they had just used a simple for loop.
I checked the htop output during the incident. One CPU core was pinned at 100% (the event loop thread), while the other 31 cores on the machine were sitting idle, mocking me.
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
14202 root 20 0 72.1G 61.4G 1240 R 99.9 96.2 0:45.12 python3 ingestor.py
If they actually needed parallelism, they should have used ProcessPoolExecutor to bypass the GIL, or better yet, kept the logic simple enough that the GIL wasn’t the bottleneck. But instead, they chose the “modern” asyncio approach because it’s the current “python best” trend, regardless of whether it fits the problem.
H2: Dependency Hell and the 400MB Docker Image.
The container that failed was a bloated monstrosity. To support this “clean” code, the team added:
– pydantic (and its dependencies)
– pandas (because someone thought they might need it later)
– httpx (because requests is “old”)
– uvloop (to try and make asyncio faster, which is like putting a spoiler on a tractor)
The resulting Docker image was 420MB. For a script that essentially moves data from point A to point B. When the OOM killer started nuking pods, the Kubernetes scheduler tried to restart them. But because the image was so large, the “ImagePullBackOff” errors started rolling in as the container registry struggled under the simultaneous pull requests from 48 nodes.
We aren’t just writing code; we are shipping artifacts. A 400MB image for a data ingestor is a sign of architectural rot. It means you don’t know what your dependencies are doing. It means you’ve prioritized developer convenience (“I’ll just pip install the whole world”) over operational stability. In a real “python best” environment, we would be using requirements.txt with strict version pinning or a pyproject.toml that doesn’t include the kitchen sink. We would be using slim base images (like python:3.11-slim-bookworm) and multi-stage builds to keep the runtime footprint small.
H2: Logging as a Denial of Service Attack.
The final nail in the coffin was the logging. In an attempt to be “thorough,” the developers added a log line for every transformation failure.
logger.info(f"Successfully processed item {item.sensor_id}")
When you have 10 million items, and you’re using the standard logging module, each log call is a synchronous I/O operation. Even if you’re logging to stdout, that output has to be captured by the Docker logging driver, written to a JSON file on disk, and then picked up by a log aggregator (like Fluentd or Vector).
As the system started to struggle with memory, the I/O wait (iowait) spiked. The event loop, already overwhelmed by 10 million coroutines, was now blocking on every logger.info call. This is a self-inflicted Denial of Service. The system was spending more time formatting strings and waiting for the disk than it was processing data.
I saw the strace output. The process was spending 60% of its time in write() syscalls.
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
62.12 0.842102 12 68201 write
15.40 0.208712 8 25102 read
This is what happens when you apply “best practices” from a web tutorial to a high-throughput data pipeline. You don’t log every item in a 10-million-row batch. You log the start, the end, and a summary of errors. You use sampled logging. You use non-blocking, asynchronous logging if you absolutely must log during the loop.
4. The Fix
After 72 hours of cleaning up the mess, I rewrote the ingestor. It’s not “pretty.” It doesn’t use Pydantic. It doesn’t use asyncio.gather. It uses 1/100th of the memory and runs 10x faster.
import sys
import json
import time
from typing import Iterable
# Use slots to save memory: no __dict__, no __weakref__
class TelemetryData:
__slots__ = ('sensor_id', 'value', 'timestamp')
def __init__(self, sensor_id: int, value: float, timestamp: int):
self.sensor_id = sensor_id
self.value = value
self.timestamp = timestamp
def get_raw_data(limit: int) -> Iterable[dict]:
"""A generator that yields data one by one."""
for i in range(limit):
yield {
"sensor_id": i,
"value": i * 1.5,
"timestamp": int(time.time())
}
def transform_stream(data_stream: Iterable[dict]) -> Iterable[TelemetryData]:
"""Process data as a stream, keeping memory usage constant."""
for item in data_stream:
try:
# Manual validation is faster than Pydantic in a hot loop
obj = TelemetryData(
sensor_id=int(item['sensor_id']),
value=float(item['value']) * 1.0000001,
timestamp=int(item['timestamp'])
)
yield obj
except (KeyError, ValueError) as e:
# Log errors, but don't kill the process
continue
def main():
start_time = time.perf_counter()
# The pipeline is a chain of generators
raw_stream = get_raw_data(10_000_000)
processed_stream = transform_stream(raw_stream)
count = 0
for _ in processed_stream:
count += 1
if count % 1_000_000 == 0:
print(f"Processed {count} items...")
end_time = time.perf_counter()
print(f"Finished {count} items in {end_time - start_time:.2f} seconds")
if __name__ == "__main__":
main()
Why this works:
1. Generators: The memory footprint is constant. Whether we process 10 items or 10 billion, the RSS stays under 100MB.
2. __slots__: We eliminated the overhead of the instance dictionary.
3. No asyncio overhead: We aren’t managing millions of task objects. We’re just iterating.
4. No Pydantic in the hot path: We do basic type casting where it’s needed.
5. No eager list creation: We never call list() on our data.
5. The “Never Again” List
If you are a developer at this company, read these rules. Memorize them. If I see a PR that violates these without a damn good reason, I will reject it so hard your keyboard will shake.
- Stop Eagerly Loading Data: If your dataset is larger than 10,000 items, you use a generator. You do not use a list comprehension. You do not use
asyncio.gatheron a list of 10 million tasks. - Pydantic is for Boundaries, Not Loops: Use Pydantic for validating incoming JSON at the API layer. Do not use it for internal data structures that are instantiated millions of times per second.
__slots__is Your Friend: If you are defining a class that will have millions of instances, use__slots__. It’s not “un-Pythonic”; it’s “not-crashing-the-server-ic.”- Understand the GIL: Before you reach for
asyncioorthreading, ask yourself if your task is actually I/O bound. If it’s math, it’s CPU bound.asynciowill only make it slower. - Log with Purpose: Do not log inside a high-frequency loop. If you must, use a counter and log every 10,000th iteration, or use a buffered, non-blocking logger.
- Pin Your Versions: I don’t care if a new version of a library came out yesterday. We use specific, tested versions. Your “clean” update to Pydantic v2 during a minor release is what started this cascade.
- Profile Before You “Optimize”: If you think your code is slow, use
cProfileorpy-spy. Don’t just addasyncand hope for the best.
I’m going home now. I’m going to sleep for 14 hours. When I come back, I expect to see the memory usage graphs trending downward, or I’m starting to look at Go jobs.
Report End.
Signed,
The SRE who actually had to fix it.
Related Articles
Explore more insights and best practices: