{"id":4856,"date":"2026-08-11T21:35:04","date_gmt":"2026-08-11T16:05:04","guid":{"rendered":"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/"},"modified":"2026-08-11T21:35:04","modified_gmt":"2026-08-11T16:05:04","slug":"python-best-practices-write-cleaner-professional-code","status":"publish","type":"post","link":"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/","title":{"rendered":"Python Best Practices: Write Cleaner, Professional Code"},"content":{"rendered":"<p><strong>INCIDENT REPORT: #882-BRAVO-TANGO<\/strong><br \/>\n<strong>TIMESTAMP:<\/strong> 2023-10-14 03:14:22 UTC<br \/>\n<strong>STATUS:<\/strong> RESOLVED (After 72 hours of manual state recovery)<br \/>\n<strong>SYSTEM:<\/strong> Real-time Analytics Ingestion Engine (The &#8220;Data-Sucker-9000&#8221;)<br \/>\n<strong>PRIMARY ON-CALL:<\/strong> Senior SRE (Current Mood: Resignation Letter is drafted)<\/p>\n<hr \/>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_80 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a7b817a74af9\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a7b817a74af9\"  aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#1_Incident_Summary\" >1. Incident Summary<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#2_The_%E2%80%9CCrime_Scene%E2%80%9D_Broken_Code\" >2. The &#8220;Crime Scene&#8221; (Broken Code)<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#3_The_Investigation\" >3. The Investigation<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#H2_The_3_00_AM_Wake-up_Call_and_the_OOM_Killer\" >H2: The 3:00 AM Wake-up Call and the OOM Killer.<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#H2_Why_Your_%E2%80%98Elegant_List_Comprehensions_Ate_16GB_of_RAM\" >H2: Why Your &#8216;Elegant&#8217; List Comprehensions Ate 16GB of RAM.<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#H2_The_%E2%80%98Python_Best_Fallacy_Type_Hinting_Isnt_a_Safety_Net\" >H2: The &#8216;Python Best&#8217; Fallacy: Type Hinting Isn&#8217;t a Safety Net.<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#H2_Threading_vs_Multiprocessing_A_Comedy_of_Errors\" >H2: Threading vs. Multiprocessing: A Comedy of Errors.<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#H2_Dependency_Hell_and_the_400MB_Docker_Image\" >H2: Dependency Hell and the 400MB Docker Image.<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#H2_Logging_as_a_Denial_of_Service_Attack\" >H2: Logging as a Denial of Service Attack.<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#4_The_Fix\" >4. The Fix<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#5_The_%E2%80%9CNever_Again%E2%80%9D_List\" >5. The &#8220;Never Again&#8221; List<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#Related_Articles\" >Related Articles<\/a><\/li><\/ul><\/nav><\/div>\n<h3><span class=\"ez-toc-section\" id=\"1_Incident_Summary\"><\/span>1. Incident Summary<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>At 03:14 UTC, the primary ingestion cluster for our &#8220;Real-time Analytics Dashboard&#8221; didn&#8217;t just fail; it committed ritualistic suicide. Within forty seconds, the Resident Set Size (RSS) across all 48 nodes in the production-A pool spiked from a healthy 4.2GB to the hard limit of 64GB. The Linux Kernel 6.2 OOM (Out of Memory) killer did exactly what it was programmed to do: it started hunting. It didn&#8217;t just kill the Python processes; it nuked the SSH daemon, the monitoring agents, and eventually caused a kernel panic on three nodes because the system was so starved for pages it couldn&#8217;t even handle an interrupt.<\/p>\n<p>The cause? A &#8220;clean code&#8221; refactor pushed by the &#8220;Platform Optimization Team&#8221; (a group of people who have clearly never seen a production workload larger than a TodoMVC app). They replaced a battle-tested, albeit &#8220;ugly,&#8221; generator-based pipeline with what they called &#8220;modern, idiomatic Python.&#8221; They used Pydantic v2.5 for everything, heavy decorators, and list comprehensions that would make a functional programming enthusiast weep with joy\u2014until they saw the bill from AWS. This report is being published internally and externally because I am tired of explaining why &#8220;pretty&#8221; code that breaks at scale is just expensive garbage.<\/p>\n<hr \/>\n<h3><span class=\"ez-toc-section\" id=\"2_The_%E2%80%9CCrime_Scene%E2%80%9D_Broken_Code\"><\/span>2. The &#8220;Crime Scene&#8221; (Broken Code)<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Below is the &#8220;modern&#8221; Python 3.11.4 script that brought a multi-million dollar infrastructure to its knees. It looks &#8220;clean.&#8221; It has type hints. It uses the &#8220;latest&#8221; patterns. It is also a technical debt bomb.<\/p>\n<pre class=\"codehilite\"><code class=\"language-python\"># The &quot;Clean&quot; Version that killed the cluster\nimport asyncio\nfrom pydantic import BaseModel, Field\nfrom typing import List, Optional\nimport time\n\nclass TelemetryData(BaseModel):\n    sensor_id: int\n    value: float\n    timestamp: int\n    metadata: Optional[dict] = Field(default_factory=dict)\n\nclass DataProcessor:\n    def __init__(self, raw_payloads: List[dict]):\n        # The first sin: Loading everything into memory at once\n        self.data = [TelemetryData(**item) for item in raw_payloads]\n\n    async def process_all(self):\n        # The second sin: Creating a massive list of coroutines\n        tasks = [self.transform(item) for item in self.data]\n        return await asyncio.gather(*tasks)\n\n    async def transform(self, item: TelemetryData):\n        # The third sin: Blocking the event loop with &quot;clean&quot; logic\n        item.value = item.value * 1.0000001  # Simulated &quot;complex&quot; math\n        await asyncio.sleep(0.0001) # Artificial &quot;io&quot;\n        return item\n\nasync def main():\n    # Simulated 10 million rows from a &quot;clean&quot; API call\n    raw_data = [{&quot;sensor_id&quot;: i, &quot;value&quot;: i * 1.5, &quot;timestamp&quot;: int(time.time())} for i in range(10_000_000)]\n    processor = DataProcessor(raw_data)\n    results = await processor.process_all()\n    print(f&quot;Processed {len(results)} items&quot;)\n\nif __name__ == &quot;__main__&quot;:\n    asyncio.run(main())\n<\/code><\/pre>\n<hr \/>\n<h3><span class=\"ez-toc-section\" id=\"3_The_Investigation\"><\/span>3. The Investigation<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<h2><span class=\"ez-toc-section\" id=\"H2_The_3_00_AM_Wake-up_Call_and_the_OOM_Killer\"><\/span>H2: The 3:00 AM Wake-up Call and the OOM Killer.<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>When my pager went off, the first thing I saw in the logs wasn&#8217;t a Python traceback. It was the silence of a dead node. I logged into the jump box and ran <code>dmesg -T<\/code>. The output was a graveyard of memory addresses:<\/p>\n<pre class=\"codehilite\"><code class=\"language-text\">[Oct 14 03:15:22] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=\/,mems_allowed=0,global_oom,task_memcg=\/system.slice\/docker.service,task=python3,pid=14202,uid=1000\n[Oct 14 03:15:22] Out of memory: Killed process 14202 (python3) total-vm:72142016kB, anon-rss:62142016kB, file-rss:0kB, shmem-rss:0kB\n[Oct 14 03:15:22] oom_reaper: reaped process 14202 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB\n<\/code><\/pre>\n<p>Python 3.11.4 is fast, but it isn&#8217;t magic. The garbage collector (GC) in Python works on reference counting supplemented by a generational collector for detecting cycles. However, when you allocate 10 million Pydantic objects in a single list comprehension, you aren&#8217;t just creating data; you are creating a massive, contiguous block of pointers in the heap. <\/p>\n<p>The <code>TelemetryData<\/code> class, despite being &#8220;modern,&#8221; is a memory hog. Each instance of a Pydantic model carries with it a <code>__dict__<\/code> (unless <code>__slots__<\/code> is used, which it wasn&#8217;t) and a significant amount of metadata for validation. In our case, each <code>TelemetryData<\/code> object was taking up approximately 160 bytes. Multiply that by 10 million, and you&#8217;re looking at 1.6GB just for the objects. But wait, there&#8217;s more. The list itself\u2014the container\u2014needs to store the pointers. That\u2019s another 80MB. Then there&#8217;s the <code>raw_payloads<\/code> list which was <em>also<\/em> in memory. We had three copies of the data in various states of &#8220;transformation&#8221; before the first line of actual processing even started. The OOM killer didn&#8217;t stand a chance. It saw a process asking for 72GB of virtual memory on a 64GB machine and did its job.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"H2_Why_Your_%E2%80%98Elegant_List_Comprehensions_Ate_16GB_of_RAM\"><\/span>H2: Why Your &#8216;Elegant&#8217; List Comprehensions Ate 16GB of RAM.<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The junior developers love list comprehensions. They think it&#8217;s &#8220;python best&#8221; practice because it&#8217;s concise. It&#8217;s not. It&#8217;s a memory allocation trap. <\/p>\n<p>When you write <code>[TelemetryData(**item) for item in raw_payloads]<\/code>, Python must evaluate the entire expression and build the full list in memory before it can move to the next line of code. This is &#8220;eager evaluation.&#8221; For a small script, it&#8217;s fine. For a production pipeline handling millions of telemetry points, it&#8217;s catastrophic. <\/p>\n<p>Compare this to a generator expression: <code>(TelemetryData(**item) for item in raw_payloads)<\/code>. The difference is a single character\u2014a bracket vs a parenthesis\u2014but the architectural difference is the gap between a functioning system and a 72-hour outage. A generator is &#8220;lazy.&#8221; It yields one item at a time, keeping only the current item in memory. <\/p>\n<p>In the &#8220;Crime Scene&#8221; code, the <code>DataProcessor.__init__<\/code> method forced the entire dataset into a list. Then, the <code>process_all<\/code> method created <em>another<\/em> list of coroutines using another list comprehension: <code>tasks = [self.transform(item) for item in self.data]<\/code>. At this point, we have the raw data, the Pydantic objects, and the coroutine objects all co-existing in the heap. <\/p>\n<p>I ran a <code>tracemalloc<\/code> on a subset of this logic. The results were sickening:<\/p>\n<pre class=\"codehilite\"><code class=\"language-text\">python3.11\/site-packages\/pydantic\/main.py:314: size=4200 MiB, count=1000000, average=4404 B\ninfrastructure\/ingestor.py:15: size=1200 MiB, count=1000000, average=1258 B\n<\/code><\/pre>\n<p>The overhead of the &#8220;pretty&#8221; abstraction was 4x the size of the actual data.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"H2_The_%E2%80%98Python_Best_Fallacy_Type_Hinting_Isnt_a_Safety_Net\"><\/span>H2: The &#8216;Python Best&#8217; Fallacy: Type Hinting Isn&#8217;t a Safety Net.<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>There is a pervasive myth that following &#8220;python best&#8221; practices like heavy type hinting and Pydantic validation makes your code &#8220;production-ready.&#8221; Let me be clear: <code>mypy<\/code> is a static analysis tool. It does nothing for you at 3:00 AM when your logic is flawed. <\/p>\n<p>In the incident code, the developers used Pydantic for &#8220;safety.&#8221; But Pydantic&#8217;s strength\u2014runtime validation\u2014is its greatest weakness in a hot loop. Every time <code>TelemetryData(**item)<\/code> was called, Pydantic performed a suite of checks: Is <code>sensor_id<\/code> an int? Can <code>value<\/code> be coerced to a float? This is fine for an API endpoint receiving one JSON object. It is a performance nightmare when done 10 million times in a row.<\/p>\n<p>Furthermore, the type hints gave the developers a false sense of security. They thought that because the code passed <code>flake8<\/code> and <code>mypy<\/code>, it was &#8220;correct.&#8221; They ignored the fundamental physics of the machine. They forgot that Python is an interpreted language where every abstraction has a cost. They used <code>Optional[dict] = Field(default_factory=dict)<\/code>, which sounds great until you realize that for every one of those 10 million objects, a new dictionary is being instantiated and tracked by the GC. <\/p>\n<p>The &#8220;python best&#8221; way to handle this would have been to use <code>__slots__<\/code> to prevent the creation of <code>__dict__<\/code> and <code>__weakref__<\/code> for every instance, saving roughly 40-50 bytes per object. But that isn&#8217;t &#8220;pretty,&#8221; is it? It doesn&#8217;t look like the examples in the Pydantic documentation.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"H2_Threading_vs_Multiprocessing_A_Comedy_of_Errors\"><\/span>H2: Threading vs. Multiprocessing: A Comedy of Errors.<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The &#8220;Crime Scene&#8221; code used <code>asyncio.gather(*tasks)<\/code>. This is a classic mistake. <code>asyncio<\/code> is designed for I\/O-bound tasks\u2014waiting for a database, waiting for a network socket, waiting for a file. It is <em>not<\/em> a tool for parallelizing CPU-bound data transformation. <\/p>\n<p>Because Python has a Global Interpreter Lock (GIL), only one thread can execute Python bytecode at a time. When the developers ran <code>await asyncio.gather(*tasks)<\/code>, they weren&#8217;t running 10 million transformations in parallel. They were telling the single-threaded event loop to manage 10 million coroutine objects. The overhead of the event loop trying to schedule 10 million tiny tasks actually took longer than if they had just used a simple <code>for<\/code> loop.<\/p>\n<p>I checked the <code>htop<\/code> output during the incident. One CPU core was pinned at 100% (the event loop thread), while the other 31 cores on the machine were sitting idle, mocking me. <\/p>\n<pre class=\"codehilite\"><code class=\"language-text\">  PID USER      PRI  NI  VIRT   RES   SHR S  CPU% MEM%   TIME+  Command\n14202 root      20   0 72.1G 61.4G  1240 R  99.9 96.2  0:45.12 python3 ingestor.py\n<\/code><\/pre>\n<p>If they actually needed parallelism, they should have used <code>ProcessPoolExecutor<\/code> to bypass the GIL, or better yet, kept the logic simple enough that the GIL wasn&#8217;t the bottleneck. But instead, they chose the &#8220;modern&#8221; <code>asyncio<\/code> approach because it&#8217;s the current &#8220;python best&#8221; trend, regardless of whether it fits the problem.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"H2_Dependency_Hell_and_the_400MB_Docker_Image\"><\/span>H2: Dependency Hell and the 400MB Docker Image.<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The container that failed was a bloated monstrosity. To support this &#8220;clean&#8221; code, the team added:<br \/>\n&#8211; <code>pydantic<\/code> (and its dependencies)<br \/>\n&#8211; <code>pandas<\/code> (because someone thought they might need it later)<br \/>\n&#8211; <code>httpx<\/code> (because <code>requests<\/code> is &#8220;old&#8221;)<br \/>\n&#8211; <code>uvloop<\/code> (to try and make <code>asyncio<\/code> faster, which is like putting a spoiler on a tractor)<\/p>\n<p>The resulting Docker image was 420MB. For a script that essentially moves data from point A to point B. When the OOM killer started nuking pods, the Kubernetes scheduler tried to restart them. But because the image was so large, the &#8220;ImagePullBackOff&#8221; errors started rolling in as the container registry struggled under the simultaneous pull requests from 48 nodes.<\/p>\n<p>We aren&#8217;t just writing code; we are shipping artifacts. A 400MB image for a data ingestor is a sign of architectural rot. It means you don&#8217;t know what your dependencies are doing. It means you&#8217;ve prioritized developer convenience (&#8220;I&#8217;ll just pip install the whole world&#8221;) over operational stability. In a real &#8220;python best&#8221; environment, we would be using <code>requirements.txt<\/code> with strict version pinning or a <code>pyproject.toml<\/code> that doesn&#8217;t include the kitchen sink. We would be using slim base images (like <code>python:3.11-slim-bookworm<\/code>) and multi-stage builds to keep the runtime footprint small.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"H2_Logging_as_a_Denial_of_Service_Attack\"><\/span>H2: Logging as a Denial of Service Attack.<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The final nail in the coffin was the logging. In an attempt to be &#8220;thorough,&#8221; the developers added a log line for every transformation failure. <\/p>\n<pre class=\"codehilite\"><code class=\"language-python\">logger.info(f&quot;Successfully processed item {item.sensor_id}&quot;)\n<\/code><\/pre>\n<p>When you have 10 million items, and you&#8217;re using the standard <code>logging<\/code> module, each log call is a synchronous I\/O operation. Even if you&#8217;re logging to <code>stdout<\/code>, that output has to be captured by the Docker logging driver, written to a JSON file on disk, and then picked up by a log aggregator (like Fluentd or Vector). <\/p>\n<p>As the system started to struggle with memory, the I\/O wait (iowait) spiked. The event loop, already overwhelmed by 10 million coroutines, was now blocking on every <code>logger.info<\/code> call. This is a self-inflicted Denial of Service. The system was spending more time formatting strings and waiting for the disk than it was processing data. <\/p>\n<p>I saw the <code>strace<\/code> output. The process was spending 60% of its time in <code>write()<\/code> syscalls. <\/p>\n<pre class=\"codehilite\"><code class=\"language-text\">% time     seconds  usecs\/call     calls    errors syscall\n------ ----------- ----------- --------- --------- ----------------\n 62.12    0.842102          12     68201           write\n 15.40    0.208712           8     25102           read\n<\/code><\/pre>\n<p>This is what happens when you apply &#8220;best practices&#8221; from a web tutorial to a high-throughput data pipeline. You don&#8217;t log every item in a 10-million-row batch. You log the start, the end, and a summary of errors. You use sampled logging. You use non-blocking, asynchronous logging if you absolutely must log during the loop.<\/p>\n<hr \/>\n<h3><span class=\"ez-toc-section\" id=\"4_The_Fix\"><\/span>4. The Fix<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>After 72 hours of cleaning up the mess, I rewrote the ingestor. It\u2019s not &#8220;pretty.&#8221; It doesn&#8217;t use Pydantic. It doesn&#8217;t use <code>asyncio.gather<\/code>. It uses 1\/100th of the memory and runs 10x faster.<\/p>\n<pre class=\"codehilite\"><code class=\"language-python\">import sys\nimport json\nimport time\nfrom typing import Iterable\n\n# Use slots to save memory: no __dict__, no __weakref__\nclass TelemetryData:\n    __slots__ = ('sensor_id', 'value', 'timestamp')\n    def __init__(self, sensor_id: int, value: float, timestamp: int):\n        self.sensor_id = sensor_id\n        self.value = value\n        self.timestamp = timestamp\n\ndef get_raw_data(limit: int) -&gt; Iterable[dict]:\n    &quot;&quot;&quot;A generator that yields data one by one.&quot;&quot;&quot;\n    for i in range(limit):\n        yield {\n            &quot;sensor_id&quot;: i,\n            &quot;value&quot;: i * 1.5,\n            &quot;timestamp&quot;: int(time.time())\n        }\n\ndef transform_stream(data_stream: Iterable[dict]) -&gt; Iterable[TelemetryData]:\n    &quot;&quot;&quot;Process data as a stream, keeping memory usage constant.&quot;&quot;&quot;\n    for item in data_stream:\n        try:\n            # Manual validation is faster than Pydantic in a hot loop\n            obj = TelemetryData(\n                sensor_id=int(item['sensor_id']),\n                value=float(item['value']) * 1.0000001,\n                timestamp=int(item['timestamp'])\n            )\n            yield obj\n        except (KeyError, ValueError) as e:\n            # Log errors, but don't kill the process\n            continue\n\ndef main():\n    start_time = time.perf_counter()\n\n    # The pipeline is a chain of generators\n    raw_stream = get_raw_data(10_000_000)\n    processed_stream = transform_stream(raw_stream)\n\n    count = 0\n    for _ in processed_stream:\n        count += 1\n        if count % 1_000_000 == 0:\n            print(f&quot;Processed {count} items...&quot;)\n\n    end_time = time.perf_counter()\n    print(f&quot;Finished {count} items in {end_time - start_time:.2f} seconds&quot;)\n\nif __name__ == &quot;__main__&quot;:\n    main()\n<\/code><\/pre>\n<p><strong>Why this works:<\/strong><br \/>\n1. <strong>Generators:<\/strong> The memory footprint is constant. Whether we process 10 items or 10 billion, the RSS stays under 100MB.<br \/>\n2. <strong><code>__slots__<\/code>:<\/strong> We eliminated the overhead of the instance dictionary.<br \/>\n3. <strong>No <code>asyncio<\/code> overhead:<\/strong> We aren&#8217;t managing millions of task objects. We&#8217;re just iterating.<br \/>\n4. <strong>No Pydantic in the hot path:<\/strong> We do basic type casting where it&#8217;s needed.<br \/>\n5. <strong>No eager list creation:<\/strong> We never call <code>list()<\/code> on our data.<\/p>\n<hr \/>\n<h3><span class=\"ez-toc-section\" id=\"5_The_%E2%80%9CNever_Again%E2%80%9D_List\"><\/span>5. The &#8220;Never Again&#8221; List<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>If you are a developer at this company, read these rules. Memorize them. If I see a PR that violates these without a damn good reason, I will reject it so hard your keyboard will shake.<\/p>\n<ol>\n<li><strong>Stop Eagerly Loading Data:<\/strong> If your dataset is larger than 10,000 items, you use a generator. You do not use a list comprehension. You do not use <code>asyncio.gather<\/code> on a list of 10 million tasks.<\/li>\n<li><strong>Pydantic is for Boundaries, Not Loops:<\/strong> Use Pydantic for validating incoming JSON at the API layer. Do not use it for internal data structures that are instantiated millions of times per second.<\/li>\n<li><strong><code>__slots__<\/code> is Your Friend:<\/strong> If you are defining a class that will have millions of instances, use <code>__slots__<\/code>. It\u2019s not &#8220;un-Pythonic&#8221;; it\u2019s &#8220;not-crashing-the-server-ic.&#8221;<\/li>\n<li><strong>Understand the GIL:<\/strong> Before you reach for <code>asyncio<\/code> or <code>threading<\/code>, ask yourself if your task is actually I\/O bound. If it&#8217;s math, it&#8217;s CPU bound. <code>asyncio<\/code> will only make it slower.<\/li>\n<li><strong>Log with Purpose:<\/strong> Do not log inside a high-frequency loop. If you must, use a counter and log every 10,000th iteration, or use a buffered, non-blocking logger.<\/li>\n<li><strong>Pin Your Versions:<\/strong> I don&#8217;t care if a new version of a library came out yesterday. We use specific, tested versions. Your &#8220;clean&#8221; update to Pydantic v2 during a minor release is what started this cascade.<\/li>\n<li><strong>Profile Before You &#8220;Optimize&#8221;:<\/strong> If you think your code is slow, use <code>cProfile<\/code> or <code>py-spy<\/code>. Don&#8217;t just add <code>async<\/code> and hope for the best.<\/li>\n<\/ol>\n<p>I\u2019m going home now. I\u2019m going to sleep for 14 hours. When I come back, I expect to see the memory usage graphs trending downward, or I\u2019m starting to look at Go jobs.<\/p>\n<p><strong>Report End.<\/strong><br \/>\n<strong>Signed,<\/strong><br \/>\n<em>The SRE who actually had to fix it.<\/em><\/p>\n<h2><span class=\"ez-toc-section\" id=\"Related_Articles\"><\/span>Related Articles<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Explore more insights and best practices:<\/p>\n<ul>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/10-essential-cybersecurity-best-practices-to-stay-safe-5\/\">10 Essential Cybersecurity Best Practices To Stay Safe 5<\/a><\/li>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/mastering-docker-compose-simplify-multi-container-apps-2\/\">Mastering Docker Compose Simplify Multi Container Apps 2<\/a><\/li>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/machine-learning-best-practices-10-tips-for-success-2\/\">Machine Learning Best Practices 10 Tips For Success 2<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>INCIDENT REPORT: #882-BRAVO-TANGO TIMESTAMP: 2023-10-14 03:14:22 UTC STATUS: RESOLVED (After 72 hours of manual state recovery) SYSTEM: Real-time Analytics Ingestion Engine (The &#8220;Data-Sucker-9000&#8221;) PRIMARY ON-CALL: Senior SRE (Current Mood: Resignation Letter is drafted) 1. Incident Summary At 03:14 UTC, the primary ingestion cluster for our &#8220;Real-time Analytics Dashboard&#8221; didn&#8217;t just fail; it committed ritualistic suicide. &#8230; <a title=\"Python Best Practices: Write Cleaner, Professional Code\" class=\"read-more\" href=\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/\" aria-label=\"Read more  on Python Best Practices: Write Cleaner, Professional Code\">Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-4856","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.0 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Python Best Practices: Write Cleaner, Professional Code - ITSupportWale<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Python Best Practices: Write Cleaner, Professional Code - ITSupportWale\" \/>\n<meta property=\"og:description\" content=\"INCIDENT REPORT: #882-BRAVO-TANGO TIMESTAMP: 2023-10-14 03:14:22 UTC STATUS: RESOLVED (After 72 hours of manual state recovery) SYSTEM: Real-time Analytics Ingestion Engine (The &#8220;Data-Sucker-9000&#8221;) PRIMARY ON-CALL: Senior SRE (Current Mood: Resignation Letter is drafted) 1. Incident Summary At 03:14 UTC, the primary ingestion cluster for our &#8220;Real-time Analytics Dashboard&#8221; didn&#8217;t just fail; it committed ritualistic suicide. ... Read more\" \/>\n<meta property=\"og:url\" content=\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/\" \/>\n<meta property=\"og:site_name\" content=\"ITSupportWale\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Itsupportwale-298547177495978\" \/>\n<meta property=\"article:published_time\" content=\"2026-08-11T16:05:04+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2021\/05\/android-chrome-512x512-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"512\" \/>\n\t<meta property=\"og:image:height\" content=\"512\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Techie\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Techie\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"13 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/\"},\"author\":{\"name\":\"Techie\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d\"},\"headline\":\"Python Best Practices: Write Cleaner, Professional Code\",\"datePublished\":\"2026-08-11T16:05:04+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/\"},\"wordCount\":2001,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#organization\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/\",\"url\":\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/\",\"name\":\"Python Best Practices: Write Cleaner, Professional Code - ITSupportWale\",\"isPartOf\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#website\"},\"datePublished\":\"2026-08-11T16:05:04+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/itsupportwale.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Python Best Practices: Write Cleaner, Professional Code\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#website\",\"url\":\"https:\/\/itsupportwale.com\/blog\/\",\"name\":\"ITSupportWale\",\"description\":\"Tips, Tricks, Fixed-Errors, Tutorials &amp; Guides\",\"publisher\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/itsupportwale.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#organization\",\"name\":\"itsupportwale\",\"url\":\"https:\/\/itsupportwale.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png\",\"contentUrl\":\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png\",\"width\":1119,\"height\":144,\"caption\":\"itsupportwale\"},\"image\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/Itsupportwale-298547177495978\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d\",\"name\":\"Techie\",\"sameAs\":[\"https:\/\/itsupportwale.com\",\"iswblogadmin\"],\"url\":\"https:\/\/itsupportwale.com\/blog\/author\/iswblogadmin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Python Best Practices: Write Cleaner, Professional Code - ITSupportWale","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/","og_locale":"en_US","og_type":"article","og_title":"Python Best Practices: Write Cleaner, Professional Code - ITSupportWale","og_description":"INCIDENT REPORT: #882-BRAVO-TANGO TIMESTAMP: 2023-10-14 03:14:22 UTC STATUS: RESOLVED (After 72 hours of manual state recovery) SYSTEM: Real-time Analytics Ingestion Engine (The &#8220;Data-Sucker-9000&#8221;) PRIMARY ON-CALL: Senior SRE (Current Mood: Resignation Letter is drafted) 1. Incident Summary At 03:14 UTC, the primary ingestion cluster for our &#8220;Real-time Analytics Dashboard&#8221; didn&#8217;t just fail; it committed ritualistic suicide. ... Read more","og_url":"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/","og_site_name":"ITSupportWale","article_publisher":"https:\/\/www.facebook.com\/Itsupportwale-298547177495978","article_published_time":"2026-08-11T16:05:04+00:00","og_image":[{"width":512,"height":512,"url":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2021\/05\/android-chrome-512x512-1.png","type":"image\/png"}],"author":"Techie","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Techie","Est. reading time":"13 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#article","isPartOf":{"@id":"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/"},"author":{"name":"Techie","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d"},"headline":"Python Best Practices: Write Cleaner, Professional Code","datePublished":"2026-08-11T16:05:04+00:00","mainEntityOfPage":{"@id":"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/"},"wordCount":2001,"commentCount":0,"publisher":{"@id":"https:\/\/itsupportwale.com\/blog\/#organization"},"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/","url":"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/","name":"Python Best Practices: Write Cleaner, Professional Code - ITSupportWale","isPartOf":{"@id":"https:\/\/itsupportwale.com\/blog\/#website"},"datePublished":"2026-08-11T16:05:04+00:00","breadcrumb":{"@id":"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/itsupportwale.com\/blog\/python-best-practices-write-cleaner-professional-code\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/itsupportwale.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Python Best Practices: Write Cleaner, Professional Code"}]},{"@type":"WebSite","@id":"https:\/\/itsupportwale.com\/blog\/#website","url":"https:\/\/itsupportwale.com\/blog\/","name":"ITSupportWale","description":"Tips, Tricks, Fixed-Errors, Tutorials &amp; Guides","publisher":{"@id":"https:\/\/itsupportwale.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/itsupportwale.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/itsupportwale.com\/blog\/#organization","name":"itsupportwale","url":"https:\/\/itsupportwale.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png","contentUrl":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png","width":1119,"height":144,"caption":"itsupportwale"},"image":{"@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Itsupportwale-298547177495978"]},{"@type":"Person","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d","name":"Techie","sameAs":["https:\/\/itsupportwale.com","iswblogadmin"],"url":"https:\/\/itsupportwale.com\/blog\/author\/iswblogadmin\/"}]}},"_links":{"self":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts\/4856","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/comments?post=4856"}],"version-history":[{"count":0,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts\/4856\/revisions"}],"wp:attachment":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/media?parent=4856"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/categories?post=4856"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/tags?post=4856"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}