{"id":4787,"date":"2026-05-12T22:49:42","date_gmt":"2026-05-12T17:19:42","guid":{"rendered":"https:\/\/itsupportwale.com\/blog\/machine-learning-best-practices-a-guide-to-success\/"},"modified":"2026-05-12T22:49:42","modified_gmt":"2026-05-12T17:19:42","slug":"machine-learning-best-practices-a-guide-to-success","status":"publish","type":"post","link":"https:\/\/itsupportwale.com\/blog\/machine-learning-best-practices-a-guide-to-success\/","title":{"rendered":"Machine Learning Best Practices: A Guide to Success"},"content":{"rendered":"

[2023-10-14 03:14:22.891] ERROR: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.50 GiB (GPU 0; 15.78 GiB total capacity; 11.20 GiB already allocated; 2.45 GiB free; 12.10 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
\n[2023-10-14 03:14:22.892] TRACEBACK:
\n File “\/opt\/venv\/lib\/python3.10\/site-packages\/torch\/nn\/modules\/module.py”, line 1501, in _call_impl
\n return forward_call(args, <\/em>*kwargs)
\n File “\/app\/models\/transformer_v4_final_FINAL_v2.py”, line 442, in forward
\n x = self.attention(x)
\n File “\/app\/utils\/dirty_hacks.py”, line 12, in attention
\n return torch.matmul(q, k.transpose(-2, -1)) \/ math.sqrt(d_k)
\n[2023-10-14 03:14:23.004] CRITICAL: Worker process (PID 4421) exited with code 1.
\n[2023-10-14 03:14:23.005] MONITORING: Alert ‘Inference_Service_Down’ fired. Severity: P0.
\n[2023-10-14 03:14:23.005] LOG: Attempting to dump local variables to \/tmp\/crash_dump_031422.json…
\n[2023-10-14 03:14:23.006] ERROR: [Errno 28] No space left on device: ‘\/tmp\/crash_dump_031422.json’<\/p>\n

I didn't get a call at 3:14 AM because the system was "intelligent." I got a call because the system was a bloated, unmonitored corpse of a project that finally stopped twitching. \n\nFor six months, the "Research Team" had been patting themselves on the back for their "machine learning" breakthroughs. They had a Jupyter Notebook that produced a beautiful ROC curve. They had a slide deck that promised 99.2% accuracy on fraud detection. What they didn't have was a single line of production-ready code, a stable environment, or any understanding of how data actually moves through a network.\n\nThis is the post-mortem of Project Sentinel. It\u2019s a story about how "machine learning" is 10% math and 90% plumbing\u2014and how, if you ignore the plumbing, you end up drowning in technical debt and broken CUDA kernels.\n\n## The Dependency Hell of Python 3.8.10\n\nThe nightmare started with the environment. Or rather, the lack of one. When I was handed the repository, the `README.md` was a single sentence: "Run the notebook." There was no `requirements.txt`, no `pyproject.toml`, and certainly no `Dockerfile`. \n\nI spent the first forty-eight hours trying to reconstruct the environment. The researchers had been using a mix of `conda` and `pip` on their local MacBooks, installing packages at random. When I finally managed to extract a `pip freeze` from one of their machines, it looked like a suicide note.\n\n```text\n# Partial output from the 'research_env_v1'\nnumpy==1.21.0\npandas==1.3.5\ntorch==1.10.0+cu111\ntorchvision==0.11.1+cu111\nscikit-learn==0.24.2\nscipy==1.7.3\nmatplotlib==3.4.3\n# Why is this here?\ntensorflow-gpu==2.6.0\n# And this?\nprotobuf==3.19.1\n# This version of urllib3 has a known CVE\nurllib3==1.26.7\n<\/code><\/pre>\n
The first thing I did was burn it down. You cannot build a reliable “machine learning” pipeline on Python 3.8.10 in 2023. We migrated to Python 3.11.4. Why? Because the performance improvements in the runtime are non-negotiable when you\u2019re processing 50,000 events per second. <\/p>\n
We moved to PyTorch 2.1.0 to leverage the torch.compile<\/code> feature, which promised a 20% speedup on our inference kernels. But moving versions isn’t just about changing a number in a file. It\u2019s about the cascading failures of every sub-dependency. protobuf 3.20.3<\/code> doesn’t like tensorflow 2.14.0<\/code> unless you pin the specific C++ implementation. scikit-learn 1.3.0<\/code> changed the way it handles certain array inputs, breaking the preprocessing scripts that the researchers had “carefully” crafted.<\/p>\n
If you aren’t pinning your versions to the third decimal point, you aren’t doing “machine learning”; you’re playing Russian Roulette with a fully loaded cylinder.<\/p>\n
\n
Table of Contents<\/p>\n