Machine Learning Best Practices: 7 Tips for Success

by

INTERNAL POST-MORTEM: PROJECT “ICARUS” / INCIDENT REPORT #8842-B
TO: Engineering Leadership, DevOps, and anyone else who thinks they can “just run a script”
FROM: Silas Thorne, Principal Systems Architect (Infrastructure & Recovery)
SUBJECT: The Smoldering Remains of our “Machine Learning” Pipeline

It is 4:42 AM. I have been awake for thirty-eight hours. The air in the server room is thick with the smell of overstressed silicon and the bitter ozone of a failing UPS. I am currently staring at a terminal window that represents the professional epitaph of our former “Rockstar” Data Scientist, Chad. Chad has moved on to a “stealth-mode AI startup” in Palo Alto, leaving us with a repository that is less of a software project and more of a crime scene.

If you are reading this, it means I have successfully stabilized the production environment, or at least I’ve managed to stop the $450-per-hour bleeding from our AWS account. This document is not a suggestion. It is a mandatory autopsy of why our machine learning efforts failed and a manifesto for how we will operate moving forward. If you disagree with anything here, my office door is locked, and I am ignoring all Slack notifications until next Tuesday.

THE INCIDENT: 03:14 AM

The following log is the last thing our primary node saw before it decided to commit digital seppuku.

[2024-05-20 03:14:18] INFO: Starting epoch 42...
[2024-05-20 03:14:19] DEBUG: Loading batch 14400/50000
[2024-05-20 03:14:21] WARNING: Memory pressure detected. System RAM at 98.4%.
[2024-05-20 03:14:22] ERROR: torch.cuda.OutOfMemoryError: CUDA out of memory. 
Tried to allocate 12.50 GiB (GPU 0; 40.00 GiB total capacity; 38.22 GiB already allocated; 
1.12 GiB free; 38.50 GiB reserved in total by PyTorch 2.2.0)
[2024-05-20 03:14:22] CRITICAL: Kernel Panic - not syncing: Fatal exception in interrupt
[2024-05-20 03:14:23] CONNECTION_LOST: Worker node ip-10-0-42-11.ec2.internal unreachable.

The “Rockstar” forgot that tensors don’t magically vanish when you’re done with them if you keep them in a global list for “later visualization.” He was caching every single intermediate activation from a transformer model during a production inference run. On a 40GB A100. At 3:00 AM, the garbage collector finally gave up, and the OOM killer took the entire API gateway down with it.

1. The Fallacy of the Infinite Cloud Budget

Chad’s philosophy was simple: if the code is slow, throw more compute at it. I found a Terraform script that was spinning up p4d.24xlarge instances for “exploratory data analysis.” We were spending $32 an hour so a guy could run a regex on a 2GB CSV file.

In this machine learning pipeline, the data loading was so inefficient that the GPUs were idling 85% of the time, waiting for the CPU to unpickle Python objects. He was using pickle for data serialization. In 2024. Not only is that a security nightmare, but it’s also incredibly slow. I’ve replaced this with Apache Arrow and Parquet, but the damage to our quarterly budget is already done.

We are not a charity for NVIDIA. From this point forward, every model training run must have a projected cost-benefit analysis. If you cannot explain why you need 96GB of VRAM to classify customer support tickets, you don’t get the keys to the cluster. We are moving back to spot instances with aggressive checkpointing. If your code can’t handle a SIGTERM and resume from a saved state, your code isn’t production-ready.

2. Dependency Hell is a Choice (and You Chose Poorly)

I spent six hours yesterday trying to replicate the environment. Chad’s requirements.txt was a work of fiction. It contained torch==2.2.0, but the code relied on a specific bug in torchvision 0.17.0 that was patched three weeks ago. Even worse, he had manually compiled a custom C++ extension against a version of the LLVM compiler that only exists on his specific MacBook Pro.

Here is a snippet of the pip freeze I managed to scrape from the dying container:

numpy==1.26.4
pandas==2.2.2
scikit-learn==1.4.2
scipy==1.13.0
torch==2.2.0
torchvision==0.17.0
# The following were installed via direct git links with no commit hashes:
git+https://github.com/some-random-repo/experimental-layers.git

Do you see that last line? That is a ticking time bomb. That repository was updated two days ago, breaking the API. Because there was no commit hash, the CI/CD pipeline pulled the “latest” version, which was incompatible with our inference logic.

The New Rule: Every dependency must be pinned. Not just the version, but the hash. We are moving to Poetry or uv for dependency management. If I see a pip install without a version number in a Dockerfile, I will personally revoke your sudo access. We are also standardizing on Python 3.11.8. No more “I’m using 3.12 for the speed” while the rest of the stack is on 3.9.

3. Why Your Notebook is a Liability, Not an Asset

I found a folder named final_models/ containing 42 Jupyter notebooks. They were named train_v1.ipynb, train_v2_FIXED.ipynb, train_v2_FIXED_FINAL.ipynb, and my personal favorite, train_v2_FIXED_FINAL_USE_THIS_ONE_FOR_REAL.ipynb.

Notebooks are for sketching. They are not for production. The “Rockstar” was running cells out of order, creating a hidden state that made it impossible to reproduce his results. He would run cell 1, then cell 5, then cell 2, and then wonder why the model accuracy was 99% (spoiler: he was leaking the label into the feature set).

When we tried to export this to a .py script, the model performance dropped to 60%. Why? Because the notebook had a global variable X_train that had been modified in a cell he deleted, but the kernel hadn’t been restarted in three weeks.

The New Rule: No code goes to production unless it is a modular, testable Python package. If it’s in a .ipynb file, it doesn’t exist. I want to see pytest suites for your data transformations. I want to see type hints. If you can’t run mypy on your machine learning code without it lighting up like a Christmas tree, you aren’t done.

4. Data Versioning (DVC) or Death

The most terrifying part of this “machine learning” pipeline was the data. Or rather, the lack of it. Chad had a script that pulled data from a production SQL database with a WHERE created_at > '2023-01-01' clause.

Every time that script ran, the dataset changed. There was no snapshot. No versioning. No way to go back and see what data produced the “Model_v4” that is currently hallucinating prices for our European customers. We have no reproducibility. We are essentially practicing alchemy, not engineering.

I have spent the last 12 hours setting up DVC (Data Version Control) version 3.50.1. 

# dvc.yaml
stages:
  process_data:
    cmd: python src/process.py data/raw data/processed
    deps:
      - data/raw
      - src/process.py
    outs:
      - data/processed
  train_model:
    cmd: python src/train.py data/processed models/model.pkl
    deps:
      - data/processed
      - src/train.py
    outs:
      - models/model.pkl

If you change a single row in the training set, I want a new hash. I want to be able to git checkout a specific commit and have the exact dataset, the exact code, and the exact model weights appear. If we can’t reproduce a failure, we can’t fix it.

5. Feature Drift and the Prometheus/Grafana Altar

The “Rockstar” told management the model was “self-correcting.” That is a lie. Models don’t self-correct; they degrade.

I checked the logs. Our input data distribution shifted three months ago when the marketing team changed the lead-gen form. The model, built on scikit-learn 1.4.2, was expecting a normalized range between 0 and 1. Marketing started sending integers between 1 and 100. The model didn’t crash; it just started giving garbage outputs. And because we had no monitoring, we’ve been serving garbage to our users for 90 days.

We are now implementing a full observability stack. I don’t care about your “accuracy” on a static test set from last year. I care about the Kolmogorov-Smirnov test results on our live features.

# prometheus_exporter_config.yaml
metrics:
  - name: feature_drift_score
    type: gauge
    help: "Distance between training and serving data distributions"
    labels: [feature_name, model_version]
  - name: prediction_latency_ms
    type: histogram
    help: "Time taken for model inference"
    buckets: [10, 50, 100, 500, 1000]

Every model endpoint will now export metrics to Prometheus. We will have Grafana dashboards that scream at us when the input distribution changes. If the model starts predicting “0” for 95% of requests, I want a PagerDuty alert to wake you up, not me.

6. The Physical Reality of the Rack

You people think the “cloud” is some ethereal dimension where logic lives. It’s not. It’s a series of Dell PowerEdge R750s in a data center in Northern Virginia that are currently running so hot they could bake bread.

When you write an inefficient loop in Python that iterates over a pandas 2.2.2 DataFrame instead of using vectorized operations, you are physically heating up a room. You are consuming real-world electricity. You are contributing to the heat death of the universe because you were too lazy to learn how numpy broadcasting works.

The “Rockstar” had a nested loop that was $O(n^2)$ for a join operation that could have been a single hash map lookup. This isn’t just “bad code.” It’s an insult to the engineers who built the hardware you’re abusing. We are going to start auditing the computational complexity of our training scripts. If your “machine learning” training job takes 48 hours, I will be looking at your code to see if it should take 48 hours, or if you’re just a bad programmer.

7. “It Worked on My Laptop” is a Fireable Offense

The final straw was when I told Chad the production build was failing. His response? “That’s weird, it worked on my laptop.”

Your laptop has 64GB of RAM, a different version of GLIBC, and you’re running macOS. The production environment is a stripped-down Alpine Linux container running on an EPYC processor. Your laptop is irrelevant. If it doesn’t work in the container, it doesn’t work.

We are moving to a “Container-First” development workflow. You will use DevContainers. You will test your code in an environment that mirrors production. If I hear the words “on my machine” one more time, I will personally delete your ~/.ssh folder.

Checklist for the Next Person Who Tries to Break My Production Environment

Before you even think about pushing a “machine learning” model to the staging branch, you will verify the following:

  1. Deterministic Seeding: Have you set random.seed(), np.random.seed(), and torch.manual_seed()? If I run your script twice, do I get the exact same weights? If not, go back to the drawing board.
  2. Dependency Lockdown: Is your pyproject.toml or requirements.txt fully pinned? Does it include the specific version of CUDA and cuDNN required?
  3. Memory Profiling: Have you run mprof or a similar tool to check for memory leaks? Does your memory usage scale linearly with batch size, or is there a hidden leak in your evaluation loop?
  4. Data Integrity: Is the data pulled via a DVC-tracked hash? Is there a schema validation step (using Pydantic or Pandera) to ensure the input features haven’t changed?
  5. Logging vs. Printing: Did you remove all print("here") statements and replace them with structured logging? Do the logs include the model version and the request ID?
  6. Unit Tests for Logic: Do you have tests for your custom loss functions? Do you have tests for your data augmentation pipeline? (Hint: If your augmentation flips an image but doesn’t flip the bounding box, your model is learning nonsense).
  7. Resource Limits: Have you defined resources: limits: and requests: in your Kubernetes manifest? If your pod gets killed for exceeding its limit, do you have a plan for how the system recovers?
  8. The “Chad” Test: If I delete your entire home directory right now, can I still rebuild and deploy the model using only what is in the Git repository?

I am going home now. I am going to sleep for fourteen hours. When I come back, I expect to see this repository cleaned up. I have left a script in the root directory called cleanup_mess.sh. It will delete every .ipynb file it finds. You have until I wake up to save your work.

This is not a “vibrant” community of “rockstars.” This is an engineering department. Start acting like it.

Silas Thorne
Principal Systems Architect
Department of Fixing Other People’s Mistakes

Technical Specs of the Recovered System (for the record):
– OS: Ubuntu 22.04.4 LTS (Jammy Jellyfish)
– Kernel: 5.15.0-101-generic
– NVIDIA Driver: 550.54.14
– Python: 3.11.8
– PyTorch: 2.2.0+cu121
– NumPy: 1.26.4
– Pandas: 2.2.2
– Scikit-Learn: 1.4.2
– DVC: 3.50.1
– Prometheus Client: 0.20.0

Related Articles

Explore more insights and best practices:

Leave a Comment