Machine Learning Best Practices – Guide

by

Stop Treating Your Models Like Precious Snowflakes: A No-Nonsense Guide to Machine Learning in Production

In 2021, I watched a $50,000-a-day ad-spend budget evaporate over a single weekend because of a pandas version mismatch. We had a “state-of-the-art” recommendation engine running on a cluster of 20 nodes. The data science team had updated their local environment to pandas 1.3.0 to use some new grouping feature, but the production environment was pinned to 1.1.5. Because of the way Python handles silent failures in certain vectorized operations, the model didn’t crash. It just started outputting zeros for every user’s “interest score.” The system defaulted to showing generic ads for industrial-grade cat litter to everyone, including teenagers and corporate CEOs. By the time the alerts fired, we’d burned through the quarterly experimental budget.

That wasn’t a “machine learning” failure. It was a boring, predictable, and entirely avoidable engineering failure. We spent three days post-mortem-ing the “math” when the culprit was a simple requirements.txt drift. This is the reality of machine learning at scale. It’s 10% math and 90% plumbing, and most of your plumbing is probably leaking. If you’re here for a tutorial on how to tune hyperparameters or choose between a Random Forest and a Transformer, leave. This is about the unglamorous work of making sure your model doesn’t OOM-kill your Kubelet at 3:00 AM on a Sunday.

The “Data Science” Mirage

Most documentation for machine learning libraries is written for researchers. They assume you have an infinite amount of RAM, a single GPU that you own exclusively, and that your data lives in a pristine CSV file on your desktop. In the real world, your data is a messy stream from a Kafka topic, your RAM is shared with three other microservices, and your GPU is a precious resource managed by a Kubernetes scheduler that hates you.

The industry is obsessed with “accuracy.” I don’t care about your 99% F1 score if your model takes 400ms to respond. In a high-frequency trading environment or a real-time bidding system, a 90% accurate model that responds in 10ms is infinitely more valuable than a “perfect” model that introduces a half-second of tail latency. We need to stop optimizing for the leaderboard and start optimizing for the 99th percentile latency (p99).

The Serialization Trap: Why Pickle is a Security Risk

If you are still using pickle to save your models, you are essentially leaving your front door unlocked in a bad neighborhood. pickle is not just inefficient; it’s a literal remote code execution (RCE) vulnerability. When you call pickle.load(), you are telling Python to execute whatever instructions are in that file. If an attacker swaps your model file for a malicious payload, they own your production server.

Beyond security, pickle is brittle. It stores references to classes and modules, not the actual code. If you move your model from app.models.classifier to app.ml.classifier, pickle will break. Use safetensors for deep learning or ONNX (Open Neural Network Exchange) for general interoperability. 


# The wrong way (Pickle)
import pickle
with open("model_v1.pkl", "wb") as f:
    pickle.dump(my_expensive_model, f)

# The right way (ONNX)
import torch.onnx
torch.onnx.export(model, dummy_input, "model.onnx", 
                  input_names=['input'], 
                  output_names=['output'],
                  dynamic_axes={'input': {0: 'batch_size'}, 'output': {0: 'batch_size'}})

ONNX allows you to run your model in a C++ runtime or even in the browser. It forces you to define your inputs and outputs strictly. This prevents the “it worked on my machine” syndrome where a data scientist passes a list of strings but the production API expects a NumPy array of floats.

Pro-tip: Always use mmap (memory mapping) when loading large model weights. It allows the OS to swap parts of the model in and out of memory, preventing the entire 8GB file from bloating your RSS (Resident Set Size) immediately.

Containerization: Debian-Slim over Alpine

The “hype” says use Alpine Linux for everything because it’s small. For machine learning, Alpine is a nightmare. Most ML libraries (NumPy, PyTorch, TensorFlow) rely on glibc. Alpine uses musl. When you pip install these libraries on Alpine, you can’t use the pre-compiled wheels. Your Docker build will spend 45 minutes compiling C++ code from source, only to fail because of a missing header file.

Use python:3.11-slim-bookworm. It’s slightly larger, but it’s based on Debian, uses glibc, and just works. Your CI/CD pipeline will thank you.

  • Layer Caching: Copy your requirements.txt first, run pip install, and then copy your source code. This ensures that a one-line change in a README doesn’t trigger a 10-minute re-download of PyTorch.
  • Multi-stage Builds: Use a “build” stage to compile any custom C extensions and a “run” stage to keep the final image lean.
  • Non-root Users: Never run your model server as root. USER 1001 is your friend.
  • CUDA Versions: If you’re using GPUs, match your nvidia-container-toolkit version on the host with the devel image version in your Dockerfile. A mismatch between libcuda.so.1 and the driver version is the #1 cause of “GPU not found” errors in production.

# Example Dockerfile for a pragmatic ML service
FROM python:3.11-slim-bookworm AS builder

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential gcc && \
    rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --user --no-cache-dir -r requirements.txt

FROM python:3.11-slim-bookworm

WORKDIR /app
COPY --from=builder /root/.local /root/.local
COPY . .

ENV PATH=/root/.local/bin:$PATH
USER 1001

CMD ["uvicorn", "api.main:app", "--host", "0.0.0.0", "--port", "8080"]

The Training-Serving Skew: Your Silent Killer

The most dangerous bug in machine learning is the “Training-Serving Skew.” This happens when the features you use during training are calculated differently than the features you use during inference. I once saw a model that used “average user spend over 30 days” as a feature. During training, this was calculated using a SQL query that looked at historical logs. In production, it was calculated using a Redis cache that only updated once every 24 hours. The model was making decisions based on data that was 23 hours stale, but the “accuracy” metrics looked perfect because the training data was “fresh.”

To fix this, you need a unified feature engineering pipeline. If you use a library like Pandas for training, you should probably be using something like Pydantic to validate the incoming JSON in production before it ever touches your model.


from pydantic import BaseModel, Field, validator

class InferenceRequest(BaseModel):
    user_id: int
    session_duration: float = Field(..., gt=0)
    last_purchase_amount: float

    @validator('session_duration')
    def check_sanity(cls, v):
        if v > 86400: # More than 24 hours? Something is wrong.
            raise ValueError('Session duration exceeds one day')
        return v

Don’t trust the data coming from the frontend. Don’t trust the data coming from the database. Validate everything. If a feature is missing, don’t just fill it with a 0 or a -1 unless your model was specifically trained to handle those as “missing” indicators. A NaN in production can propagate through a neural network and turn your entire output vector into [NaN, NaN, NaN], which usually results in a 500 Internal Server Error or, worse, a silent failure.

Monitoring: Beyond the 200 OK

Standard SRE monitoring focuses on the “Golden Signals”: Latency, Traffic, Errors, and Saturation. For machine learning, these are necessary but insufficient. You need to monitor the distribution of your data. This is called “Drift Detection.”

If your model was trained on data from users in the US, and suddenly you launch in Japan, your model’s performance will likely crater. Your HTTP status codes will still be 200 OK. Your latency will be fine. But your predictions will be garbage. You need to export custom metrics to Prometheus that track the mean and variance of your input features and your output predictions.

  • Prediction Histograms: If your model usually predicts a “0.1” probability and suddenly starts predicting “0.9,” something has changed in the world (or your data pipeline).
  • Confidence Scores: If you’re using a classifier, track the entropy of the output. High entropy means the model is “confused.”
  • Feature Null Rates: If a feature that is usually present 99% of the time suddenly drops to 50%, an upstream data source is broken.
  • Model Versioning: Always include the model version as a label in your Prometheus metrics. predictions_total{model_version="v1.2.4", status="success"}.

from prometheus_client import Counter, Histogram

PREDICTION_SCORE = Histogram('model_prediction_score', 'Distribution of model scores', ['model_version'])
INPUT_VALUE = Histogram('model_input_feature_value', 'Distribution of input feature X', ['feature_name'])

def predict(data):
    score = model.predict(data)
    PREDICTION_SCORE.labels(model_version="2.4.1").observe(score)
    return score

The “Shadow Mode” Deployment

Never, ever do a “Big Bang” deployment of a new machine learning model. I don’t care how much backtesting you’ve done. The real world is weirder than your test set. Use “Shadow Mode” (also known as “Dark Launching” or “Teeing”).

In Shadow Mode, your application sends the incoming request to both the current production model and the new candidate model. You return the production model’s result to the user, but you log the candidate model’s result. After 24-48 hours, you compare the two. Did the new model predict significantly different outcomes? Did it crash on edge cases that weren’t in the training set? Did it double the memory usage of the pod?

This is much more effective than a Canary deployment for ML. In a Canary, if the new model is “bad” but doesn’t crash, you might be giving 5% of your users a terrible experience for hours. In Shadow Mode, the risk to the user is zero.

Note to self: When implementing Shadow Mode, make sure the shadow call is asynchronous or wrapped in a tight timeout. You don’t want a slow “experimental” model to drag down the latency of your actual production response.

The Infrastructure Overhead: Why FastAPI isn’t always enough

FastAPI is great for quick APIs. But for high-throughput machine learning, it can become a bottleneck. Python’s Global Interpreter Lock (GIL) is a constant thorn in our side. If your model is CPU-bound (like many Scikit-Learn models), a single FastAPI worker will only use one core. If you spin up 10 workers, you now have 10 copies of your 2GB model in RAM, which is a great way to get OOM-killed by Kubernetes.

Consider using a dedicated model server like NVIDIA Triton or TorchServe. These tools are written in C++/Java and handle things like:

  • Dynamic Batching: They wait for a few milliseconds to group multiple individual requests into a single batch, which is much more efficient for GPUs.
  • Model Versioning: They can host multiple versions of a model simultaneously and allow you to route traffic via API headers.
  • Shared Memory: They load the model once and share it across multiple worker threads.
  • Multi-framework support: You can run a PyTorch model and a TensorFlow model in the same process.

If you must stay in Python-land, use gunicorn with the uvicorn.workers.UvicornWorker and carefully tune your --workers count. A good rule of thumb is (2 x num_cores) + 1, but for ML, you often have to go lower to avoid memory exhaustion.

The “Real World” Gotcha: The Cold Start Problem

Here is something they don’t tell you in the “Intro to ML” courses: loading a model takes time. A 5GB model might take 30 seconds to load from a standard SSD and another 10 seconds to move to GPU memory. If your Kubernetes HPA (Horizontal Pod Autoscaler) triggers a scale-up event because of a traffic spike, those new pods won’t be ready to handle traffic for nearly a minute. By then, your existing pods might have already crashed under the load.

You need robust readinessProbes and livenessProbes. A readinessProbe should actually check if the model is loaded in memory and ready to predict, not just if the web server is running.


# Kubernetes snippet
readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 5
  failureThreshold: 10

Also, consider “Pre-warming.” After the model is loaded, run a few “dummy” predictions through it. This initializes the CUDA kernels and fills the caches. The first prediction on a fresh model is always significantly slower than the subsequent ones. Don’t let your first actual user be the one to pay that “init tax.”

YAML-Hell and Reproducibility

If I ask you to redeploy the model you had in production six months ago, could you do it? Most people can’t. They have the code in Git, but the model weights are in an S3 bucket named model-final-v2-fixed-REALLY-FINAL.onnx, and they don’t remember which version of the preprocessing script was used.

You need a “Model Registry.” This doesn’t have to be a fancy paid product. It can be a structured S3 bucket where every model is stored as /models/{model_name}/{git_sha}/{timestamp}/model.onnx. Along with the model, store a metadata.json that includes the exact versions of every library used during training.

Stop using pip install [package]. Use pip install [package]==[version]. Better yet, use a lockfile (poetry.lock or requirements.txt generated by pip-compile). In machine learning, a minor version bump in a dependency can change the output of a floating-point calculation just enough to degrade your model’s performance in a way that is nearly impossible to debug.

The Wrap-up

Machine learning is not a magic wand; it is a complex, stateful, and resource-heavy dependency that you are introducing into your stack. Treat it with the same skepticism you would treat a legacy C++ library or a temperamental third-party API. Prioritize observability over complexity, favor safetensors over pickle, and never trust a model that hasn’t survived 24 hours of Shadow Mode. Stop chasing the state-of-the-art and start chasing the state-of-the-stable.

Related Articles

Explore more insights and best practices:

Leave a Comment