Machine Learning Best Practices: 10 Tips for Success

by

[2023-10-27 03:14:22,891] ERROR: [Worker-7] MainProcess: Uncaught exception: MemoryError
File “/usr/local/lib/python3.11/site-packages/pandas/core/internals/blocks.py”, line 393, in grow_labels
new_labels = np.empty(new_shape, dtype=labels.dtype)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 14.2 GiB for an array with shape (1905421, 1000) and data type float64
File “/app/models/recommender.py”, line 142, in predict
X_transformed = self.pipeline.transform(raw_data)
File “/usr/local/lib/python3.11/site-packages/sklearn/pipeline.py”, line 549, in transform
Xt = transform.transform(Xt)
File “/usr/local/lib/python3.11/site-packages/sklearn/preprocessing/_encoders.py”, line 987, in transform
return self._transform(X, handle_unknown=self.handle_unknown)
File “/usr/local/lib/python3.11/site-packages/sklearn/preprocessing/_encoders.py”, line 455, in _transform
X_int, X_mask = self._transform_selected_batch(
… [TRUNCATED BY KERNEL OOM KILLER] …
[2023-10-27 03:14:23,002] CRITICAL: [Kernel] Out of memory: Killed process 142 (python3) total-vm:28452104kB, anon-rss:26142104kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:56212kB oom_score_adj:0

TO: Engineering All, Data Science Team, Leadership
FROM: Senior Site Reliability Engineer (On-Call Rotation 4)
SUBJECT: POST-MORTEM: Incident #8842 – The “Clever” Model That Ate the Cluster
STATUS: Resolved (Temporarily, until someone touches the code again)

I’ve been awake for 72 hours. My eyes feel like they’ve been scrubbed with industrial-grade steel wool, and my blood is approximately 40% espresso by volume. While most of you were enjoying your weekend, I was watching our Kubernetes nodes drop like flies because someone thought it was a “great idea” to push a 14GB unoptimized feature matrix into a production environment with a 16GB RAM limit.

Let’s be crystal clear: Machine Learning is not magic. It is not a “vibrant” new frontier. It is a dangerous, stateful, resource-heavy liability that you have collectively decided to treat like a sandbox. This is a survival guide. Read it, or I will personally revoke your kubectl access and make you hand-calculate your gradients on a chalkboard.

1. INCIDENT-8842: THE FALLACY OF THE JUPYTER NOTEBOOK

The root cause of this weekend’s catastrophe was a fundamental misunderstanding of the difference between a research environment and a production system. In your Jupyter notebooks, you have the luxury of “restarting the kernel.” In production, the kernel is the Linux OS, and when it runs out of memory, it doesn’t give you a polite little red box. It kills the process.

The model in question used pandas==2.1.1 and scikit-learn==1.3.2. The “clever” part? A OneHotEncoder that wasn’t configured with sparse_output=True. On the training set of 10,000 rows, this worked fine. When it hit the production stream of 2 million concurrent users, the resulting dense matrix attempted to allocate 14.2 GiB of contiguous memory.

Python’s memory management is already a joke, but when you combine it with numpy==1.26.2‘s demand for contiguous blocks, you aren’t just asking for a crash—you’re demanding one. A notebook is a lie. It’s a curated, stateful snapshot of a moment in time where you ignored technical debt because the “accuracy” looked good. Accuracy doesn’t pay the bills when the p99 latency spikes to 45 seconds because the garbage collector is fighting for its life.

2. DEPENDENCY HELL IS A CHOICE (AND YOU CHOSE WRONG)

I looked at the requirements.txt for this service. It wasn’t a requirement file; it was a suicide note. Half the libraries weren’t pinned, and the other half were pinned to versions that haven’t been patched since the Obama administration.

Here is a snippet of the pip freeze I pulled from the dying container:

# The "I don't care about security or stability" starter pack
numpy==1.26.2
pandas==2.1.1
scikit-learn==1.3.2
scipy==1.11.3
torch==2.1.2+cu121
tensorflow==2.15.0
requests==2.31.0
urllib3==2.0.7
# Why is this even here?
matplotlib==3.8.2
jupyter-client==8.6.0

Why is matplotlib in a production inference container? Are we plotting graphs for the CPU to look at while it dies? Why is torch AND tensorflow in the same environment? This is 4GB of dead weight in the container image before we even load a single weight file.

Every unpinned dependency is a ticking time bomb. When pip decides to resolve a sub-dependency and pulls in a new version of urllib3 that breaks your requests call, that’s on you. We use poetry or pip-compile for a reason. If I see a requirements.txt without hashes in the next PR, I’m rejecting it without looking at the code.

3. SERIALIZATION: PICKLE IS A SECURITY VULNERABILITY, NOT A FORMAT

We found a 2.4GB .pkl file in the root directory of the container. Not only is pickle notoriously insecure—allowing for arbitrary code execution if someone poisons the model file—but it is also incredibly inefficient for large-scale data.

Worse, you’re tracking this in Git. Git is for code. Git is for text. Git is not for 2GB binary blobs of “cleverness.” Every time someone runs git clone, they are downloading every version of that model you’ve ever pushed. Our CI/CD pipeline slowed down by 400% this month because the runner has to pull 15GB of historical garbage just to run a linter.

Use Data Versioning (DVC). Use S3 or an artifact registry. Use joblib with compression or, better yet, export to ONNX or TensorRT if you actually care about performance. A .pkl file is a lazy solution for a lazy engineer. It couples your model to the specific version of the library that created it. If we upgrade scikit-learn from 1.3.2 to 1.4.0, your pickle might not even load. That’s not a feature; that’s a bug.

4. THE LATENCY TAX AND THE p99 LIE

The “Data Science” report said this model had a 98% accuracy. Great. Fantastic. Do you know what the p99 latency was? 1,200ms.

In a microservices architecture, a 1.2-second block is an eternity. It causes upstream connection pooling to exhaust, it triggers retries that lead to a “thundering herd” effect, and it eventually brings down the API gateway.

You cannot treat a model like a black box. You need to understand the computational complexity of your transform calls.
– pandas operations are single-threaded.
– The Python Global Interpreter Lock (GIL) is still a thing in 3.11.
– Every time you call .apply(lambda x: ...) on a DataFrame, a part of my soul dies, and a CPU core sits idle while one core screams.

If your model can’t return a prediction in under 50ms, it doesn’t belong in the synchronous request path. Put it in a Celery worker, use a message queue like RabbitMQ, or optimize the math. Stop blaming the “infrastructure” for your slow code. The infrastructure is fine; your O(n^2) feature engineering is the problem.

5. STATEFULNESS: THE SILENT KILLER

Production systems should be as stateless as possible. Your model, however, is a giant ball of state. You’re loading weights into memory, caching feature lookups in global variables, and—in the most egregious case I saw this weekend—writing temporary files to /tmp without cleaning them up.

The incident was exacerbated because the model was “warming up” on startup. It took 4 minutes for the container to become “Ready.” In Kubernetes, if your readinessProbe fails for 4 minutes, the orchestrator thinks the pod is dead and kills it. Then it starts a new one. Which also takes 4 minutes to warm up.

This is called a “CrashLoopBackoff.” We had 50 pods stuck in this loop, each consuming 12GB of RAM during their “warm-up” phase, which eventually starved the kubelet itself.

Cold Start Requirements:
1. Your model should load in under 30 seconds.
2. If it needs more time, you must use a sidecar or a pre-stop/post-start hook that doesn’t block the main thread.
3. Use mmap for loading weights if possible so the OS can handle page faults instead of you loading the whole thing into the heap.

6. MONITORING IS NOT JUST A DASHBOARD

I saw the Grafana dashboard you built. It shows “Model Accuracy” and “Number of Predictions.” That’s not monitoring; that’s vanity.

Real monitoring for ML in production requires:
– Memory RSS vs. Limit: I want to see how close we are to the OOM killer at all times.
– Feature Drift: If the input data distribution changes, your model is hallucinating. We need alerts for that.
– Inference Latency Percentiles: p50, p90, p99, and p99.9.
– Saturation: How many requests are waiting for a thread to become available?

We had no visibility into why the memory was climbing. We just saw the “Signal 9” and the silence that followed. We need telemetry inside the code. Use prometheus_client. Log the shape of your input tensors. If you’re sending a 1,000-column matrix when the model expects 10, I want to see an error log, not a memory overflow.

7. THE “IDEMPOTENCY” PROBLEM

During the recovery, we tried to replay the failed requests. Because your inference logic wasn’t idempotent—it was updating a “feature store” database on every read—we ended up double-counting thousands of events.

A prediction should be a pure function: f(x) = y. It should not have side effects. If you need to log the prediction, do it asynchronously. If you need to update a feature, do it in a separate process. Do not mix your “clever” math with our system of record.

THE “DEFINITION OF DONE” CHECKLIST (OR: WHY YOUR PR IS GETTING REJECTED)

If you want to ship a model to my production environment, you will complete this checklist. If any item is unchecked, don’t even bother tagging me for a review.

  • [ ] Environment Isolation: Does the pyproject.toml or requirements.txt contain ONLY the libraries needed for inference? (No matplotlib, pytest, or jupyter).
  • [ ] Memory Profiling: Have you run memory_profiler on a production-sized batch? What is the peak RSS?
  • [ ] Load Testing: Have you run a Locust or k6 test for 30 minutes at 2x expected peak load?
  • [ ] Serialization: Is the model stored in a non-executable format (ONNX, TFLite, or at least a compressed joblib)?
  • [ ] Artifact Management: Is the model pulled from an S3 bucket at runtime using a versioned hash, rather than being baked into the Git repo?
  • [ ] Error Handling: Does the code catch numpy.core._exceptions._ArrayMemoryError and return a 503 instead of letting the kernel kill the whole process?
  • [ ] Observability: Are there Prometheus metrics for inference time and input feature shapes?
  • [ ] Graceful Degradation: If the model service is down, does the application have a fallback (e.g., a simple heuristic or a cached result)?
  • [ ] Container Hygiene: Is the final Docker image under 2GB? (Hint: Use multi-stage builds and python:3.11-slim).
  • [ ] Documentation: Is there a README.md that explains what the features are, without using the word “leverage” or “synergy”?

I’m going home now. I’m going to sleep for 14 hours. When I come back, I expect to see a plan for refactoring the recommender service. If I see another MemoryError in the logs because someone forgot how math works, I’m moving my desk to the basement and changing my name.

Engineering is about constraints. Machine Learning is not an excuse to forget that.

Regards,

The SRE who kept your “clever” code alive.

APPENDIX A: THE RECOVERY SCRIPT (FOR WHEN YOU INEVITABLY IGNORE THIS)

#!/bin/bash
# Emergency script to clear stuck pods and reset the node pressure
# Usage: ./i_ignored_the_sre_and_broke_prod.sh

NAMESPACE="production-ml"

echo "Checking for OOM-killed pods..."
kubectl get pods -n $NAMESPACE | grep -i "OOMKilled"

echo "Scaling down the offender to 0 to save the cluster..."
kubectl scale deployment recommender-v2 --replicas=0 -n $NAMESPACE

echo "Cleaning up orphaned volumes..."
# [REDACTED: SRE INTERNAL ONLY]

echo "Cluster stabilized. Waiting for Data Science to apologize."

APPENDIX B: PIP FREEZE COMPARISON

What you gave us:
pandas==2.1.1
numpy==1.26.2
scikit-learn==1.3.2
torch==2.1.2
tensorflow==2.15.0
matplotlib==3.8.2
scipy==1.11.3

What you actually needed:
pandas==2.1.1 (with pyarrow backend for memory efficiency)
numpy==1.26.2
scikit-learn==1.3.2 (only for the Pipeline object)
onnxruntime==1.16.3 (for actual inference)
fastapi==0.104.1
uvicorn==0.24.0

Total size difference: 3.4 GB.
Total memory overhead reduction: 65%.
Total SRE sanity regained: Minimal, but it’s a start.

Related Articles

Explore more insights and best practices:

Leave a Comment