Stop Treating Your Models Like Precious Snowflakes: A No-Nonsense Guide to Machine Learning in Production In 2021, I watched a $50,000-a-day ad-spend budget evaporate over a single weekend because of a pandas version mismatch. We had a “state-of-the-art” recommendation engine running on a cluster of 20 nodes. The data science team had updated their local … Read more
text 2024-05-14T03:02:11.492Z [ERROR] [worker-7f9b] – Internal Server Error: Traceback (most recent call last): File “/usr/local/lib/python3.11/site-packages/sklearn/utils/_set_output.py”, line 142, in _wrap_method_output AttributeError: ‘NoneType’ object has no attribute ‘get’ 2024-05-14T03:02:12.101Z [WARN] [ingress-nginx] – Upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.42.0.1, server: api.internal.prod 2024-05-14T03:02:14.550Z [CRIT] [kernel] – [68293.120] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/kubepods/besteffort/pod-ml-inference,task=python3,pid=1422,uid=1000 2024-05-14T03:02:14.800Z [INFO] … Read more
THE MANIFESTO OF THE SCARRED: WHY YOUR CODE IS A CRIME AGAINST ENGINEERING Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/service/processor.py", line 442, in process_payload result = core_logic.calculate_weighted_offset(payload[‘delta’], config.STRATEGY_ALPHA) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/core/logic.py", line 89, in calculate_weighted_offset return (delta * self.multiplier) / (config_val – 1) ~~~~~~^~~~~~~~~~~~~~~~~ TypeError: unsupported operand type(s) for *: ‘NoneType’ and ‘float’ >>> # … Read more
[ 259200.412098] nvidia-uvm: Found 81920MB VRAM on Device 0 [ 259200.412105] out_of_memory: Kill process 12849 (python3) score 999 or sacrifice child [ 259200.412110] Kernel panic – not syncing: Fatal exception in interrupt [ 259200.412115] CPU: 48 PID: 12849 Comm: python3 Tainted: P OE 5.15.0-101-generic #111-Ubuntu [ 259200.412118] Hardware name: Supermicro AS -4124GS-TNR/H12DSG-O-MS, BIOS 2.4 05/24/2023 … Read more
text $ az network vnet peering show –name PeerToProd –resource-group rg-core-connectivity –vnet-name vnet-hub –output json Deployment failed. Correlation ID: 4e52-b91a-8823471bc. { “error”: { “code”: “InternalServerError”, “message”: “An error occurred while processing your request. Please try again in a few minutes.”, “details”: [] } } $ az network vnet list -g rg-core-connectivity –query “[].{Name:name, State:provisioningState}” [ … Read more
text [2024-05-22T03:14:02.891Z] SEVERE: [MainThread] Uncaught RangeError: Maximum update depth exceeded. at Object.updateContainer (react-dom.development.js:25331:7) at Object.scheduleUpdateOnFiber (react-dom.development.js:21835:3) at dispatchSetState (react-dom.development.js:15853:7) at onAuthChange (AuthContext.tsx:42:5) [2024-05-22T03:14:03.112Z] FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed – JavaScript heap out of memory [2024-05-22T03:14:03.445Z] Process exited with code 137 (OOM) My eyes are vibrating. Not from the four double-shots of … Read more
React is Not a Framework, It’s a Debt Generator (And How to Stop the Bleeding) Three years ago, I was paged at 2:14 AM on a Tuesday. Our main logistics dashboard, the one the warehouse teams use to track every single shipment across the Atlantic, had turned into a space heater. The browser was consuming … Read more
POST-MORTEM INCIDENT REPORT #ML-FAIL-0909 DATE: October 14, 2023 INCIDENT DURATION: 48 Hours, 12 Minutes STATUS: Critical / Post-Recovery AUTHOR: Senior SRE (Infrastructure & Operations) I’ve spent the last 48 hours staring at a terminal buffer that smells like burnt silicon and hubris. While the rest of the “Data Science” team was likely dreaming of neural … Read more
THE HEAT DEATH OF THE ALPHA: A POST-MORTEM ON SILICON HUBRIS LOG_ID_0001: THE TRIGGER [2024-05-14 03:14:02.881] FATAL: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.50 GiB (GPU 0; 80.00 GiB total capacity; 64.21 GiB already allocated; 2.14 GiB free; 70.12 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try … Read more
text [2023-10-27 03:14:22,891] ERROR: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.50 GiB (GPU 0; 80.00 GiB total capacity; 72.14 GiB already allocated; 4.12 GiB free; 74.50 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF Traceback … Read more