[2024-05-22 03:14:22] ERROR: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.50 GiB (GPU 0; 79.35 GiB total capacity; 64.12 GiB already allocated; 10.23 GiB free; 66.12 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF [2024-05-22 03:14:22] … Read more
text $ kubectl get pods -n prod NAME READY STATUS RESTARTS AGE api-gateway-v2-7f8d9b4c-xhq2z 0/1 CrashLoopBackOff 42 (3m ago) 14h order-processor-66d5f4e3-99abc 0/1 OOMKilled 12 (1m ago) 14h payment-service-55c2a1b0-zxy98 1/1 Running 0 14h marketing-tracker-88f123a4-bbbbb 1/1 Running 0 14h $ kubectl logs -f api-gateway-v2-7f8d9b4c-xhq2z –previous {“level”:”fatal”,”ts”:1715432100.123,”caller”:”main.go:45″,”msg”:”failed to connect to redis”,”error”:”dial tcp 10.96.0.15:6379: i/o timeout”} {“level”:”info”,”ts”:1715432105.456,”msg”:”Attempting reconnection… (Attempt 43)”} … Read more
text [2024-05-22 03:14:02] INFO: Deployment triggered by ‘Rockstar_Dev_69’ [2024-05-22 03:15:44] ERROR: Failed to pull image “registry.internal/awesome-app:latest” [2024-05-22 03:15:44] ERROR: RPC error: code = Unknown desc = failed to register layer: Error processing tar file(exit status 1): write /usr/src/app/node_modules/huge-useless-library/dist/bundle.js: no space left on device [2024-05-22 03:16:10] CRITICAL: Node ip-10-0-42-12.ec2.internal is DiskPressure [2024-05-22 03:16:15] CRITICAL: Kubelet stopped … Read more
May 14 02:14:01 prod-web-04 sshd[14202]: Failed password for root from 192.168.1.45 port 54322 ssh2 May 14 02:14:05 prod-web-04 sshd[14202]: Accepted password for root from 192.168.1.45 port 54322 ssh2 May 14 02:14:06 prod-web-04 systemd[1]: Starting User Manager for UID 0… May 14 02:14:06 prod-web-04 systemd-logind[645]: New session 142 of user root. whoami root uname -a Linux … Read more
INCIDENT REPORT #882-B: Why our heap looks like a disaster zone. DATE: October 14, 2023 TO: Engineering Lead, CTO, and anyone else who thinks “it’s just a script” FROM: Senior Systems Engineer (Level 4, Infrastructure) SUBJECT: Post-Mortem of the Node-04 Memory Exhaustion Event (Python 3.11.4) SYSTEM INCIDENT LOG: 2023-10-12T03:14:22Z 03:14:22 – CRITICAL – kernel: [99283.12] … Read more
POST-MORTEM: THE BLACK SUNDAY CATASTROPHE TIMESTAMP: 2024-05-19 04:12:09 UTC INCIDENT DURATION: 72 Hours, 14 Minutes STATUS: Partially Restored / Total Mental Collapse OPERATOR: Senior SRE (Employee #402, currently fueled by spite and expired Ibuprofen) 1. Summary of the Carnage At 04:00 UTC on Sunday, while the rest of the world was sleeping or having a … Read more
TIMESTAMP: 03:14 AM. The pagers started screaming, and that’s when I realized nobody in this building actually knows what is devops. My eyes are vibrating. I’ve been awake for forty-eight hours, fueled by lukewarm espresso and the bitter realization that our “cloud-native” architecture is actually just three raccoons in a trench coat holding a soldering … Read more
In today’s fast-paced world of cloud-native development, microservices architecture has become the standard for building scalable and maintainable applications. Managing these microservices on Kubernetes can be complex, but with Argo CD, you can simplify deployment and continuous delivery through a GitOps approach. This guide will walk you through the process of installing and configuring Argo … Read more
In today’s fast-paced development landscape, microservices architecture has become a go-to approach for building scalable and maintainable applications. However, deploying a microservices-based application can be complex, especially when considering factors like scalability, orchestration, and resilience. Kubernetes, an open-source container orchestration platform, has emerged as the preferred choice for managing microservices in production environments. In this … Read more
In the ever-evolving landscape of cloud computing, serverless architecture has emerged as a transformative approach that is reshaping how applications are developed and deployed. As businesses increasingly seek agility, scalability, and cost efficiency, serverless computing offers a compelling solution that addresses these needs while simplifying the development process. In this blog post, we’ll explore the … Read more