text $ kubectl get pods -n prod-core NAME READY STATUS RESTARTS AGE api-gateway-7f5d8b9-x2k9l 0/1 CrashLoopBackOff 42 14m order-processor-0 0/1 Pending 0 72h order-processor-1 0/1 Pending 0 72h payment-service-5d4f9c8-m1n2b 0/1 CreateContainerConfigError 0 12m $ kubectl describe pod order-processor-0 Events: Type Reason Age From Message —- —— —- —- ——- Warning FailedScheduling 3m (x450 over 72h) default-scheduler … Read more
text $ npx react-native run-ios –scheme “FlagshipApp” info Found Xcode workspace “FlagshipApp.xcworkspace” info Building (fe) error Failed to build iOS project. We ran “xcodebuild” command but it exited with error code 65. To debug build logs further, consider building your app via Xcode.app directly. ** BUILD FAILED ** The following build commands failed: PhaseScriptExecution [CP-User]\ … Read more
Sit down, kid. We’re going to talk about why your “magic box” is just a fancy wrapper for a process. Stop looking at your watch. It’s 3:00 AM, the AC in this rack room has been dead since Tuesday, and that blinking amber light on the SAN is the only thing keeping me awake. You’ve … Read more
INTERNAL POST-MORTEM: INCIDENT #8842-BRAVO DATE: Monday, October 16, 2023 DURATION: 72 Hours, 14 Minutes TOTAL ESTIMATED LOSS: $85,422.19 (Infrastructure Egress + Compute Over-provisioning + Lost Revenue) STATUS: SEV-0 (Mitigated, Not Resolved) AUTHOR: Senior SRE (Platform Reliability Team) 1. THE INCIDENT SUMMARY TIMESTAMP: 2023-10-13 23:43:12 UTC The pager went off. I was three sips into a … Read more
json { “Version”: “2012-10-17”, “Statement”: [ { “Sid”: “ExplicitDenyAllExceptVPC”, “Effect”: “Deny”, “Principal”: ““, “Action”: “s3:“, “Resource”: [ “arn:aws:s3:::critical-prod-assets/“, “arn:aws:s3:::critical-prod-assets” ], “Condition”: { “StringNotEquals”: { “aws:SourceVpc”: “vpc-0a1b2c3d4e5f6g7h8” }, “Bool”: { “aws:PrincipalIsAWSService”: “false” } } }, { “Sid”: “AllowAppRoleAccess”, “Effect”: “Allow”, “Principal”: { “AWS”: “arn:aws:iam::123456789012:role/application-server-role” }, “Action”: [ “s3:GetObject”, “s3:PutObject” ], “Resource”: “arn:aws:s3:::critical-prod-assets/” } ] } The … Read more
TIMESTAMP: 2024-05-22 04:12:08 UTC STATUS: CRITICAL / DEGRADED INCIDENT ID: #8829-BETA-CASCADING-FAILURE OPERATOR: SRE_042 (COFFEE_LEVEL: CRITICAL) The hum of the data center fans is a sound I can hear even in my own apartment now. It’s a low-frequency vibration that lives in the base of my skull. I’ve been staring at a Grafana dashboard for forty-eight … Read more
text $ docker compose up -d [+] Running 5/5 ⠿ Network infrastructure_default Created 0.1s ⠿ Container postgres-db Started 0.5s ⠿ Container redis-cache Started 0.4s ⠿ Container backend-api Error 2.1s ⠿ Container frontend-app Started 0.8s Error response from daemon: failed to mount local volume: mount /home/sre_hell/app/config:/etc/app/config, flags: 0x1000: no such file or directory $ docker compose … Read more
INCIDENT REPORT: POST-MORTEM #882-B (THE “FRIDAY AFTERNOON ARCHITECT SPECIAL”) TIMESTAMP: 2024-05-17T03:04:12Z INITIAL ALERT: CRITICAL – PagerDuty – Service: checkout-api – Severity: 1 – Status: FAILED FIRST LOG ENTRY: kubelet[1024]: E0517 03:04:12.442103 1024 remote_runtime.go:116] “RunPodSandbox from runtime service failed” err=”rpc error: code = Unknown desc = failed to setup network for sandbox: failed to allocate for … Read more
[2024-05-22 03:14:22] ERROR: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.50 GiB (GPU 0; 79.35 GiB total capacity; 64.12 GiB already allocated; 10.23 GiB free; 66.12 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF [2024-05-22 03:14:22] … Read more
text $ kubectl get pods -n prod NAME READY STATUS RESTARTS AGE api-gateway-v2-7f8d9b4c-xhq2z 0/1 CrashLoopBackOff 42 (3m ago) 14h order-processor-66d5f4e3-99abc 0/1 OOMKilled 12 (1m ago) 14h payment-service-55c2a1b0-zxy98 1/1 Running 0 14h marketing-tracker-88f123a4-bbbbb 1/1 Running 0 14h $ kubectl logs -f api-gateway-v2-7f8d9b4c-xhq2z –previous {“level”:”fatal”,”ts”:1715432100.123,”caller”:”main.go:45″,”msg”:”failed to connect to redis”,”error”:”dial tcp 10.96.0.15:6379: i/o timeout”} {“level”:”info”,”ts”:1715432105.456,”msg”:”Attempting reconnection… (Attempt 43)”} … Read more