Techie

text 2024-05-21T03:14:07.892Z [ERROR] [BatchID: 9921-X] FATAL: Connection reset by peer. 2024-05-21T03:14:08.001Z [WARN] [Kinesis-Consumer] Shard iterator expired. Retrying… 2024-05-21T03:14:08.445Z [CRITICAL] [Billing-Alert] Estimated charges for current month: $42,901.12 (Threshold $5,000 exceeded). 2024-05-21T03:14:10.112Z [SYS] Kernel Panic – not syncing: Fatal exception in interrupt. $ aws ec2 describe-instances –filters “Name=instance-state-name,Values=running” –query “Reservations[].Instances[].[InstanceId, State.Name, PrivateIpAddress, InstanceType]” –output table | DescribeInstances … Read more

INTERNAL MEMO: AETHER-SCALE POST-MORTEM LOG #88-B FROM: Silas, Lead Systems Architect (Bare-Metal Division) TO: Junior Engineering Cohort (Cloud-Native/API-First Teams) SUBJECT: The Manifesto of Regret Listen up. I’ve spent thirty years in the trenches, from the days of hand-soldering serial cables to the current nightmare of virtualized abstractions. I’ve watched the industry trade stability for “velocity” … Read more

text $ npx react-native run-android info Starting JS server… info Building and installing the app on the device (failed reverse-proxy)… Task :app:installDebug FAILED FAILURE: Build failed with an exception. What went wrong: Execution failed for task ‘:app:mergeLibDexDebug’. A failure occurred while executing com.android.build.gradle.internal.tasks.CheckDuplicatesRunnable Duplicate class com.facebook.jni.HybridData found in modules jetified-fbjni-0.5.1 (com.facebook.fbjni:fbjni:0.5.1) and jetified-hermes-android-0.74.2-debug (com.facebook.react:hermes-android:0.74.2) Go … Read more

[2023-10-24 03:14:22.891] ERROR: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.50 GiB (GPU 0; 80.00 GiB total capacity; 64.22 GiB already allocated; 1.12 GiB free; 72.45 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF [2023-10-24 03:14:25.102] … Read more

Stop Buying Security Tools and Start Fixing Your SSH Config It was 2017. I was working at a fintech startup that was “disrupting” the lending space. We were moving fast, which is industry shorthand for “we didn’t have a staging environment that matched production.” I was tasked with spinning up a Redis instance for a … Read more

The rain is drumming against the corrugated tin roof of the shed. It’s a cold, rhythmic sound that reminds me why I left Palo Alto. Out here, if a fence post rots, the cows get out. There’s no “fail-fast” philosophy when you’re chasing a half-ton of beef through a muddy creek at 3:00 AM. There … Read more

text [ 259200.412834] Out of memory: Killed process 124092 (python3) total-vm:154234824kB, anon-rss:82341204kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:302144kB oom_score_adj:0 [ 259200.413102] oom_reaper: reaped process 124092 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [ 259200.415001] Kernel panic – not syncing: Fatal exception in interrupt [ 259200.415050] Kernel Offset: 0x24000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 259200.415110] —[ end Kernel panic … Read more

AWS is a Collection of Primitives, Not a Magic Wand I once cost a former employer $42,000 in a single weekend because I misunderstood how AWS NAT Gateways handle data processing. I was migrating a high-throughput logging cluster—think 50TB of JSON blobs a day—from a co-located data center into a VPC in us-east-1. I followed … Read more

INCIDENT LOG: 2024-05-20T03:04:12Z LOCATION: us-east-1 (Production Cluster ‘Ares-01’) STATUS: CRITICAL FAILURE 03:04:12Z – PagerDuty alert: KubePodNotReady (Namespace: checkout-prod). 03:05:45Z – Node ip-10-244-12-84.ec2.internal enters NotReady state. 03:06:10Z – Cascading failure detected. kube-dns latency spikes to 4500ms. 03:07:30Z – API Server unresponsive. kubectl commands timing out. 03:10:00Z – On-call engineer attempts manual recovery. $ kubectl get pods … Read more

TO: Engineering All FROM: Senior SRE (On-Call Rotation 4) DATE: 04:12 AM – Day 3 of Incident #8842 SUBJECT: The Corpse of the Production Cluster and the Myth of Automation It’s 4:12 AM. I haven’t slept since Tuesday. My vision is tunneling, and the blue light from my third monitor is starting to feel like … Read more