INCIDENT REPORT #882-B: Why our migration to the ‘Modern Stack’ nearly bankrupted the company. DATE: October 14, 2023 TO: Executive Steering Committee, Engineering Leads FROM: Lead Architect (Recovery Team) SUBJECT: Post-Mortem of “Project Phoenix” (The Great Migration) TOTAL LOSS: $50,420,000 USD (Estimated) TIMELINE: January 2022 – June 2023 1. Incident Report Summary Project Phoenix was … Read more
text [ 2592.148203] NVRM: GPU at PCI:0000:01:00: GPU-8d7f4a2b-3c1e-4f5d-9a8b-7c6d5e4f3a2b [ 2592.148210] NVRM: Xid (PCI:0000:01:00): 31, GPU memory page fault in isolation [ 2592.148215] nvidia-nvlink: Unregistered the Nvlink Core, major device number 234 [ 2592.148222] NVRM: os_schedule: Attempted to yield the CPU while holding a spinlock! [ 2592.148230] BUG: kernel NULL pointer dereference, address: 0000000000000000 [ 2592.148235] … Read more
text Uncaught Error: Hydration failed because the initial UI does not match what was rendered on the server. at makeError (react-dom.development.js:22839:1) at validateDOMNesting (react-dom.development.js:10842:1) at attemptHydration (react-dom.development.js:12344:1) at mountHydratableComponent (react-dom.development.js:14522:1) … [48 more lines of minified agony] … at Page (./src/app/dashboard/page.tsx:42:11) Node v20.11.0 React 18.3.1 Next.js 14.2.3 I’m sitting here at 4:14 AM, staring at … Read more
INCIDENT REPORT #882-ALPHA. Status: Resolved (Barely). Subject: Why your ‘artificial intelligence’ strategy is a ticking time bomb. TIMESTAMP LOG: THE 72 HOURS OF RADIOLOGICAL FALLOUT T-Minus 03:14:00 (Friday, 18:00): Deployment of “Project Prometheus” (the internal name for our ‘artificial intelligence’ recommendation engine v4.2) goes live. Data Science team leaves for a “celebratory happy hour.” T-00:00:00 … Read more
Stop Building Pipelines and Start Building Systems: A Decade of DevOps Regrets It was 3:15 AM on a Tuesday in 2017. I was staring at a Grafana dashboard that looked like a heart monitor for a patient in cardiac arrest. We had just “automated” our deployment pipeline using a series of nested Jenkins Groovy scripts … Read more
text May 14 03:12:04 web-prod-01 sshd[29481]: Invalid user admin from 193.161.193.193 port 54228 May 14 03:12:06 web-prod-01 sshd[29483]: Invalid user support from 193.161.193.193 port 54230 May 14 03:12:09 web-prod-01 sshd[29485]: Accepted password for jdoe from 193.161.193.193 port 54232 ssh2 May 14 03:12:09 web-prod-01 systemd-logind[652]: New session 482 of user jdoe. May 14 03:12:10 web-prod-01 sudo: … Read more
AWS Best Practices: Why Your Infrastructure is a Money Pit and How to Stop the Bleeding I once worked at a fintech startup where we “followed the docs” to the letter. We set up a multi-AZ deployment for a high-traffic microservice, thinking we were being smart about availability. Three days later, the Head of Engineering … Read more
HTTP/2 200 content-type: text/html; charset=UTF-8 vary: Accept-Encoding server: cloudflare cf-ray: 892349823498234-SJC cache-control: max-age=0, no-cache, no-store, must-revalidate x-frame-options: DENY x-xss-protection: 1; mode=block content-security-policy: default-src ‘self’; script-src ‘self’ ‘unsafe-inline’ ‘unsafe-eval’ https://cdn.segment.com https://www.google-analytics.com; style-src ‘self’ ‘unsafe-inline’ https://fonts.googleapis.com; set-cookie: _session_id=89234jksdf89234; path=/; expires=Tue, 19 Jan 2038 03:14:07 GMT; secure; HttpOnly; SameSite=Lax strict-transport-security: max-age=31536000; includeSubDomains; preload x-content-type-options: nosniff content-length: 842903 Look … Read more
Your “Cybersecurity Best Practices” Are Probably Just Security Theater I once took down an entire payment processing cluster because I thought I was being clever with iptables. We were trying to implement “cybersecurity best” practices—specifically, egress filtering. I pushed a script to 400 nodes that blocked all outbound traffic except for a whitelist of IPs. … Read more
INCIDENT REPORT #882-B: THE DAY THE ‘AI’ FORGOT HOW TO DO MATH. [2023-10-24 03:14:22] ERROR: worker-7 terminated with signal 9 (SIGKILL) [2023-10-24 03:14:23] Traceback (most recent call last): File "/opt/analytics/smart_scaler_v2.py", line 442, in <module> model.fit(X_train, y_train) File "/usr/local/lib/python3.11/site-packages/sklearn/utils/_set_output.py", line 140, in wrapped data_to_wrap = f(self, X, *args, **kwargs) File "/usr/local/lib/python3.11/site-packages/sklearn/linear_model/_base.py", line 678, in fit X, … Read more