Stop Building Pipelines and Start Building Systems: A Decade of DevOps Regrets It was 3:15 AM on a Tuesday in 2017. I was staring at a Grafana dashboard that looked like a heart monitor for a patient in cardiac arrest. We had just “automated” our deployment pipeline using a series of nested Jenkins Groovy scripts … Read more
text May 14 03:12:04 web-prod-01 sshd[29481]: Invalid user admin from 193.161.193.193 port 54228 May 14 03:12:06 web-prod-01 sshd[29483]: Invalid user support from 193.161.193.193 port 54230 May 14 03:12:09 web-prod-01 sshd[29485]: Accepted password for jdoe from 193.161.193.193 port 54232 ssh2 May 14 03:12:09 web-prod-01 systemd-logind[652]: New session 482 of user jdoe. May 14 03:12:10 web-prod-01 sudo: … Read more
AWS Best Practices: Why Your Infrastructure is a Money Pit and How to Stop the Bleeding I once worked at a fintech startup where we “followed the docs” to the letter. We set up a multi-AZ deployment for a high-traffic microservice, thinking we were being smart about availability. Three days later, the Head of Engineering … Read more
HTTP/2 200 content-type: text/html; charset=UTF-8 vary: Accept-Encoding server: cloudflare cf-ray: 892349823498234-SJC cache-control: max-age=0, no-cache, no-store, must-revalidate x-frame-options: DENY x-xss-protection: 1; mode=block content-security-policy: default-src ‘self’; script-src ‘self’ ‘unsafe-inline’ ‘unsafe-eval’ https://cdn.segment.com https://www.google-analytics.com; style-src ‘self’ ‘unsafe-inline’ https://fonts.googleapis.com; set-cookie: _session_id=89234jksdf89234; path=/; expires=Tue, 19 Jan 2038 03:14:07 GMT; secure; HttpOnly; SameSite=Lax strict-transport-security: max-age=31536000; includeSubDomains; preload x-content-type-options: nosniff content-length: 842903 Look … Read more
Your “Cybersecurity Best Practices” Are Probably Just Security Theater I once took down an entire payment processing cluster because I thought I was being clever with iptables. We were trying to implement “cybersecurity best” practices—specifically, egress filtering. I pushed a script to 400 nodes that blocked all outbound traffic except for a whitelist of IPs. … Read more
INCIDENT REPORT #882-B: THE DAY THE ‘AI’ FORGOT HOW TO DO MATH. [2023-10-24 03:14:22] ERROR: worker-7 terminated with signal 9 (SIGKILL) [2023-10-24 03:14:23] Traceback (most recent call last): File "/opt/analytics/smart_scaler_v2.py", line 442, in <module> model.fit(X_train, y_train) File "/usr/local/lib/python3.11/site-packages/sklearn/utils/_set_output.py", line 140, in wrapped data_to_wrap = f(self, X, *args, **kwargs) File "/usr/local/lib/python3.11/site-packages/sklearn/linear_model/_base.py", line 678, in fit X, … Read more
Your JavaScript Code is Killing Your Infrastructure (And You’re Probably Fine With It) It was 3:14 AM on a Tuesday in 2019. I was the on-call SRE for a fintech startup that processed payments via api.stripe.com. Suddenly, PagerDuty started screaming. Our main API gateway—a Node.js service—was hitting 95% memory utilization across 40 pods. Then the … Read more
AWS is Just Someone Else’s Data Center (And Why You’re Paying Too Much for It) It was 3:14 AM on a Tuesday when my PagerDuty went off. The alert was a generic TargetResponseTime high on our main production ALB. I logged in, eyes blurry, and saw that our NAT Gateway was processing 4.5 Gbps of … Read more
The smell of ozone and stale coffee is the only thing that stays the same. It was 03:14 AM on a Tuesday in 2014. I was standing in a data center in Secaucus, the floor tiles vibrating under my boots from the sheer CFM of the cooling fans. We had a cluster of HP ProLiant … Read more
text $ node -v v14.17.0 (Warning: Node.js version is end-of-life. Security patches are non-existent.) $ npm audit ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Scanning 2,482 dependencies for vulnerabilities… Critical: 14 High: 89 Moderate: 156 Low: 412 Run npm audit fix to do absolutely nothing because the dependency tree is a circular nightmare of peer-dependency conflicts. $ vite build vite v2.9.15 … Read more