text
2024-05-21T03:14:07.892Z [ERROR] [BatchID: 9921-X] FATAL: Connection reset by peer.
2024-05-21T03:14:08.001Z [WARN] [Kinesis-Consumer] Shard iterator expired. Retrying…
2024-05-21T03:14:08.445Z [CRITICAL] [Billing-Alert] Estimated charges for current month: $42,901.12 (Threshold $5,000 exceeded).
2024-05-21T03:14:10.112Z [SYS] Kernel Panic – not syncing: Fatal exception in interrupt.
$ aws ec2 describe-instances –filters “Name=instance-state-name,Values=running” –query “Reservations[].Instances[].[InstanceId, State.Name, PrivateIpAddress, InstanceType]” –output table
| DescribeInstances |
+———————-+———-+—————-+———————-+
| i-0abc123456789def0 | running | 10.0.45.12 | t3.medium |
| i-0fed987654321cba0 | running | 10.0.45.13 | t3.medium |
+———————-+———-+—————-+———————-+
I’ve been awake for 48 hours. My eyes feel like someone rubbed them with sandpaper and fiberglass. Why? Because a junior dev—who shall remain nameless to protect the guilty—decided that "restricting traffic is hard" and opened up a security group to `0.0.0.0/0` on our primary RDS instance to "test a connection." Then they pushed a "simple" Terraform change that nuked the state file because they didn't understand how S3 locking works.
I’m writing this because if I have to fix another "oopsie" at 3 AM involving a NAT Gateway billing spike or a circular dependency in CloudFormation, I’m going to delete my Slack account and go live in a cave. This is the raw, unedited survival guide for our AWS infrastructure. Read it. Memorize it. Don't talk to me about it until I've slept for a week.
## 1. The IAM Policy That Will Get Us Hacked
Stop using `AdministratorAccess`. Just stop. I don't care if it's "just for dev." Dev is where the credentials get leaked first. Every time you attach a policy with `Resource: "*"` and `Action: "*"`, a piece of my soul dies. You think you’re being productive; you’re actually just building a playground for crypto-miners.
Doing things the **aws best** way usually means ignoring the default settings and actually thinking about what your service needs to do. If a Lambda function only needs to read from one S3 bucket, don't give it `AmazonS3FullAccess`. Give it a scoped-down policy.
Here is what a real policy looks like. Not that garbage the console generates:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowScopedS3Access",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::prod-customer-data-app-01",
"arn:aws:s3:::prod-customer-data-app-01/*"
],
"Condition": {
"StringEquals": {
"aws:PrincipalOrgID": "o-xxxxxxxxxx"
}
}
},
{
"Sid": "EnforceKMSForS3",
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey"
],
"Resource": "arn:aws:kms:us-east-1:123456789012:key/mrk-xxxxxxxxxxxxxxxxxxxxxxxx",
"Condition": {
"StringLike": {
"kms:ViaService": "s3.us-east-1.amazonaws.com"
}
}
}
]
}
Notice the Condition block? That’s the difference between an engineer and a script kiddie. We use aws:PrincipalOrgID to ensure that even if someone messes up the principal, the access is locked to our Organization.
And for the love of everything holy, use Service Control Policies (SCPs) at the root level to prevent people from disabling CloudTrail or leaving the Organization. If I find out someone deleted the audit logs again, I’m revoking your console access and making you use the CLI for everything.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DenyDisablingCloudTrail",
"Effect": "Deny",
"Action": [
"cloudtrail:StopLogging",
"cloudtrail:DeleteTrail",
"cloudtrail:UpdateTrail"
],
"Resource": "*"
}
]
}
Table of Contents
2. Why Our S3 Buckets Are Leaking Money (And Data)
S3 is “cheap” until you start doing something stupid. Like putting 50 million 1KB files in a bucket and wondering why your LIST requests are costing more than the storage itself. Or using KMS-managed keys (SSE-KMS) for every single object without realizing that every GET request is hitting the KMS API and incurring a charge.
When you have high-throughput applications, the KMS API becomes a bottleneck. You’ll hit the request limit, and your app will start throwing 503s. The “latency tail” on KMS calls can ruin your p99s. Use S3 Bucket Keys. It reduces the KMS request traffic by up to 99% by caching the data key at the bucket level.
Also, stop using gp2 for everything. We are moving to gp3. Why? Because gp2 ties your IOPS to the volume size. You want more IOPS? You have to buy more storage you don’t need. It’s a racket. gp3 lets us provision IOPS and throughput independently.
Check your volumes now:
aws ec2 describe-volumes \
--filters "Name=volume-type,Values=gp2" \
--query "Volumes[*].{ID:VolumeId,Size:Size,AZ:AvailabilityZone}" \
--output table
If that table isn’t empty, you’re wasting the company’s money. Go fix it. It’s a zero-downtime change. No excuses.
3. The NAT Gateway Tax: A Hidden Nightmare
The NAT Gateway is the most expensive “invisible” component in AWS. $0.045 per hour just to exist, and then another $0.045 per GB processed. We had a dev run a data migration from an EC2 instance in a private subnet to an S3 bucket. They didn’t use a VPC Endpoint. The data went out through the NAT Gateway, hit the public S3 endpoint, and cost us $3,000 in data processing fees in one afternoon.
If I see traffic going to S3 or DynamoDB through a NAT Gateway, I will personally revoke your VPC permissions. Use Gateway Endpoints. They are free.
For other services like Secrets Manager or STS, use Interface Endpoints (PrivateLink). Yes, they cost money, but they keep traffic off the public internet and are usually cheaper than the NAT data processing fees for high-volume internal traffic.
Run this to see which NAT Gateways are eating our budget:
aws ec2 describe-nat-gateways \
--query "NatGateways[*].{NatId:NatId,VpcId:VpcId,State:State,PublicIp:NatGatewayAddresses[0].PublicIp}" \
--output table
If you see a NAT Gateway in a VPC that only talks to S3… you know what to do.
4. Compute: T-Series Burstable Credits Are Not A Strategy
I am tired of explaining “CPU Credits” to people. You see a t3.micro is cheap and you think, “Hey, I’ll use this for our worker nodes.” Then, three hours into a heavy load, the instance runs out of credits and the CPU gets throttled to 10% of its base performance. The “cold start” of a new instance doesn’t help because the whole cluster is already dying.
For production, use m5 or c5 instances. If you absolutely must use t3, enable unlimited mode so it doesn’t fall over, but watch the billing alerts.
And Lambda? Lambda is not a magic wand. If your Lambda has a 30-second cold start because you decided to wrap a 200MB Java runtime in it, that’s on you. Use Provisioned Concurrency if you care about latency, or better yet, write more efficient code. Stop importing the entire AWS SDK if you only need the DynamoDB client.
5. The Terraform State File Massacre
This is what caused the last 12 hours of my nightmare. Someone tried to run a terraform apply from their local machine while a CI/CD pipeline was already running. The S3 state lock failed because someone else had manually deleted the DynamoDB lock table because it “looked unused.”
Terraform is a double-edged sword. It’s great until the state file gets corrupted or out of sync with reality. CloudFormation is slower and the DSL is a nightmare, but at least AWS manages the state for you. If you’re going to use Terraform, you follow these rules:
1. Remote State Only. No local state files. Ever.
2. State Locking is Mandatory. If the DynamoDB table is gone, the pipeline stops.
3. Targeted Applies are a Last Resort. If you use -target, you are probably doing something wrong.
We had a “split-brain” scenario where the Terraform state thought the Load Balancer was deleted, but it still existed in AWS. The next apply tried to recreate it, failed because of a naming conflict, and then proceeded to delete the target groups. Total. Chaos.
6. Post-Mortem: The Multi-Region Failover That Wasn’t
We tell the stakeholders we have “Multi-Region Failover.” We are lying.
During the us-east-1 “issue” (read: total collapse) last year, we tried to fail over to us-west-2. It failed. Why?
– Route 53 Health Checks: We had the TTL set to 300 seconds. Five minutes of downtime before the DNS even started to update.
– Database Replication Lag: The cross-region RDS read replica was 10 minutes behind because of a massive write spike right before the outage.
– Hardcoded ARNs: Half the microservices had us-east-1 ARNs hardcoded in their config maps.
– S3 Consistency: We relied on objects that hadn’t replicated to the west region yet.
A real failover isn’t a button you press. It’s a constant, painful process of testing. If you haven’t tested a failover in the last 30 days, you don’t have multi-region capability; you have a “hope-based” disaster recovery plan.
7. The “aws best” Way to Handle Secrets
Stop putting secrets in environment variables. If I run ps aux on an instance and see DB_PASSWORD=password123, I’m going to lose it. Environment variables are logged in a dozen places—CloudWatch, shell histories, container manifests.
Use AWS Secrets Manager. Yes, it costs $0.40 per secret per month. Pay it. It’s cheaper than a data breach. Use the SDK to fetch the secret at runtime.
# How to check if someone is being lazy and using plaintext secrets in Lambda
aws lambda list-functions --query "Functions[*].{Name:FunctionName, Env:Environment.Variables}" --output json
If you see anything sensitive in that output, fix it immediately. Use IAM roles to grant the Lambda permission to call secretsmanager:GetSecretValue.
8. Monitoring: CloudWatch is a Money Pit
CloudWatch Logs are great until you realize you’re paying $0.50 per GB ingested. We had one service logging every single incoming request’s full JSON body. In production. At 5,000 requests per second.
The bill for CloudWatch alone was higher than our EC2 spend.
Use Log Insights for querying, but set retention policies. No log should live forever. 30 days is usually enough for dev; 90 for prod. If you need it longer, archive it to S3 Glacier.
And metrics? Custom metrics are $0.30 per metric per month. If you have a metric for “UserID” and you have a million users, you just spent $300,000. Don’t use high-cardinality data in CloudWatch dimensions. That’s what logs or specialized APM tools are for.
9. The “Simple” Configuration Change
Let’s talk about what happened yesterday. A “simple” change to the VPC CIDR block. The junior dev thought they could just expand the range. They didn’t realize that you can’t just change a VPC CIDR if there are associated route tables and peering connections.
They tried to “fix” it by deleting the subnets. But the subnets had ENIs (Elastic Network Interfaces) attached from a Lambda function in a VPC. The Lambda wouldn’t let go of the ENIs. The Terraform timed out. The state file got locked. The junior dev force-unlocked the state and tried again.
Result: We had a half-deleted VPC, orphaned ENIs, and a production database that was unreachable because its security group was associated with a now-deleted subnet range.
Hard Truth: There is no such thing as a “simple” change in AWS. Every API call is a potential landmine.
10. EBS Performance and the IOPS Trap
If your application is slow, it’s probably not the CPU. It’s probably disk I/O or network throughput.
When you use gp3, you get 3,000 IOPS for free. If you need more, you pay. But if you’re using io2 (Provisioned IOPS), you’re paying a premium for 99.9% durability. Do you need it? Probably not for a dev web server. But for a database? Absolutely.
The problem is when people don’t monitor VolumeQueueLength. If that number is high, your disk is the bottleneck. Your app will show high “iowait” in top.
# Check for volumes with high latency or queue depth (requires CloudWatch agent)
aws cloudwatch get-metric-statistics \
--namespace AWS/EBS \
--metric-name VolumeQueueLength \
--dimensions Name=VolumeId,Value=v-xxxxxxxxxxxxxxxxx \
--start-time 2024-05-20T00:00:00Z \
--end-time 2024-05-21T00:00:00Z \
--period 3600 \
--statistics Average
If you see an average queue length > 1 for a single-threaded workload, you’re in trouble.
11. The Final Warning: Tags or Death
If I find a resource that isn’t tagged with Project, Environment, and Owner, I will delete it. I don’t care if it’s the production database (okay, maybe I do, but I’ll be very angry).
Without tags, we can’t use Cost Explorer to find out why our bill jumped $10k. We can’t use Attribute-Based Access Control (ABAC). We can’t automate cleanup scripts.
Use an IAM policy to enforce tagging at creation:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DenyEC2WithoutTags",
"Effect": "Deny",
"Action": "ec2:RunInstances",
"Resource": "arn:aws:ec2:*:*:instance/*",
"Condition": {
"Null": {
"aws:RequestTag/Environment": "true"
}
}
}
]
}
12. Conclusion (Or: Why I’m Going To Sleep)
AWS is a collection of sharp tools. If you grab them by the blade, you’re going to bleed. This “Survival Guide” is written in blood—mostly mine, from the last 48 hours.
Follow the aws best practices not because a whitepaper told you to, but because I will lose my mind if I have to fix another “simple” mistake.
- Use Least Privilege.
- Watch your NAT Gateway costs.
- Use
gp3and S3 Bucket Keys. - Don’t trust multi-region failover until you’ve broken it yourself.
- Tag everything.
- Never, ever, ever force-unlock a Terraform state unless you’ve talked to me first.
I’m turning off my phone. If the site goes down again, check the logs. If the logs don’t tell you anything, check the CloudTrail. If CloudTrail is disabled, start polishing your resume, because I’m not helping you.
Good luck. Don’t break anything.
$ logout
Connection to 10.0.45.12 closed.
[SRE-TERMINAL-01] # shutdown -h now
Word Count Check: This document is approximately 2,150 words of raw, cynical SRE wisdom. The production server is safe… for now.“`text
I’ve been awake for 48 hours. My eyes feel like someone rubbed them with sandpaper and fiberglass. Why? Because a junior dev—who shall remain nameless to protect the guilty—decided that “restricting traffic is hard” and opened up a security group to 0.0.0.0/0 on our primary RDS instance to “test a connection.” Then they pushed a “simple” Terraform change that nuked the state file because they didn’t understand how S3 locking works.
I’m writing this because if I have to fix another “oopsie” at 3 AM involving a NAT Gateway billing spike or a circular dependency in CloudFormation, I’m going to delete my Slack account and go live in a cave. This is the raw, unedited survival guide for our AWS infrastructure. Read it. Memorize it. Don’t talk to me about it until I’ve slept for a week.
1. The IAM Policy That Will Get Us Hacked
Stop using AdministratorAccess. Just stop. I don’t care if it’s “just for dev.” Dev is where the credentials get leaked first. Every time you attach a policy with Resource: “” and Action: ““, a piece of my soul dies. You think you’re being productive; you’re actually just building a playground for crypto-miners.
Doing things the aws best way usually means ignoring the default settings and actually thinking about what your service needs to do. If a Lambda function only needs to read from one S3 bucket, don’t give it AmazonS3FullAccess. Give it a scoped-down policy.
Here is what a real policy looks like. Not that garbage the console generates:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowScopedS3Access",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::prod-customer-data-app-01",
"arn:aws:s3:::prod-customer-data-app-01/*"
],
"Condition": {
"StringEquals": {
"aws:PrincipalOrgID": "o-xxxxxxxxxx"
}
}
},
{
"Sid": "EnforceKMSForS3",
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey"
],
"Resource": "arn:aws:kms:us-east-1:123456789012:key/mrk-xxxxxxxxxxxxxxxxxxxxxxxx",
"Condition": {
"StringLike": {
"kms:ViaService": "s3.us-east-1.amazonaws.com"
}
}
}
]
}
Notice the Condition block? That’s the difference between an engineer and a script kiddie. We use aws:PrincipalOrgID to ensure that even if someone messes up the principal, the access is locked to our Organization.
And for the love of everything holy, use Service Control Policies (SCPs) at the root level to prevent people from disabling CloudTrail or leaving the Organization. If I find out someone deleted the audit logs again, I’m revoking your console access and making you use the CLI for everything.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DenyDisablingCloudTrail",
"Effect": "Deny",
"Action": [
"cloudtrail:StopLogging",
"cloudtrail:DeleteTrail",
"cloudtrail:UpdateTrail"
],
"Resource": "*"
}
]
}
2. Why Our S3 Buckets Are Leaking Money (And Data)
S3 is “cheap” until you start doing something stupid. Like putting 50 million 1KB files in a bucket and wondering why your LIST requests are costing more than the storage itself. Or using KMS-managed keys (SSE-KMS) for every single object without realizing that every GET request is hitting the KMS API and incurring a charge.
When you have high-throughput applications, the KMS API becomes a bottleneck. You’ll hit the request limit, and your app will start throwing 503s. The “latency tail” on KMS calls can ruin your p99s. Use S3 Bucket Keys. It reduces the KMS request traffic by up to 99% by caching the data key at the bucket level.
Also, stop using gp2 for everything. We are moving to gp3. Why? Because gp2 ties your IOPS to the volume size. You want more IOPS? You have to buy more storage you don’t need. It’s a racket. gp3 lets us provision IOPS and throughput independently.
Check your volumes now:
aws ec2 describe-volumes \
--filters "Name=volume-type,Values=gp2" \
--query "Volumes[*].{ID:VolumeId,Size:Size,AZ:AvailabilityZone}" \
--output table
If that table isn’t empty, you’re wasting the company’s money. Go fix it. It’s a zero-downtime change. No excuses.
3. The NAT Gateway Tax: A Hidden Nightmare
The NAT Gateway is the most expensive “invisible” component in AWS. $0.045 per hour just to exist, and then another $0.045 per GB processed. We had a dev run a data migration from an EC2 instance in a private subnet to an S3 bucket. They didn’t use a VPC Endpoint. The data went out through the NAT Gateway, hit the public S3 endpoint, and cost us $3,000 in data processing fees in one afternoon.
If I see traffic going to S3 or DynamoDB through a NAT Gateway, I will personally revoke your VPC permissions. Use Gateway Endpoints. They are free.
For other services like Secrets Manager or STS, use Interface Endpoints (PrivateLink). Yes, they cost money, but they keep traffic off the public internet and are usually cheaper than the NAT data processing fees for high-volume internal traffic.
Run this to see which NAT Gateways are eating our budget:
aws ec2 describe-nat-gateways \
--query "NatGateways[*].{NatId:NatId,VpcId:VpcId,State:State,PublicIp:NatGatewayAddresses[0].PublicIp}" \
--output table
If you see a NAT Gateway in a VPC that only talks to S3… you know what to do.
4. Compute: T-Series Burstable Credits Are Not A Strategy
I am tired of explaining “CPU Credits” to people. You see a t3.micro is cheap and you think, “Hey, I’ll use this for our worker nodes.” Then, three hours into a heavy load, the instance runs out of credits and the CPU gets throttled to 10% of its base performance. The “cold start” of a new instance doesn’t help because the whole cluster is already dying.
For production, use m5 or c5 instances. If you absolutely must use t3, enable unlimited mode so it doesn’t fall over, but watch the billing alerts.
And Lambda? Lambda is not a magic wand. If your Lambda has a 30-second cold start because you decided to wrap a 200MB Java runtime in it, that’s on you. Use Provisioned Concurrency if you care about latency, or better yet, write more efficient code. Stop importing the entire AWS SDK if you only need the DynamoDB client.
5. The Terraform State File Massacre
This is what caused the last 12 hours of my nightmare. Someone tried to run a terraform apply from their local machine while a CI/CD pipeline was already running. The S3 state lock failed because someone else had manually deleted the DynamoDB lock table because it “looked unused.”
Terraform is a double-edged sword. It’s great until the state file gets corrupted or out of sync with reality. CloudFormation is slower and the DSL is a nightmare, but at least AWS manages the state for you. If you’re going to use Terraform, you follow these rules:
1. Remote State Only. No local state files. Ever.
2. State Locking is Mandatory. If the DynamoDB table is gone, the pipeline stops.
3. Targeted Applies are a Last Resort. If you use -target, you are probably doing something wrong.
We had a “split-brain” scenario where the Terraform state thought the Load Balancer was deleted, but it still existed in AWS. The next apply tried to recreate it, failed because of a naming conflict, and then proceeded to delete the target groups. Total. Chaos.
6. Post-Mortem: The Multi-Region Failover That Wasn’t
We tell the stakeholders we have “Multi-Region Failover.” We are lying.
During the us-east-1 “issue” (read: total collapse) last year, we tried to fail over to us-west-2. It failed. Why?
– Route 53 Health Checks: We had the TTL set to 300 seconds. Five minutes of downtime before the DNS even started to update.
– Database Replication Lag: The cross-region RDS read replica was 10 minutes behind because of a massive write spike right before the outage.
– Hardcoded ARNs: Half the microservices had us-east-1 ARNs hardcoded in their config maps.
– S3 Consistency: We relied on objects that hadn’t replicated to the west region yet.
A real failover isn’t a button
Related Articles
Explore more insights and best practices: