text
[root@ops-bastion-01 ~]# aws ce get-cost-and-usage \
–time-period Start=2024-05-18,End=2024-05-21 \
–granularity DAILY \
–metrics “UnblendedCost” \
–group-by Type=DIMENSION,Key=SERVICE
{
“ResultsByTime”: [
{
“TimePeriod”: {“Start”: “2024-05-18”, “End”: “2024-05-19”},
“Groups”: [
{“Keys”: [“Amazon Elastic Compute Cloud – Compute”], “Metrics”: {“UnblendedCost”: {“Amount”: “412.45”, “Unit”: “USD”}}},
{“Keys”: [“Amazon Simple Storage Service”], “Metrics”: {“UnblendedCost”: {“Amount”: “89.12”, “Unit”: “USD”}}},
{“Keys”: [“Amazon VPC”], “Metrics”: {“UnblendedCost”: {“Amount”: “12.50”, “Unit”: “USD”}}}
]
},
{
“TimePeriod”: {“Start”: “2024-05-19”, “End”: “2024-05-20”},
“Groups”: [
{“Keys”: [“Amazon Elastic Compute Cloud – Compute”], “Metrics”: {“UnblendedCost”: {“Amount”: “84200.15”, “Unit”: “USD”}}},
{“Keys”: [“Amazon Simple Storage Service”], “Metrics”: {“UnblendedCost”: {“Amount”: “112450.30”, “Unit”: “USD”}}},
{“Keys”: [“Amazon VPC”], “Metrics”: {“UnblendedCost”: {“Amount”: “53120.44”, “Unit”: “USD”}}}
]
}
]
}
[root@ops-bastion-01 ~]# tail -n 50 /var/log/k8s-autoscaler.log
E0519 03:14:22.123456 static_autoscaler.go:214] Failed to scale up: InsufficientInstanceCapacity for c5.9xlarge in us-east-1a
W0519 03:14:25.987654 cluster_state.go:120] Node group ‘worker-pool-heavy’ is at max size 500, but pending pods exist.
[root@ops-bastion-01 ~]# date
Mon May 21 04:12:09 UTC 2024
I haven't slept since Saturday. My keyboard is sticky with spilled espresso and the bitter residue of a "cloud-native" architecture designed by a consultant who charged $400 an hour to build a system that nearly bankrupted us in a single weekend. The Slack alerts started at 3:00 AM on Sunday—the kind of alerts that don't just ping, they scream. PagerDuty was essentially a funeral dirge for our burn rate.
The "Incident" wasn't a hack. It wasn't a DDoS. It was a perfectly functioning, "infinitely scalable" pipeline doing exactly what it was told to do. It was a monument to the hubris of ignoring the underlying physics of networking and the predatory pricing models of managed services.
## Ticket #9902: The $10,000-an-Hour NAT Gateway
The first thing I saw in the Cost Explorer was a vertical line for VPC costs. In a sane world, VPC costs are a rounding error. In this world—the one the "Architects" built—we were routing 400TB of raw telemetry data from a fleet of EKS worker nodes in a private subnet through a single pair of NAT Gateways to reach an S3 bucket in the same region.
The consultant didn't believe in VPC Endpoints. "Too much complexity," they said. "Just use the default route," they said.
Here is the math they ignored: NAT Gateway processing charges are $0.045 per GB. That sounds small until you realize that 400TB through a NAT Gateway is $18,000 just in processing fees, not counting the standard data transfer rates. Because the worker nodes were in `us-east-1a` and the NAT Gateway was in `us-east-1b` (don't ask), we were also getting hit with cross-AZ transfer taxes of $0.01 per GB in both directions.
I had to rip out the routing table and replace it with a Gateway Endpoint while the system was under load. This is the Terraform I had to hot-patch into production at 4:00 AM while my hands were shaking:
```hcl
# Terraform 1.7.x - Fixing the "Consultant Special"
resource "aws_vpc_endpoint" "s3_gateway" {
vpc_id = var.vpc_id
service_name = "com.amazonaws.us-east-1.s3"
vpc_endpoint_type = "Gateway"
# Attach to all private route tables to bypass the NAT Gateway
route_table_ids = data.aws_route_tables.private.ids
tags = {
Name = "s3-gateway-endpoint"
Environment = "production"
Reason = "Stopping the NAT Gateway bleeding"
}
}
# We also had to kill the NAT Gateways once the routes propagated
# But first, we had to ensure the Interface Endpoints for STS and Kinesis were there
resource "aws_vpc_endpoint" "kinesis_interface" {
vpc_id = var.vpc_id
service_name = "com.amazonaws.us-east-1.kinesis-streams"
vpc_endpoint_type = "Interface"
private_dns_enabled = true
subnet_ids = var.private_subnet_ids
security_group_ids = [aws_security_group.vpc_endpoint_sg.id]
}
The “aws best” practices—the ones they actually put in the whitepapers but nobody reads because they’re too busy looking at “Serverless” infographics—explicitly state that you should use Gateway Endpoints for S3 and DynamoDB. It’s free. It’s a routing table entry. But no, we paid $53,000 for a NAT Gateway to act as a glorified, expensive straw for our data.
Table of Contents
Ticket #9905: S3 Egress and the “Infinite” Data Lake Mirage
The second spike was S3. Specifically, S3 Egress. The “Data Science” team—bless their hearts—decided to run a “global aggregation” job. They had a fleet of c5.9xlarge instances in us-west-2 (Oregon) pulling the entire 400TB dataset from us-east-1 (N. Virginia).
They thought “The Cloud” was just one big happy computer. They didn’t realize that crossing the continental United States with a bitstream costs $0.02 per GB.
400,000 GB * $0.02 = $8,000.
But it wasn’t just one pass. Their Spark job was misconfigured, leading to massive “shuffling” and re-reads. They were calling GetObject on the same 100MB Parquet files thousands of times because they hadn’t implemented any local caching or used S3 Select to filter the data at the source.
I looked at the S3 access logs. It was a horror show of GET requests.
{
"eventTime": "2024-05-19T04:00:01Z",
"eventSource": "s3.amazonaws.com",
"eventName": "GetObject",
"requestParameters": {
"bucketName": "prod-telemetry-lake",
"key": "year=2024/month=05/day=19/data.parquet"
},
"sourceIPAddress": "54.x.x.x",
"userAgent": "aws-sdk-java/1.12.x (Spark-Runtime)"
}
The “S3 Egress Trap” is real. You put data in for free, but you pay a ransom to get it out, especially if you’re crossing regions. I had to kill their IAM roles mid-job to stop the bleeding. They complained about “broken experiments.” I showed them the $112,000 S3 bill. They went quiet.
We had to implement S3 Lifecycle policies immediately. We had “zombie data”—petabytes of logs from 2021 that were still sitting in S3 Standard.
resource "aws_s3_bucket_lifecycle_configuration" "telemetry_lifecycle" {
bucket = aws_s3_bucket.telemetry.id
rule {
id = "archive-old-telemetry"
status = "Enabled"
transition {
days = 30
storage_class = "INTELLIGENT_TIERING"
}
transition {
days = 90
storage_class = "GLACIER_IR"
}
expiration {
days = 365
}
}
}
If you aren’t using Intelligent-Tiering, you are essentially donating money to Jeff Bezos’s next rocket launch. The “consultant-grade” architecture just left everything in Standard because “it’s simpler for the developers.” Simplicity is a luxury we can no longer afford.
Ticket #9908: IAM Spaghetti and the AdministratorAccess Crutch
While I was digging through the wreckage, I found the IAM roles. It was a nightmare of “IAM Spaghetti.” Every single Lambda function, every EC2 instance, and every developer had AdministratorAccess or some variation of a Resource: "*" policy.
The “Architect” had argued that “fine-grained IAM policies slow down velocity.” You know what else slows down velocity? A $250,000 bill and a security audit that takes six months.
I found a “Cleanup Script” running on a t3.medium that had this policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*",
"ec2:*",
"rds:*",
"lambda:*"
],
"Resource": "*"
}
]
}
This is “dangerously permissive” doesn’t even begin to cover it. This is an invitation for a total account takeover. If that instance was compromised, the attacker wouldn’t even need to escalate privileges; they were already the king of the castle.
I spent six hours writing “least privilege” policies using the Access Analyzer, which, by the way, is the only thing that kept me from quitting on the spot. I had to move us to something that actually resembles a security posture:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowS3ReadWriteSpecificBucket",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::prod-telemetry-lake",
"arn:aws:s3:::prod-telemetry-lake/*"
]
},
{
"Sid": "EnforceEncryptionAndVPC",
"Effect": "Deny",
"Action": "s3:*",
"Resource": "arn:aws:s3:::prod-telemetry-lake/*",
"Condition": {
"Bool": {
"aws:SecureTransport": "false"
},
"StringNotEquals": {
"aws:SourceVpc": "vpc-0a1b2c3d4e5f6g7h8"
}
}
}
]
}
The “aws best” way to handle this is to use Permission Boundaries and Service Control Policies (SCPs) at the Organization level, but we were too “agile” for that. We were so agile we ran straight off a cliff.
Ticket #9912: The x86 Legacy Tax and the Graviton Migration That Wasn’t
As I was looking at the EC2 fleet, I realized we were still running almost entirely on c5.4xlarge and m5.large instances. These are Intel-based machines. They are the “legacy tax” of the cloud.
AWS has been screaming about Graviton (ARM-based) for years. 20% cheaper, 40% better performance. But the developers didn’t want to “deal with cross-compiling.” They just wanted to push their bloated Docker images and go to happy hour.
During the spike, the c5.9xlarge instances were struggling with the I/O wait times. I did a quick test—I spun up a c6g.8xlarge (Graviton3) and ran the same ingestion script. Not only did it handle the throughput better, but the hourly rate was significantly lower.
We are literally burning money to support x86 instructions that our Python and Go code doesn’t even care about.
# The old, expensive way
resource "aws_instance" "legacy_worker" {
instance_type = "c5.4xlarge" # $0.68 per hour
ami = data.aws_ami.intel_ami.id
}
# The "I want to keep my job" way
resource "aws_instance" "modern_worker" {
instance_type = "c7g.4xlarge" # $0.58 per hour (and faster)
ami = data.aws_ami.graviton_ami.id
}
The transition to Graviton isn’t just a “nice to have” anymore. It’s a survival tactic. If you’re still on x86 for general-purpose workloads, you’re paying a 20% premium for the privilege of being lazy.
Ticket #9915: Auto-Scaling or Auto-Bankruptcy?
The “Infinite Scalability” myth is the most dangerous lie in tech. The consultant set up the Cluster Autoscaler on EKS with a maximum node count of 500. They thought they were being “resilient.”
When the Spark job went haywire and started spawning thousands of pods due to a recursive retry logic error, the Autoscaler did exactly what it was told. It started spinning up c5.9xlarge instances like it was trying to win a prize.
The problem is that there is no “circuit breaker” in the standard AWS scaling logic. If your code starts looping and requesting more resources, AWS will happily provide them until your credit line is exhausted.
We saw “InsufficientInstanceCapacity” errors in us-east-1a, which means we actually exhausted the physical supply of those instances in that AZ. We were so “scalable” we broke the region’s inventory.
I had to implement hard limits and “Scale-In” protections that should have been there from day one. I also had to deal with “zombie instances”—nodes that the Autoscaler thought were healthy but were actually stuck in a NotReady state due to the NAT Gateway saturation, yet they were still racking up charges.
# Finding the zombies
for node in $(kubectl get nodes | grep NotReady | awk '{print $1}'); do
INSTANCE_ID=$(kubectl get node $node -o jsonpath='{.spec.providerID}' | cut -d'/' -f5)
echo "Terminating zombie instance: $INSTANCE_ID"
aws ec2 terminate-instances --instance-ids $INSTANCE_ID
done
And then there are the “cold starts.” The consultant decided that every single micro-task should be a Lambda function. We had 50,000 Lambdas firing per minute. The “Provisioned Concurrency” costs alone were $12,000 because they wanted to avoid the 200ms cold start.
If you have that much consistent traffic, you don’t use Lambda. You use a long-running process on Fargate or EKS. Lambda is for intermittent, event-driven tasks. Using it for a high-throughput data pipeline is like using a fleet of Ferraris to move gravel. It’s flashy, it’s expensive, and it’s the wrong tool for the job.
Ticket #9920: Remediation and the Terraform State of Despair
The sun is coming up now. The bill has stabilized. The NAT Gateways are gone, replaced by Gateway Endpoints. The cross-region egress has been throttled. The IAM roles are being systematically gutted and replaced with least-privilege policies.
But the damage is done. $250,000. That’s a couple of senior engineer salaries gone in a weekend because of “architectural best practices” that were anything but.
I’m looking at the Terraform state file now. It’s 15MB of JSON misery. I’m using the new removed block in Terraform 1.7 to clean up the mess without destroying the world:
removed {
from = aws_nat_gateway.expensive_mistake
lifecycle {
destroy = true
}
}
# Moving to a more sane architecture
module "vpc_endpoints" {
source = "terraform-aws-modules/vpc/aws//modules/vpc-endpoints"
version = "5.8.1"
vpc_id = var.vpc_id
security_group_ids = [aws_security_group.endpoints.id]
endpoints = {
s3 = {
service = "s3"
service_type = "Gateway"
route_table_ids = data.aws_route_tables.private.ids
},
dynamodb = {
service = "dynamodb"
service_type = "Gateway"
route_table_ids = data.aws_route_tables.private.ids
},
ecr_api = {
service = "ecr.api"
private_dns_enabled = true
subnet_ids = var.private_subnet_ids
},
ecr_dkr = {
service = "ecr.dkr"
private_dns_enabled = true
subnet_ids = var.private_subnet_ids
}
}
}
The “aws best” practices aren’t just suggestions. They are the difference between a successful product and a “Post-Mortem of My Sanity.”
If I see one more “Serverless-First” diagram that doesn’t include a VPC Endpoint or a cost estimation for data transfer, I’m going to throw my MacBook into the nearest river.
The “Infinite Scalability” of the cloud is a lie. It’s only as infinite as your company’s bank account. And right now, our account is looking very, very finite.
I’m going home. I’m turning off my phone. If the cluster scales up again, let it. Let the whole thing burn. At least the fire will be warm, which is more than I can say for this office.
[EOF] – SRE Out.
Related Articles
Explore more insights and best practices: