Table of Contents
AWS is a Collection of Primitives, Not a Magic Wand
I once cost a former employer $42,000 in a single weekend because I misunderstood how AWS NAT Gateways handle data processing. I was migrating a high-throughput logging cluster—think 50TB of JSON blobs a day—from a co-located data center into a VPC in us-east-1. I followed the “best practice” architecture: private subnets for the workers, a NAT Gateway in the public subnet for outbound traffic. I didn’t realize that NAT Gateways charge $0.045 per GB of data processed. By Sunday night, the billing alert hit my inbox like a physical punch to the gut. The “cloud” wasn’t just someone else’s computer; it was someone else’s computer with a very expensive, very invisible toll booth at every intersection.
That’s the reality of AWS. It isn’t a “solution.” It’s a massive, sprawling, often contradictory collection of low-level primitives. If you treat it like a magic black box that handles your scaling and reliability for you, it will bankrupt you or break your application in ways that the documentation never explicitly warns you about. This post is about what AWS actually is when you strip away the marketing fluff and the “certified architect” exam questions. It’s about the trade-offs, the hidden costs, and the technical debt you’re signing up for.
What is AWS, Really?
At its core, AWS is a programmable interface for hardware. Before 2006, if you wanted a server, you talked to a guy named Dave in a cage who would rack a Dell PowerEdge. Now, you send a POST request to an API endpoint. That’s the only fundamental difference. Everything else—the “serverless” hype, the “managed” databases, the “AI-driven” insights—is just layers of abstraction built on top of Xen or KVM virtualization, S3 storage buckets, and a very complex networking layer called the VPC.
Most people get AWS wrong because they think it’s a platform. It’s not. Heroku is a platform. AWS is a hardware store. If you go to a hardware store expecting to buy a house, you’re going to be disappointed when they hand you a pile of 2x4s and a bucket of nails. AWS gives you the 2x4s (EC2), the nails (S3), and the plumbing (VPC), but you still have to build the house. And if you don’t know how to swing a hammer, the house is going to fall down the first time a single Availability Zone (AZ) has a power blip.
Pro-tip: Stop looking at the “Services” menu. 90% of those 200+ services are wrappers around the core five: EC2, S3, IAM, VPC, and RDS. Master those, and the rest is just learning new API parameters.
The Foundation: VPC and the Networking Tax
The Virtual Private Cloud (VPC) is where every AWS journey begins and where most of them fail. It is a logically isolated section of the AWS Cloud where you launch your resources. But the documentation makes it sound simpler than it is. You aren’t just picking an IP range like 10.0.0.0/16; you are designing a network topology that must account for latency, blast radius, and data transfer costs.
In the real world, networking is the most expensive thing you don’t plan for. AWS charges you for “Data Transfer Out” (DTO) to the internet, but they also charge you for data moving between Availability Zones. If your application server in us-east-1a talks to your database in us-east-1b, you’re paying $0.01 per GB in both directions. It sounds small. It isn’t. For a high-traffic microservices architecture, this “AZ Tax” can easily become 20% of your total bill.
- Subnetting: Don’t just create one big subnet. Use /24s. It gives you 251 usable IP addresses per subnet (AWS reserves 5). It’s enough for most workloads and keeps your routing tables manageable.
- Endpoints: If your instances need to talk to S3, do not go over the public internet or through a NAT Gateway. Use a VPC Gateway Endpoint. It’s free. It keeps traffic on the internal AWS backbone. I’ve seen companies save $5k a month just by adding a single line of Terraform for an S3 endpoint.
# Example: A simple VPC Endpoint for S3 in Terraform
resource "aws_vpc_endpoint" "s3" {
vpc_id = aws_vpc.main_prod.id
service_name = "com.amazonaws.us-east-1.s3"
route_table_ids = [aws_route_table.private_rt.id]
}
Compute: EC2 vs. Lambda (The “Serverless” Lie)
Compute is the heart of AWS. You have two main paths: Elastic Compute Cloud (EC2) or AWS Lambda. EC2 is a virtual machine. You choose your CPU, your RAM, and your OS. Lambda is “Serverless,” which is a marketing term that means “someone else manages the OS and the scaling, but you pay a premium for the privilege.”
I am opinionated about this: Lambda is for glue code, not for core business logic. If you have a steady-state workload, Lambda is significantly more expensive than EC2 or Fargate. Plus, you have to deal with “Cold Starts.” When a Lambda hasn’t been used in a while, AWS has to spin up a new Firecracker microVM to run your code. If you’re running a Java runtime on Lambda, that cold start can be 5-10 seconds. Your users will hate you.
EC2, on the other hand, is predictable. But you have to understand the instance types. If you use a t3.medium, you are using “burstable” CPU. You earn CPU credits when the instance is idle and spend them when it’s busy. If you run out of credits, AWS throttles your CPU to a tiny fraction of its capacity. I once spent 6 hours debugging why a Go binary was suddenly taking 200ms to process a request that usually took 5ms. The answer? CPUCreditBalance was zero. We switched to a c5.large (fixed CPU) and the problem vanished.
- Nitro System: Modern instance types (C5, M5, R5, etc.) run on the Nitro System. It offloads networking and storage tasks to dedicated hardware. Use these. Avoid the old T2 or M3 instances; they are legacy tech with higher overhead.
- IMDSv2: The Instance Metadata Service is how your code gets its IAM credentials. Always enforce IMDSv2. IMDSv1 is vulnerable to SSRF (Server-Side Request Forgery) attacks. If an attacker can make your server perform a GET request to
http://169.254.169.254/latest/meta-data/iam/security-credentials/, they own your account. - Spot Instances: If your workload is fault-tolerant (like a CI/CD runner or a stateless worker), use Spot. You can get up to 90% off the on-demand price. Just be ready for the instance to be terminated with a 2-minute warning.
Storage: S3 is the Only Perfect Service
If AWS disappeared tomorrow and I could only save one service, it would be S3 (Simple Storage Service). It is the most robust, reliable piece of distributed systems engineering ever made publicly available. It promises 11 nines of durability (99.999999999%). That means if you store 10,000,000 objects, you might lose one every 10,000 years.
But S3 is not a file system. It is an object store. This is a crucial distinction. You cannot “append” to a file in S3. You have to overwrite the whole object. And while S3 now has strong read-after-write consistency, it didn’t used to. We used to have to build complex “wait-and-retry” logic because if you uploaded config.json and immediately tried to read it, you might get a 404 or an old version.
The “Gotcha” with S3 is API costs. Everyone looks at the storage cost ($0.023 per GB), but they forget the PUT and LIST costs. If you have a script that does a LIST operation on a bucket with millions of small files every minute, your bill will explode. I’ve seen a “cleanup script” cost more in API calls than the storage it was trying to save.
Note to self: Always use S3 Lifecycle policies. Don’t write a Python script to delete old logs. Let AWS do it at the infrastructure level. It’s cheaper and it won’t fail because of a
TimeoutError.
{
"Rules": [
{
"ID": "MoveToGlacierAfter30Days",
"Status": "Enabled",
"Filter": { "Prefix": "logs/" },
"Transitions": [
{
"Days": 30,
"StorageClass": "GLACIER"
}
]
}
]
}
IAM: The Real Boss of AWS
Identity and Access Management (IAM) is the most difficult part of AWS. It is the gatekeeper. If you get IAM wrong, you either have a massive security hole or a broken application. Most people default to AdministratorAccess because they are frustrated. This is how you end up in the news after a leaked access key results in a $100k crypto-mining bill.
IAM is based on the Principle of Least Privilege. Your application should only have the exact permissions it needs. If it only needs to read from one S3 bucket, don’t give it s3:*. Give it s3:GetObject on that specific ARN.
The policy language is a JSON-based DSL that is notoriously finicky. One missing comma or a misplaced wildcard and nothing works. And the error messages from the AWS SDK are often useless. “Access Denied” doesn’t tell you which permission you’re missing. You have to use the IAM Policy Simulator or dig through CloudTrail logs to find the specific API call that failed.
Here is what a “real” policy looks like. It restricts access to a specific bucket and requires the user to be coming from a specific office IP range:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::my-secure-data-prod",
"arn:aws:s3:::my-secure-data-prod/*"
],
"Condition": {
"IpAddress": {
"aws:SourceIp": "203.0.113.0/24"
}
}
}
]
}
The Managed Service Trap: RDS and Beyond
AWS loves to sell you “Managed Services.” The pitch is: “Don’t run your own Postgres, use RDS! We handle backups, patching, and failover.”
This is mostly true, but “Managed” does not mean “Hands-off.” You still have to manage your database. You still have to tune your autovacuum settings in Postgres. You still have to monitor your IOPS. If you choose a db.t3.micro for your production database, it will fall over the moment you have more than 10 concurrent connections, managed or not.
The biggest lie in RDS is “Multi-AZ.” It provides high availability by keeping a synchronous standby in a different AZ. If the primary dies, AWS flips the DNS record to the standby. This is great, but it is not a silver bullet for performance. Because the replication is synchronous, every COMMIT on your primary has to wait for an acknowledgement from the standby. This adds latency. If your app is write-heavy, Multi-AZ will slow you down. You need to weigh that against the cost of downtime.
And then there’s EBS (Elastic Block Store), the “disks” for your instances. Most people use gp2. Use gp3 instead. gp2 ties your IOPS (input/output operations per second) to the size of the disk. If you want more speed, you have to buy more space. gp3 decouples them. You can have a small 20GB disk with 12,000 IOPS. It’s cheaper and more flexible. Switching from gp2 to gp3 is a zero-downtime operation. Do it today.
The “Real World” Gotchas
After a decade in the trenches, you start to see the patterns of how AWS fails. It’s rarely a total region outage (though us-east-1 tries its best once a year). It’s usually a “gray failure.”
1. The “Kinesis Wedge”: You’re using Kinesis for data streaming. One of your shards gets a “hot key”—too much data for one shard to handle. The entire consumer group slows down because it’s stuck processing that one shard. AWS won’t auto-split that shard for you. You have to monitor IncomingRecords per shard and manually trigger a split. If you don’t, your data lag will grow until your consumers OOM-kill themselves trying to catch up.
2. The “CloudWatch Delay”: CloudWatch is the monitoring service. It is slow. Standard metrics have a 1-minute granularity, but they often lag by 2-3 minutes. If you are relying on CloudWatch Alarms to trigger an Auto Scaling Group (ASG) during a traffic spike, your site will be down before the new instances even start booting. You need to scale proactively or use custom metrics with high-resolution (1-second) reporting.
3. The “NAT Gateway Bottleneck”: A single NAT Gateway is limited to 45 Gbps. That sounds like a lot, but it also has a limit on concurrent connections (55,000 to a single destination). If you have a fleet of microservices all talking to the same external API (like api.stripe.com), you can exhaust the NAT Gateway’s ports. Your application will start throwing “Connection Timeout” errors, and you’ll spend hours looking at your code when the problem is actually a managed networking component you don’t even control.
The Hidden Complexity of “What is AWS”
When someone asks “what is AWS,” they usually want a simple answer. The simple answer is “a cloud provider.” The real answer is “a distributed systems nightmare disguised as a user-friendly web console.”
To use AWS effectively, you have to stop thinking like a developer and start thinking like a systems engineer. You have to care about packet loss. You have to care about disk throughput. You have to care about the CAP theorem. AWS doesn’t solve these problems; it just gives you a different set of tools to fail with.
Take “Global Accelerator” as an example. It’s a service that gives you static IP addresses that act as anycast entry points into the AWS network. It’s brilliant for reducing latency for global users. But if you don’t understand how BGP (Border Gateway Protocol) works, you won’t understand why your traffic is suddenly taking a detour through Tokyo when your users are in London. You are still bound by the laws of physics and the quirks of the internet’s routing tables.
The Cost of Convenience
The “Cloud Tax” is real. You are paying a 30-50% premium over raw hardware for the ability to spin up a server in 30 seconds. For a startup, this is a no-brainer. Speed to market is everything. But for a mature company with predictable workloads, the “What is AWS” question eventually turns into “Why are we still on AWS?”
I’ve seen companies move back to on-prem or colocation (the “Cloud Exit”) and save millions. But they only succeed if they have a team that knows how to manage hardware. Most modern SREs don’t know how to replace a failed DIMM or configure a top-of-rack switch. AWS has abstracted that knowledge away. That is the true “lock-in.” It’s not just the proprietary APIs like DynamoDB; it’s the atrophy of low-level engineering skills.
Final Advice for the Weary SRE
If you’re tasked with “moving to the cloud” or managing an existing AWS footprint, ignore the certifications. Don’t spend your time memorizing the names of every ML service they announced at re:Invent. Instead, do this:
- Learn the CLI: The console is a lie. It hides complexity and encourages manual “click-ops.” If you can’t do it via the AWS CLI or Terraform, it doesn’t exist.
- Monitor the Bill: Your billing dashboard is your most important architectural diagram. If a service’s cost is spiking, it means your architecture is inefficient. Cost is a proxy for technical health.
- Assume Failure: Every service in AWS will fail. S3 will have an outage. EBS volumes will degrade. AZs will go dark. Design your system so that you can lose an entire data center and your users don’t notice. If you haven’t tested a failover this quarter, you don’t have high availability; you have a hope and a prayer.
- Read the “Limits” Page: Every AWS service has service quotas. Some are soft (you can ask for an increase), some are hard. Knowing that an NLB has a limit of 50 listeners per load balancer before you design your multi-tenant architecture will save you months of refactoring.
AWS is a powerful, dangerous tool. It’s a Ferrari with no brakes and a very expensive gas tank. Drive it carefully, keep your eyes on the gauges, and never, ever trust the default settings.
Stop treating AWS as a platform and start treating it as a vendor of volatile, expensive, but incredibly flexible raw materials. Build your own platform on top of it, keep it simple, and for the love of everything holy, set up your billing alerts before you push to prod.
Related Articles
Explore more insights and best practices: