Table of Contents
AWS is Not a Cloud; It’s a Billable API for Someone Else’s Technical Debt
I once cost a former employer $14,200 in forty-eight hours because I misunderstood a single checkbox in the VPC console. I was setting up a “highly available” log processing cluster in us-east-1. I provisioned a NAT Gateway in each Availability Zone to ensure that if one rack caught fire, our logs would still flow. What I didn’t realize was that our log-shipper, a poorly configured Fluentd daemon, was sending 400TB of raw JSON to an external endpoint over the public internet instead of using a VPC Endpoint. The NAT Gateway data processing charge is $0.045 per GB. Do the math. By the time the billing alert hit my inbox, the damage was done. I didn’t get fired, but I did get a permanent twitch in my left eye whenever I see the word “Managed.”
That is the reality of AWS. It isn’t a “seamless transition to the cloud.” It is a sprawling, inconsistent, and often hostile collection of primitives that will happily let you bankrupt your company if you don’t understand the underlying hardware. If you’re looking for a definition of what is AWS that sounds like a marketing brochure, go to their homepage. If you want to know what it actually is—a massive API for renting virtualized components—keep reading.
The Identity Crisis: IAM is the Only Service That Matters
Most people think AWS is about servers. They’re wrong. AWS is an identity management platform that happens to sell compute on the side. If you don’t get Identity and Access Management (IAM) right, nothing else matters. You can have the most resilient Kubernetes cluster in the world, but if your s3:PutObject permission is scoped to *, you’re one leaked environment variable away from a data breach.
IAM is where the “what is” of AWS becomes painfully clear. It is a system of JSON policies that define who can do what. It is notoriously difficult to debug. You will spend hours staring at a 403 Forbidden error, wondering why your Lambda function can’t read from a DynamoDB table, only to realize you forgot the kms:Decrypt permission for the underlying encryption key.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::prod-customer-data-9921",
"arn:aws:s3:::prod-customer-data-9921/*"
],
"Condition": {
"IpAddress": {
"aws:SourceIp": "203.0.113.0/24"
}
}
}
]
}
Look at that policy. It looks simple, but it’s a landmine. If you omit the /* on the resource ARN, you can list the bucket but you can’t read the files. If you forget the Condition block, any compromised credential with this role can pull your data from a Starbucks Wi-Fi. This is the granularity AWS demands. It doesn’t hold your hand. It expects you to be a security engineer, a network architect, and a systems administrator simultaneously.
Pro-tip: Never use the
AdministratorAccessmanaged policy for your CI/CD runners. Use a tool likeiamliveto track the actual calls your Terraform or Pulumi code makes and generate a least-privilege policy. It’s tedious, but it’s better than a post-mortem.
The Networking Tax: VPCs and the Lie of “The Cloud”
When people ask “what is” the cloud, they usually imagine some ethereal space where data floats freely. In AWS, the cloud is a Virtual Private Cloud (VPC), and data movement is the most expensive thing you will ever do. AWS charges you for data leaving a region. They charge you for data crossing an Availability Zone (AZ). They even charge you for data moving between services in the same region if you don’t use a VPC Endpoint.
The VPC is the foundation. You carve out an IPv4 CIDR block—usually something like 10.0.0.0/16—and then you start slicing it into subnets. This is where the 1990s return to haunt you. You have to care about routing tables, internet gateways, and CIDR math. If you pick a CIDR block that overlaps with your corporate VPN, you are in for a world of “YAML-hell” trying to refactor your infrastructure later.
- Public Subnets: They have a route to an Internet Gateway. Use these for load balancers and nothing else.
- Private Subnets: No direct internet access. This is where your databases and application servers live. They talk to the world via a NAT Gateway (the $14k mistake mentioned earlier).
- VPC Endpoints (PrivateLink): These allow your VPC to talk to S3 or DynamoDB without the traffic ever hitting the public internet. They cost money per hour, but they save you from the data egress tax.
- Security Groups: These are stateful firewalls. They are not “rules”; they are a distributed firewall layer that lives at the ENI (Elastic Network Interface) level.
- NACLs: Network Access Control Lists. They are stateless and a nightmare to manage. Most SREs I know set them to
ALLOW ALLand handle everything in Security Groups because life is too short for stateless debugging.
The biggest “gotcha” in AWS networking is the “Inter-AZ Data Transfer” fee. If your application server in us-east-1a talks to your database in us-east-1b, you pay $0.01 per GB in both directions. For a high-throughput microservices architecture, this can easily become 30% of your total bill. You end up designing “AZ-affinity” into your service mesh just to avoid paying Jeff Bezos for the privilege of moving bits across a fiber optic cable in Virginia.
Compute: From Rented Metal to Ephemeral Functions
What is AWS compute? It’s a spectrum of control versus convenience. At one end, you have EC2 (Elastic Compute Cloud). These are just virtual machines running on Xen or Nitro hypervisors. You pick an instance type—like a c5.xlarge if you need CPU or an r5.2xlarge if you’re running a memory-hungry Java app—and you manage it. You patch the OS. You rotate the SSH keys. You worry about disk pressure.
Then there’s Lambda. The “Serverless” dream. You upload a ZIP file or a container image, and AWS runs it in response to an event. No servers to manage, right? Wrong. Now you have to manage “Cold Starts.” If your Lambda hasn’t run in a while, the first request will take 2 seconds to initialize the runtime. If you’re building a high-frequency trading platform, Lambda is a joke. If you’re processing image uploads to S3, it’s a godsend.
The middle ground is Fargate. It’s “Serverless” containers. You don’t manage the underlying EC2 instances, but you still have to define CPU and Memory limits. If your app leaks memory and hits that limit, the kernel will OOM-kill your process, and Fargate will just restart the container in a loop. I’ve seen teams spend weeks debugging “random” restarts only to find they set their memory limit to 512MB for a Node.js app that needed 1GB just to start up.
# A snippet of a Task Definition that will probably fail
{
"containerDefinitions": [
{
"name": "api-service",
"image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/api:v1.0.4",
"cpu": 256,
"memory": 512,
"essential": true,
"portMappings": [
{
"containerPort": 8080,
"hostPort": 8080
}
]
}
]
}
In the real world, compute is about trade-offs. EC2 gives you the best price-to-performance ratio if you have a steady load and use Reserved Instances or Savings Plans. Lambda is the cheapest for intermittent workloads but the most expensive for sustained high throughput. Fargate is for people who have more money than time and just want their Docker containers to run without thinking about yum update.
Storage: S3 is the Only Magic Left
If AWS has one “killer app,” it’s S3 (Simple Storage Service). It is the only service that actually feels like the future. It’s an object store with “eleven nines” of durability (99.999999999%). That means if you store 10,000 objects, you might lose one every 10,000,000 years. I trust S3 more than I trust my own ability to remember my mother’s birthday.
But even S3 has its quirks. It’s an object store, not a file system. You can’t “rename” a folder because folders don’t exist. There are only keys and values. If you want to rename a “folder” with 1TB of data, you have to copy every single object to a new key and delete the old ones. That’s a lot of PUT and DELETE requests, and yes, AWS charges you for every single one of them.
Then there’s the consistency model. For years, S3 was “eventually consistent” for overwrites. If you updated config.json and immediately read it back, you might get the old version. They fixed this in 2020 to be “strong read-after-write consistency,” which solved a decade of distributed systems headaches. But the scars remain. Old-school S3 users still build retry logic into their code out of pure habit.
And we have to talk about Glacier. It’s the “cold storage” for S3. It’s incredibly cheap to store data, but it’s incredibly expensive and slow to get it back. I once saw a junior dev move 50TB of logs to Glacier Deep Archive to save money, not realizing that the “expedited retrieval” cost to get those logs back for a compliance audit would cost more than his annual salary. S3 is a tiered system; you have Standard, Intelligent-Tiering, Standard-IA, and Glacier. Choosing the wrong one is a financial decision, not a technical one.
The Database Dilemma: RDS vs. DynamoDB
What is AWS when it comes to data? It’s a choice between the familiar and the scalable. RDS (Relational Database Service) is just Postgres, MySQL, or SQL Server managed by AWS. They handle the backups, the patching, and the multi-AZ failover. It’s great until you hit the limits of vertical scaling. When your db.m5.24xlarge is hitting 90% CPU, you can’t just “add more servers.” You have to start sharding or move to Aurora.
Aurora is AWS’s proprietary “cloud-native” database. It separates storage from compute. It’s faster, more resilient, and significantly more expensive. It’s also a “black box.” When Aurora has a performance hiccup, you can’t just check the disk latency of the underlying EBS volume. You have to rely on “Enhanced Monitoring” and hope the metrics give you a clue.
On the other side is DynamoDB. It’s a NoSQL database that can handle millions of requests per second with single-digit millisecond latency. It’s incredible. It’s also a trap for anyone who likes SQL. There are no joins. There are no complex queries. You have to model your data based on your access patterns. If you realize six months into production that you need to query your data by a different attribute, you have to create a Global Secondary Index (GSI), which—you guessed it—costs more money and adds replication lag.
Note to self: Always enable “Point-in-Time Recovery” (PITR) on DynamoDB. It’s not enabled by default, and without it, a
DELETE *from a buggy script is permanent. There is no “undo” in NoSQL.
The Hidden Complexity of “Managed” Services
The marketing says AWS manages the “undifferentiated heavy lifting.” This is a half-truth. They manage the hardware and the base software, but they don’t manage the configuration. Managed services like EKS (Elastic Kubernetes Service) are a prime example. AWS gives you a managed Control Plane, but you still have to manage the worker nodes, the CNI plugin, the CoreDNS configuration, and the IAM roles for service accounts (IRSA).
I’ve spent more time debugging the interaction between the AWS VPC CNI and the Kubelet than I ever did managing on-prem servers. In EKS, you can run out of IP addresses because every Pod gets a real IP from your VPC CIDR. If you picked a small subnet (like a /24), you can only run about 250 Pods across your entire cluster before the VPC is “full.” This is the kind of “managed” complexity that kills productivity.
Then there’s MSK (Managed Streaming for Kafka). Kafka is notoriously hard to run. AWS makes it “easier” by giving you a managed cluster. But they don’t give you access to the underlying brokers. If a partition gets stuck or a disk fills up, you’re stuck waiting for AWS Support to look at it, or you’re digging through CloudWatch logs that are 5 minutes delayed. “Managed” often means “I have fewer knobs to turn when things go wrong.”
The Console is a Lie, Terraform is the Truth
If you are using the AWS Web Console to manage your infrastructure, you are not doing SRE; you are doing “Click-Ops.” The console is fine for exploring a new service, but it’s a disaster for production. It’s inconsistent—some services use the new “Polaris” design system, while others look like they haven’t been updated since 2012 (looking at you, Simple Workflow Service).
The real AWS is defined in code. Whether it’s Terraform, CloudFormation, or the CDK (Cloud Development Kit), infrastructure-as-code is the only way to survive. But even here, AWS shows its cracks. CloudFormation is slow and often gets stuck in “UPDATE_ROLLBACK_FAILED” state, which requires a support ticket to fix. Terraform is better, but it’s a third-party tool that has to play catch-up every time AWS releases a new feature.
# Terraform example for a simple S3 bucket
resource "aws_s3_bucket" "data_lake" {
bucket = "company-data-lake-prod"
tags = {
Environment = "Prod"
ManagedBy = "Terraform"
CostCenter = "DataEngineering"
}
}
resource "aws_s3_bucket_public_access_block" "data_lake_privacy" {
bucket = aws_s3_bucket.data_lake.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
Notice how it takes two separate resources just to make a bucket private? That’s AWS in a nutshell. The default state of many older services was “public” or “permissive,” and AWS has had to bolt on security layers over time. If you don’t use a tool like Terraform to enforce these standards, you will eventually leak data. It’s not a matter of if, but when.
The Real-World Gotcha: The “Soft” Limits
Every AWS account comes with “Service Quotas.” These are limits on how many resources you can create. Some are “hard” limits (you can’t have more than 5 VPCs per region), and some are “soft” (you can only have 20 EC2 instances). You will hit these limits at 3:00 AM during an auto-scaling event. Your application will try to scale up to handle a traffic spike, AWS will say “LimitExceeded,” and your site will go down.
You have to proactively request limit increases. But you can’t just request “infinity.” You have to justify it. And for some limits, like the number of API calls you can make to the EC2 service (Rate Limiting), there is no official quota you can see. You just start getting ThrottlingException errors. If you’re running a large-scale Kubernetes cluster or a massive Terraform deployment, you will hit the “DescribeInstances” rate limit, and your entire automation pipeline will grind to a halt.
The Bill: Where the Magic Dies
What is AWS at the end of the month? It’s a PDF that is impossible to read. The AWS Bill is a 10,000-line CSV file if you enable the Cost and Usage Report (CUR). It breaks down every single cent. You’ll see things like “$0.000001 for 732 PutRequests in us-east-1.”
The complexity of the billing is a feature, not a bug. It makes it very difficult to see where you are wasting money. You need a whole sub-industry of tools (like CloudHealth or Vantage) just to understand what you’re paying for. The most common sources of waste I see are:
- Unattached EBS Volumes: You delete an EC2 instance, but the virtual hard drive (EBS) stays behind, costing you $0.10 per GB per month forever.
- Old Snapshots: You took a backup of a database three years ago. It’s still there. You’re still paying for it.
- Idle Load Balancers: An ALB costs about $16/month just to exist, even if it’s not processing a single request.
- NAT Gateway Idle Charges: $32/month per AZ, regardless of traffic.
- Over-provisioned Instances: Running a
t3.largewhen at3.microwould do.
AWS is a game of margins. They make their money on the people who forget to turn things off. As an SRE, half my job is just being the person who turns things off.
The “What Is” Summary That Isn’t a Summary
AWS is a collection of low-level building blocks that require a high level of expertise to assemble safely. It is not a “platform as a service” in the way Heroku is. It is a programmable data center. It gives you the power to build global-scale infrastructure in minutes, but it also gives you the power to lose a million dollars or leak a billion records with a single API call. If you treat it like a magic black box, it will eventually explode. If you treat it like a complex, fragile system of interconnected hardware and software primitives, you might just make it work.
Stop looking for the “Easy Button.” In AWS, the only way out is through the documentation, the CLI, and a very, very careful look at your Cost Explorer every Monday morning.
Related Articles
Explore more insights and best practices: