{"id":4496,"date":"2026-02-08T21:05:53","date_gmt":"2026-02-08T15:35:53","guid":{"rendered":"https:\/\/itsupportwale.com\/blog\/aws-best-practices-the-ultimate-guide-to-cloud-success\/"},"modified":"2026-02-08T21:05:53","modified_gmt":"2026-02-08T15:35:53","slug":"aws-best-practices-the-ultimate-guide-to-cloud-success","status":"publish","type":"post","link":"https:\/\/itsupportwale.com\/blog\/aws-best-practices-the-ultimate-guide-to-cloud-success\/","title":{"rendered":"AWS Best Practices: The Ultimate Guide to Cloud Success"},"content":{"rendered":"

text
\n$ kubectl get pods -n prod-core
\nNAME READY STATUS RESTARTS AGE
\napi-gateway-7f5d8b9-x2k9l 0\/1 CrashLoopBackOff 42 14m
\norder-processor-0 0\/1 Pending 0 72h
\norder-processor-1 0\/1 Pending 0 72h
\npayment-service-5d4f9c8-m1n2b 0\/1 CreateContainerConfigError 0 12m<\/p>\n

$ kubectl describe pod order-processor-0
\nEvents:
\n Type Reason Age From Message
\n —- —— —- —- ——-
\n Warning FailedScheduling 3m (x450 over 72h) default-scheduler 0\/3 nodes are available: 3 node(s) had volume node affinity conflict.
\n Normal NotSpecified 12m ebs.csi.aws.com Failed to attach volume “vol-0a1b2c3d4e5f6g7h8”: rpc error: code = Internal desc = Could not attach volume “vol-0a1b2c3d4e5f6g7h8” to node “i-0987654321fedcba”: AccessDenied: User: arn:aws:sts::123456789012:assumed-role\/eks-node-role\/i-0987654321fedcba is not authorized to perform: ec2:AttachVolume on resource: arn:aws:ec2:us-east-1:123456789012:volume\/vol-0a1b2c3d4e5f6g7h8<\/p>\n

$ aws sts get-caller-identity –query “Arn” –output text
\narn:aws:iam::123456789012:user\/exhausted-sre-who-wants-to-go-home<\/p>\n

I\u2019m writing this because if I don\u2019t, the "Cloud Architect" who caused this will probably get promoted for "proactive incident response" while I\u2019m still picking the shrapnel of our production environment out of my teeth. We just spent 72 hours on a bridge call because someone thought a Medium article from 2019 was a valid replacement for reading the actual AWS documentation. \n\nOur "architecture" was a house of cards built on burstable instances, a misunderstanding of how physics works in a multi-AZ environment, and an IAM policy that was basically a "Please Hack Me" sign. If you\u2019re reading this and you\u2019re the one who committed that Terraform code, consider this your formal invitation to stay the hell away from my VPCs.\n\n## The T3\/T2 Burst Credit Trap: Why Your Production Cluster is a Zombie\n\nLet\u2019s start with the most basic, amateur-hour mistake: running production EKS worker nodes on `t3.medium` instances. I don\u2019t care what the "aws best" practices guide for cost-optimization says in the "Getting Started" section; if you are running a stateful workload on a burstable instance, you are gambling with the stability of the entire stack.\n\nHere is what happened: Kevin (our architect) saw that `t3.mediums` were cheaper than `m5.large` instances. He figured, "Hey, our average CPU utilization is only 15%, so we\u2019ll save thousands." What he didn't account for\u2014because he doesn't live in the real world\u2014is the CPU Credit Balance. When the order-processor service hit a minor spike, the T3 instances exhausted their credits. In AWS-land, when a T3 runs out of credits, it doesn't just "slow down." It throttles you to a baseline performance that is so abysmal the Kubelet stops responding to the control plane.\n\nThe control plane, thinking the node is dead, marks it as `NotReady` and tries to reschedule 400 pods onto the remaining two nodes. Those nodes, already sweating, immediately burn through their own credit balances and die. It\u2019s a cascading failure that no amount of "cloud-native" magic can fix.\n\nIf you\u2019re running production, you use M5, C5, or R5 instances. Period. You want predictable performance. You don\u2019t want your infrastructure to turn into a pumpkin at 3:00 AM because of a "credit exhaustion" event.\n\n```bash\n# How to check if your "architect" is sabotaging you with burstable instances\naws ec2 describe-instances \\\n    --filters "Name=instance-type,Values=t2.*,t3.*" \\\n    --query "Reservations[*].Instances[*].{ID:InstanceId,Type:InstanceType,State:State.Name}" \\\n    --output table\n<\/code><\/pre>\n
If that command returns anything in your production account, you have a ticking time bomb. Switch to m5.large<\/code> or higher. The $20 you save on the instance bill will be dwarfed by the $50,000 in lost revenue when the cluster enters a death spiral.<\/p>\n
\n
Table of Contents<\/p>\n