Table of Contents
Kubernetes: The Distributed System You Probably Don’t Need (But Are Stuck With Anyway)
It was 3:14 AM on a Tuesday in 2021. I was staring at a Grafana dashboard that looked like a heart monitor for someone having a massive coronary. We were running a fleet of microservices on Kubernetes v1.20 in us-east-1. I had just pushed a change to the ingress-nginx controller config to “optimize” the buffer sizes for a new API endpoint. Within ninety seconds, the kube-apiserver latency spiked to thirty seconds. The cluster was effectively lobotomized. Nodes started dropping into NotReady status because the kubelet couldn’t report heartbeats. The controller-manager, sensing blood in the water, started rescheduling pods from the “dead” nodes onto the “live” ones, which promptly collapsed under the sudden surge of traffic and I/O pressure.
The culprit wasn’t the config change itself. It was a cascading failure triggered by a misconfigured livenessProbe. When the ingress controller reloaded, it momentarily stopped responding to health checks. Kubernetes, being the dutiful soldier it is, killed the pods. But because we hadn’t set proper readinessProbes, the service endpoints were removed before the new pods were ready. Traffic hit a black hole. The retry logic in our checkout-service was too aggressive, creating a thundering herd that saturated the conntrack tables on every worker node. I spent four hours manually scaling the aws-node daemonset just to get the CNI to stop choking on its own tongue. That is the reality of Kubernetes: it is a force multiplier for both your productivity and your ability to destroy your own infrastructure.
The “Hello World” Lie
Most Kubernetes documentation is written by people who want to sell you a managed service or a YAML-templating tool. They show you a 10-line Deployment manifest and tell you that you’ve achieved “web-scale.” They don’t tell you about the ndots: 5 issue in /etc/resolv.conf that adds 20ms of latency to every external DNS lookup. They don’t mention that Resources.Limits.CPU is implemented via CFS quotas, which will throttle your application into the dirt even if the node has 90% idle CPU. They definitely don’t talk about the nightmare of managing stateful sets when your EBS volume is stuck in attaching state for twenty minutes because of a race condition in the CSI driver.
We use Kubernetes because we want an API for our infrastructure. That’s it. It’s not about “containers” anymore; it’s about having a standardized way to describe a desired state and letting a control loop try—and often fail—to reach it. If you are running three Go binaries and a Postgres instance, you are paying a massive “complexity tax” for features you will never use. You are managing a distributed database (etcd), a complex overlay network (Overlay/VxLAN), and a sophisticated scheduler just to do what a systemd unit and a bash script could do in 50 lines of code.
The Control Plane: Where the Bodies are Buried
The kube-apiserver is the only thing that matters. Everything else—the kube-scheduler, the kube-controller-manager, the kubelet—is just a client of the API. When people talk about “Kubernetes scaling,” they are usually talking about etcd performance. If your etcd fsync latency climbs above 10ms, your cluster is a ticking time bomb.
Pro-tip: Never, ever run etcd on the same disks as your application logs. I’ve seen
fluentbitsaturate disk I/O during a log spike, causing etcd to lose quorum, which resulted in the entire cluster entering a read-only state. Use dedicated NVMe drives for etcd.
The scheduler is another area where “magic” happens. It’s essentially a massive for loop that matches pods to nodes. But it’s a greedy algorithm. It doesn’t look at actual usage; it looks at Requests. If you have a node with 32GB of RAM and you have 4 pods requesting 8GB each, that node is “full,” even if those pods are only actually using 512MB. This leads to the “Bin Packing” problem. You end up with 40% cluster utilization but you can’t schedule a new pod because of your Requests settings.
# A typical "I don't know what I'm doing" resource block
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2" # This will cause CFS throttling. Avoid it for latency-sensitive apps.
memory: "1Gi" # Keep requests and limits equal for Memory to get 'Guaranteed' QoS.
If you set limits.cpu, the Linux kernel will enforce a quota. If your app is multi-threaded (like a Java or Go runtime), it will burst, hit the quota, and the kernel will stop it from running for the remainder of the period. This looks like “random” latency spikes in your APM. My opinion? Don’t set CPU limits unless you’re in a multi-tenant environment where you don’t trust the developers. Set requests to what you actually need and let the scheduler do its job.
Networking: The CNI and the ndots Disaster
Kubernetes networking is a lie built on top of iptables or IPVS. When a pod at 10.2.4.5 wants to talk to api.stripe.com, it doesn’t just go out. It goes through a series of transformations that would make a mathematician weep. If you’re using the default bridge mode, you’re losing performance. If you’re using Calico with BGP, you’re managing a network topology you probably aren’t qualified for. If you’re using Cilium, you’re using eBPF, which is brilliant until you need to debug why a packet is being dropped and your standard tcpdump tools show you nothing.
Let’s talk about ndots. This is the single most common “hidden” performance killer. By default, Kubernetes sets ndots: 5 in /etc/resolv.conf. This means if you try to resolve api.stripe.com, the resolver will try:
api.stripe.com.namespace.svc.cluster.localapi.stripe.com.svc.cluster.localapi.stripe.com.cluster.localapi.stripe.com.us-east-1.compute.internal- And finally:
api.stripe.com
That is four failed DNS lookups before it even tries the real domain. If your CoreDNS is under load, or if you’re hitting AWS’s 1024 packets-per-second limit on the VPC DNS, your application will start throwing UnknownHostException or ErrImagePull. The fix? Use a fully qualified domain name (FQDN) with a trailing dot: api.stripe.com. or manually override the dnsConfig in your pod spec.
spec:
dnsConfig:
options:
- name: ndots
value: "1"
Storage: The “Stateful” Fallacy
I have a very simple rule: Do not run your primary database on Kubernetes. Yes, I know about the Postgres Operator. Yes, I know about Rook/Ceph. I don’t care. The complexity of managing a distributed storage layer on top of a distributed container orchestrator on top of a distributed cloud provider’s block storage is a recipe for data loss.
When a node dies in Kubernetes, the PersistentVolume (PV) is still attached to that dead node. The AttachDetachController has to realize the node is gone, wait for the timeout, call the AWS/GCP API to detach the volume, and then attach it to the new node. This can take anywhere from 2 to 15 minutes. If your database is down for 15 minutes because a node failed, your “High Availability” is a joke. Use RDS. Use Cloud SQL. Use a managed service until you are at a scale where the cost of the managed service is higher than the salary of two full-time DBAs. Most of you are not at that scale.
If you must run stateful workloads, use LocalPersistentVolumes with NVMe drives. You lose the ability to move pods between nodes easily, but you gain predictable I/O and you don’t have to deal with the “Stuck EBS Volume” dance. But then you have to manage replication at the application level. Are you ready to manage Patroni or Galera? Probably not.
The OOMKiller and the Memory Trap
Memory is not like CPU. You can’t “throttle” memory. When you run out, someone has to die. In Kubernetes, this is the OOMKiller. But it’s not just about your pod. There is a hierarchy of who gets killed first based on the oom_score_adj.
- Guaranteed: (Requests == Limits). These are the last to be killed.
- Burstable: (Requests < Limits). These are killed if the node is under pressure.
- BestEffort: (No requests or limits). These are the first to be sacrificed to the chaos gods.
I once saw a cluster where the kubelet itself was OOM-killed because a developer deployed a “log-scraper” with no limits that leaked memory. Because the kubelet was in the same cgroup as other system processes but didn’t have its memory properly reserved via --kube-reserved, the kernel killed the most important process on the node. The node went NotReady, the pods were rescheduled, and the cycle repeated on the next node. A literal “virus” of a pod that killed every node it touched.
Note to self: Always set
--system-reservedand--kube-reservedon your worker nodes. If you don’t, the Linux kernel will prioritize a random Python script over the Kubelet when things get tight.
YAML-Hell and the Configuration Gap
We’ve traded 50 lines of bash for 50,000 lines of YAML. Helm is a band-aid on a bullet wound. It’s a text-templating engine that doesn’t understand Kubernetes semantics. You end up with values.yaml files that are 2,000 lines long, and a single indentation error causes a production outage.
The real problem is that YAML is static, but infrastructure is dynamic. We use tools like Kustomize or CDK8s to try and manage the mess, but we’re just adding layers of abstraction. The “Pragmatic” approach? Keep it as flat as possible. Avoid deeply nested Helm charts. If you can’t explain what a manifest does by looking at it for 30 seconds, it’s too complex.
Consider the terminationGracePeriodSeconds. The default is 30. If your app takes 35 seconds to flush its buffers to the database, you are losing data every time you deploy. Kubernetes sends a SIGTERM, waits 30 seconds, and then sends a SIGKILL. Most people don’t even have a signal handler in their code, so the app just dies immediately anyway. You need to handle SIGTERM gracefully.
// Example of what your code SHOULD do
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGTERM)
<-sigChan
log.Println("Received SIGTERM, shutting down gracefully...")
server.Shutdown(ctx) // This gives the app time to finish active requests
The "Real World" Gotcha: Image Pull Secrets and Rate Limits
Here is a fun one that only happens when you're already having a bad day. You have a massive outage. You need to scale your pods from 10 to 100 to handle the recovery traffic. But you're using Docker Hub, and you haven't configured imagePullSecrets with a paid account. Suddenly, half your pods are stuck in ImagePullBackOff because you've been rate-limited. Or worse, you're using imagePullPolicy: Always, and your private registry is down. Even though the image is already on the node, the kubelet refuses to start the pod because it can't check if there's a newer version.
The Expert Move: Use imagePullPolicy: IfNotPresent. Use a local registry (like ECR, GCR, or an internal Artifactory). Never depend on the public internet for your production availability. If Docker Hub goes down, your cluster should still be able to scale.
RBAC: The Illusion of Security
Most companies have "Cluster Admin" for everyone because RBAC is hard. This is how you get a junior dev accidentally deleting the kube-system namespace because they thought they were in minikube.
Kubernetes RBAC is verbose and easy to get wrong. You have Roles, ClusterRoles, RoleBindings, and ClusterRoleBindings. If you give a service account the ability to create pods, you have effectively given it root on the node, because it can just create a privileged pod that mounts the host's / filesystem.
# This is a security nightmare. Don't do this.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: "allow-everything"
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
Instead, use tools like Gatekeeper or Kyverno to enforce policies. Block privileged containers. Block hostPath mounts. Block containers running as root. If you don't enforce these at the admission controller level, your RBAC is just theater.
The Cost of "Cloud Native"
We need to talk about the cloud bill. Managed Kubernetes (EKS, GKE, AKS) isn't expensive because of the control plane fee ($70-$100/month). It's expensive because of the "hidden" costs:
- Inter-AZ Data Transfer: Kubernetes doesn't care about Availability Zones by default. A pod in
us-east-1awill happily talk to a service inus-east-1b, and AWS will charge you $0.01 per GB. At scale, this can be thousands of dollars. UsetopologyKeysorService Topologyto keep traffic local. - Load Balancers: Every
type: LoadBalancerservice creates a new cloud LB. At $20/month per LB, plus data processing, this adds up. Use an Ingress Controller and a single LB. - NAT Gateway: If your nodes are in private subnets, all their traffic (including pulling images!) goes through a NAT Gateway. This is often the most expensive item on an AWS bill. Use VPC Endpoints for ECR, S3, and STS.
- Empty Nodes: If your HPA (Horizontal Pod Autoscaler) scales down pods but your Cluster Autoscaler doesn't scale down nodes, you're paying for idle compute.
The Reality Check
Kubernetes is a tool for managing complexity with more complexity. It solves the "it works on my machine" problem by making it "it doesn't work in production for reasons I don't understand." If you are a Senior SRE, your job isn't to "install Kubernetes." Your job is to protect the business from the inherent instability of a distributed system. This means setting up PodDisruptionBudgets so a node upgrade doesn't take down your entire API. It means setting priorityClass so your critical payment service kicks off the "cat-picture-generator" pod when resources are low. It means knowing that iptables -L is still your best friend when the network goes sideways.
I've spent a decade in this industry, and I've seen the hype cycle move from VMs to Mesos to Swarm to Kubernetes. The tech changes, but the failure modes remain the same: exhausted resources, unhandled signals, and a lack of understanding of the underlying primitives. Kubernetes is just a very fancy way to run exec() on a remote machine. Don't let the YAML fool you into thinking it's anything more than that.
Stop trying to build the "perfect" platform. Build a platform that is observable enough that you can fix it when it inevitably breaks at 3 AM. Because it will break. And when it does, no amount of "transformative" cloud-native AI-driven auto-scaling will save you. Only a deep understanding of the Linux kernel and a very fast kubectl finger will.
If you're still using latest tags in production, you deserve the outage you're about to have.
Related Articles
Explore more insights and best practices: