{"id":4867,"date":"2026-08-25T21:21:30","date_gmt":"2026-08-25T15:51:30","guid":{"rendered":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/"},"modified":"2026-08-25T21:21:30","modified_gmt":"2026-08-25T15:51:30","slug":"10-kubernetes-best-practices-for-production-success-4","status":"publish","type":"post","link":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/","title":{"rendered":"10 Kubernetes Best Practices for Production Success"},"content":{"rendered":"<p>The Graveyard of Our Infrastructure: A Handover Memo for the Unfortunate<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\">$ kubectl get nodes\nNAME             STATUS     ROLES    AGE   VERSION\nip-10-0-42-12    NotReady   worker   89d   v1.29.2\nip-10-0-42-13    NotReady   worker   89d   v1.29.2\nip-10-0-42-14    Ready      worker   89d   v1.29.2\nip-10-0-42-15    NotReady   worker   89d   v1.29.2\n\n$ kubectl describe pod checkout-api-7f8d9b6c5-x4z2l\nName:           checkout-api-7f8d9b6c5-x4z2l\nNamespace:      prod-main\nStatus:         Running\nIP:             10.2.14.155\nContainers:\n  checkout-svc:\n    State:          Waiting\n      Reason:       CrashLoopBackOff\n    Last State:     Terminated\n      Reason:       OOMKilled\n      Exit Code:    137\n      Started:      Mon, 12 Aug 2024 03:14:22 -0400\n      Finished:     Mon, 12 Aug 2024 03:15:01 -0400\n<\/code><\/pre>\n<p>You\u2019re reading this because I\u2019m gone. By the time you\u2019ve decoded my Git history, I\u2019ll be somewhere without cell service, trying to forget the sound of a PagerDuty alert screaming at 3:14 AM. You\u2019ve inherited the &#8220;Phoenix&#8221; cluster. It was named ironically. It doesn&#8217;t rise from the ashes; it just stays on fire.<\/p>\n<p>The terminal output above is your new reality. Three out of four nodes are <code>NotReady<\/code> because the Kubelet has effectively given up on life. The one &#8220;Ready&#8221; node is currently being hammered by every single pod in the <code>prod-main<\/code> namespace, which is why the <code>checkout-api<\/code> is getting OOMKilled every forty seconds. <\/p>\n<p>We ignored every <strong>kubernetes best<\/strong> practice because &#8220;the business needed to ship.&#8221; Now the business is losing $40k an hour, and you\u2019re the one holding the shovel.<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_80 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a8dc4eae969d\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a8dc4eae969d\"  aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#The_Resource_Limit_Lie_and_the_OOM_Killers_Wrath\" >The Resource Limit Lie and the OOM Killer\u2019s Wrath<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#The_Wrong_Way_What_we_did\" >The Wrong Way (What we did):<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#The_Fixed_Way\" >The Fixed Way:<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#The_Readiness_Probe_Suicide_Pact\" >The Readiness Probe Suicide Pact<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#The_Wrong_Way\" >The Wrong Way:<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#The_Fixed_Way-2\" >The Fixed Way:<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#CoreDNS_and_the_Five-Second_Search_Penalty\" >CoreDNS and the Five-Second Search Penalty<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#RBAC_The_Key_to_the_Kingdom_Under_the_Mat\" >RBAC: The Key to the Kingdom Under the Mat<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#The_Wrong_Way-2\" >The Wrong Way:<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#The_Fixed_Way-3\" >The Fixed Way:<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#Sidecars_and_the_129_Lifecycle_Revolution\" >Sidecars and the 1.29 Lifecycle Revolution<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#The_Fixed_Way_129_Native_Sidecars\" >The Fixed Way (1.29 Native Sidecars):<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-13\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#The_Ephemeral_Storage_Death_Spiral\" >The Ephemeral Storage Death Spiral<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-14\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#The_Silence_of_the_Alerts\" >The Silence of the Alerts<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-15\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#The_Final_Descent\" >The Final Descent<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-16\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#Related_Articles\" >Related Articles<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"The_Resource_Limit_Lie_and_the_OOM_Killers_Wrath\"><\/span>The Resource Limit Lie and the OOM Killer\u2019s Wrath<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>We treated resource requests and limits like they were optional suggestions. They aren&#8217;t. In Kubernetes 1.29, the interaction between cgroups v2 and the Linux Out-Of-Memory (OOM) Killer is precise and unforgiving. We didn&#8217;t set limits on the <code>checkout-api<\/code>. We let it &#8220;burst.&#8221; <\/p>\n<p>When a container doesn&#8217;t have a memory limit, it thinks it owns the entire node. The Java heap, being the greedy pig it is, expanded until it hit the node&#8217;s physical capacity. At that point, the Linux kernel stepped in. The OOM Killer doesn&#8217;t care about your microservices architecture. It looks at the <code>oom_score<\/code>. Since we didn&#8217;t follow the <strong>kubernetes best<\/strong> practice of defining <code>Guaranteed<\/code> Quality of Service (QoS) classes, our critical pods had the same priority as a cronjob that calculates employee of the month.<\/p>\n<p>The kernel calculates <code>oom_score_adj<\/code> based on the ratio of memory requested to memory used. Because we set <code>requests<\/code> to 256Mi but let the app use 8Gi, the score was astronomical. The kernel killed the process, the container exited with code 137, and the Kubelet\u2014already struggling with disk pressure\u2014just stopped responding.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"The_Wrong_Way_What_we_did\"><\/span>The Wrong Way (What we did):<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<pre class=\"codehilite\"><code class=\"language-yaml\"># This is a suicide note in YAML format\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\n  name: checkout-api\nspec:\n  template:\n    spec:\n      containers:\n      - name: checkout-svc\n        image: checkout:latest\n        resources:\n          requests:\n            memory: &quot;256Mi&quot;\n            cpu: &quot;100m&quot;\n          # No limits. &quot;Let it scale,&quot; they said.\n<\/code><\/pre>\n<h3><span class=\"ez-toc-section\" id=\"The_Fixed_Way\"><\/span>The Fixed Way:<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<pre class=\"codehilite\"><code class=\"language-yaml\"># This is how you stop the bleeding\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\n  name: checkout-api\nspec:\n  template:\n    spec:\n      containers:\n      - name: checkout-svc\n        image: checkout:v1.29.4 # Use specific tags, for the love of god\n        resources:\n          requests:\n            memory: &quot;2Gi&quot;\n            cpu: &quot;1000m&quot;\n          limits:\n            memory: &quot;2Gi&quot; # Limits = Requests for Guaranteed QoS\n            cpu: &quot;1000m&quot;\n<\/code><\/pre>\n<p>By setting limits equal to requests, you ensure the pod is in the <code>Guaranteed<\/code> class. The OOM Killer will target almost everything else before it touches this pod. Also, notice the CPU. We didn&#8217;t use limits there either, which led to &#8220;CPU Throttling.&#8221; When the CFS (Completely Fair Scheduler) quota kicks in because you didn&#8217;t define a limit, your 10ms latency turns into 500ms.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The_Readiness_Probe_Suicide_Pact\"><\/span>The Readiness Probe Suicide Pact<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Last Tuesday, we had a &#8220;partial&#8221; outage that turned into a total blackout. Why? Because we misconfigured readiness probes. A readiness probe tells Kubernetes when a pod is ready to accept traffic. If it fails, the pod is removed from the Service\u2019s endpoints.<\/p>\n<p>Our genius move was making the <code>\/health<\/code> endpoint of the <code>checkout-api<\/code> dependent on the database connection. When the database got slightly slow, the readiness probe failed. Kubernetes, doing exactly what we told it to do, pulled the pod out of rotation. This increased the load on the remaining pods, which then slowed down, failed <em>their<\/em> probes, and were also pulled. Within three minutes, we had 50 healthy pods running, but 0 endpoints in the LoadBalancer. The cluster was a ghost town.<\/p>\n<p>In a RollingUpdate, this is fatal. Kubernetes starts a new pod, waits for it to be &#8220;Ready,&#8221; then kills an old one. If the new pod never becomes &#8220;Ready&#8221; because the database is under load, the update hangs, or worse, you end up with no pods at all if <code>maxUnavailable<\/code> is set poorly.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"The_Wrong_Way\"><\/span>The Wrong Way:<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<pre class=\"codehilite\"><code class=\"language-yaml\"># A recipe for cascading failure\nreadinessProbe:\n  httpGet:\n    path: \/health # This endpoint checks DB, Redis, and an external API\n    port: 8080\n  initialDelaySeconds: 0\n  periodSeconds: 1\n<\/code><\/pre>\n<h3><span class=\"ez-toc-section\" id=\"The_Fixed_Way-2\"><\/span>The Fixed Way:<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<pre class=\"codehilite\"><code class=\"language-yaml\"># Decouple your health from your dependencies\nreadinessProbe:\n  httpGet:\n    path: \/ready # Only checks if the app server is up\n    port: 8080\n  initialDelaySeconds: 5\n  periodSeconds: 10\n  failureThreshold: 3\nstartupProbe: # Use this for slow-starting legacy junk\n  httpGet:\n    path: \/health\n    port: 8080\n  failureThreshold: 30\n  periodSeconds: 10\n<\/code><\/pre>\n<p>Use <code>startupProbes<\/code> for your slow-ass Java apps. It prevents the Kubelet from killing the container before it has even finished loading the Spring Boot context. This is a basic <strong>kubernetes best<\/strong> practice we ignored because we thought <code>initialDelaySeconds<\/code> was enough. It wasn&#8217;t.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"CoreDNS_and_the_Five-Second_Search_Penalty\"><\/span>CoreDNS and the Five-Second Search Penalty<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>If you look at the logs for the <code>payment-gateway<\/code> service, you\u2019ll see thousands of <code>EAI_AGAIN<\/code> errors. You might think the network was down. It wasn&#8217;t. We were just DDoSing our own CoreDNS.<\/p>\n<p>By default, Linux&#8217;s <code>resolver<\/code> uses a setting called <code>ndots:5<\/code>. This means if your app tries to resolve <code>google.com<\/code>, it first tries:<br \/>\n1. <code>google.com.prod-main.svc.cluster.local<\/code><br \/>\n2. <code>google.com.svc.cluster.local<\/code><br \/>\n3. <code>google.com.cluster.local<\/code><br \/>\n4. <code>google.com.us-east-1.compute.internal<\/code><br \/>\n5. And finally, <code>google.com<\/code>.<\/p>\n<p>Each one of those is a DNS query. Our apps make hundreds of external API calls. CoreDNS was processing 50,000 queries per second, most of them for junk domains that didn&#8217;t exist. The latency spiked, the UDP packets dropped, and the apps timed out. <\/p>\n<p>We should have used <code>NodeLocal DNSCache<\/code>. It runs a thin agent on every node that intercepts these queries so they don&#8217;t have to hop across the network to the CoreDNS pods. But we didn&#8217;t. We just scaled the CoreDNS deployment to 20 replicas and hoped for the best. It wasn&#8217;t enough.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"RBAC_The_Key_to_the_Kingdom_Under_the_Mat\"><\/span>RBAC: The Key to the Kingdom Under the Mat<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Security was an afterthought. We gave the <code>default<\/code> service account in the <code>prod-main<\/code> namespace <code>cluster-admin<\/code> privileges because one developer couldn&#8217;t figure out why their Prometheus sidecar wasn&#8217;t scraping metrics. <\/p>\n<p>In Kubernetes 1.29, the attack surface is huge. By leaving that RoleBinding in place, any pod that gets compromised\u2014like that unpatched WordPress site the marketing team insisted on running in the same cluster\u2014has full control over the API server. They can delete namespaces, steal secrets, or spin up Monero miners on our GPU nodes.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"The_Wrong_Way-2\"><\/span>The Wrong Way:<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<pre class=\"codehilite\"><code class=\"language-yaml\"># How to get fired by an auditor\napiVersion: rbac.authorization.k8s.io\/v1\nkind: ClusterRoleBinding\nmetadata:\n  name: allow-everything-to-everyone\nsubjects:\n- kind: ServiceAccount\n  name: default\n  namespace: prod-main\nroleRef:\n  kind: ClusterRole\n  name: cluster-admin\n  apiGroup: rbac.authorization.k8s.io\n<\/code><\/pre>\n<h3><span class=\"ez-toc-section\" id=\"The_Fixed_Way-3\"><\/span>The Fixed Way:<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<pre class=\"codehilite\"><code class=\"language-yaml\"># Least privilege is not a suggestion\napiVersion: rbac.authorization.k8s.io\/v1\nkind: Role\nmetadata:\n  namespace: prod-main\n  name: pod-reader\nrules:\n- apiGroups: [&quot;&quot;]\n  resources: [&quot;pods&quot;]\n  verbs: [&quot;get&quot;, &quot;watch&quot;, &quot;list&quot;]\n---\napiVersion: rbac.authorization.k8s.io\/v1\nkind: RoleBinding\nmetadata:\n  name: read-pods\n  namespace: prod-main\nsubjects:\n- kind: ServiceAccount\n  name: my-app-sa\nroleRef:\n  kind: Role\n  name: pod-reader\n  apiGroup: rbac.authorization.k8s.io\n<\/code><\/pre>\n<p>Stop using the <code>default<\/code> service account. Create a specific <code>ServiceAccount<\/code> for every deployment. It takes ten extra lines of YAML, but it prevents a single compromised pod from becoming a cluster-wide catastrophe.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Sidecars_and_the_129_Lifecycle_Revolution\"><\/span>Sidecars and the 1.29 Lifecycle Revolution<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>We had a massive problem with our logging sidecars. The main app would finish its job and exit, but the logging sidecar would keep running, keeping the pod in a <code>Running<\/code> state forever and preventing Job completion. Or, the sidecar would take too long to start, and the main app would crash because it couldn&#8217;t find the local log socket.<\/p>\n<p>Kubernetes 1.29 finally moved <code>SidecarContainers<\/code> to GA. You can now define a container in the <code>initContainers<\/code> list with a <code>restartPolicy: Always<\/code>. This ensures the sidecar starts <em>before<\/em> your main app and shuts down <em>after<\/em> it. We didn&#8217;t implement this. We were still using the old, broken pattern of just stuffing two containers in a pod and praying to the scheduler.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"The_Fixed_Way_129_Native_Sidecars\"><\/span>The Fixed Way (1.29 Native Sidecars):<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<pre class=\"codehilite\"><code class=\"language-yaml\">apiVersion: v1\nkind: Pod\nmetadata:\n  name: telemetry-pod\nspec:\n  initContainers:\n  - name: network-proxy\n    image: proxy:v1\n    restartPolicy: Always # This makes it a formal sidecar\n  containers:\n  - name: main-app\n    image: app:v1\n<\/code><\/pre>\n<p>This ensures the <code>network-proxy<\/code> is up before <code>main-app<\/code> starts. If the proxy dies, it\u2019s restarted. If the main app exits, the proxy is sent a SIGTERM. This is the <strong>kubernetes best<\/strong> practice for any auxiliary process. Use it, or enjoy your zombie pods.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The_Ephemeral_Storage_Death_Spiral\"><\/span>The Ephemeral Storage Death Spiral<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>We didn&#8217;t set <code>ephemeral-storage<\/code> limits. One of the microservices had a bug where it would dump its entire debug log to a local file instead of stdout. It filled up the node&#8217;s root partition in twenty minutes.<\/p>\n<p>When a node runs out of disk space, the Kubelet starts evicting pods. But it doesn&#8217;t do it gracefully. It panics. It starts killing pods to save itself. Because we didn&#8217;t have <code>Taints<\/code> or <code>Tolerations<\/code> set up to protect the system critical services, the Kubelet ended up evicting the <code>aws-node<\/code> CNI plugin and the <code>kube-proxy<\/code>. <\/p>\n<p>Suddenly, the node wasn&#8217;t just out of disk; it was off the network. The API server marked it as <code>NotReady<\/code>. The scheduler saw 200 pods that needed a home and tried to cram them onto the remaining three nodes. Those nodes then ran out of memory, and the &#8220;Great Collapse of 2024&#8221; began.<\/p>\n<p>You need to set <code>requests<\/code> and <code>limits<\/code> for <code>ephemeral-storage<\/code> just like you do for memory. And for the love of all that is holy, use a <code>logrotate<\/code> sidecar or stream everything to a centralized collector. Do not trust the developers to manage their own file handles.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The_Silence_of_the_Alerts\"><\/span>The Silence of the Alerts<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>If you look at Alertmanager, you\u2019ll find a silence titled &#8220;Temporary &#8211; Fix later.&#8221; It was created eight months ago. It silences all <code>KubeNodeNotReady<\/code> alerts for the <code>prod-main<\/code> cluster. <\/p>\n<p>The previous lead (my boss, before he &#8220;retired&#8221; to a goat farm) got tired of the alerts firing every time we did a node rotation. He silenced it for &#8220;two hours&#8221; and forgot. We\u2019ve been flying blind for the better part of a year. <\/p>\n<p>The Prometheus instance is also currently OOMing because we\u2019re trying to ingest 2 million cardinality metrics from a defunct A\/B testing framework that someone forgot to turn off. The <code>prometheus-k8s<\/code> pods are stuck in a loop because the Write-Ahead Log (WAL) is corrupted from the last three hard reboots of the node they were on.<\/p>\n<p>To fix this, you\u2019ll need to:<br \/>\n1. Delete the corrupted WAL volume (yes, you\u2019ll lose data, no one cares).<br \/>\n2. Increase the memory limit to at least 16Gi.<br \/>\n3. Remove the silence in Alertmanager.<br \/>\n4. Prepare for the 500 emails you\u2019re about to get.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The_Final_Descent\"><\/span>The Final Descent<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The cluster isn&#8217;t broken because Kubernetes is bad. It\u2019s broken because we treated it like a giant VPS instead of a distributed orchestrator. We ignored the <strong>kubernetes best<\/strong> practices because they felt like &#8220;overhead.&#8221; <\/p>\n<p>We didn&#8217;t use <code>PodDisruptionBudgets<\/code>, so when the cloud provider performed maintenance on the underlying instances, Kubernetes killed all replicas of the <code>auth-service<\/code> at the same time.<br \/>\nWe didn&#8217;t use <code>TopologySpreadConstraints<\/code>, so all our API pods ended up on the same physical rack in the same Availability Zone. When that AZ had a power flicker, the entire stack went dark.<br \/>\nWe didn&#8217;t use <code>NetworkPolicies<\/code>, so when a dev&#8217;s laptop was compromised via a phishing link and they had <code>kubectl<\/code> access, the attacker could scan the entire internal network from a pod in the <code>dev<\/code> namespace.<\/p>\n<p>The documentation is all there. The 1.29 release notes are clear. The tools exist. But you won&#8217;t have time to read them because the <code>checkout-api<\/code> just crashed again. <\/p>\n<p>I left a bottle of high-proof bourbon in the bottom drawer of the desk. You\u2019re going to need it when you realize that the <code>etcd<\/code> backup script has been failing since February because the S3 bucket it was writing to was deleted to &#8220;save costs.&#8221;<\/p>\n<p>Good luck. You\u2019re going to need more than luck, actually. You\u2019re going to need a miracle and a lot of YAML. <\/p>\n<p>Signed,<br \/>\nThe SRE who saw too much.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Related_Articles\"><\/span>Related Articles<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Explore more insights and best practices:<\/p>\n<ul>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/10-essential-python-best-practices-for-clean-code\/\">10 Essential Python Best Practices For Clean Code<\/a><\/li>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/react-native-guide-build-high-performance-mobile-apps\/\">React Native Guide Build High Performance Mobile Apps<\/a><\/li>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/what-is-docker-a-beginners-guide-to-containerization\/\">What Is Docker A Beginners Guide To Containerization<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>The Graveyard of Our Infrastructure: A Handover Memo for the Unfortunate $ kubectl get nodes NAME STATUS ROLES AGE VERSION ip-10-0-42-12 NotReady worker 89d v1.29.2 ip-10-0-42-13 NotReady worker 89d v1.29.2 ip-10-0-42-14 Ready worker 89d v1.29.2 ip-10-0-42-15 NotReady worker 89d v1.29.2 $ kubectl describe pod checkout-api-7f8d9b6c5-x4z2l Name: checkout-api-7f8d9b6c5-x4z2l Namespace: prod-main Status: Running IP: 10.2.14.155 Containers: checkout-svc: &#8230; <a title=\"10 Kubernetes Best Practices for Production Success\" class=\"read-more\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/\" aria-label=\"Read more  on 10 Kubernetes Best Practices for Production Success\">Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-4867","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.0 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>10 Kubernetes Best Practices for Production Success - ITSupportWale<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"10 Kubernetes Best Practices for Production Success - ITSupportWale\" \/>\n<meta property=\"og:description\" content=\"The Graveyard of Our Infrastructure: A Handover Memo for the Unfortunate $ kubectl get nodes NAME STATUS ROLES AGE VERSION ip-10-0-42-12 NotReady worker 89d v1.29.2 ip-10-0-42-13 NotReady worker 89d v1.29.2 ip-10-0-42-14 Ready worker 89d v1.29.2 ip-10-0-42-15 NotReady worker 89d v1.29.2 $ kubectl describe pod checkout-api-7f8d9b6c5-x4z2l Name: checkout-api-7f8d9b6c5-x4z2l Namespace: prod-main Status: Running IP: 10.2.14.155 Containers: checkout-svc: ... Read more\" \/>\n<meta property=\"og:url\" content=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/\" \/>\n<meta property=\"og:site_name\" content=\"ITSupportWale\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Itsupportwale-298547177495978\" \/>\n<meta property=\"article:published_time\" content=\"2026-08-25T15:51:30+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2021\/05\/android-chrome-512x512-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"512\" \/>\n\t<meta property=\"og:image:height\" content=\"512\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Techie\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Techie\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/\"},\"author\":{\"name\":\"Techie\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d\"},\"headline\":\"10 Kubernetes Best Practices for Production Success\",\"datePublished\":\"2026-08-25T15:51:30+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/\"},\"wordCount\":1717,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#organization\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/\",\"url\":\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/\",\"name\":\"10 Kubernetes Best Practices for Production Success - ITSupportWale\",\"isPartOf\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#website\"},\"datePublished\":\"2026-08-25T15:51:30+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/itsupportwale.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"10 Kubernetes Best Practices for Production Success\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#website\",\"url\":\"https:\/\/itsupportwale.com\/blog\/\",\"name\":\"ITSupportWale\",\"description\":\"Tips, Tricks, Fixed-Errors, Tutorials &amp; Guides\",\"publisher\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/itsupportwale.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#organization\",\"name\":\"itsupportwale\",\"url\":\"https:\/\/itsupportwale.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png\",\"contentUrl\":\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png\",\"width\":1119,\"height\":144,\"caption\":\"itsupportwale\"},\"image\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/Itsupportwale-298547177495978\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d\",\"name\":\"Techie\",\"sameAs\":[\"https:\/\/itsupportwale.com\",\"iswblogadmin\"],\"url\":\"https:\/\/itsupportwale.com\/blog\/author\/iswblogadmin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"10 Kubernetes Best Practices for Production Success - ITSupportWale","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/","og_locale":"en_US","og_type":"article","og_title":"10 Kubernetes Best Practices for Production Success - ITSupportWale","og_description":"The Graveyard of Our Infrastructure: A Handover Memo for the Unfortunate $ kubectl get nodes NAME STATUS ROLES AGE VERSION ip-10-0-42-12 NotReady worker 89d v1.29.2 ip-10-0-42-13 NotReady worker 89d v1.29.2 ip-10-0-42-14 Ready worker 89d v1.29.2 ip-10-0-42-15 NotReady worker 89d v1.29.2 $ kubectl describe pod checkout-api-7f8d9b6c5-x4z2l Name: checkout-api-7f8d9b6c5-x4z2l Namespace: prod-main Status: Running IP: 10.2.14.155 Containers: checkout-svc: ... Read more","og_url":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/","og_site_name":"ITSupportWale","article_publisher":"https:\/\/www.facebook.com\/Itsupportwale-298547177495978","article_published_time":"2026-08-25T15:51:30+00:00","og_image":[{"width":512,"height":512,"url":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2021\/05\/android-chrome-512x512-1.png","type":"image\/png"}],"author":"Techie","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Techie","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#article","isPartOf":{"@id":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/"},"author":{"name":"Techie","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d"},"headline":"10 Kubernetes Best Practices for Production Success","datePublished":"2026-08-25T15:51:30+00:00","mainEntityOfPage":{"@id":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/"},"wordCount":1717,"commentCount":0,"publisher":{"@id":"https:\/\/itsupportwale.com\/blog\/#organization"},"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/","url":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/","name":"10 Kubernetes Best Practices for Production Success - ITSupportWale","isPartOf":{"@id":"https:\/\/itsupportwale.com\/blog\/#website"},"datePublished":"2026-08-25T15:51:30+00:00","breadcrumb":{"@id":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-production-success-4\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/itsupportwale.com\/blog\/"},{"@type":"ListItem","position":2,"name":"10 Kubernetes Best Practices for Production Success"}]},{"@type":"WebSite","@id":"https:\/\/itsupportwale.com\/blog\/#website","url":"https:\/\/itsupportwale.com\/blog\/","name":"ITSupportWale","description":"Tips, Tricks, Fixed-Errors, Tutorials &amp; Guides","publisher":{"@id":"https:\/\/itsupportwale.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/itsupportwale.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/itsupportwale.com\/blog\/#organization","name":"itsupportwale","url":"https:\/\/itsupportwale.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png","contentUrl":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png","width":1119,"height":144,"caption":"itsupportwale"},"image":{"@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Itsupportwale-298547177495978"]},{"@type":"Person","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d","name":"Techie","sameAs":["https:\/\/itsupportwale.com","iswblogadmin"],"url":"https:\/\/itsupportwale.com\/blog\/author\/iswblogadmin\/"}]}},"_links":{"self":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts\/4867","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/comments?post=4867"}],"version-history":[{"count":0,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts\/4867\/revisions"}],"wp:attachment":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/media?parent=4867"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/categories?post=4867"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/tags?post=4867"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}