{"id":4885,"date":"2026-09-18T00:26:41","date_gmt":"2026-09-17T18:56:41","guid":{"rendered":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/"},"modified":"2026-09-18T00:26:41","modified_gmt":"2026-09-17T18:56:41","slug":"10-kubernetes-best-practices-for-scalable-production","status":"publish","type":"post","link":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/","title":{"rendered":"10 Kubernetes Best Practices for Scalable Production"},"content":{"rendered":"<p><strong>INCIDENT REPORT: POST-MORTEM #8842-B<\/strong><br \/>\n<strong>TIMESTAMP:<\/strong> 2024-05-14T03:14:15Z<br \/>\n<strong>STATUS:<\/strong> CRITICAL \/ DEGRADED<br \/>\n<strong>INCIDENT LEAD:<\/strong> Senior SRE (Grizzled)<br \/>\n<strong>SUBJECT:<\/strong> The &#8220;Great Cloud Meltdown&#8221; of the Production Cluster<\/p>\n<hr \/>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_80 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6aaff4583abdd\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6aaff4583abdd\"  aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#1_THE_INCIDENT_SUMMARY\" >1. THE INCIDENT SUMMARY<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#2_THE_ROOT_CAUSE_ANALYSIS_RCA\" >2. THE ROOT CAUSE ANALYSIS (RCA)<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#I_The_OOMKillers_Scythe_and_the_Myth_of_Unlimited_Memory\" >I. The OOMKiller\u2019s Scythe and the Myth of Unlimited Memory<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#II_Liveness_Probes_The_Thundering_Herd_and_the_Death_Spiral\" >II. Liveness Probes: The Thundering Herd and the Death Spiral<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#III_Istio_120_and_the_Sidecar_Memory_Tax\" >III. Istio 1.20 and the Sidecar Memory Tax<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#IV_The_Schedulers_Nightmare_Lack_of_Pod_Anti-Affinity\" >IV. The Scheduler\u2019s Nightmare: Lack of Pod Anti-Affinity<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#V_Network_Policies_The_Wild_West_of_Internal_Traffic\" >V. Network Policies: The Wild West of Internal Traffic<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#VI_Helm_Template_Hell_and_Hardcoded_Secrets\" >VI. Helm Template Hell and Hardcoded Secrets<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#3_FIXING_THE_MESS\" >3. FIXING THE MESS<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#4_LETTER_TO_THE_JUNIOR_DEVS\" >4. LETTER TO THE JUNIOR DEVS<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#Related_Articles\" >Related Articles<\/a><\/li><\/ul><\/nav><\/div>\n<h3><span class=\"ez-toc-section\" id=\"1_THE_INCIDENT_SUMMARY\"><\/span>1. THE INCIDENT SUMMARY<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>At 03:14:15Z, my pager didn&#8217;t just beep; it screamed. I was three hours into a four-hour sleep cycle, dreaming about a world where people actually read documentation. Instead, I woke up to a Slack channel that looked like a digital war zone. The monitoring dashboard, usually a boring sea of green, was a pulsating, angry neon red. Our primary production cluster\u2014running Kubernetes v1.29.2\u2014wasn&#8217;t just failing; it was undergoing a systematic, cascading collapse.<\/p>\n<p>The first log I pulled from the jump box told the whole story. It wasn&#8217;t a subtle bug. It was a suicide pact between the microservices.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\">$ kubectl get pods -n production -o wide\nNAME                                READY   STATUS             RESTARTS         AGE     IP            NODE\napi-gateway-7f8d9b6c5-2x9wl         0\/1     CrashLoopBackOff   42 (3m ago)      14h     10.2.4.15     node-01\npayment-processor-5d4f3e2-m8zqp     0\/1     OOMKilled          12 (1m ago)      2h      10.2.4.88     node-01\ninventory-service-9a1b2c3-k4j5h     1\/1     Running            0                14h     10.2.4.92     node-01\nauth-service-1a2b3c4-p9o8i          0\/1     Error              5                10m     10.2.4.101    node-01\n<\/code><\/pre>\n<p>I checked the Kubelet logs on <code>node-01<\/code>. The kernel was screaming for mercy.<\/p>\n<pre class=\"codehilite\"><code class=\"language-text\">May 14 03:16:22 node-01 kernel: [12445.678] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=\/,mems_allowed=0,oom_memcg=\/kubepods.slice\/kubepods-burstable.slice\/kubepods-burstable-pod5d4f3e2.slice,task_memcg=\/kubepods.slice\/kubepods-burstable.slice\/kubepods-burstable-pod5d4f3e2.slice,task=java,pid=12345,uid=1000\nMay 14 03:16:22 node-01 kernel: [12445.679] Memory cgroup out of memory: Killed process 12345 (java) total-vm:8542304kB, anon-rss:4194304kB, file-rss:0kB, shmem-rss:0kB, spid:12345, oom_score_adj:985\nMay 14 03:16:23 node-01 kubelet[1022]: E0514 03:16:23.123456 1022 pod_workers.go:1294] &quot;Error syncing pod, skipping&quot; err=&quot;failed to \\&quot;StartContainer\\&quot; for \\&quot;payment-processor\\&quot; with CrashLoopBackOff: back-off 5m0s restarting failed container=payment-processor pod=payment-processor-5d4f3e2-m8zqp_production&quot; pod=&quot;production\/payment-processor-5d4f3e2-m8zqp&quot;\n<\/code><\/pre>\n<p>The &#8220;kubernetes best&#8221; practices we\u2019ve been preaching for three years were ignored like a &#8220;No Smoking&#8221; sign in a 1970s bowling alley. We didn&#8217;t have a cluster; we had a high-speed collision of YAML files written by people who think &#8220;memory limits&#8221; are just suggestions.<\/p>\n<hr \/>\n<h3><span class=\"ez-toc-section\" id=\"2_THE_ROOT_CAUSE_ANALYSIS_RCA\"><\/span>2. THE ROOT CAUSE ANALYSIS (RCA)<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<h2><span class=\"ez-toc-section\" id=\"I_The_OOMKillers_Scythe_and_the_Myth_of_Unlimited_Memory\"><\/span>I. The OOMKiller\u2019s Scythe and the Myth of Unlimited Memory<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The primary failure point was the <code>payment-processor<\/code> service. Looking at the Helm v3.14 charts used for deployment, I found that the <code>resources<\/code> block was either missing or set to values that a toddler would find optimistic. In Kubernetes v1.29.2, the scheduler relies on these numbers to make adult decisions. When you leave them blank, you\u2019re telling the Kubelet, &#8220;Hey, feel free to let this Java app eat the entire node&#8217;s RAM.&#8221;<\/p>\n<p>The Linux kernel uses cgroups v2 to manage these boundaries. When the <code>payment-processor<\/code> hit its limit\u2014or rather, when it exhausted the node&#8217;s physical memory because no limit was set\u2014the kernel&#8217;s OOM (Out Of Memory) Killer stepped in. It doesn&#8217;t ask questions. It doesn&#8217;t send a polite email. It looks for the process with the highest <code>oom_score_adj<\/code> and puts a bullet in its head. Because these pods were in the <code>BestEffort<\/code> QoS class (the lowest tier), they were the first to be executed.<\/p>\n<p>To follow &#8220;kubernetes best&#8221; practices, every single container must have <code>requests<\/code> and <code>limits<\/code> explicitly defined. <code>Requests<\/code> are what the scheduler uses to find a home for the pod; <code>limits<\/code> are the hard ceiling enforced by the kernel. Without these, you aren&#8217;t running a distributed system; you&#8217;re running a game of Russian Roulette where every chamber is loaded. We saw the <code>oom_score_adj<\/code> at 985\u2014that\u2019s basically a neon sign saying &#8220;Kill Me First.&#8221;<\/p>\n<h2><span class=\"ez-toc-section\" id=\"II_Liveness_Probes_The_Thundering_Herd_and_the_Death_Spiral\"><\/span>II. Liveness Probes: The Thundering Herd and the Death Spiral<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Once the <code>payment-processor<\/code> died, the <code>api-gateway<\/code> started failing. Why? Because the liveness probes were configured by someone who apparently hates our infrastructure. Here is the <code>kubectl describe<\/code> output for the failing gateway:<\/p>\n<pre class=\"codehilite\"><code class=\"language-text\">Liveness:   http-get http:\/\/:8080\/healthz delay=0s timeout=1s period=2s #success=1 #failure=3\nReadiness:  http-get http:\/\/:8080\/ready delay=0s timeout=1s period=2s #success=1 #failure=3\n<\/code><\/pre>\n<p>A <code>delay=0s<\/code> on a liveness probe is a death sentence. The moment the container starts, the Kubelet starts hammering the <code>\/healthz<\/code> endpoint. If the application takes more than two seconds to initialize its internal connection pools\u2014which it does\u2014the Kubelet decides the pod is dead and kills it. This is called a &#8220;death spiral.&#8221; <\/p>\n<p>The &#8220;kubernetes best&#8221; approach here is to use <code>startupProbes<\/code> for heavy lifting and give the <code>livenessProbe<\/code> enough breathing room to account for temporary network jitters. By setting the <code>periodSeconds<\/code> to 2 and the <code>failureThreshold<\/code> to 3, we gave the app exactly 6 seconds to be perfect. In the real world, things are never perfect. The gateway was being killed while it was still trying to connect to the database, leading to the <code>CrashLoopBackOff<\/code> we saw in the logs. It\u2019s like shooting a marathon runner because they didn&#8217;t finish the first mile in ten seconds.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"III_Istio_120_and_the_Sidecar_Memory_Tax\"><\/span>III. Istio 1.20 and the Sidecar Memory Tax<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>We are running Istio 1.20 for our service mesh. It\u2019s a powerful tool, but it\u2019s also a hungry one. Every pod has an <code>istio-proxy<\/code> (Envoy) sidecar. In this incident, we discovered that the sidecars were consuming more memory than the actual applications. This happens because, by default, Envoy is sent the configuration for every single service in the entire cluster. <\/p>\n<p>As our cluster grew to 500+ services, the Envoy memory footprint ballooned. We hadn&#8217;t implemented <code>Sidecar<\/code> resources to limit the scope of configuration discovery. Each proxy was holding a massive, redundant map of the entire network in its memory. When the node started feeling memory pressure from the OOM-happy Java apps, the sidecars pushed the node over the edge. <\/p>\n<p>The &#8220;kubernetes best&#8221; practice for service meshes is to use the <code>Sidecar<\/code> Custom Resource Definition (CRD) to restrict the namespace visibility. You don&#8217;t need the payment service to know the IP address of the internal Jenkins runner. By failing to prune these configurations, we turned our networking layer into a memory-hogging anchor that dragged the whole ship down.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"IV_The_Schedulers_Nightmare_Lack_of_Pod_Anti-Affinity\"><\/span>IV. The Scheduler\u2019s Nightmare: Lack of Pod Anti-Affinity<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>When I looked at the node distribution, I realized that four of our most critical services were all scheduled on <code>node-01<\/code>. This is a classic &#8220;all your eggs in one rusted bucket&#8221; scenario. When <code>node-01<\/code> started thrashing due to the OOM issues, it didn&#8217;t just take down one service; it took down the entire checkout flow.<\/p>\n<p>Kubernetes is supposed to be a distributed system, but the scheduler isn&#8217;t a psychic. If you don&#8217;t tell it to spread pods across different physical nodes or availability zones, it will bin-pack them onto the first node it finds with available &#8220;paper&#8221; capacity. <\/p>\n<p>Following &#8220;kubernetes best&#8221; practices means using <code>podAntiAffinity<\/code> with <code>topologyKey: kubernetes.io\/hostname<\/code>. This forces the scheduler to act like a sensible person and put replicas on different hardware. We had three replicas of the <code>auth-service<\/code>, and all three were sitting on the same dying node. When that node\u2019s Kubelet stopped responding because the kernel was too busy killing processes, the entire auth layer vanished. It\u2019s the equivalent of building a three-story house and putting all the support beams in the same corner.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"V_Network_Policies_The_Wild_West_of_Internal_Traffic\"><\/span>V. Network Policies: The Wild West of Internal Traffic<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>During the meltdown, we noticed a spike in traffic to the internal database that didn&#8217;t originate from the <code>payment-processor<\/code>. It turns out a compromised dev-tool pod in the same namespace was able to reach out and touch the production DB. Why? Because we had zero <code>NetworkPolicies<\/code> in place.<\/p>\n<p>In a default Kubernetes environment, the network is flat. Every pod can talk to every other pod. It\u2019s a plumber\u2019s nightmare where every pipe is connected to every other pipe, regardless of whether it\u2019s carrying fresh water or raw sewage. <\/p>\n<p>The &#8220;kubernetes best&#8221; way to handle this is a &#8220;default-deny&#8221; ingress and egress policy. You should have to explicitly permit every single connection. If the <code>api-gateway<\/code> needs to talk to the <code>payment-processor<\/code>, you write a policy for it. You don&#8217;t just leave the door open and hope for the best. Our lack of isolation meant that the &#8220;noise&#8221; from the failing services was able to saturate the network interfaces of unrelated pods, turning a localized fire into a forest fire.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"VI_Helm_Template_Hell_and_Hardcoded_Secrets\"><\/span>VI. Helm Template Hell and Hardcoded Secrets<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Finally, we have to talk about the configuration. The Helm v3.14 charts used for this deployment were a mess of hardcoded values and missing abstractions. When we tried to scale up the <code>inventory-service<\/code> to handle the load, the new pods failed because they couldn&#8217;t pull their configuration. The <code>ConfigMap<\/code> was updated, but the pods didn&#8217;t restart because there was no checksum annotation in the deployment template.<\/p>\n<p>In Kubernetes, a <code>ConfigMap<\/code> update doesn&#8217;t automatically trigger a rolling update of a <code>Deployment<\/code>. This is a known behavior, yet we keep falling for it. The &#8220;kubernetes best&#8221; practice is to include a hash of the configuration in the pod&#8217;s annotations. When the config changes, the hash changes, and Kubernetes realizes it needs to roll out new pods. <\/p>\n<p>Instead, we had half the pods running on &#8220;Config A&#8221; and the other half failing on &#8220;Config B.&#8221; It was a split-brain scenario that made debugging nearly impossible. We were trying to fix a leak in a pipe while the pipe was constantly changing its diameter.<\/p>\n<hr \/>\n<h3><span class=\"ez-toc-section\" id=\"3_FIXING_THE_MESS\"><\/span>3. FIXING THE MESS<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>We aren&#8217;t just going to patch this; we\u2019re going to rebuild it correctly. Here is the corrected manifest for the <code>payment-processor<\/code>. If I see another deployment without resource limits, I\u2019m revoking your <code>kubectl<\/code> access and sending you back to manual VM provisioning.<\/p>\n<p><strong>Corrected Deployment Manifest (<code>payment-processor-v2.yaml<\/code>):<\/strong><\/p>\n<pre class=\"codehilite\"><code class=\"language-yaml\">apiVersion: apps\/v1\nkind: Deployment\nmetadata:\n  name: payment-processor\n  namespace: production\n  labels:\n    app: payment-processor\n    version: &quot;2.1.0&quot;\nspec:\n  replicas: 3\n  selector:\n    matchLabels:\n      app: payment-processor\n  template:\n    metadata:\n      annotations:\n        # This ensures a rollout when the config changes\n        checksum\/config: ${CONFIG_HASH}\n        sidecar.istio.io\/inject: &quot;true&quot;\n      labels:\n        app: payment-processor\n    spec:\n      affinity:\n        podAntiAffinity:\n          requiredDuringSchedulingIgnoredDuringExecution:\n          - labelSelector:\n              matchExpressions:\n              - key: app\n                operator: In\n                values:\n                - payment-processor\n            topologyKey: &quot;kubernetes.io\/hostname&quot;\n      containers:\n      - name: payment-processor\n        image: our-registry.io\/payment-processor:v2.1.0\n        ports:\n        - containerPort: 8080\n        resources:\n          requests:\n            memory: &quot;1Gi&quot;\n            cpu: &quot;500m&quot;\n          limits:\n            memory: &quot;2Gi&quot;\n            cpu: &quot;1000m&quot;\n        livenessProbe:\n          httpGet:\n            path: \/healthz\n            port: 8080\n          initialDelaySeconds: 30 # Give the JVM time to wake up\n          periodSeconds: 15\n          failureThreshold: 3\n        readinessProbe:\n          httpGet:\n            path: \/ready\n            port: 8080\n          initialDelaySeconds: 10\n          periodSeconds: 10\n        securityContext:\n          runAsNonRoot: true\n          runAsUser: 1000\n          allowPrivilegeEscalation: false\n          readOnlyRootFilesystem: true\n<\/code><\/pre>\n<p><strong>Enforcing Network Isolation:<\/strong><\/p>\n<pre class=\"codehilite\"><code class=\"language-yaml\">apiVersion: networking.k8s.io\/v1\nkind: NetworkPolicy\nmetadata:\n  name: allow-gateway-to-payment\n  namespace: production\nspec:\n  podSelector:\n    matchLabels:\n      app: payment-processor\n  policyTypes:\n  - Ingress\n  ingress:\n  - from:\n    - podSelector:\n        matchLabels:\n          app: api-gateway\n    ports:\n    - protocol: TCP\n      port: 8080\n<\/code><\/pre>\n<p><strong>Terminal Commands for Recovery:<\/strong><\/p>\n<p>To get the cluster back into a sane state, I had to manually prune the dead weight and force a clean rollout.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\"># 1. Delete the pods that are stuck in the death spiral\nkubectl delete pods -n production -l app=payment-processor --force --grace-period=0\n\n# 2. Apply the new resource-constrained manifests\nkubectl apply -f payment-processor-v2.yaml\n\n# 3. Verify the QoS Class is 'Guaranteed' or 'Burstable' (not 'BestEffort')\nkubectl get pod -n production -l app=payment-processor -o jsonpath='{.items[0].status.qosClass}'\n\n# 4. Check the Istio sidecar logs for configuration sync issues\nistioctl proxy-status\n<\/code><\/pre>\n<hr \/>\n<h3><span class=\"ez-toc-section\" id=\"4_LETTER_TO_THE_JUNIOR_DEVS\"><\/span>4. LETTER TO THE JUNIOR DEVS<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Listen closely, because I\u2019m only going to say this once before I go back to my cave and wait for the next pager alert.<\/p>\n<p>I don&#8217;t care that &#8220;it worked on your machine.&#8221; Your machine is a controlled environment with 64GB of RAM and a single user. Production is a chaotic, hostile environment where the network is unreliable, the hardware is shared, and the kernel is looking for any excuse to kill your process. <\/p>\n<p>When you omit resource limits, you aren&#8217;t being &#8220;flexible.&#8221; You\u2019re being a bad neighbor. You\u2019re saying that your service is more important than every other service on that node. When you write a liveness probe with a zero-second delay, you\u2019re essentially building a self-destruct button and asking the Kubelet to press it every time the wind blows.<\/p>\n<p>Kubernetes is not a magic wand that fixes bad code. It is an orchestrator. If you give it a bad score, it will play a bad symphony. The &#8220;Great Cloud Meltdown&#8221; wasn&#8217;t a failure of the cloud; it was a failure of engineering discipline. We treat YAML like it\u2019s just configuration, but in this world, YAML is the infrastructure. A single misplaced space or a missing <code>resources<\/code> block is the equivalent of a plumber using duct tape on a high-pressure steam line.<\/p>\n<p>Next time you open a Pull Request, don&#8217;t tell me about the new features. Show me the <code>resources<\/code> block. Show me the <code>securityContext<\/code>. Show me the <code>podAntiAffinity<\/code>. If you can&#8217;t tell me how your service will behave when it\u2019s under 90% load and the network is dropping 5% of packets, then you aren&#8217;t ready to ship to my cluster.<\/p>\n<p>Now, if you&#8217;ll excuse me, I have a date with a bottle of ibuprofen and a very long nap. Don&#8217;t touch anything.<\/p>\n<p><strong>\u2014 The SRE<\/strong><\/p>\n<h2><span class=\"ez-toc-section\" id=\"Related_Articles\"><\/span>Related Articles<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Explore more insights and best practices:<\/p>\n<ul>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/docker-best-practices-build-efficient-secure-containers\/\">Docker Best Practices Build Efficient Secure Containers<\/a><\/li>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/machine-learning-best-practices-guide\/\">Machine Learning Best Practices Guide<\/a><\/li>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/html-guide\/\">Html Guide<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>INCIDENT REPORT: POST-MORTEM #8842-B TIMESTAMP: 2024-05-14T03:14:15Z STATUS: CRITICAL \/ DEGRADED INCIDENT LEAD: Senior SRE (Grizzled) SUBJECT: The &#8220;Great Cloud Meltdown&#8221; of the Production Cluster 1. THE INCIDENT SUMMARY At 03:14:15Z, my pager didn&#8217;t just beep; it screamed. I was three hours into a four-hour sleep cycle, dreaming about a world where people actually read documentation. &#8230; <a title=\"10 Kubernetes Best Practices for Scalable Production\" class=\"read-more\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/\" aria-label=\"Read more  on 10 Kubernetes Best Practices for Scalable Production\">Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-4885","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.0 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>10 Kubernetes Best Practices for Scalable Production - ITSupportWale<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"10 Kubernetes Best Practices for Scalable Production - ITSupportWale\" \/>\n<meta property=\"og:description\" content=\"INCIDENT REPORT: POST-MORTEM #8842-B TIMESTAMP: 2024-05-14T03:14:15Z STATUS: CRITICAL \/ DEGRADED INCIDENT LEAD: Senior SRE (Grizzled) SUBJECT: The &#8220;Great Cloud Meltdown&#8221; of the Production Cluster 1. THE INCIDENT SUMMARY At 03:14:15Z, my pager didn&#8217;t just beep; it screamed. I was three hours into a four-hour sleep cycle, dreaming about a world where people actually read documentation. ... Read more\" \/>\n<meta property=\"og:url\" content=\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/\" \/>\n<meta property=\"og:site_name\" content=\"ITSupportWale\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Itsupportwale-298547177495978\" \/>\n<meta property=\"article:published_time\" content=\"2026-09-17T18:56:41+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2021\/05\/android-chrome-512x512-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"512\" \/>\n\t<meta property=\"og:image:height\" content=\"512\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Techie\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Techie\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/\"},\"author\":{\"name\":\"Techie\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d\"},\"headline\":\"10 Kubernetes Best Practices for Scalable Production\",\"datePublished\":\"2026-09-17T18:56:41+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/\"},\"wordCount\":1751,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#organization\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/\",\"url\":\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/\",\"name\":\"10 Kubernetes Best Practices for Scalable Production - ITSupportWale\",\"isPartOf\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#website\"},\"datePublished\":\"2026-09-17T18:56:41+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/itsupportwale.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"10 Kubernetes Best Practices for Scalable Production\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#website\",\"url\":\"https:\/\/itsupportwale.com\/blog\/\",\"name\":\"ITSupportWale\",\"description\":\"Tips, Tricks, Fixed-Errors, Tutorials &amp; Guides\",\"publisher\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/itsupportwale.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#organization\",\"name\":\"itsupportwale\",\"url\":\"https:\/\/itsupportwale.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png\",\"contentUrl\":\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png\",\"width\":1119,\"height\":144,\"caption\":\"itsupportwale\"},\"image\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/Itsupportwale-298547177495978\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d\",\"name\":\"Techie\",\"sameAs\":[\"https:\/\/itsupportwale.com\",\"iswblogadmin\"],\"url\":\"https:\/\/itsupportwale.com\/blog\/author\/iswblogadmin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"10 Kubernetes Best Practices for Scalable Production - ITSupportWale","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/","og_locale":"en_US","og_type":"article","og_title":"10 Kubernetes Best Practices for Scalable Production - ITSupportWale","og_description":"INCIDENT REPORT: POST-MORTEM #8842-B TIMESTAMP: 2024-05-14T03:14:15Z STATUS: CRITICAL \/ DEGRADED INCIDENT LEAD: Senior SRE (Grizzled) SUBJECT: The &#8220;Great Cloud Meltdown&#8221; of the Production Cluster 1. THE INCIDENT SUMMARY At 03:14:15Z, my pager didn&#8217;t just beep; it screamed. I was three hours into a four-hour sleep cycle, dreaming about a world where people actually read documentation. ... Read more","og_url":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/","og_site_name":"ITSupportWale","article_publisher":"https:\/\/www.facebook.com\/Itsupportwale-298547177495978","article_published_time":"2026-09-17T18:56:41+00:00","og_image":[{"width":512,"height":512,"url":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2021\/05\/android-chrome-512x512-1.png","type":"image\/png"}],"author":"Techie","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Techie","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#article","isPartOf":{"@id":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/"},"author":{"name":"Techie","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d"},"headline":"10 Kubernetes Best Practices for Scalable Production","datePublished":"2026-09-17T18:56:41+00:00","mainEntityOfPage":{"@id":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/"},"wordCount":1751,"commentCount":0,"publisher":{"@id":"https:\/\/itsupportwale.com\/blog\/#organization"},"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/","url":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/","name":"10 Kubernetes Best Practices for Scalable Production - ITSupportWale","isPartOf":{"@id":"https:\/\/itsupportwale.com\/blog\/#website"},"datePublished":"2026-09-17T18:56:41+00:00","breadcrumb":{"@id":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/itsupportwale.com\/blog\/10-kubernetes-best-practices-for-scalable-production\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/itsupportwale.com\/blog\/"},{"@type":"ListItem","position":2,"name":"10 Kubernetes Best Practices for Scalable Production"}]},{"@type":"WebSite","@id":"https:\/\/itsupportwale.com\/blog\/#website","url":"https:\/\/itsupportwale.com\/blog\/","name":"ITSupportWale","description":"Tips, Tricks, Fixed-Errors, Tutorials &amp; Guides","publisher":{"@id":"https:\/\/itsupportwale.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/itsupportwale.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/itsupportwale.com\/blog\/#organization","name":"itsupportwale","url":"https:\/\/itsupportwale.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png","contentUrl":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png","width":1119,"height":144,"caption":"itsupportwale"},"image":{"@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Itsupportwale-298547177495978"]},{"@type":"Person","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d","name":"Techie","sameAs":["https:\/\/itsupportwale.com","iswblogadmin"],"url":"https:\/\/itsupportwale.com\/blog\/author\/iswblogadmin\/"}]}},"_links":{"self":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts\/4885","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/comments?post=4885"}],"version-history":[{"count":0,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts\/4885\/revisions"}],"wp:attachment":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/media?parent=4885"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/categories?post=4885"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/tags?post=4885"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}