{"id":4831,"date":"2026-07-08T22:15:28","date_gmt":"2026-07-08T16:45:28","guid":{"rendered":"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/"},"modified":"2026-07-08T22:15:28","modified_gmt":"2026-07-08T16:45:28","slug":"kubernetes-best-practices-guide","status":"publish","type":"post","link":"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/","title":{"rendered":"Kubernetes Best Practices &#8211; Guide"},"content":{"rendered":"<p><strong>INCIDENT LOG: 2024-05-20T03:04:12Z<\/strong><br \/>\n<strong>LOCATION: us-east-1 (Production Cluster &#8216;Ares-01&#8217;)<\/strong><br \/>\n<strong>STATUS: CRITICAL FAILURE<\/strong><\/p>\n<p>03:04:12Z &#8211; PagerDuty alert: <code>KubePodNotReady<\/code> (Namespace: <code>checkout-prod<\/code>).<br \/>\n03:05:45Z &#8211; Node <code>ip-10-244-12-84.ec2.internal<\/code> enters <code>NotReady<\/code> state.<br \/>\n03:06:10Z &#8211; Cascading failure detected. <code>kube-dns<\/code> latency spikes to 4500ms.<br \/>\n03:07:30Z &#8211; API Server unresponsive. <code>kubectl<\/code> commands timing out.<br \/>\n03:10:00Z &#8211; On-call engineer attempts manual recovery.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\">$ kubectl get pods -n checkout-prod -o wide\nNAME                             READY   STATUS             RESTARTS      AGE   IP            NODE\ncheckout-api-7f8d9b6c5d-2v8xl    0\/1     OOMKilled          4 (12s ago)   14m   10.244.2.45   ip-10-244-12-84\ncheckout-api-7f8d9b6c5d-9z2pq    0\/1     ImagePullBackOff   0             2m    10.244.3.11   ip-10-244-13-19\npayment-gw-5d4f3a2b1c-m4n5b      1\/1     Running            0             12h   10.244.2.46   ip-10-244-12-84\nredis-master-0                   0\/1     Pending            0             1m    &lt;none&gt;        &lt;none&gt;\n<\/code><\/pre>\n<p>03:12:45Z &#8211; Root cause identified: A &#8220;Cloud Native&#8221; developer pushed a deployment with no memory limits and a broken sidecar configuration, triggering a node-level OOM event that took down the CNI plugin.<\/p>\n<hr \/>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_80 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a4ea7641c116\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a4ea7641c116\"  aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/#Your_Resource_Limits_are_a_Mathematical_Lie\" >Your Resource Limits are a Mathematical Lie<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/#Stop_Blindly_Trusting_Your_Cloud_Providers_Default_CNI\" >Stop Blindly Trusting Your Cloud Provider&#8217;s Default CNI<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/#Your_Sidecars_are_Parasites_Not_Features\" >Your Sidecars are Parasites, Not Features<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/#DNS_is_Always_the_Culprit_Because_Youre_Lazy\" >DNS is Always the Culprit Because You&#8217;re Lazy<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/#NetworkPolicies_are_Mandatory_Not_Optional\" >NetworkPolicies are Mandatory, Not Optional<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/#Admission_Controllers_Your_Last_Line_of_Defense_Against_Junior_Devs\" >Admission Controllers: Your Last Line of Defense Against Junior Devs<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/#Storage_is_Not_a_Magic_Bucket\" >Storage is Not a Magic Bucket<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/#PodDisruptionBudgets_and_the_Illusion_of_Availability\" >PodDisruptionBudgets and the Illusion of Availability<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/#The_Kernel_is_the_Source_of_Truth\" >The Kernel is the Source of Truth<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/#Related_Articles\" >Related Articles<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"Your_Resource_Limits_are_a_Mathematical_Lie\"><\/span>Your Resource Limits are a Mathematical Lie<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>If you think setting <code>resources.limits.cpu: \"1\"<\/code> means your pod gets exactly one core, you shouldn&#8217;t be touching a production cluster. You are confusing abstraction with reality. Kubernetes v1.29 and v1.30 rely heavily on Linux cgroups v2, and if you haven&#8217;t bothered to understand the unified hierarchy, you are just guessing.<\/p>\n<p>In cgroups v1, memory and CPU were separate silos. In cgroups v2, they are integrated. When your pod hits a memory limit, the kernel doesn&#8217;t just kill the process; it looks at the <code>memory.high<\/code> and <code>memory.max<\/code> settings. If you haven&#8217;t configured your <code>kubelet<\/code> with <code>MemoryQoS=true<\/code>, you are leaving your node&#8217;s stability to the whims of the OOM killer, which is about as reliable as a coin toss in a hurricane.<\/p>\n<p>The CPU limit is even worse. It\u2019s implemented via the Completely Fair Scheduler (CFS) quota. If you set a limit of 100ms per 100ms period, and your application is multi-threaded, you will hit that quota in 10ms and be throttled for the remaining 90ms. Your &#8220;low latency&#8221; Java app is now stuttering because you don&#8217;t understand how <code>cpu.cfs_quota_us<\/code> works.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\">$ kubectl describe node ip-10-244-12-84.ec2.internal\nConditions:\n  Type                 Status  LastHeartbeatTime                 Reason\n  ----                 ------  -----------------                 ------\n  MemoryPressure       True    Mon, 20 May 2024 03:15:22 +0000   KubeletHasInsufficientMemory\n  DiskPressure         False   Mon, 20 May 2024 03:15:22 +0000   KubeletHasNoDiskPressure\n  PIDPressure          False   Mon, 20 May 2024 03:15:22 +0000   KubeletHasSufficientPID\n  Ready                False   Mon, 20 May 2024 03:15:22 +0000   KubeletNotReady\nEvents:\n  Type     Reason               Age                From        Message\n  ----     ------               ----               ----        -------\n  Warning  SystemOOM            2m14s              kubelet     System OOM encountered, victim process: checkout-api\n  Normal   NodeNotReady         1m58s              kubelet     Node ip-10-244-12-84.ec2.internal status is now. NotReady\n<\/code><\/pre>\n<p>To fix this, you need to stop using limits as a &#8220;safety net&#8221; and start using them as a hard constraint based on actual profiling. Use <code>GOMEMLIMIT<\/code> for Go apps or <code>MaxRAMPercentage<\/code> for JVM. If you don&#8217;t align your application&#8217;s internal runtime awareness with the cgroup limits, you are just begging for a 3:00 AM wake-up call. This is &#8220;kubernetes best&#8221; practice: alignment, not guesswork.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Stop_Blindly_Trusting_Your_Cloud_Providers_Default_CNI\"><\/span>Stop Blindly Trusting Your Cloud Provider&#8217;s Default CNI<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Most of you are running the default AWS VPC CNI or Azure CNI and wondering why your pod networking feels like it&#8217;s running over a 56k modem. These plugins are designed for the lowest common denominator. They eat IP addresses like candy and introduce massive overhead in the kernel&#8217;s routing table.<\/p>\n<p>When you have 200 pods on a node, and each pod has its own ENI or secondary IP, the <code>iptables<\/code> rules grow exponentially. Every single packet has to traverse a chain of thousands of rules. If you haven&#8217;t switched to eBPF-based networking like Cilium, you are living in 2015. eBPF bypasses the <code>iptables<\/code> bottleneck by hooking directly into the kernel&#8217;s socket layer.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\">$ istioctl proxy-status\nNAME                                                   CLUSTER        CDS                LDS                EDS                RDS          ECDS         ISTIOD                      VERSION\ncheckout-api-7f8d9b6c5d-2v8xl.checkout-prod            Ares-01        STALE (14m)        STALE (14m)        STALE (14m)        NOT SENT     NOT SENT     istiod-789456-abcde         1.21.0\npayment-gw-5d4f3a2b1c-m4n5b.checkout-prod              Ares-01        SYNCED             SYNCED             SYNCED             SYNCED       NOT SENT     istiod-789456-abcde         1.21.0\n<\/code><\/pre>\n<p>Look at that <code>STALE<\/code> status. That\u2019s what happens when your CNI can&#8217;t handle the churn. The control plane is trying to push XDS updates to Envoy, but the network is so congested with ARP requests and <code>iptables<\/code> lookups that the sidecar just gives up. You need to tune your <code>conntrack<\/code> tables. If <code>net.netfilter.nf_conntrack_max<\/code> isn&#8217;t set to at least 1,048,576 on a high-traffic node, you&#8217;re an amateur.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Your_Sidecars_are_Parasites_Not_Features\"><\/span>Your Sidecars are Parasites, Not Features<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The &#8220;Service Mesh&#8221; hype train has convinced you that every pod needs a sidecar. Congratulations, you&#8217;ve just increased your memory footprint by 50% and added 10ms of latency to every hop. In the incident above, the <code>istio-proxy<\/code> was consuming 150MB of RAM just to route traffic to a local Redis instance.<\/p>\n<p>If you are using Istio, you must use <code>Sidecar<\/code> resources to limit the configuration sent to each proxy. By default, every sidecar knows about every other service in the cluster. That is thousands of endpoints being shoved into a tiny Envoy process.<\/p>\n<pre class=\"codehilite\"><code class=\"language-yaml\"># A production-grade Deployment that doesn't suck\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\n  name: checkout-api\n  namespace: checkout-prod\n  labels:\n    app: checkout-api\n    tier: backend\nspec:\n  replicas: 3\n  selector:\n    matchLabels:\n      app: checkout-api\n  strategy:\n    rollingUpdate:\n      maxSurge: 25%\n      maxUnavailable: 1\n  template:\n    metadata:\n      labels:\n        app: checkout-api\n      annotations:\n        sidecar.istio.io\/inject: &quot;true&quot;\n        sidecar.istio.io\/cpuLimit: &quot;200m&quot;\n        sidecar.istio.io\/memoryLimit: &quot;256Mi&quot;\n    spec:\n      securityContext:\n        runAsNonRoot: true\n        runAsUser: 1000\n        seccompProfile:\n          type: RuntimeDefault\n      containers:\n      - name: checkout-api\n        image: internal-reg.io\/checkout-api:v1.4.2\n        imagePullPolicy: IfNotPresent\n        ports:\n        - containerPort: 8080\n        resources:\n          requests:\n            cpu: &quot;500m&quot;\n            memory: &quot;512Mi&quot;\n          limits:\n            cpu: &quot;1000m&quot;\n            memory: &quot;1Gi&quot;\n        livenessProbe:\n          httpGet:\n            path: \/healthz\n            port: 8080\n          initialDelaySeconds: 15\n          periodSeconds: 20\n          failureThreshold: 3\n        readinessProbe:\n          httpGet:\n            path: \/readyz\n            port: 8080\n          initialDelaySeconds: 5\n          periodSeconds: 10\n<\/code><\/pre>\n<p>Notice the <code>securityContext<\/code>. If you&#8217;re running as root, you&#8217;ve already lost. Notice the <code>initialDelaySeconds<\/code>. If you set this to 0, the <code>kubelet<\/code> will kill your app before it even finishes loading its config, leading to a <code>CrashLoopBackOff<\/code>. This is &#8220;kubernetes best&#8221; practice: give your application room to breathe before you start poking it with sticks.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"DNS_is_Always_the_Culprit_Because_Youre_Lazy\"><\/span>DNS is Always the Culprit Because You&#8217;re Lazy<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>When the API server went down at 03:07Z, it wasn&#8217;t because of a bug in Kubernetes. It was because <code>kube-dns<\/code> (CoreDNS) was being hammered by 50,000 requests per second from pods trying to resolve <code>redis-master.checkout-prod.svc.cluster.local<\/code>.<\/p>\n<p>By default, the <code>ndots<\/code> setting in <code>\/etc\/resolv.conf<\/code> is set to 5. This means for every DNS lookup of <code>google.com<\/code>, the resolver tries:<br \/>\n1. <code>google.com.checkout-prod.svc.cluster.local<\/code><br \/>\n2. <code>google.com.svc.cluster.local<\/code><br \/>\n3. <code>google.com.cluster.local<\/code><br \/>\n&#8230;and so on.<\/p>\n<p>You are effectively quintupling your DNS traffic for no reason. Use <code>NodeLocal DNSCache<\/code>. It runs a DNS caching agent on every node as a DaemonSet, intercepting these requests before they ever hit the network.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\">$ kubectl get svc -n kube-system\nNAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE\nkube-dns         ClusterIP   10.96.0.10   &lt;none&gt;        53\/UDP,53\/TCP,9153\/TCP   180d\nnode-local-dns   ClusterIP   10.96.0.11   &lt;none&gt;        53\/UDP,53\/TCP,9153\/TCP   180d\n\n$ kubectl top pods -n kube-system | grep coredns\ncoredns-64444d474-8vdfv   15m   42Mi\ncoredns-64444d474-9zpxl   12m   38Mi\n<\/code><\/pre>\n<p>If your CoreDNS pods are using more than 100MB of RAM, you have a leak or a massive misconfiguration in your <code>Corefile<\/code>. Stop using the <code>autopath<\/code> plugin if you don&#8217;t know exactly what it does; it\u2019s a notorious source of race conditions in high-concurrency environments.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"NetworkPolicies_are_Mandatory_Not_Optional\"><\/span>NetworkPolicies are Mandatory, Not Optional<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Running a cluster without <code>NetworkPolicies<\/code> is like leaving your front door open and putting a sign out front that says &#8220;Free Money.&#8221; By default, every pod in Kubernetes can talk to every other pod. If your frontend web server gets compromised, the attacker has a direct line to your internal database and your metadata service (169.254.169.254).<\/p>\n<p>You need a &#8220;Default Deny&#8221; policy in every namespace. Period. No exceptions.<\/p>\n<pre class=\"codehilite\"><code class=\"language-yaml\"># Production-grade NetworkPolicy: Deny All except what is explicitly allowed\napiVersion: networking.k8s.io\/v1\nkind: NetworkPolicy\nmetadata:\n  name: default-deny-all\n  namespace: checkout-prod\nspec:\n  podSelector: {}\n  policyTypes:\n  - Ingress\n  - Egress\n---\napiVersion: networking.k8s.io\/v1\nkind: NetworkPolicy\nmetadata:\n  name: allow-checkout-to-redis\n  namespace: checkout-prod\nspec:\n  podSelector:\n    matchLabels:\n      app: checkout-api\n  policyTypes:\n  - Egress\n  egress:\n  - to:\n    - podSelector:\n        matchLabels:\n          app: redis\n    ports:\n    - protocol: TCP\n      port: 6379\n<\/code><\/pre>\n<p>This isn&#8217;t just about security; it&#8217;s about blast radius. When a service goes rogue and starts scanning the network, a <code>NetworkPolicy<\/code> will drop those packets at the kernel level (via <code>iptables<\/code> or eBPF), preventing the &#8220;thundering herd&#8221; from taking down your entire infrastructure.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Admission_Controllers_Your_Last_Line_of_Defense_Against_Junior_Devs\"><\/span>Admission Controllers: Your Last Line of Defense Against Junior Devs<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>You cannot rely on &#8220;documentation&#8221; or &#8220;best practices&#8221; wikis to keep your cluster stable. People will ignore them. You must use Admission Controllers to enforce your standards. Whether it&#8217;s OPA Gatekeeper or Kyverno, you need to block any deployment that doesn&#8217;t have resource requests and limits, or that tries to run as root.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\">$ kubectl get events -n checkout-prod --field-selector involvedObject.name=checkout-api\nLAST SEEN   TYPE      REASON             OBJECT              MESSAGE\n2m          Warning   FailedCreate       ReplicaSet          Error creating: admission webhook &quot;validation.gatekeeper.sh&quot; denied the request: Container &lt;checkout-api&gt; must have memory limits defined.\n<\/code><\/pre>\n<p>If your CI\/CD pipeline is the only thing checking your YAML, you are vulnerable. Someone will eventually use <code>kubectl edit<\/code> or a manual <code>helm install<\/code> and bypass your checks. The Admission Controller is the only way to ensure that what is running in the cluster actually matches your &#8220;kubernetes best&#8221; standards.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Storage_is_Not_a_Magic_Bucket\"><\/span>Storage is Not a Magic Bucket<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>If you are still using <code>In-Tree<\/code> storage drivers, you are living in a museum. Kubernetes v1.29+ has fully migrated to the Container Storage Interface (CSI). If your <code>kube-controller-manager<\/code> is still trying to talk to the AWS API to attach an EBS volume, you are asking for deadlocks.<\/p>\n<p>The most common failure mode in storage is the <code>Multi-Attach<\/code> error. A node dies, the controller tries to move the volume to a new node, but the old node still has a lock on it.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\">$ kubectl describe pod checkout-api-7f8d9b6c5d-9z2pq\nEvents:\n  Type     Reason              Age   From                     Message\n  ----     ------              ----  ----                     -------\n  Warning  FailedMount         4m    kubelet                  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data]: timed out waiting for the condition\n  Warning  FailedAttachVolume  4m    attachdetach-controller  Multi-Attach error for volume &quot;pvc-12345&quot; Volume is already used by pod checkout-api-7f8d9b6c5d-2v8xl on node ip-10-244-12-84\n<\/code><\/pre>\n<p>To mitigate this, you must use <code>ReadWriteOncePod<\/code> access mode (introduced in v1.22 and stable in v1.27) if you are on a supported CSI driver. Also, ensure your <code>StorageClass<\/code> has <code>volumeBindingMode: WaitForFirstConsumer<\/code>. If you let Kubernetes pick a zone for your volume before it knows where the pod will be scheduled, you will end up with a pod in <code>us-east-1a<\/code> trying to mount a volume in <code>us-east-1b<\/code>. It will never work.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"PodDisruptionBudgets_and_the_Illusion_of_Availability\"><\/span>PodDisruptionBudgets and the Illusion of Availability<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>You claim your app is &#8220;highly available&#8221; because you have 3 replicas. Then, the cloud provider performs a scheduled maintenance on the underlying nodes, or your cluster autoscaler decides to bin-pack the nodes, and it terminates all 3 replicas at the same time.<\/p>\n<p>A <code>PodDisruptionBudget<\/code> (PDB) is the only way to tell the Kubernetes scheduler: &#8220;I don&#8217;t care what you&#8217;re doing, you are not allowed to have fewer than 2 replicas of this service online.&#8221;<\/p>\n<pre class=\"codehilite\"><code class=\"language-yaml\"># Production-grade PodDisruptionBudget\napiVersion: policy\/v1\nkind: PodDisruptionBudget\nmetadata:\n  name: checkout-api-pdb\n  namespace: checkout-prod\nspec:\n  minAvailable: 2\n  selector:\n    matchLabels:\n      app: checkout-api\n<\/code><\/pre>\n<p>Without a PDB, your &#8220;availability&#8221; is just a suggestion. During the 03:00 AM incident, the cluster autoscaler tried to drain node <code>ip-10-244-12-84<\/code> because it was under pressure. Because there was no PDB, it killed the only healthy <code>payment-gw<\/code> pod, turning a minor slowdown into a total outage.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\">$ kubectl get pdb -n checkout-prod\nNAME               MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE\ncheckout-api-pdb   2               N\/A               1                     14d\npayment-gw-pdb     1               N\/A               0                     12h\n<\/code><\/pre>\n<p>If <code>ALLOWED DISRUPTIONS<\/code> is 0, the <code>kubectl drain<\/code> command will hang until you manually intervene or until the timeout is reached. This is exactly what you want. It forces the system to wait until a new replica is <code>Ready<\/code> (passing its <code>readinessProbe<\/code>) before killing the old one.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The_Kernel_is_the_Source_of_Truth\"><\/span>The Kernel is the Source of Truth<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Finally, stop treating the node as a black box. You need to monitor kernel metrics. If your <code>node-exporter<\/code> isn&#8217;t scraping <code>node_vmstat_oom_kill<\/code> or <code>node_sched_stat_waiting_seconds_total<\/code>, you are flying blind.<\/p>\n<p>In Kubernetes v1.30, the interaction between the <code>kubelet<\/code> and the kernel&#8217;s <code>oom_score_adj<\/code> has been refined. The <code>kubelet<\/code> assigns a score to each process based on its QoS class (<code>Guaranteed<\/code>, <code>Burstable<\/code>, or <code>BestEffort<\/code>). If you don&#8217;t set both requests and limits to the same value, your pod is <code>Burstable<\/code>. In a memory crunch, the kernel will kill <code>BestEffort<\/code> pods first, then <code>Burstable<\/code>. If you want your database to stay alive, it must be in the <code>Guaranteed<\/code> class.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\">$ kubectl get nodes -o wide\nNAME                        STATUS   ROLES    AGE    VERSION   INTERNAL-IP   OS-IMAGE                         KERNEL-VERSION                 CONTAINER-RUNTIME\nip-10-244-12-84.ec2.internal Ready    &lt;none&gt;   180d   v1.30.1   10.244.12.84  Ubuntu 22.04.4 LTS               5.15.0-1058-aws                containerd:\/\/1.7.13\nip-10-244-13-19.ec2.internal Ready    &lt;none&gt;   180d   v1.30.1   10.244.13.19  Ubuntu 22.04.4 LTS               5.15.0-1058-aws                containerd:\/\/1.7.13\n<\/code><\/pre>\n<p>Check your kernel version. If you are running anything older than 5.10, you are missing out on critical cgroups v2 improvements and eBPF features that make Kubernetes actually manageable at scale.<\/p>\n<p>Your cluster is not a &#8220;platform.&#8221; It is a complex, fragile orchestration of Linux processes, network namespaces, and filesystem mounts. If you continue to treat it like a magic cloud bucket where you just &#8220;deploy code,&#8221; it will continue to burn. Stop reading marketing blogs and start reading the kernel documentation.<\/p>\n<p>Fix your YAML. Tune your sysctls. Or get used to the sound of PagerDuty.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Related_Articles\"><\/span>Related Articles<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Explore more insights and best practices:<\/p>\n<ul>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/install-and-secure-phpmyadmin-with-nginx-on-ubuntu-18-04\/\">Install And Secure Phpmyadmin With Nginx On Ubuntu 18 04<\/a><\/li>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/how-to-install-open-source-zimbra-8-8-mail-server-zcs-8-8-12-on-ubuntu-16-04-lts\/\">How To Install Open Source Zimbra 8 8 Mail Server Zcs 8 8 12 On Ubuntu 16 04 Lts<\/a><\/li>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/top-machine-learning-algorithms-a-comprehensive-guide\/\">Top Machine Learning Algorithms A Comprehensive Guide<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>INCIDENT LOG: 2024-05-20T03:04:12Z LOCATION: us-east-1 (Production Cluster &#8216;Ares-01&#8217;) STATUS: CRITICAL FAILURE 03:04:12Z &#8211; PagerDuty alert: KubePodNotReady (Namespace: checkout-prod). 03:05:45Z &#8211; Node ip-10-244-12-84.ec2.internal enters NotReady state. 03:06:10Z &#8211; Cascading failure detected. kube-dns latency spikes to 4500ms. 03:07:30Z &#8211; API Server unresponsive. kubectl commands timing out. 03:10:00Z &#8211; On-call engineer attempts manual recovery. $ kubectl get pods &#8230; <a title=\"Kubernetes Best Practices &#8211; Guide\" class=\"read-more\" href=\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/\" aria-label=\"Read more  on Kubernetes Best Practices &#8211; Guide\">Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-4831","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.0 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Kubernetes Best Practices - Guide - ITSupportWale<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Kubernetes Best Practices - Guide - ITSupportWale\" \/>\n<meta property=\"og:description\" content=\"INCIDENT LOG: 2024-05-20T03:04:12Z LOCATION: us-east-1 (Production Cluster &#8216;Ares-01&#8217;) STATUS: CRITICAL FAILURE 03:04:12Z &#8211; PagerDuty alert: KubePodNotReady (Namespace: checkout-prod). 03:05:45Z &#8211; Node ip-10-244-12-84.ec2.internal enters NotReady state. 03:06:10Z &#8211; Cascading failure detected. kube-dns latency spikes to 4500ms. 03:07:30Z &#8211; API Server unresponsive. kubectl commands timing out. 03:10:00Z &#8211; On-call engineer attempts manual recovery. $ kubectl get pods ... Read more\" \/>\n<meta property=\"og:url\" content=\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/\" \/>\n<meta property=\"og:site_name\" content=\"ITSupportWale\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Itsupportwale-298547177495978\" \/>\n<meta property=\"article:published_time\" content=\"2026-07-08T16:45:28+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2021\/05\/android-chrome-512x512-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"512\" \/>\n\t<meta property=\"og:image:height\" content=\"512\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Techie\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Techie\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/\"},\"author\":{\"name\":\"Techie\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d\"},\"headline\":\"Kubernetes Best Practices &#8211; Guide\",\"datePublished\":\"2026-07-08T16:45:28+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/\"},\"wordCount\":1580,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#organization\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/\",\"url\":\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/\",\"name\":\"Kubernetes Best Practices - Guide - ITSupportWale\",\"isPartOf\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#website\"},\"datePublished\":\"2026-07-08T16:45:28+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/itsupportwale.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Kubernetes Best Practices &#8211; Guide\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#website\",\"url\":\"https:\/\/itsupportwale.com\/blog\/\",\"name\":\"ITSupportWale\",\"description\":\"Tips, Tricks, Fixed-Errors, Tutorials &amp; Guides\",\"publisher\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/itsupportwale.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#organization\",\"name\":\"itsupportwale\",\"url\":\"https:\/\/itsupportwale.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png\",\"contentUrl\":\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png\",\"width\":1119,\"height\":144,\"caption\":\"itsupportwale\"},\"image\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/Itsupportwale-298547177495978\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d\",\"name\":\"Techie\",\"sameAs\":[\"https:\/\/itsupportwale.com\",\"iswblogadmin\"],\"url\":\"https:\/\/itsupportwale.com\/blog\/author\/iswblogadmin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Kubernetes Best Practices - Guide - ITSupportWale","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/","og_locale":"en_US","og_type":"article","og_title":"Kubernetes Best Practices - Guide - ITSupportWale","og_description":"INCIDENT LOG: 2024-05-20T03:04:12Z LOCATION: us-east-1 (Production Cluster &#8216;Ares-01&#8217;) STATUS: CRITICAL FAILURE 03:04:12Z &#8211; PagerDuty alert: KubePodNotReady (Namespace: checkout-prod). 03:05:45Z &#8211; Node ip-10-244-12-84.ec2.internal enters NotReady state. 03:06:10Z &#8211; Cascading failure detected. kube-dns latency spikes to 4500ms. 03:07:30Z &#8211; API Server unresponsive. kubectl commands timing out. 03:10:00Z &#8211; On-call engineer attempts manual recovery. $ kubectl get pods ... Read more","og_url":"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/","og_site_name":"ITSupportWale","article_publisher":"https:\/\/www.facebook.com\/Itsupportwale-298547177495978","article_published_time":"2026-07-08T16:45:28+00:00","og_image":[{"width":512,"height":512,"url":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2021\/05\/android-chrome-512x512-1.png","type":"image\/png"}],"author":"Techie","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Techie","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/#article","isPartOf":{"@id":"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/"},"author":{"name":"Techie","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d"},"headline":"Kubernetes Best Practices &#8211; Guide","datePublished":"2026-07-08T16:45:28+00:00","mainEntityOfPage":{"@id":"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/"},"wordCount":1580,"commentCount":0,"publisher":{"@id":"https:\/\/itsupportwale.com\/blog\/#organization"},"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/","url":"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/","name":"Kubernetes Best Practices - Guide - ITSupportWale","isPartOf":{"@id":"https:\/\/itsupportwale.com\/blog\/#website"},"datePublished":"2026-07-08T16:45:28+00:00","breadcrumb":{"@id":"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/itsupportwale.com\/blog\/kubernetes-best-practices-guide\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/itsupportwale.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Kubernetes Best Practices &#8211; Guide"}]},{"@type":"WebSite","@id":"https:\/\/itsupportwale.com\/blog\/#website","url":"https:\/\/itsupportwale.com\/blog\/","name":"ITSupportWale","description":"Tips, Tricks, Fixed-Errors, Tutorials &amp; Guides","publisher":{"@id":"https:\/\/itsupportwale.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/itsupportwale.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/itsupportwale.com\/blog\/#organization","name":"itsupportwale","url":"https:\/\/itsupportwale.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png","contentUrl":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png","width":1119,"height":144,"caption":"itsupportwale"},"image":{"@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Itsupportwale-298547177495978"]},{"@type":"Person","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d","name":"Techie","sameAs":["https:\/\/itsupportwale.com","iswblogadmin"],"url":"https:\/\/itsupportwale.com\/blog\/author\/iswblogadmin\/"}]}},"_links":{"self":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts\/4831","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/comments?post=4831"}],"version-history":[{"count":0,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts\/4831\/revisions"}],"wp:attachment":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/media?parent=4831"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/categories?post=4831"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/tags?post=4831"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}