{"id":4920,"date":"2026-10-31T01:24:21","date_gmt":"2026-10-30T19:54:21","guid":{"rendered":"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/"},"modified":"2026-10-31T01:24:21","modified_gmt":"2026-10-30T19:54:21","slug":"mastering-your-kubernetes-cluster-a-step-by-step-guide","status":"publish","type":"post","link":"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/","title":{"rendered":"Mastering Your Kubernetes Cluster: A Step-by-Step Guide"},"content":{"rendered":"<p>It was 3:14 AM on a Tuesday when the first 503 errors started hitting the ingress-nginx logs, and by 3:20, the entire <strong>kubernetes cluster<\/strong> was a graveyard of CrashLoopBackOffs.<\/p>\n<p>I\u2019m writing this because if I don\u2019t document this nightmare now, I\u2019ll probably hallucinate that it was all a bad dream induced by too much caffeine and not enough REM sleep. I\u2019ve been staring at a terminal for 72 hours. My eyes feel like they\u2019ve been scrubbed with steel wool. The &#8220;Architects&#8221; are all tucked away in their beds, dreaming of microservices and &#8220;agility,&#8221; while I\u2019m sitting here wondering why we ever thought running a stateful database on a distributed scheduler with a flaky CNI was a good idea.<\/p>\n<p>This isn&#8217;t a &#8220;success story.&#8221; This is a post-mortem of a cluster that decided to commit ritual suicide because of a single misconfigured YAML indentation and a kernel parameter that everyone ignored.<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_89 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6ae73e5f958a0\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6ae73e5f958a0\"  aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#Incident_Timeline_The_72-Hour_Descent\" >Incident Timeline: The 72-Hour Descent<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#Root_Cause_Analysis_The_Etcd_Quorum_Collapse\" >Root Cause Analysis: The Etcd Quorum Collapse<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#Networking_Hell_CNI_Encapsulation_and_MTU_Mismatches\" >Networking Hell: CNI Encapsulation and MTU Mismatches<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#The_PVC_Deadlock_Finalizers_and_the_CSI_Ghost_in_the_Machine\" >The PVC Deadlock: Finalizers and the CSI Ghost in the Machine<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#The_%E2%80%9CFix%E2%80%9D_Patching_the_Control_Plane_with_Duct_Tape\" >The &#8220;Fix&#8221;: Patching the Control Plane with Duct Tape<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#Post-Mortem_Architecture_Critique_The_Cost_of_Complexity\" >Post-Mortem Architecture Critique: The Cost of Complexity<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#Stabilization_The_Long_Road_Back\" >Stabilization: The Long Road Back<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#The_Warning\" >The Warning<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#Technical_Appendix_Sub-System_Analysis\" >Technical Appendix: Sub-System Analysis<\/a><ul class='ez-toc-list-level-4' ><li class='ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#The_CNI_Calico_Deep-Dive\" >The CNI (Calico) Deep-Dive<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#The_CSI_EBS_Deep-Dive\" >The CSI (EBS) Deep-Dive<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#Kube-Proxy_and_Iptables\" >Kube-Proxy and Iptables<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-4'><a class=\"ez-toc-link ez-toc-heading-13\" href=\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#Resource_Limits_and_the_OOMKiller\" >Resource Limits and the OOMKiller<\/a><\/li><\/ul><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-14\" href=\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#Related_Articles\" >Related Articles<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"Incident_Timeline_The_72-Hour_Descent\"><\/span>Incident Timeline: The 72-Hour Descent<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><strong>03:14 UTC<\/strong>: PagerDuty triggers. <code>IngressHighErrorRate<\/code>. I check the logs. <code>ingress-nginx<\/code> is throwing 503s like it\u2019s going out of style. I figure it\u2019s just a pod that got OOMKilled. I was wrong.<\/p>\n<p><strong>03:22 UTC<\/strong>: <code>kubectl get nodes<\/code> returns nothing. The API server is timing out. I try to SSH into the master node. Connection refused.<\/p>\n<p><strong>03:45 UTC<\/strong>: I finally get into the control plane. The load average is 450 on a 16-core machine. <code>etcd<\/code> is screaming. The logs are filled with <code>apply entries took too long<\/code>.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\"># etcdctl --endpoints=https:\/\/127.0.0.1:2379 --cacert=\/etc\/kubernetes\/pki\/etcd\/ca.crt \\\n--cert=\/etc\/kubernetes\/pki\/etcd\/server.crt --key=\/etc\/kubernetes\/pki\/etcd\/server.key \\\nendpoint health\n\n127.0.0.1:2379 is unhealthy: failed to commit proposal: context deadline exceeded\nError: unhealthy cluster\n<\/code><\/pre>\n<p><strong>04:10 UTC<\/strong>: The entire cluster is now in a <code>NotReady<\/code> state. The kubelets have lost contact with the API server.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\"># kubectl get nodes -o wide\nNAME             STATUS     ROLES           AGE   VERSION   INTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME\nprod-worker-01   NotReady   &lt;none&gt;          142d  v1.29.2   10.0.12.4     Ubuntu 22.04.3 LTS   5.15.0-101-generic  containerd:\/\/1.7.1\nprod-worker-02   NotReady   &lt;none&gt;          142d  v1.29.2   10.0.12.5     Ubuntu 22.04.3 LTS   5.15.0-101-generic  containerd:\/\/1.7.1\nprod-master-01   NotReady   control-plane   142d  v1.29.2   10.0.12.1     Ubuntu 22.04.3 LTS   5.15.0-101-generic  containerd:\/\/1.7.1\n<\/code><\/pre>\n<p><strong>06:00 UTC<\/strong>: I realize the <code>etcd<\/code> database has hit the 8GB quota limit. Why? Because some &#8220;Senior Developer&#8221; decided to enable debug logging for a service that stores its entire state in a ConfigMap, which was being updated every 500ms. The raft log was bloated beyond recovery.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Root_Cause_Analysis_The_Etcd_Quorum_Collapse\"><\/span>Root Cause Analysis: The Etcd Quorum Collapse<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The core of the failure was the <code>etcd<\/code> cluster. In Kubernetes v1.29.2, <code>etcd<\/code> is the brain. If the brain has a stroke, the body starts twitching. We were seeing disk fdatasync latencies of over 500ms. For those of you who don&#8217;t live in the real world, <code>etcd<\/code> needs sub-10ms latency to maintain quorum.<\/p>\n<p>The disk I\/O was saturated because we were running the control plane on the same SSD array as the Prometheus WAL (Write-Ahead Log). When the metrics spike happened, Prometheus started churning IOPS, starving <code>etcd<\/code>. <code>etcd<\/code> lost quorum, the API server stopped responding, and the Kubelet on every node decided that since it couldn&#8217;t talk to the master, it should just stop working.<\/p>\n<p>But it gets worse. Because the nodes went <code>NotReady<\/code>, the controller-manager started trying to reschedule 4,000 pods simultaneously. This created a thundering herd problem that crushed the API server the moment I tried to bring it back up.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Networking_Hell_CNI_Encapsulation_and_MTU_Mismatches\"><\/span>Networking Hell: CNI Encapsulation and MTU Mismatches<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>While I was fighting <code>etcd<\/code>, the CNI (Container Network Interface) decided to join the party. We use Calico with VXLAN encapsulation. Somewhere in the chaos, the MTU (Maximum Transmission Unit) settings on the virtual interfaces got out of sync with the underlying physical network.<\/p>\n<p>The physical network was set to 1500, but with VXLAN overhead, the packets were hitting 1550 and getting dropped. This meant that even when a pod was &#8220;Running,&#8221; it couldn&#8217;t talk to the database. The <code>iptables<\/code> rules were a mess. <code>kube-proxy<\/code> had crashed, leaving behind a legacy of 15,000 orphaned rules that were being evaluated for every single packet.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\"># iptables -nvL | grep KUBE-SERVICES | wc -l\n15422\n<\/code><\/pre>\n<p>The latency for a simple <code>curl<\/code> between two pods was hitting 2000ms. I had to manually go into each node and flush the <code>conntrack<\/code> tables because they were full. The kernel parameter <code>net.netfilter.nf_conntrack_max<\/code> was set to the default 262144, which is a joke for a cluster this size. We were dropping packets at the edge because the connection tracking table was overflowing.<\/p>\n<p>I also found that <code>net.ipv4.ip_forward<\/code> had been disabled on two of the worker nodes by a &#8220;security hardening&#8221; script that someone ran via Ansible without testing it. You can&#8217;t run a Kubernetes node without IP forwarding. It\u2019s like trying to run a car without a fuel pump.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The_PVC_Deadlock_Finalizers_and_the_CSI_Ghost_in_the_Machine\"><\/span>The PVC Deadlock: Finalizers and the CSI Ghost in the Machine<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>By hour 30, I had <code>etcd<\/code> back up after a manual defrag and a quota increase. But then I hit the Persistent Volume Claim (PVC) wall. <\/p>\n<p>When the nodes went down, the pods were stuck in <code>Terminating<\/code>. Because the nodes were <code>NotReady<\/code>, the CSI (Container Storage Interface) couldn&#8217;t unmount the volumes. We had 200 volumes stuck in a &#8220;Multi-Attach&#8221; error state. The AWS EBS API was refusing to attach the volumes to new nodes because it thought they were still attached to the dead ones.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\"># kubectl describe pod api-service-7f5d6b8d-9z2x4\nEvents:\n  Type     Reason              Age    From                         Message\n  ----     ------              ----   ----                         -------\n  Warning  FailedAttachVolume  14m    attachdetach-controller      Multi-Attach error for volume &quot;pvc-0a1b2c3d&quot; Volume is already used by pod...\n  Warning  FailedMount         2m     kubelet, prod-worker-03      MountVolume.SetUp failed for volume &quot;pvc-0a1b2c3d&quot; : rpc error: code = Internal desc = volume is in use\n<\/code><\/pre>\n<p>The only way to fix this was a manual, surgical strike on the finalizers in the API. I had to write a bash loop to patch the JSON of every stuck PVC and Pod to remove the <code>kubernetes.io\/pvc-protection<\/code> and <code>foregroundDeletion<\/code> finalizers. This is dangerous. If you do this wrong, you end up with orphaned resources that you have to clean up in the cloud provider&#8217;s console like a janitor.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\"># Forcing deletion of stuck pods\nfor pod in $(kubectl get pods --all-namespaces | grep Terminating | awk '{print $2}'); do\n    kubectl patch pod $pod -p '{&quot;metadata&quot;:{&quot;finalizers&quot;:null}}' --type=merge\ndone\n<\/code><\/pre>\n<h2><span class=\"ez-toc-section\" id=\"The_%E2%80%9CFix%E2%80%9D_Patching_the_Control_Plane_with_Duct_Tape\"><\/span>The &#8220;Fix&#8221;: Patching the Control Plane with Duct Tape<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The &#8220;fix&#8221; wasn&#8217;t some elegant architectural shift. It was a series of desperate hacks to keep the cluster from collapsing again. <\/p>\n<p>First, I moved the <code>etcd<\/code> data directory to a dedicated NVMe drive. I had to manually edit the <code>\/etc\/kubernetes\/manifests\/etcd.yaml<\/code> file on all three master nodes. One tiny typo\u2014a single extra space before the <code>hostPath<\/code>\u2014and the kubelet would refuse to start the static pod. I spent forty minutes debugging a YAML indentation error while the CTO was breathing down my neck on Zoom.<\/p>\n<p>Second, I had to fix the OOMKiller behavior. The developers hadn&#8217;t set any <code>limits<\/code> on their sidecars. A logging agent was consuming 4GB of RAM because it was trying to buffer all the errors from the crash. I had to force-apply a <code>LimitRange<\/code> to the namespace to stop the bleeding.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\"># kubectl describe node prod-worker-01\nNon-terminated Pods:          (12 in total)\n  Namespace                   Name                                   CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age\n  ---------                   ----                                   ------------  ----------  ---------------  -------------  ---\n  production                  heavy-service-54d9f7-x89               2 (25%)       4 (50%)     8Gi (50%)        16Gi (100%)    12m\n  monitoring                  prometheus-node-exporter-2z4           100m (1%)     200m (2%)   256Mi (1%)       512Mi (3%)     142d\n<\/code><\/pre>\n<p>The <code>heavy-service<\/code> pod was set to a memory limit of 100% of the node&#8217;s capacity. When the kernel hit a low-memory condition, it didn&#8217;t just kill the pod; it killed the <code>containerd<\/code> process because it had a lower <code>oom_score_adj<\/code>. I had to manually tune the <code>kube-reserved<\/code> and <code>system-reserved<\/code> settings in the Kubelet configuration to ensure the OS had enough breathing room to actually function.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Post-Mortem_Architecture_Critique_The_Cost_of_Complexity\"><\/span>Post-Mortem Architecture Critique: The Cost of Complexity<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Why did this happen? It happened because we\u2019ve built a Rube Goldberg machine and called it &#8220;infrastructure.&#8221; <\/p>\n<p>We are using Kubernetes v1.29.2, which is great, but we are treating it like a magic black box. The initial design was flawed because it assumed the network was reliable and that disk I\/O was infinite. We put our control plane on a shared storage network. We didn&#8217;t set <code>PriorityClasses<\/code> for critical components, so when the cluster got squeezed, the API server was evicted to make room for a dev-environment Jenkins runner.<\/p>\n<p>The CNI was configured with default settings that work in a lab but fail in a high-throughput production environment. We didn&#8217;t account for the <code>iptables<\/code> overhead. With 1,500 services, every packet has to traverse a chain of rules that grows linearly. We should have moved to IPVS mode months ago, but &#8220;it wasn&#8217;t a priority.&#8221;<\/p>\n<p>And don&#8217;t get me started on the &#8220;Managed&#8221; Kubernetes service we use. They manage the control plane, sure, but they don&#8217;t manage the underlying VPC routing or the disk throughput limits. When <code>etcd<\/code> started failing, their &#8220;health check&#8221; just kept rebooting the nodes, which made the raft election cycle even more unstable. It was a feedback loop of failure.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Stabilization_The_Long_Road_Back\"><\/span>Stabilization: The Long Road Back<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>To stabilize the cluster, I had to perform a rolling restart of every single node, but not before I manually cleared the <code>containerd<\/code> shim leaks. <\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\"># Cleaning up orphaned shims\nps aux | grep containerd-shim | awk '{print $2}' | xargs kill -9\n<\/code><\/pre>\n<p>I then had to re-apply the CoreDNS ConfigMap. During the outage, CoreDNS had cached the NXDOMAIN responses for our internal services. Even when the pods came back up, the services couldn&#8217;t find each other because the DNS cache was poisoned with failure. I had to scale the CoreDNS deployment to zero and back up to three to clear the internal cache.<\/p>\n<p>The final step was fixing the <code>kube-proxy<\/code> mess. I switched the mode from <code>iptables<\/code> to <code>ipvs<\/code>. This required loading the <code>ip_vs<\/code>, <code>ip_vs_rr<\/code>, <code>ip_vs_wrr<\/code>, and <code>ip_vs_sh<\/code> kernel modules on every single worker node.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\"># Verifying IPVS modules\nlsmod | grep -i ip_vs\nip_vs_sh               16384  0\nip_vs_wrr              16384  0\nip_vs_rr               16384  0\nip_vs                 176128  6 ip_vs_rr,ip_vs_sh,ip_vs_wrr\nnf_conntrack          172032  1 ip_vs\n<\/code><\/pre>\n<p>Once IPVS was active, the CPU load on the worker nodes dropped by 20% instantly. The <code>KUBE-SERVICES<\/code> chain was no longer being evaluated for every packet.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The_Warning\"><\/span>The Warning<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>If you\u2019re reading this and you think your cluster is &#8220;safe&#8221; because you\u2019re using a cloud provider and a fancy CNI, you\u2019re delusional. You are one bad YAML commit away from a 72-hour shift.<\/p>\n<p>Stop ignoring your <code>etcd<\/code> metrics. If your <code>wal_fsync_duration_seconds<\/code> is spiking, you are on borrowed time. Stop letting developers deploy pods without resource requests and limits. An <code>OOMKiller<\/code> is not a cleanup strategy; it\u2019s a failure of governance.<\/p>\n<p>Check your kernel parameters. If <code>net.ipv4.ip_forward<\/code> isn&#8217;t 1, your cluster is a paperweight. If your <code>nf_conntrack_max<\/code> is still at the default, you\u2019re going to drop traffic the moment you get a marketing spike.<\/p>\n<p>And for the love of everything holy, stop putting your Prometheus data on the same physical disks as your <code>etcd<\/code> state. You\u2019re building a suicide pact, not a platform.<\/p>\n<p>I\u2019m going to sleep now. If the pager goes off again, I\u2019m throwing it into the ocean.<\/p>\n<hr \/>\n<h3><span class=\"ez-toc-section\" id=\"Technical_Appendix_Sub-System_Analysis\"><\/span>Technical Appendix: Sub-System Analysis<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<h4><span class=\"ez-toc-section\" id=\"The_CNI_Calico_Deep-Dive\"><\/span>The CNI (Calico) Deep-Dive<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>The failure of the CNI was exacerbated by the BGP peering flapping. When the nodes went <code>NotReady<\/code>, the BGP sessions timed out. Calico tried to withdraw the routes, but because the API server was down, the <code>calico-node<\/code> pods couldn&#8217;t update the IPAM (IP Address Management) state. This led to &#8220;IP exhaustion&#8221; errors even though we had thousands of available IPs. The fix required a manual reset of the BGP speaker nodes and a forced flush of the IPAM block affinities.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"The_CSI_EBS_Deep-Dive\"><\/span>The CSI (EBS) Deep-Dive<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>The EBS CSI driver (v1.25.0) has a known issue where it can get stuck in a loop if the <code>VolumeAttachment<\/code> object isn&#8217;t cleaned up by the external-attacher. In our case, the <code>external-attacher<\/code> was itself stuck in a <code>CrashLoopBackOff<\/code> because it couldn&#8217;t reach the API server. This is a circular dependency that Kubernetes documentation rarely mentions. If the thing that fixes the storage needs the storage to run, you\u2019re stuck in a deadlock.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"Kube-Proxy_and_Iptables\"><\/span>Kube-Proxy and Iptables<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>The transition to IPVS is not just a performance boost; it\u2019s a necessity for clusters exceeding 500 services. In <code>iptables<\/code> mode, the kernel uses a sequential search through the rules. In <code>ipvs<\/code> mode, it uses a hash table. The difference in O(n) vs O(1) complexity is the difference between a functional cluster and a pile of smoking silicon. We were at 1,500 services. We were already dead; we just hadn&#8217;t fallen over yet.<\/p>\n<h4><span class=\"ez-toc-section\" id=\"Resource_Limits_and_the_OOMKiller\"><\/span>Resource Limits and the OOMKiller<span class=\"ez-toc-section-end\"><\/span><\/h4>\n<p>When a pod exceeds its memory limit, the kernel sends a <code>SIGKILL<\/code>. However, if the node itself is out of memory, the kernel&#8217;s OOMKiller logic takes over. It looks at the <code>oom_score<\/code>. By default, the <code>kubelet<\/code> and <code>containerd<\/code> have a very low <code>oom_score_adj<\/code> to protect them. But if you don&#8217;t configure <code>eviction-hard<\/code> thresholds in your kubelet config, the node will lock up before the OOMKiller can even decide what to kill. We had no <code>eviction-hard<\/code> settings, so the kernel just panicked.<\/p>\n<p><strong>Final Stats:<\/strong><br \/>\n&#8211; <strong>Total Downtime<\/strong>: 14 hours (partial), 4 hours (total).<br \/>\n&#8211; <strong>Coffee Consumed<\/strong>: 18 cups.<br \/>\n&#8211; <strong>YAML Files Edited<\/strong>: 42.<br \/>\n&#8211; <strong>Will to Live<\/strong>: 0.2%.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Related_Articles\"><\/span>Related Articles<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Explore more insights and best practices:<\/p>\n<ul>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-to-streamline-your-workflow-2\/\">10 Devops Best Practices To Streamline Your Workflow 2<\/a><\/li>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/ai-cybersecurity-protecting-your-business-from-new-threats\/\">Ai Cybersecurity Protecting Your Business From New Threats<\/a><\/li>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/artificial-intelligence-best-practices-a-complete-guide-3\/\">Artificial Intelligence Best Practices A Complete Guide 3<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>It was 3:14 AM on a Tuesday when the first 503 errors started hitting the ingress-nginx logs, and by 3:20, the entire kubernetes cluster was a graveyard of CrashLoopBackOffs. I\u2019m writing this because if I don\u2019t document this nightmare now, I\u2019ll probably hallucinate that it was all a bad dream induced by too much caffeine &#8230; <a title=\"Mastering Your Kubernetes Cluster: A Step-by-Step Guide\" class=\"read-more\" href=\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/\" aria-label=\"Read more  on Mastering Your Kubernetes Cluster: A Step-by-Step Guide\">Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-4920","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.0 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Mastering Your Kubernetes Cluster: A Step-by-Step Guide - ITSupportWale<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Mastering Your Kubernetes Cluster: A Step-by-Step Guide - ITSupportWale\" \/>\n<meta property=\"og:description\" content=\"It was 3:14 AM on a Tuesday when the first 503 errors started hitting the ingress-nginx logs, and by 3:20, the entire kubernetes cluster was a graveyard of CrashLoopBackOffs. I\u2019m writing this because if I don\u2019t document this nightmare now, I\u2019ll probably hallucinate that it was all a bad dream induced by too much caffeine ... Read more\" \/>\n<meta property=\"og:url\" content=\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/\" \/>\n<meta property=\"og:site_name\" content=\"ITSupportWale\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Itsupportwale-298547177495978\" \/>\n<meta property=\"article:published_time\" content=\"2026-10-30T19:54:21+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2021\/05\/android-chrome-512x512-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"512\" \/>\n\t<meta property=\"og:image:height\" content=\"512\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Techie\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Techie\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/\"},\"author\":{\"name\":\"Techie\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d\"},\"headline\":\"Mastering Your Kubernetes Cluster: A Step-by-Step Guide\",\"datePublished\":\"2026-10-30T19:54:21+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/\"},\"wordCount\":1906,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#organization\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/\",\"url\":\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/\",\"name\":\"Mastering Your Kubernetes Cluster: A Step-by-Step Guide - ITSupportWale\",\"isPartOf\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#website\"},\"datePublished\":\"2026-10-30T19:54:21+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/itsupportwale.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Mastering Your Kubernetes Cluster: A Step-by-Step Guide\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#website\",\"url\":\"https:\/\/itsupportwale.com\/blog\/\",\"name\":\"ITSupportWale\",\"description\":\"Tips, Tricks, Fixed-Errors, Tutorials &amp; Guides\",\"publisher\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/itsupportwale.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#organization\",\"name\":\"itsupportwale\",\"url\":\"https:\/\/itsupportwale.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png\",\"contentUrl\":\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png\",\"width\":1119,\"height\":144,\"caption\":\"itsupportwale\"},\"image\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/Itsupportwale-298547177495978\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d\",\"name\":\"Techie\",\"sameAs\":[\"https:\/\/itsupportwale.com\",\"iswblogadmin\"],\"url\":\"https:\/\/itsupportwale.com\/blog\/author\/iswblogadmin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Mastering Your Kubernetes Cluster: A Step-by-Step Guide - ITSupportWale","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/","og_locale":"en_US","og_type":"article","og_title":"Mastering Your Kubernetes Cluster: A Step-by-Step Guide - ITSupportWale","og_description":"It was 3:14 AM on a Tuesday when the first 503 errors started hitting the ingress-nginx logs, and by 3:20, the entire kubernetes cluster was a graveyard of CrashLoopBackOffs. I\u2019m writing this because if I don\u2019t document this nightmare now, I\u2019ll probably hallucinate that it was all a bad dream induced by too much caffeine ... Read more","og_url":"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/","og_site_name":"ITSupportWale","article_publisher":"https:\/\/www.facebook.com\/Itsupportwale-298547177495978","article_published_time":"2026-10-30T19:54:21+00:00","og_image":[{"width":512,"height":512,"url":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2021\/05\/android-chrome-512x512-1.png","type":"image\/png"}],"author":"Techie","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Techie","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#article","isPartOf":{"@id":"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/"},"author":{"name":"Techie","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d"},"headline":"Mastering Your Kubernetes Cluster: A Step-by-Step Guide","datePublished":"2026-10-30T19:54:21+00:00","mainEntityOfPage":{"@id":"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/"},"wordCount":1906,"commentCount":0,"publisher":{"@id":"https:\/\/itsupportwale.com\/blog\/#organization"},"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/","url":"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/","name":"Mastering Your Kubernetes Cluster: A Step-by-Step Guide - ITSupportWale","isPartOf":{"@id":"https:\/\/itsupportwale.com\/blog\/#website"},"datePublished":"2026-10-30T19:54:21+00:00","breadcrumb":{"@id":"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/itsupportwale.com\/blog\/mastering-your-kubernetes-cluster-a-step-by-step-guide\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/itsupportwale.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Mastering Your Kubernetes Cluster: A Step-by-Step Guide"}]},{"@type":"WebSite","@id":"https:\/\/itsupportwale.com\/blog\/#website","url":"https:\/\/itsupportwale.com\/blog\/","name":"ITSupportWale","description":"Tips, Tricks, Fixed-Errors, Tutorials &amp; Guides","publisher":{"@id":"https:\/\/itsupportwale.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/itsupportwale.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/itsupportwale.com\/blog\/#organization","name":"itsupportwale","url":"https:\/\/itsupportwale.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png","contentUrl":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png","width":1119,"height":144,"caption":"itsupportwale"},"image":{"@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Itsupportwale-298547177495978"]},{"@type":"Person","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d","name":"Techie","sameAs":["https:\/\/itsupportwale.com","iswblogadmin"],"url":"https:\/\/itsupportwale.com\/blog\/author\/iswblogadmin\/"}]}},"_links":{"self":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts\/4920","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/comments?post=4920"}],"version-history":[{"count":0,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts\/4920\/revisions"}],"wp:attachment":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/media?parent=4920"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/categories?post=4920"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/tags?post=4920"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}