{"id":4853,"date":"2026-08-07T21:31:32","date_gmt":"2026-08-07T16:01:32","guid":{"rendered":"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/"},"modified":"2026-08-07T21:31:32","modified_gmt":"2026-08-07T16:01:32","slug":"10-devops-best-practices-for-faster-software-delivery-5","status":"publish","type":"post","link":"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/","title":{"rendered":"10 DevOps Best Practices for Faster Software Delivery"},"content":{"rendered":"<p><code>2024-10-24T03:14:07.821Z [ERROR] controller-runtime.manager.controller.pod-lifecycle-controller: Reconciler error {\"error\": \"failed to allocate IP from CIDR block 10.128.0.0\/24: address already in use by peer-vpc-02-us-east-1\", \"stacktrace\": \"github.com\/kubernetes-sigs\/controller-runtime\/pkg\/internal\/controller.(*Controller).reconcileHandler\\n\\t\/go\/pkg\/mod\/github.com\/kubernetes-sigs\/controller-runtime@v0.17.2\/pkg\/internal\/controller\/controller.go:329\"}<br \/>\n2024-10-24T03:14:08.001Z [WARN] kubelet: Readiness probe failed: HTTP probe failed with statuscode: 503 for container \"api-gateway\" (pod \"api-gateway-7f8d9b4c5-xk2m9_prod\")<br \/>\n2024-10-24T03:14:08.112Z [FATAL] kernel: [192837.441] Out of memory: Kill process 29384 (envoy) score 942 or sacrifice child<\/code><\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_80 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6a776873456bc\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6a776873456bc\"  aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#INCIDENT-882_The_Ghost_of_Overlapping_Subnets\" >INCIDENT-882: The Ghost of Overlapping Subnets<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#The_Readiness_Probe_Death_Spiral\" >The Readiness Probe Death Spiral<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#Why_Your_CICD_Pipeline_is_a_Glorified_Bash_Script\" >Why Your CI\/CD Pipeline is a Glorified Bash Script<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#The_Silent_Failure_of_Silent_Retries\" >The Silent Failure of Silent Retries<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#The_Containerd_Transition_and_the_Loss_of_Tooling\" >The Containerd Transition and the Loss of Tooling<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#Persistent_Volume_Claims_and_the_Lie_of_Statelessness\" >Persistent Volume Claims and the Lie of Statelessness<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#The_Service_Mesh_Tax_Istio_Envoy_and_the_50ms_Latency_Floor\" >The Service Mesh Tax: Istio, Envoy, and the 50ms Latency Floor<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#The_Silent_Failure_of_%E2%80%9CSelf-Healing%E2%80%9D_Systems\" >The Silent Failure of &#8220;Self-Healing&#8221; Systems<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#Technical_Debt_as_a_Physical_Force\" >Technical Debt as a Physical Force<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#The_Fallacy_of_the_%E2%80%9CSingle_Pane_of_Glass%E2%80%9D\" >The Fallacy of the &#8220;Single Pane of Glass&#8221;<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#A_Note_to_the_Junior_Who_Triggered_This\" >A Note to the Junior Who Triggered This<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#Related_Articles\" >Related Articles<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"INCIDENT-882_The_Ghost_of_Overlapping_Subnets\"><\/span>INCIDENT-882: The Ghost of Overlapping Subnets<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>It\u2019s 3:15 AM. I\u2019m staring at a monitor that\u2019s too bright for my aging retinas, smelling the phantom scent of ozone and dust that used to permeate the data centers where I started my career. Back then, if a Sun Fire V240 went down, you knew why. You could hear the fans screaming or see the &#8220;Service Required&#8221; LED glowing like a malevolent eye. Today, I\u2019m chasing ghosts in a virtualized, abstracted, containerized hellscape where the &#8220;hardware&#8221; is just a line item on a bill I\u2019m not allowed to see.<\/p>\n<p>The log above is the result of what happens when &#8220;marketing-led engineering&#8221; meets the reality of the networking stack. Someone in Product decided we needed a &#8220;multi-region global footprint&#8221; by the end of Q3. So, a junior developer\u2014bless their heart\u2014fired up a Terraform module they found on a medium post and applied it to our production environment. They didn&#8217;t check the existing VPC peering routes. They didn&#8217;t look at the routing tables. They just assumed the cloud would &#8220;handle it.&#8221;<\/p>\n<p>The result? A CIDR block collision that effectively black-holed half of our internal API traffic. The CNI plugin, trying to be helpful, kept attempting to assign IPs from a range that was already being advertised by a legacy VPC we use for database replication.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\"># Checking the routing table on the node... or what's left of it.\n$ ip route show table main\ndefault via 10.128.0.1 dev eth0 proto dhcp src 10.128.0.45 metric 100 \n10.128.0.0\/24 dev eth0 proto kernel scope link src 10.128.0.45 \n10.128.0.0\/24 via 10.200.1.1 dev tun0  # &lt;--- There's the culprit. Overlapping route from the VPN tunnel.\n169.254.169.254 dev eth0 proto php scope link \n<\/code><\/pre>\n<p>We followed every devops best practice listed in the glossy brochures. We had CI\/CD. We had automated testing. We had &#8220;Infrastructure as Code.&#8221; But Terraform 1.5+, for all its fancy <code>import<\/code> blocks and <code>check<\/code> assertions, doesn&#8217;t know that your legacy VPN gateway is going to squat on a subnet you just tried to provision for a new EKS node group. <\/p>\n<p>The <code>terraform plan<\/code> looked clean because the VPN gateway isn&#8217;t managed by Terraform. It\u2019s a &#8220;manual hotfix&#8221; from 2019 that became permanent infrastructure. This is the reality of the &#8220;modern&#8221; stack: a thin veneer of YAML covering a mountain of technical debt and undocumented manual changes.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The_Readiness_Probe_Death_Spiral\"><\/span>The Readiness Probe Death Spiral<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>While the networking team (which is just me and a guy named Dave who is currently on a flight to Tokyo) tries to unfurl the routing mess, the Kubernetes scheduler is making things worse. Because the CIDR collision is dropping packets, the readiness probes for the <code>api-gateway<\/code> pods are failing. <\/p>\n<p>In K8s 1.30, the kubelet is more aggressive than ever. It sees the 503, marks the pod as Unready, and pulls it from the Service endpoint list. Now, the remaining pods\u2014which are also struggling with intermittent packet loss\u2014are being slammed with 100% of the traffic. <\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\">$ kubectl describe pod api-gateway-7f8d9b4c5-xk2m9\nEvents:\n  Type     Reason     Age                   From               Message\n  ----     ------     ----                  ----               -------\n  Warning  Unhealthy  2m14s (x24 over 12m)  kubelet            Readiness probe failed: Get &quot;http:\/\/10.128.0.88:8080\/healthz&quot;: dial tcp 10.128.0.88:8080: connect: connection refused\n  Normal   Killing    2m14s                 kubelet            Container api-gateway failed liveness probe, will be restarted\n<\/code><\/pre>\n<p>The &#8220;self-healing&#8221; nature of Kubernetes is currently a circular firing squad. The liveness probe fails because the application is too busy handling a massive spike in retries from the frontend, which is also failing because the backend is unreachable. We\u2019ve built a system that is so &#8220;resilient&#8221; it will happily kill itself trying to stay alive.<\/p>\n<p>I\u2019ve seen this before. In 2005, we called it a &#8220;thundering herd.&#8221; In 2024, we call it &#8220;cloud-native scaling.&#8221; It\u2019s the same physics, just with more layers of indirection. We\u2019ve replaced simple socket errors with complex, multi-layered failure modes that require a PhD in distributed systems to debug at 3 AM.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Why_Your_CICD_Pipeline_is_a_Glorified_Bash_Script\"><\/span>Why Your CI\/CD Pipeline is a Glorified Bash Script<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The &#8220;devops best&#8221; crowd loves to talk about &#8220;shifting left.&#8221; They want developers to own the infrastructure. But when you give a developer a Terraform manifest, they don&#8217;t see a representation of physical hardware, subnets, and BGP sessions. They see a configuration file. They treat it like a JSON object.<\/p>\n<p>We\u2019ve moved from &#8220;Infrastructure as Code&#8221; to &#8220;Infrastructure as Manual Hotfixes&#8221; disguised as code. Look at our current pipeline. It\u2019s 4,000 lines of YAML spread across twenty different repositories. It uses &#8220;reusable workflows&#8221; that are so abstracted no one knows what they actually do. <\/p>\n<p>When the deployment failed tonight, the CI\/CD pipeline reported a &#8220;Success.&#8221; Why? Because the <code>helm upgrade<\/code> command returned an exit code of 0. Helm doesn&#8217;t care if the pods actually start; it only cares that the Tiller-less API server accepted the new manifest. <\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\"># The &quot;Successful&quot; deployment log\n$ helm upgrade --install api-gateway .\/charts\/api-gateway --namespace prod --wait --timeout 5m\nRelease &quot;api-gateway&quot; has been upgraded. Happy Helming!\nNAME: api-gateway\nLAST DEPLOYED: Thu Oct 24 03:10:02 2024\nNAMESPACE: prod\nSTATUS: deployed\nREVISIONS: 142\n<\/code><\/pre>\n<p>&#8220;Happy Helming!&#8221; The sheer audacity of that message while the production environment is actively melting down. The <code>--wait<\/code> flag timed out, but because of a bug in the wrapper script someone wrote three years ago, the error was swallowed and the pipeline moved on to the next stage: &#8220;Send Slack Notification.&#8221;<\/p>\n<p>So, while I\u2019m digging through <code>tcpdump<\/code> outputs, the rest of the company is getting a message saying the new feature was successfully deployed. This is the &#8220;synergy&#8221; of modern engineering.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The_Silent_Failure_of_Silent_Retries\"><\/span>The Silent Failure of Silent Retries<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>One of the most insidious things we\u2019ve introduced in the last five years is the service mesh. We\u2019re running Istio because someone read a whitepaper about &#8220;zero trust networking&#8221; and &#8220;observability.&#8221; Now, every single pod has an Envoy sidecar. <\/p>\n<p>When the CIDR collision started dropping packets, Envoy did what it was programmed to do: it started retrying. Exponential backoff, jitter, the whole nine yards. But it did it silently. The application code didn&#8217;t see a &#8220;Connection Refused&#8221; or a &#8220;Timeout.&#8221; It just saw increased latency.<\/p>\n<p>By the time the latency hit the 99th percentile and triggered an alert, the Envoy sidecars had already opened ten thousand connections to a non-existent endpoint, exhausting the conntrack table on the underlying worker nodes.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\"># Checking conntrack on the node\n$ sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max\nnet.netfilter.nf_conntrack_count = 262144\nnet.netfilter.nf_conntrack_max = 262144\n# We are at the limit. No new connections can be tracked. Everything drops.\n<\/code><\/pre>\n<p>This is the &#8220;observability&#8221; we were promised. We have dashboards that show &#8220;Gold Signals,&#8221; but they don&#8217;t show the kernel-level exhaustion that\u2019s actually killing the system. We\u2019ve built so many layers of abstraction that we\u2019ve lost sight of the machine. We\u2019re debugging YAML when we should be debugging the kernel.<\/p>\n<p>Back in the Solaris days, I could run <code>dtrace<\/code> and see exactly what the kernel was doing with a thread. Now, I have to jump through three different jump-hosts, get temporary IAM credentials, exec into a &#8220;debug container&#8221; that has the right tools, and hope that the <code>containerd<\/code> runtime hasn&#8217;t already recycled the namespace I&#8217;m trying to inspect.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The_Containerd_Transition_and_the_Loss_of_Tooling\"><\/span>The Containerd Transition and the Loss of Tooling<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Speaking of <code>containerd<\/code>, the transition away from the Docker shim was supposed to be &#8220;seamless&#8221; (a word I loathe). In reality, it broke every single one of our legacy monitoring scripts that relied on the Docker socket. <\/p>\n<p>We\u2019re running K8s 1.30 now. The <code>dockershim<\/code> is a distant memory, and we\u2019re stuck with <code>crictl<\/code>. It\u2019s technically superior, sure. It\u2019s more &#8220;standard-compliant.&#8221; But it\u2019s also another layer of friction when you\u2019re trying to figure out why a process is stuck in <code>D<\/code> state (uninterruptible sleep).<\/p>\n<p>I tried to <code>strace<\/code> the hanging process in the <code>api-gateway<\/code> pod. In the old days, I\u2019d just find the PID on the host and attach. Now, with cgroups v2 and the way <code>containerd<\/code> handles namespaces, it\u2019s a chore.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\"># Trying to find the actual PID of the process inside the container\n$ crictl inspectp 7f8d9b4c5 | grep pid\n&quot;pid&quot;: 29384\n$ strace -p 29384\nstrace: attach: ptrace(PTRACE_SEIZE, 29384): Operation not permitted\n# Ah, right. SecurityContext. AllowPrivilegeEscalation: false. \n# I have to edit the deployment YAML and restart the pod to debug it. \n# Which I can't do because the CIDR block is full.\n<\/code><\/pre>\n<p>This is the &#8220;devops best&#8221; practice of &#8220;Security by Default.&#8221; It\u2019s great for preventing hackers, but it\u2019s even better at preventing SREs from doing their jobs. We\u2019ve locked the doors so tightly that we can\u2019t get in to put out the fire.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Persistent_Volume_Claims_and_the_Lie_of_Statelessness\"><\/span>Persistent Volume Claims and the Lie of Statelessness<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The &#8220;marketing-led&#8221; push for &#8220;cloud-native&#8221; everything has led us to try and run stateful workloads in Kubernetes. We have a Kafka cluster running on top of EBS volumes managed by the AWS EBS CSI driver. <\/p>\n<p>When the nodes started failing due to the conntrack exhaustion, the scheduler tried to move the Kafka pods to &#8220;healthy&#8221; nodes. But the EBS volumes were still &#8220;attached&#8221; to the dead nodes. <\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\">$ kubectl get events -n kafka\nWarning  FailedAttachVolume  4m  attachdetach-controller  Multi-Attach error for volume &quot;pvc-88234-...&quot; Volume is already used by pod kafka-0 on node ip-10-128-0-45.ec2.internal\n<\/code><\/pre>\n<p>This is the &#8220;Multi-Attach&#8221; error that has haunted my dreams for the last three years. The cloud provider\u2019s API is slow, the CSI driver is buggy, and the Kubernetes controller is impatient. The result is a &#8220;zombie&#8221; volume that is stuck in &#8220;Attaching&#8221; state forever. <\/p>\n<p>The only way to fix it is to manually go into the AWS Console\u2014the ultimate admission of failure for an SRE\u2014and force-detach the volume. So much for &#8220;Infrastructure as Code.&#8221; It\u2019s &#8220;Infrastructure as a Series of Desperate Clicks in a Web UI.&#8221;<\/p>\n<p>We\u2019re told that containers are ephemeral. That we should treat our servers like cattle, not pets. But a 2TB Kafka partition isn&#8217;t a cow. It\u2019s a building. You can\u2019t just &#8220;reschedule&#8221; it and expect it to be fine. We\u2019ve taken the most difficult part of systems engineering\u2014state management\u2014and tried to hide it under a layer of YAML. It didn&#8217;t work. It just made the failures more opaque.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The_Service_Mesh_Tax_Istio_Envoy_and_the_50ms_Latency_Floor\"><\/span>The Service Mesh Tax: Istio, Envoy, and the 50ms Latency Floor<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>We were promised that Istio would give us &#8220;insights.&#8221; What it actually gave us was a 50ms latency floor on every internal request and a memory footprint that rivals the actual application. <\/p>\n<p>Every time a packet travels from Service A to Service B, it has to go through:<br \/>\n1. The application&#8217;s network stack.<br \/>\n2. The Envoy sidecar&#8217;s network stack.<br \/>\n3. The host&#8217;s iptables (which are a mess of <code>ISTIO_INBOUND<\/code> and <code>ISTIO_OUTPUT<\/code> chains).<br \/>\n4. The virtual ethernet pair.<br \/>\n5. The bridge.<br \/>\n6. The physical (virtual) NIC.<br \/>\n7. And then the whole thing in reverse on the other side.<\/p>\n<p>We\u2019ve added six layers of indirection to a simple TCP handshake. And for what? So we can have a pretty graph in Kiali that shows us the traffic is failing? I could have told you the traffic was failing by looking at the <code>stderr<\/code> logs, which are free.<\/p>\n<p>The &#8220;devops best&#8221; approach would be to simplify. To use standard Linux tools. To trust the kernel. But there\u2019s no money in simplicity. There are no &#8220;Certified Simplicity Architect&#8221; badges you can put on your LinkedIn profile. So we build these Rube Goldberg machines and act surprised when they break in ways we didn&#8217;t anticipate.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The_Silent_Failure_of_%E2%80%9CSelf-Healing%E2%80%9D_Systems\"><\/span>The Silent Failure of &#8220;Self-Healing&#8221; Systems<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The most dangerous part of the modern stack is the belief that the system can fix itself. The &#8220;Horizontal Pod Autoscaler&#8221; (HPA) is a perfect example. <\/p>\n<p>Tonight, as the latency increased, the HPA saw the CPU usage spike (because Envoy was burning cycles on retries). It decided we needed more pods. So it started spinning up new <code>api-gateway<\/code> instances. <\/p>\n<p>But each new pod needs an IP address. And where do those IP addresses come from? The CIDR block that is already exhausted. <\/p>\n<p>So the HPA is trying to scale up, which is triggering more CNI errors, which is putting more pressure on the Kube-API server, which is already struggling because the etcd cluster is on the same overloaded network. It\u2019s a feedback loop of failure.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\"># HPA trying to be helpful\n$ kubectl get hpa\nNAME          REFERENCE                TARGETS    MINPODS   MAXPODS   REPLICAS   AGE\napi-gateway   Deployment\/api-gateway   240%\/50%   5         50        32         142d\n<\/code><\/pre>\n<p>32 replicas. All of them &#8220;Pending&#8221; or &#8220;CrashLoopBackOff.&#8221; All of them consuming IP addresses that don&#8217;t exist. All of them contributing to the noise in the logs. <\/p>\n<p>In the old days, if a server was overloaded, it just got slow. You could see the load average climb. You could see the swap space filling up. Now, the system tries to &#8220;help&#8221; by adding more fuel to the fire. We\u2019ve automated our own destruction.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Technical_Debt_as_a_Physical_Force\"><\/span>Technical Debt as a Physical Force<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>People talk about technical debt like it\u2019s an abstract concept, like a credit card balance you\u2019ll eventually pay off. It\u2019s not. It\u2019s a physical force. It\u2019s the friction that makes every change harder. It\u2019s the &#8220;manual hotfix&#8221; from five years ago that is now a &#8220;critical dependency.&#8221;<\/p>\n<p>Tonight\u2019s incident wasn&#8217;t caused by a single bug. It was caused by the accumulation of &#8220;good enough&#8221; decisions.<br \/>\n&#8211; The &#8220;good enough&#8221; VPC design from 2019.<br \/>\n&#8211; The &#8220;good enough&#8221; CI\/CD pipeline that swallows errors.<br \/>\n&#8211; The &#8220;good enough&#8221; monitoring that doesn&#8217;t look at the kernel.<br \/>\n&#8211; The &#8220;good enough&#8221; Terraform module that doesn&#8217;t check for existing routes.<\/p>\n<p>When you follow &#8220;devops best&#8221; practices without understanding the underlying systems, you\u2019re just building a faster way to fail. You\u2019re automating the chaos instead of eliminating it.<\/p>\n<p>I\u2019m looking at the <code>strace<\/code> output now. I finally got into a debug container.<\/p>\n<pre class=\"codehilite\"><code class=\"language-bash\">$ strace -f -p 29384\n[pid 29384] epoll_wait(6, [], 1024, 0)  = 0\n[pid 29384] epoll_wait(6, [], 1024, 0)  = 0\n[pid 29384] epoll_wait(6, [], 1024, 0)  = 0\n[pid 29384] epoll_wait(6, [{events=EPOLLIN, data={u32=11, u64=11}}], 1024, 100) = 1\n[pid 29384] read(11, 0xc000456000, 4096) = -1 EAGAIN (Resource temporarily unavailable)\n<\/code><\/pre>\n<p><code>EAGAIN<\/code>. The universal sign that the system is screaming &#8220;I can&#8217;t keep up!&#8221; The application is fine. The code is perfect. The &#8220;business logic&#8221; is flawless. But the environment it\u2019s running in is a toxic wasteland of overlapping subnets and exhausted tables.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The_Fallacy_of_the_%E2%80%9CSingle_Pane_of_Glass%E2%80%9D\"><\/span>The Fallacy of the &#8220;Single Pane of Glass&#8221;<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Marketing loves to sell the &#8220;single pane of glass.&#8221; The one dashboard that tells you everything. Tonight, that pane of glass is a lie. <\/p>\n<p>The Datadog dashboard says the &#8220;Service Health&#8221; is 99.9%. Why? Because it\u2019s measuring the success rate of the load balancer, which is successfully returning 503s. To the load balancer, a 503 is a &#8220;successful&#8221; response\u2014it successfully communicated that the backend is dead.<\/p>\n<p>The &#8220;Error Rate&#8221; widget is green because we\u2019re filtering out 5xx errors that are &#8220;expected&#8221; during a deployment. <\/p>\n<p>We\u2019ve spent millions of dollars on &#8220;observability&#8221; tools, and I\u2019m still sitting here using <code>ip route<\/code> and <code>tcpdump<\/code> like it\u2019s 1998. Because at the end of the day, the only thing that matters is the packet. The packet doesn&#8217;t care about your &#8220;holistic&#8221; view or your &#8220;synergy.&#8221; It only cares if there\u2019s a route to the destination.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"A_Note_to_the_Junior_Who_Triggered_This\"><\/span>A Note to the Junior Who Triggered This<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>You\u2019re probably sleeping right now. You\u2019ll wake up tomorrow, see the Slack thread with 400 messages, and feel a pit in your stomach. You\u2019ll think you failed because you didn&#8217;t follow the &#8220;devops best&#8221; practices.<\/p>\n<p>You didn&#8217;t fail. The system failed you. It was designed to be too complex to understand, too fragile to change, and too opaque to debug. You were told that YAML was &#8220;code&#8221; and that the cloud was &#8220;abstracted.&#8221; You were lied to.<\/p>\n<p>The cloud is just someone else\u2019s computer, and it\u2019s running the same buggy Linux kernel we\u2019ve been using for decades. The abstractions are just masks. <\/p>\n<p>If you want to survive the next twenty years in this industry, stop reading &#8220;Thought Leadership&#8221; blogs and start reading the man pages. Learn how <code>iptables<\/code> actually works. Learn the difference between <code>L4<\/code> and <code>L7<\/code>. Learn why a <code>\/24<\/code> subnet is different from a <code>\/23<\/code>. <\/p>\n<p>And for the love of all that is holy, never trust a tool that tells you &#8220;Happy Helming!&#8221;<\/p>\n<p>Now, I have to go manually delete thirty-two &#8220;Pending&#8221; pods and force-detach an EBS volume because the &#8220;self-healing&#8221; automation is currently stuck in a loop. <\/p>\n<p>Go back to sleep. The &#8220;modern&#8221; stack will still be broken when you wake up.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Related_Articles\"><\/span>Related Articles<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Explore more insights and best practices:<\/p>\n<ul>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/machine-learning-models-a-complete-guide-for-beginners\/\">Machine Learning Models A Complete Guide For Beginners<\/a><\/li>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/3-simple-ways-to-create-bootable-usb-in-ubuntu-linux\/\">3 Simple Ways To Create Bootable Usb In Ubuntu Linux<\/a><\/li>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/10-python-best-practices-every-developer-should-know\/\">10 Python Best Practices Every Developer Should Know<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>2024-10-24T03:14:07.821Z [ERROR] controller-runtime.manager.controller.pod-lifecycle-controller: Reconciler error {&#8220;error&#8221;: &#8220;failed to allocate IP from CIDR block 10.128.0.0\/24: address already in use by peer-vpc-02-us-east-1&#8221;, &#8220;stacktrace&#8221;: &#8220;github.com\/kubernetes-sigs\/controller-runtime\/pkg\/internal\/controller.(*Controller).reconcileHandler\\n\\t\/go\/pkg\/mod\/github.com\/kubernetes-sigs\/controller-runtime@v0.17.2\/pkg\/internal\/controller\/controller.go:329&#8221;} 2024-10-24T03:14:08.001Z [WARN] kubelet: Readiness probe failed: HTTP probe failed with statuscode: 503 for container &#8220;api-gateway&#8221; (pod &#8220;api-gateway-7f8d9b4c5-xk2m9_prod&#8221;) 2024-10-24T03:14:08.112Z [FATAL] kernel: [192837.441] Out of memory: Kill process 29384 (envoy) score 942 or sacrifice child &#8230; <a title=\"10 DevOps Best Practices for Faster Software Delivery\" class=\"read-more\" href=\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/\" aria-label=\"Read more  on 10 DevOps Best Practices for Faster Software Delivery\">Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-4853","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.0 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>10 DevOps Best Practices for Faster Software Delivery - ITSupportWale<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"10 DevOps Best Practices for Faster Software Delivery - ITSupportWale\" \/>\n<meta property=\"og:description\" content=\"2024-10-24T03:14:07.821Z [ERROR] controller-runtime.manager.controller.pod-lifecycle-controller: Reconciler error {&quot;error&quot;: &quot;failed to allocate IP from CIDR block 10.128.0.0\/24: address already in use by peer-vpc-02-us-east-1&quot;, &quot;stacktrace&quot;: &quot;github.com\/kubernetes-sigs\/controller-runtime\/pkg\/internal\/controller.(*Controller).reconcileHandlernt\/go\/pkg\/mod\/github.com\/kubernetes-sigs\/controller-runtime@v0.17.2\/pkg\/internal\/controller\/controller.go:329&quot;} 2024-10-24T03:14:08.001Z [WARN] kubelet: Readiness probe failed: HTTP probe failed with statuscode: 503 for container &quot;api-gateway&quot; (pod &quot;api-gateway-7f8d9b4c5-xk2m9_prod&quot;) 2024-10-24T03:14:08.112Z [FATAL] kernel: [192837.441] Out of memory: Kill process 29384 (envoy) score 942 or sacrifice child ... Read more\" \/>\n<meta property=\"og:url\" content=\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/\" \/>\n<meta property=\"og:site_name\" content=\"ITSupportWale\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Itsupportwale-298547177495978\" \/>\n<meta property=\"article:published_time\" content=\"2026-08-07T16:01:32+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2021\/05\/android-chrome-512x512-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"512\" \/>\n\t<meta property=\"og:image:height\" content=\"512\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Techie\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Techie\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"14 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/\"},\"author\":{\"name\":\"Techie\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d\"},\"headline\":\"10 DevOps Best Practices for Faster Software Delivery\",\"datePublished\":\"2026-08-07T16:01:32+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/\"},\"wordCount\":2450,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#organization\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/\",\"url\":\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/\",\"name\":\"10 DevOps Best Practices for Faster Software Delivery - ITSupportWale\",\"isPartOf\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#website\"},\"datePublished\":\"2026-08-07T16:01:32+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/itsupportwale.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"10 DevOps Best Practices for Faster Software Delivery\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#website\",\"url\":\"https:\/\/itsupportwale.com\/blog\/\",\"name\":\"ITSupportWale\",\"description\":\"Tips, Tricks, Fixed-Errors, Tutorials &amp; Guides\",\"publisher\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/itsupportwale.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#organization\",\"name\":\"itsupportwale\",\"url\":\"https:\/\/itsupportwale.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png\",\"contentUrl\":\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png\",\"width\":1119,\"height\":144,\"caption\":\"itsupportwale\"},\"image\":{\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/Itsupportwale-298547177495978\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d\",\"name\":\"Techie\",\"sameAs\":[\"https:\/\/itsupportwale.com\",\"iswblogadmin\"],\"url\":\"https:\/\/itsupportwale.com\/blog\/author\/iswblogadmin\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"10 DevOps Best Practices for Faster Software Delivery - ITSupportWale","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/","og_locale":"en_US","og_type":"article","og_title":"10 DevOps Best Practices for Faster Software Delivery - ITSupportWale","og_description":"2024-10-24T03:14:07.821Z [ERROR] controller-runtime.manager.controller.pod-lifecycle-controller: Reconciler error {\"error\": \"failed to allocate IP from CIDR block 10.128.0.0\/24: address already in use by peer-vpc-02-us-east-1\", \"stacktrace\": \"github.com\/kubernetes-sigs\/controller-runtime\/pkg\/internal\/controller.(*Controller).reconcileHandlernt\/go\/pkg\/mod\/github.com\/kubernetes-sigs\/controller-runtime@v0.17.2\/pkg\/internal\/controller\/controller.go:329\"} 2024-10-24T03:14:08.001Z [WARN] kubelet: Readiness probe failed: HTTP probe failed with statuscode: 503 for container \"api-gateway\" (pod \"api-gateway-7f8d9b4c5-xk2m9_prod\") 2024-10-24T03:14:08.112Z [FATAL] kernel: [192837.441] Out of memory: Kill process 29384 (envoy) score 942 or sacrifice child ... Read more","og_url":"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/","og_site_name":"ITSupportWale","article_publisher":"https:\/\/www.facebook.com\/Itsupportwale-298547177495978","article_published_time":"2026-08-07T16:01:32+00:00","og_image":[{"width":512,"height":512,"url":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2021\/05\/android-chrome-512x512-1.png","type":"image\/png"}],"author":"Techie","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Techie","Est. reading time":"14 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#article","isPartOf":{"@id":"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/"},"author":{"name":"Techie","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d"},"headline":"10 DevOps Best Practices for Faster Software Delivery","datePublished":"2026-08-07T16:01:32+00:00","mainEntityOfPage":{"@id":"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/"},"wordCount":2450,"commentCount":0,"publisher":{"@id":"https:\/\/itsupportwale.com\/blog\/#organization"},"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/","url":"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/","name":"10 DevOps Best Practices for Faster Software Delivery - ITSupportWale","isPartOf":{"@id":"https:\/\/itsupportwale.com\/blog\/#website"},"datePublished":"2026-08-07T16:01:32+00:00","breadcrumb":{"@id":"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/itsupportwale.com\/blog\/10-devops-best-practices-for-faster-software-delivery-5\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/itsupportwale.com\/blog\/"},{"@type":"ListItem","position":2,"name":"10 DevOps Best Practices for Faster Software Delivery"}]},{"@type":"WebSite","@id":"https:\/\/itsupportwale.com\/blog\/#website","url":"https:\/\/itsupportwale.com\/blog\/","name":"ITSupportWale","description":"Tips, Tricks, Fixed-Errors, Tutorials &amp; Guides","publisher":{"@id":"https:\/\/itsupportwale.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/itsupportwale.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/itsupportwale.com\/blog\/#organization","name":"itsupportwale","url":"https:\/\/itsupportwale.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png","contentUrl":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png","width":1119,"height":144,"caption":"itsupportwale"},"image":{"@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Itsupportwale-298547177495978"]},{"@type":"Person","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d","name":"Techie","sameAs":["https:\/\/itsupportwale.com","iswblogadmin"],"url":"https:\/\/itsupportwale.com\/blog\/author\/iswblogadmin\/"}]}},"_links":{"self":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts\/4853","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/comments?post=4853"}],"version-history":[{"count":0,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts\/4853\/revisions"}],"wp:attachment":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/media?parent=4853"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/categories?post=4853"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/tags?post=4853"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}