json
{
“error”: {
“code”: “DeploymentFailed”,
“message”: “At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.”,
“details”: [
{
“code”: “Conflict”,
“message”: “The resource operation completed with terminal provisioning state ‘Failed’. The resource ‘appgw-prod-eastus2’ is currently busy and cannot be updated. Please try again later.”,
“target”: “/subscriptions/8888-dead-beef-9999/resourceGroups/prod-rg/providers/Microsoft.Network/applicationGateways/appgw-prod-eastus2”
}
]
}
}
I’m sitting in a chair that feels like it’s made of shards of glass. My eyes are vibrating. I’ve consumed enough caffeine to restart a dead star, and yet, here I am, staring at a terminal that’s been spitting out the same "Conflict" error for the last four hours. We just finished a 72-hour marathon. A regional outage in East US 2 turned our "highly available" infrastructure into a smoking crater.
If you’re a junior dev looking for a "cloud-native journey," go find a medium post written by a marketing intern. This isn't that. This is the reality of managing **azure** when the wheels fall off. It’s not pretty. It’s not "seamless." It’s a knife fight in a dark room with a provider that doesn't care about your sleep schedule.
## The 2:14 AM PagerDuty Heart Attack
It started with a single ping. Then a flood. The kind of flood that makes your phone sound like a Geiger counter in Chernobyl. Our AKS (Azure Kubernetes Service) clusters started dropping nodes like flies. The initial thought? A bad deployment. Maybe someone pushed a container image with a memory leak.
Wrong.
The nodes weren't just crashing; they were disappearing. The underlying Virtual Machine Scale Sets (VMSS) were entering a "Failed" state. When you see a VMSS go into a failed state during a regional surge, you’re already dead; you just haven't stopped twitching yet. I fired up the **azure** portal—which, by the way, was crawling at the speed of tectonic plates—and saw the dreaded "Service Health" warning. "Increased latency and connectivity issues." Translation: The region is melting.
I tried to scale out. I tried to move the workloads. But the CLI (Azure CLI 2.61.0) just hung there, mocking me.
```bash
# Trying to get some semblance of what's happening
az aks show --resource-group prod-rg --name k8s-cluster --query "agentPoolProfiles[].provisioningState"
# Result: ["Failed"]
# Trying to force a manual scale to kick the provider into gear
az aks scale --resource-group prod-rg --name k8s-cluster --node-count 10
# Error: Operation 'Microsoft.Compute/virtualMachineScaleSets/write' is not allowed
# because the resource is in a failed state.
Table of Contents
Stage 1: Denial and the Load Balancer’s Lies
The first stage of grief in an outage is denial. You tell yourself the status page is green, so it must be your code. You check the Application Gateway. The health probes are failing. You think, “Okay, I’ll just restart the backend pools.”
But the Application Gateway is a black box of misery. We were running the Standard_v2 SKU, which is supposed to be the “robust” version. During a regional event, the App Gateway doesn’t just fail; it gaslights you. It tells you the backend is unhealthy, but when you curl the pod IP directly from a jumpbox, it responds in 20ms.
The Load Balancer was lying. The control plane for the network stack was so saturated that the health probe signals were getting lost in the ether. I spent six hours tweaking probe timeouts and thresholds, thinking I could “optimize” my way out of a provider-level collapse.
Hard truth: If the region is under stress, your health probes are the first thing to go. If you’ve set your “Unhealthy Threshold” to 3, you’re going to flapping heaven. I had to bump it to 10 just to stop the gateway from cycling the few remaining healthy nodes.
Stage 2: Anger and the Terraform Provider v3.94.0 Purgatory
By hour 14, I was angry. I decided to redeploy the entire networking stack in a different region. We use Terraform. We’re “modern.” We have “Infrastructure as Code.”
Terraform provider v3.94.0 decided this was the perfect time to have a stroke. Because the resources in East US 2 were in a “Failed” provisioning state, Terraform couldn’t refresh the state file. It kept trying to read the properties of a Public IP that was stuck in a “Deleting” loop.
# The block that failed us
resource "azurerm_public_ip" "pip" {
name = "prod-pip"
location = var.location
resource_group_name = azurerm_resource_group.rg.name
allocation_method = "Static"
sku = "Standard" # If this was "Basic", we'd be even more screwed
}
The state was locked. I had to go into the azure blob storage where our .tfstate was kept and manually break the lease. Do you know how terrifying it is to manually break a state lease while your production environment is a ghost town? One wrong click and you’ve deleted the record of everything you own.
The “Standard” SKU for Public IPs is supposed to be zone-redundant. It doesn’t matter if the zone is redundant if the API that manages the IP is unresponsive. I sat there for two hours watching a “Destroying…” message in my terminal. It never finished. I had to go into the Resource Explorer and manually edit the JSON of the resource to strip out the finalizers just to get it to die.
Stage 3: Bargaining with Private Link and DNS
By hour 30, the anger had turned into desperate bargaining. “If I can just get the database connectivity back, we can serve a static maintenance page.”
We use Private Link for everything. It’s “secure.” It’s “best practice.” It’s also a nightmare when DNS propagation fails. When you’re using azure Private DNS Zones, you’re relying on a hidden, magical service that maps your internal names to private IPs.
During the outage, the DNS resolution for our SQL Managed Instance started failing. Not because the DB was down, but because the VNet link to the Private DNS zone had “ghosted.” I could ping the IP, but the connection string using the FQDN failed.
I tried to recreate the VNet link.
az network private-dns link vnet create \
-g prod-rg \
-n link-to-prod-vnet \
-z privatelink.database.windows.net \
-v prod-vnet \
-e false
The command returned a 500 Internal Server Error. Not a 404. Not a 403. A 500. The underlying infrastructure for DNS management was overwhelmed. I had to resort to the ultimate sin: I hardcoded the private IP addresses into the application’s host files via a Kubernetes ConfigMap. It’s disgusting. It’s brittle. It worked.
If you think your “automated” DNS is going to save you during a regional failure, you’re dreaming. You need a fallback. You need to know your IPs.
Wrestling with the Load Balancer’s Lies
Let’s talk about the SKU trap. This is where azure really gets you. We had some legacy dev environments running on the “Basic” Load Balancer SKU. When we tried to bridge them to the “Standard” VNet during the recovery, we hit a brick wall.
You cannot mix Basic and Standard SKUs in the same VNet for certain operations. You cannot upgrade a Basic Public IP to a Standard Public IP without deleting it. During an outage, “deleting” is a suggestion, not a command.
The Load Balancer (Standard) was refusing to attach to the backend pool because one of the NICs had a lingering association with a Basic SKU resource that didn’t even exist anymore. I had to use the azure CLI to manually detach the network interface cards from the backend pools, but the NICs were “locked” by the VMSS.
It’s a circular dependency from hell. The VMSS won’t update because the NIC is failed. The NIC won’t update because the Load Balancer is busy. The Load Balancer is busy because the API is throttled.
I spent four hours writing a Python script to loop through the NICs and try to force-update them every 30 seconds, hoping to catch a window where the API wasn’t choking.
import os
import time
# The "I'm desperate" loop
while True:
print("Attempting to kick the NIC...")
result = os.system("az network nic update --resource-group prod-rg --name prod-nic-01")
if result == 0:
print("Success. Finally.")
break
print("Still failing. Sleeping for 30s...")
time.sleep(30)
Stage 4: Depression and the Portal UI
Hour 50. This is when the sleep deprivation starts causing hallucinations. I thought I saw the “Credits” balance in our billing portal dropping in real-time.
The azure portal is a masterpiece of frustration. When the underlying services are failing, the UI fails in the most unhelpful ways. Blades won’t load. Buttons stay grayed out. You click “Restart” on a VM, and nothing happens for ten minutes, then you get a notification saying “Failed to restart: Unknown Error.”
I stopped using the portal entirely. It’s a resource hog that provides no real data during a crisis. I lived in the az CLI and the raw REST API. If you’re an SRE and you don’t know how to construct a raw PUT request to the Resource Manager API, you’re going to have a bad time.
The metrics were gone, too. Azure Monitor was delayed by 45 minutes. I was trying to debug a latency spike that happened nearly an hour ago. You’re flying a plane blind, and the instruments are telling you where you were, not where you are. We had to rely on our internal Prometheus/Grafana stack, which, thankfully, was running on a separate small cluster in a different region.
Pro-tip: Never, ever rely on the cloud provider’s native monitoring during an outage of that same provider. It’s like asking a fire to tell you how hot it is.
The SKU Trap: Why Basic Tier is a Death Sentence
If you are running anything in production on a “Basic” SKU, you are asking for a disaster. Basic SKUs don’t have availability zones. They don’t have the same SLA. But more importantly, they don’t have the same priority in the provider’s recovery queue.
During this 72-hour nightmare, the “Standard” resources started coming back online around hour 40. The “Basic” resources? They stayed dead until the very end.
We had a “Basic” tier VPN Gateway that we used for a legacy branch office. It didn’t just go down; it deleted its own configuration. I don’t know how. I don’t know why. But when the service finally responded, the shared key was gone.
I had to rebuild the VPN tunnel from scratch. But wait! The Public IP associated with the VPN was “Basic,” and now the region was so capacity-constrained that azure wouldn’t let me provision a new “Basic” IP. I was forced to upgrade to a “Standard” VPN Gateway, which costs four times as much and took 45 minutes to provision.
It felt like a protection racket. “Oh, your cheap gateway died? Pay us more and we might let you back online.”
Stage 5: Acceptance and the Final Stand
Hour 65. The region was stabilizing. Most of the services were “Green” on the status page (even though they were still flaky).
The final stage is acceptance. You accept that you aren’t going to get those 72 hours back. You accept that your “High Availability” architecture was a house of cards.
We had to perform a “Nuke and Pave.” We deleted the entire production resource group. It was the only way to clear the “Failed” provisioning states that were clogging up the ARM (Azure Resource Manager) engine.
We ran the Terraform apply from a clean state.
terraform destroy -target=module.networking # Praying this works
terraform apply -auto-approve
It took three hours for the App Gateway to provision. Three hours of staring at a spinning cursor. But because we were starting fresh, the circular dependencies were gone. The NICs attached to the Load Balancer. The Private Link endpoints found their DNS zones. The AKS nodes joined the cluster.
By hour 72, the last service was back online. I sent a “System Restored” email to the stakeholders, closed my laptop, and sat in the dark for a long time.
Hard-Won Wisdom
You survived the read. Now, here is the “survival guide” for the next time azure decides to take a nap.
- Region Redundancy is a Lie Unless You Test It: If you aren’t actively running in two regions simultaneously, you don’t have a DR plan. You have a “hope” plan. Moving to a new region during an outage is impossible because everyone else is trying to do the same thing. The capacity isn’t there. Provision your DR footprint now, while things are quiet.
- Standard SKUs Only: Delete every Basic SKU resource you have. They are a liability. They will fail you when you need them most, and they will be the last things fixed.
- Terraform Lifecycle Blocks are Your Friend: Use
ignore_changeson things like tag updates or minor properties that might get stuck in a “Failed” state. It allows Terraform to keep managing the resource even if the provider is acting up. - Know the REST API: When the CLI fails and the portal is dead,
curlis your only friend. Learn how to authenticate via service principal and hit the ARM API directly. - DNS Fallbacks: Don’t trust Private DNS zones to work 100% of the time. Have a way to inject host entries into your containers. It’s ugly, but it keeps the traffic flowing.
- Throttling is Real: During a regional recovery, the ARM API will throttle you. Hard. If you have scripts that poll the API, put aggressive backoffs in them. If you hit the limit, you might get blacklisted for an hour.
- The Portal is for Viewing, Not Doing: Do your work in code. The portal is too slow and too prone to state mismatches during a crisis.
I’m going to sleep now. My pager is silent, but I don’t trust it. I don’t trust the “Green” status icons. I don’t trust the “99.99%” promises. In the world of cloud infrastructure, you are only as good as your last successful terraform apply.
And mine was a nightmare.
Stay cynical. Keep your state files backed up. And for the love of everything, stop using Basic SKUs.
End of Report.
Related Articles
Explore more insights and best practices: