The Ghost in the Machine: A 72-Hour Descent into the Azure Portal
The cursor is mocking me. It’s a steady, rhythmic blink in a PuTTY window that hasn’t seen a packet in four minutes. I’m sitting in a room that is too quiet, too warm, and smells like nothing but stale coffee and dry wall. For twenty years, I lived in the hum. I lived in the 80-decibel scream of 1U fans, the biting chill of a hot-aisle/cold-aisle containment system, and the comforting, ozone-heavy scent of a short-circuiting PDU. I knew where the wires went. I could touch the silicon. If a disk died, I heard the click-of-death, I saw the amber light, and I swapped the damn thing.
Now? Now I’m “modernizing.” Now I’m “cloud-native.” Now I’m staring at a spinning blue circle in a browser tab while azure decides whether or not it feels like acknowledging my existence.
It started with a simple migration. Lift and shift, they said. It’ll be easy, they said. Seventy-two hours later, I am a shell of a man, trapped in a recursive loop of Resource Groups and Tenant IDs. This is my war journal.
Table of Contents
02:00 – The Resource Group Purgatory
The first sign of the rot appeared two hours into the migration. I had scripted the teardown of a staging environment to clear the path for the production import. I ran the command. I waited. The CLI returned a success code, but the portal—that bloated, Javascript-heavy monstrosity—insisted the resources were still there.
I tried to delete the Resource Group manually. The portal hung. I refreshed. The Resource Group was “Deleting,” a state of limbo that apparently lasts longer than a physical server’s entire lifecycle. I dropped to the terminal.
# azure-cli version 2.61.0
# Attempting to force delete a hung resource group
az group delete --name RG-Prod-Alpha --yes --no-wait
# Resulting output after 300 seconds of polling:
{
"error": {
"code": "ResourceGroupNotFound",
"message": "Resource group 'RG-Prod-Alpha' could not be found."
}
}
And yet, there it was. In the UI, RG-Prod-Alpha sat there, mocking me, containing a single, orphaned Public IP address that refused to die because it was “in use” by a Network Interface that I had already deleted. This is the azure paradox: a resource can simultaneously not exist and be busy. In a real rack, if I pull the cable, the connection is gone. In the cloud, the ghost of the cable haunts the hypervisor for an eternity.
I spent forty minutes clicking “Refresh.” Every click is a reminder that I no longer control the hardware. I am a tenant in a building where the landlord is an invisible algorithm that doesn’t answer the phone. The UI lag is palpable. When you have more than 50 subscriptions, the portal doesn’t just slow down; it begins to disintegrate. The search bar stops searching. The “All Resources” view takes ten seconds to populate. It is a bloated, fragile layer of abstraction over a system that is clearly straining under its own weight.
09:00 – The VNet Peering Hallucination
By morning, I was deep into the networking stack. On-prem, I’d just punch a hole in the firewall or trunk a VLAN. In azure, I have to deal with VNet Peering. I have two virtual networks: one in East US and one in West US 2. I need them to talk.
I set up the peering. I checked the “Allow forwarded traffic” boxes. I verified the address spaces didn’t overlap. And yet, no ping. No SSH. Nothing. I’m looking at the “Effective Routes” on the NIC of a Standard_D2s_v3 instance—a VM that costs more per month than the power bill for my old home lab—and the routes say the traffic should be flowing.
# Checking effective routes via PowerShell
# Module: Az.Network 7.5.0
Get-AzEffectiveRouteTable -NetworkInterfaceName "NIC-Web-01" -ResourceGroupName "RG-Prod-Alpha" | Select-Object Source, State, AddressPrefix, NextHopType
# Output:
# Source State AddressPrefix NextHopType
# ------ ----- ------------- -----------
# Default Active 10.0.0.0/16 VNetLocal
# VirtualNetworkPeering Active 10.1.0.0/16 VNetPeering
# Default Active 0.0.0.0/0 Internet
The routes say “Active.” The reality says “Timed Out.” I spent three hours digging through Network Security Groups (NSGs). In azure, an NSG is like a firewall designed by someone who hates sysadmins. You have to manage priority numbers. You have to remember that there’s a hidden rule at the bottom that will kill your traffic if you don’t explicitly allow it, but only if you’re coming from a specific tag.
I miss my Cisco gear. I miss the tactile click of a console cable. Here, I’m just screaming into a Log Analytics Workspace, waiting for KQL queries to tell me why my packets are being dropped by a “Global” infrastructure that seems to have the regional awareness of a goldfish. The latency between the portal’s “Success” message and the actual propagation of the rule to the underlying SDN is a yawning chasm of frustration.
15:00 – Identity Management is a Fever Dream
Then came the “Global Admin” paradox. I am the Global Admin of the Entra ID tenant. I should be a god. I should have the power to create and destroy worlds. But when I tried to look at the Cost Management blade to see why we had already burned through $400 in six hours, azure told me I didn’t have permission.
“Access Denied.”
I had to go into the Entra ID properties, find a tiny toggle at the bottom of a page that says “Access management for Azure resources,” and flip it to “Yes.” Only then could I “elevate” myself to see the subscriptions I supposedly own. It’s a security model built on top of a security model, wrapped in a layer of bureaucratic nonsense.
I tried to assign a Managed Identity to a VM so it could pull secrets from a Key Vault. Simple, right? No. The VM stayed in a “Updating” state for twenty minutes.
# Checking VM identity status
az vm show -g RG-Prod-Alpha -n Web-Server-01 --query identity
# Output:
{
"principalId": null,
"tenantId": "f8c88730-...",
"type": "SystemAssigned",
"userAssignedIdentities": null
}
# Note: principalId is null despite the type being SystemAssigned.
# The platform is still "thinking."
In the old days, I’d just give the server a service account and a keytab. Now, I have to wait for a distributed system to reach eventual consistency. “Eventual” is a word that cloud providers use to mean “whenever we feel like it.” I’m sitting here, staring at a null value, while the project manager pings me on Teams asking if the web tier is up. It’s not up. It’s waiting for an identity that doesn’t exist yet.
23:00 – The Application Gateway WAF Slog
Night fell, and I moved to the Application Gateway. This is azure‘s version of a load balancer, but it’s actually a massive, slow-moving beast that takes twenty minutes to update a single rule. I’m using the WAF v2 SKU. I wanted to enable a simple custom rule to block a specific IP range.
I updated the configuration. The portal gave me a “Running” notification. I went to the kitchen, made a pot of coffee, drank the pot of coffee, read a chapter of a book, and came back. Still “Running.”
When you configure a physical F5 or even a HAProxy instance on a bare-metal box, the change is instantaneous. You commit, and the traffic shifts. In azure, you are at the mercy of the “Deployment.” Every change to an Application Gateway feels like you’re submitting a request to a government agency in triplicate.
# Checking Application Gateway provisioning state
az network application-gateway show -g RG-Prod-Alpha -n AppGW-External --query provisioningState
# Output:
"Updating"
# (18 minutes later)
"Updating"
# (22 minutes later)
"Succeeded"
And then, the kicker: the rule didn’t work. Why? Because the “Detection” mode was on, but the “Prevention” mode wasn’t, or maybe it was because the listener wasn’t correctly associated with the backend pool, or maybe it was just the azure portal lying to me again. I had to dig into the JSON view—the only place where the truth actually lives—to find a typo in the resource ID that the UI had helpfully obscured.
The WAF logs are another nightmare. They don’t just show up. You have to pipe them to a Log Analytics Workspace, which costs money per gigabyte, and then you have to wait five to ten minutes for the logs to actually be indexed so you can query them with KQL. Real-time troubleshooting is a myth in this environment. You’re always looking at a ghost of what happened ten minutes ago.
08:00 (Day 2) – Private Link and the DNS Black Hole
I woke up at my desk to a flurry of alerts. The database—an Azure SQL instance—wasn’t reachable from the web tier. I had implemented Private Link because “public endpoints are bad,” but Private Link in azure is a dark art that involves Private DNS Zones.
If you don’t get the DNS exactly right, your traffic just vanishes. The VM tries to resolve mydb.database.windows.net, and instead of getting the private IP (10.0.5.4), it gets the public IP, which is blocked by the firewall. I spent four hours fighting with the “DNS Forwarding Ruleset.”
I miss /etc/hosts. I miss having a local BIND server that I could just reload. Here, I’m clicking through a “Private DNS Zone” UI that looks like it was designed in 2012. I’m trying to link the VNet to the zone, but the link is “Pending.” Why is it pending? There are no cables to plug in. There are no routers to configure. It’s just a database entry in Microsoft’s global ledger, and yet, it’s “Pending.”
I ran a nslookup from the VM.
C:\> nslookup mydb.database.windows.net
Server: UnKnown
Address: 168.63.129.16
Non-authoritative answer:
Name: mydb.database.windows.net
Address: 52.157.24.10 <-- THE PUBLIC IP. FAILURE.
The 168.63.129.16 address is the “Magic IP” of the azure recursive resolver. It is a black box. You can’t see into it. You can’t fix it. You just have to pray that your Private DNS Zone link eventually propagates. I felt the familiar itch of ozone-deprivation. I wanted to go to a data center, find the rack, and scream at the top-of-rack switch. At least then, I’d be doing something productive.
18:00 (Day 3) – The Cost Management Autopsy
By the third day, the migration was “functional,” in the same way a car with three wheels and a smoking engine is functional. I finally got into the Cost Management blade.
The “Actual Cost” was a horror show. We had provisioned Premium_LRS managed disks for everything, thinking we needed the IOPS. We didn’t. But the cost of those disks is fixed, whether you use the IOPS or not. On-prem, I bought the disks once. They sat in the SAN, and they cost me nothing but the electricity to spin them. Here, I’m being charged by the hour for the potential to use a disk.
The “Cost Alerts” I set up for $500 arrived at 17:45. The current spend was already $720. The alerts in azure are like a smoke detector that only goes off after the house has burned down and the ashes have cooled.
// Sample from a Cost Management Export
{
"id": "/subscriptions/sub-id/providers/Microsoft.Consumption/usageDetails/...",
"name": "usage_detail_123",
"properties": {
"billingPeriodStartDate": "2023-10-01T00:00:00Z",
"consumedService": "Microsoft.Compute",
"cost": 42.15,
"currency": "USD",
"instanceName": "Standard_D2s_v3",
"meterCategory": "Virtual Machines",
"unitOfMeasure": "10 Hours"
}
}
I looked at the bill for the “Inter-region Data Transfer.” Moving data between East US and West US 2 isn’t free. It’s a tax on your own architecture. In my old data center, I had a 10Gbps dark fiber link between sites. I paid for the light. Here, I pay for every bit that crawls across their “Global Network.”
I’m exhausted. My eyes are bloodshot from staring at the high-contrast “Dark Mode” of the portal, which is just a different shade of misery. I miss the hum. I miss the cold. I miss the certainty of a physical link light.
azure isn’t a platform; it’s a sprawling, shifting labyrinth of microservices held together by hope and expensive support contracts. It’s a world where “Global Admin” is a suggestion, where “Deleting” is a permanent state of being, and where the bill always wins.
I closed the browser tab. The PuTTY window was still disconnected. I didn’t bother to reconnect. I just sat there in the silence, smelling the lack of ozone, and wondered if it was too late to go back to fixing printers. At least with a printer, you know why it hates you. In the cloud, the hate is distributed, scalable, and billed by the hour.
Related Articles
Explore more insights and best practices: