json
{
“error”: {
“code”: “Forbidden”,
“message”: “Access denied. Caller was not found in any access policy. \nCaller: appid=––;oid=––;iss=https://sts.windows.net/***/\nVault: secret-store-prod-001; Location: eastus”,
“innererror”: {
“code”: “AccessDeniedByVNet”
}
},
“activityId”: “f47ac10b-58cc-4372-a567-0e02b2c3d479”,
“clientRequestId”: “b94d27b9-934c-4512-8058-7a3e77033cbc”,
“timestamp”: “2024-05-20T14:22:01.442Z”
}
The marketing team promised us a fortress, but they gave us a glass house. They told us that migrating to azure would "offload the security burden." What they didn't mention is that the burden hasn't been lifted; it’s just been obscured by layers of opaque abstraction and bolted-on API hooks. We aren't managing servers anymore; we are managing a sprawling, interconnected web of JSON configurations where a single character typo in a Terraform manifest can expose our entire customer database to the public internet.
This isn't "the cloud." This is just a massive, multi-tenant cluster of hardware owned by a corporation that prioritizes uptime and feature velocity over your data’s integrity. Every time we spin up a resource in azure, we are trusting that their hypervisor isolation is flawless and that their control plane isn't leaking metadata like a sieve. Spoiler alert: it is.
## H2: The Identity Crisis: Why RBAC is a Lie
In our legacy on-prem environment, identity was simple. You were on the domain, or you weren't. In azure, identity is a fragmented nightmare of Entra ID (formerly Azure AD, because rebranding solves technical debt, apparently) service principals, managed identities, and guest accounts.
The Role-Based Access Control (RBAC) system is marketed as a way to enforce least privilege, but in practice, it’s a convoluted mess of inheritance. If you grant a developer "Contributor" access at the Subscription level, they effectively own every resource within that bucket. There is no granular "don't touch the networking" flag that actually works when someone has Contributor rights. They can delete the Network Security Groups (NSGs), bypass the firewalls, and create their own backdoors.
Let’s look at what happens when we query a service principal's permissions using Azure CLI 2.60.0. The output is a wall of GUIDs that no human can audit effectively without a third-party tool—which, of course, introduces its own set of vulnerabilities.
```bash
# Attempting to audit a "limited" service principal
az ad sp permissions list --id 00000000-0000-0000-0000-000000000000 --query "[].{Resource:resourceAppId, Permission:oauth2Permissions}"
Terminal Output (Truncated):
[
{
"Permission": "Directory.Read.All, Files.ReadWrite.All, User.Read, Sites.FullControl.All",
"Resource": "00000003-0000-0000-c000-000000000000"
},
{
"Permission": "user_impersonation",
"Resource": "https://management.core.windows.net/"
}
]
Look at that Sites.FullControl.All and user_impersonation. This was supposed to be a “read-only” automation script for our CI/CD pipeline. The “Owner” role is even worse. It’s a god-mode toggle that is handed out like candy because the “Reader” role is too restrictive for anyone to actually get work done. We are one phished admin account away from a total tenant takeover.
AUDITOR’S NOTE: The Microsoft documentation for ‘Azure Built-in Roles’ (last updated 2024-04-12) claims that the ‘API Management Service Contributor’ role cannot manage secrets. However, cross-referencing this with the actual Resource Provider operations reveals that it can, in fact, list functional keys via the /listSecrets POST action. This is a blatant inconsistency that allows for lateral movement. The documentation is not a source of truth; it is a suggestion.
Table of Contents
H2: Public Endpoints and the Myth of the Perimeter
The most dangerous default in azure is the “Public Endpoint.” When you create a SQL Database or a Storage Account, the default configuration—unless you are using strict Policy definitions—is to assign it a public DNS entry. Sure, there’s a “firewall” in front of it, but that firewall is just a list of IP addresses.
We’ve moved from a hardened perimeter with significant latency overhead to a model where our most sensitive data sits on the public internet, protected by a checkbox. We are told to use Private Link to “solve” this. Private Link is a nightmare of DNS integration. If your local DNS forwarders aren’t perfectly synced with the privatelink.database.windows.net zones, your traffic will silently fail over to the public endpoint. You won’t even know it’s happening until you see the egress charges or, worse, the breach notification.
Consider the complexity of a simple Private Endpoint deployment using Terraform provider azurerm v3.105.0. You have to manage the azurerm_private_endpoint, the azurerm_private_dns_zone, and the azurerm_private_dns_zone_virtual_network_link. If any of these “bolted-on API hooks” fail to propagate, your application breaks, and the “fix” from the DevOps team is almost always to “just open the firewall for a second.”
H2: The Key Vault Mirage: Secrets in Plain Sight
Key Vault is supposed to be our Hardware Security Module (HSM) equivalent. In reality, it’s a REST API with a very thin layer of protection. The “Access Policy” model is being deprecated in favor of RBAC, but the transition is brittle.
The biggest issue is the “Managed Identity” trap. We assign a System-Assigned Identity to a Virtual Machine so it can pull secrets from the Vault. This sounds secure until you realize that any process running on that VM—including a compromised web server or a malicious cron job—can curl the Instance Metadata Service (IMDS) at 169.254.169.254 and get a full bearer token.
# Stealing a token from the IMDS endpoint on a compromised VM
curl -H "Metadata: true" "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://vault.azure.net"
Terminal Output:
{
"access_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im5iS3p...[REDACTED]",
"client_id": "55555555-4444-3333-2222-111111111111",
"expires_in": "28799",
"expires_on": "1716244921",
"ext_expires_in": "28799",
"not_before": "1716215821",
"resource": "https://vault.azure.net",
"token_type": "Bearer"
}
Once an attacker has that token, they are the VM. They can pull the database connection strings, the API keys, and the certificates. There is no secondary challenge. No MFA for service-to-service communication. It’s a flat security model disguised as a sophisticated one.
AUDITOR’S NOTE: Microsoft’s ‘Best Practices’ guide suggests using ‘Azure Key Vault’ for all secrets, yet their own ‘App Service’ configuration blade encourages developers to store secrets in ‘Application Settings’ which are stored in plaintext in the underlying Kudu filesystem. The left hand doesn’t know what the right hand is doing, and our secrets are the casualty.
H2: Network Security Groups (NSGs): Swiss Cheese as a Service
Network Security Groups are the most basic form of packet filtering in azure, and they are woefully inadequate for a modern threat landscape. They are not stateful firewalls in the traditional sense; they are distributed ACLs applied at the NIC or Subnet level.
The “Priority” system is a recipe for disaster. With rules ranging from 100 to 65000, it is trivial for a junior admin to insert an “Allow-All-From-Home” rule at priority 101 that negates every other security control we’ve implemented. Furthermore, the default rules—like AllowVnetInBound and AllowAzureLoadBalancerInBound—are far too permissive. AllowVnetInBound doesn’t just mean your VNet; if you have VNet peering or a Global VNet Peering setup, it could mean traffic from a completely different business unit or a compromised dev environment.
Let’s look at the current state of our “hardened” NSG for the production subnet:
az network nsg rule list --resource-group prod-rg --nsg-name prod-nsg --query "[].{Name:name, Priority:priority, Action:access, Port:destinationPortRange}"
Terminal Output:
[
{
"Action": "Allow",
"Name": "AllowHTTPS",
"Port": "443",
"Priority": 100
},
{
"Action": "Allow",
"Name": "TempDebugRule",
"Port": "*",
"Priority": 105
},
{
"Action": "Deny",
"Name": "DenyAllInBound",
"Port": "*",
"Priority": 4096
}
]
Note the TempDebugRule at priority 105. It’s been there for six months. It allows all traffic on all ports. Because it’s a higher priority than the DenyAllInBound, our “hardened perimeter” is effectively non-existent. This is the reality of “cloud-native” security: it’s only as strong as the most tired engineer’s last mistake.
H2: Storage Account Leakage: The Blob that Ate the Firewall
Azure Storage Accounts are perhaps the most misconfigured resource in the history of computing. Between Shared Access Signatures (SAS), Access Keys, and Public Access Levels, there are too many ways to leak data.
The SAS token is a particular nightmare. It’s a signed URL that grants access to a resource. Once generated, it cannot be revoked unless you rotate the entire Storage Account key—which will immediately break every other application using that key. There is no “revoke this specific SAS” button. It is a ticking time bomb.
We are also seeing “Storage Account Contributor” roles being assigned to developers. This role allows them to list the account keys. If you can list the keys, you can bypass every single SAS restriction and every RBAC entry. You are the root user of that data.
AUDITOR’S NOTE: The ‘Storage Explorer’ tool (v1.33.0) often caches credentials in a way that bypasses Conditional Access Policies. I have successfully accessed ‘Internal-Only’ blobs from an unmanaged device because the session token hadn’t expired and the tool didn’t re-validate the IP address. This is a massive hole in our ‘Zero Trust’ strategy.
H2: Logging and Monitoring: The Noise Floor of Despair
Azure Monitor and Log Analytics are sold as a “single pane of glass.” In reality, they are a high-latency, high-cost dumping ground for telemetry that is almost impossible to query in real-time.
If a breach occurs, we need to know now. But azure logs can take anywhere from 5 to 15 minutes to ingest into a Log Analytics Workspace. An attacker can exfiltrate a terabyte of data in 15 minutes. By the time our “near real-time” alert fires, the data is gone, the service principal has been deleted, and the attacker has moved on.
Furthermore, the cost of logging is prohibitive. To save money, the finance team has requested we only log “Security Events.” But in azure, a “Security Event” is narrowly defined. If an attacker uses a legitimate (but stolen) credential to call the ListKeys API on a storage account, that might not trigger a standard security alert. It’s just another API call. To catch that, we need full Administrative Activity logs, which are voluminous and expensive to store. We are literally paying for the privilege of seeing how we were hacked, and we’re being forced to choose between visibility and budget.
Hardening Checklist (The “Good Luck” List)
If we are going to stay on azure, we must implement the following. Be warned: these will break almost every “seamless” workflow the developers have built.
- Disable All Local Authentication: Every Storage Account, SQL Database, and CosmosDB instance must have local key/password authentication disabled. Use Entra ID authentication only. This will break legacy scripts. Do it anyway.
- Enforce Micro-Segmentation via ASGs: Application Security Groups must be used to restrict traffic between individual application tiers. No more “AllowVnetInBound.” If the Web Tier doesn’t need to talk to the Backup Tier, block it at the NIC level.
- Mandatory Private Link for Everything: No resource shall have a public IP or a public DNS suffix. All access must go through a Private Endpoint. This requires a complete redesign of our DNS architecture and the deployment of Private DNS Resolver sets in every region.
- Just-In-Time (JIT) VM Access: No SSH or RDP ports may be open, even to the internal network. All administrative access must be requested through Privileged Identity Management (PIM) and conducted via Azure Bastion.
- Immutable Infrastructure via Terraform v1.8.0+: No manual changes in the portal. Any resource not defined in code must be automatically deleted by a “Janitor” script every 60 minutes.
- IMDS Protection: Implement
iptablesrules on every Linux VM to restrict access to169.254.169.254to only therootuser or specific authorized system services. - Resource Locks: Apply
CanNotDeletelocks to all core networking and security resources. This will make updates a manual, painful process involving multiple levels of approval. - Egress Filtering: Implement a centralized Firewall (not just an NSG) to inspect all outbound traffic. If a compromised VM tries to “phone home” to a known C2 server, it must be dropped, even if the traffic is on port 443.
The cloud is not a safe haven. It is a minefield of “convenience” features that undermine security at every turn. If we don’t treat azure with the same suspicion we treat a public Wi-Fi network at a hacker convention, we deserve the breach that is inevitably coming.
Audit Status: CRITICAL FAILURE.
Recommendation: Immediate remediation or decommissioning of non-compliant workloads.
Related Articles
Explore more insights and best practices: