Table of Contents
The 3:00 AM Black Hole: When “It Works on My Minikube” Meets Production
At 03:14 local time, PagerDuty decided my sleep was offensive. The alarm wasn’t just ringing; it was screaming that the core billing service, payment-processor-v4-86c8d7b7f-9xq2m, had lost its ability to talk to the fraud-detection engine running in an adjacent namespace. Naturally, the dashboard was a sea of red. The application logs told a depressing story:
{"level":"error","ts":1718872441.102,"caller":"client/http.go:114","msg":"request failed","error":"Post \"http://fraud-engine.risk-assessment.svc.cluster.local:8443/v1/evaluate\": dial tcp 10.96.142.19:8443: connect: connection timed out"}
```
Three minutes prior, a junior developer had applied a network policy update designed to "lock down lateral movement." The YAML looked clean. The Kubernetes API server accepted it without a whimper. The security team slapped each other on the back for compliance. And production promptly flatlined. The developers swore the labels matched. The network team pointed at the Calico-to-Cilium migration we completed six months ago and muttered something about kernel headers. Nobody had any actual visibility because standard iptables troubleshooting guides were about as useful as a chocolate teapot.
This is where standard troubleshooting ends and the grim reality of debugging ebpfbased network mechanics begins. We weren't dealing with iptables rules chaining through a tangled web of filter tables anymore. We were staring down the barrel of custom bytecode injected directly into the Linux kernel networking stack via Cilium v1.14.5 running on a hardened Amazon Linux 2 kernel (5.10.205-195.776.amzn2.x86_64). Packets weren't dropping because a rule rejected them; they were vanishing into the digital ether because an eBPF map lookup returned an unexpected drop verdict, leaving zero trace in standard kernel logs. Let's pull back the curtain on this madness.
Verifying the Illusion: Is the Policy Actually Attached?
Never trust a Kubernetes resource status. Just because kubectl describe cnp payment-isolation says the network policy is enforced doesn't mean the kernel bytecode actually cares. The control plane lies. The control plane's job is to make you feel safe while the kernel quietly drops your packets on the floor.
We needed to verify if Cilium's agent daemonset pods on the respective nodes had actually compiled and loaded the BPF programs into the TC (Traffic Control) subsystem of the virtual Ethernet devices.
Inspecting the BPF Progs on the Interface
First, find the pod's veth interface on the host. Do not try to debug this from inside the container namespace unless you enjoy chasing ghosts.
# Find the container process ID
CONTAINER_PID=$(crictl inspect 8f7b4c3a2e1d0 | jq '.info.pid')
# Find the veth interface name inside the host network namespace corresponding to this pod
ip -n netns link show | grep -B 1 "link-netnsid"
```
More reliably, query the Cilium CLI directly. If you aren't using the cilium binary inside your troubleshooting toolkit, resign now.
kubectl exec -n kube-system ds/cilium -c cilium-agent -- cilium endpoint list
```
Output:
ENDPOINT POLICY IDENTITY LABELS (KEY=VALUE) IPV4 IPV6 STATUS
3821 Enforced 42011 io.kubernetes.pod.namespace=billing 10.244.1.20 ready
5102 Enforced 89312 io.kubernetes.pod.namespace=risk-assessment 10.244.3.88 ready
```
The endpoints showed as "Enforced". Good. Or bad. Mostly terrifying, because it meant the policies were active and aggressively breaking things.
Checking BPF Map Entries for Policy Enforcement
Cilium uses BPF maps to enforce policies at wire speed. If a packet cannot resolve its destination identity in the policy map, it drops. Let's dump the policy map for endpoint 3821. We had to exec into the agent pod responsible for the worker node running our payment pod (`ip-10-0-42-15.ec2.internal`).
kubectl exec -n kube-system cilium-agent-xyz4k -c cilium-agent -- cilium bpf policy list
```
If the output doesn't list the exact source and destination security identities (IDs), your policy selector is broken or the synchronization between the Kubernetes API server and the Cilium operator is stuck in an async retry loop. This is step one of debugging ebpfbased network layers: prove the maps contain the rules you think they do.
Tracing the Ghost Packet with Hubble CLI
Hubble is supposed to be the magical UI that saves you from the terminal. In reality, when you are staring at a production outage at 3:30 AM, the Hubble UI is too slow. You drop straight into the CLI, bypassing all the Grafana dashboards that are currently failing to query Prometheus anyway.
We fired up the Hubble observer stream to watch the drop events in real time. But standard observation wasn't enough; we needed the exact drop reason code.
Capturing Drops with Hubble Observe
kubectl exec -n kube-system cilium-agent-xyz4k -c cilium-agent -- \
hubble observe \
--from-pod billing/payment-processor-v4-86c8d7b7f-9xq2m \
--to-pod risk-assessment/fraud-engine-6f8c7b4-2l9tp \
--type drop
```
And there it was. Not a timeout. A deliberate, cold-blooded drop issued by the kernel map lookup.
Oct 20 03:18:42.102: billing/payment-processor-v4-86c8d7b7f-9xq2m (ID: 42011) -> risk-assessment/fraud-engine-6f8c7b4-2l9tp (ID: 89312) to-overlay POLICY_DENIED (TCP Flags: SYN)
```
POLICY_DENIED. Wonderful. The policy matched, but it matched a deny rule instead of an allow rule. Why?
Decoding the Identity Mismatch
Cilium translates Kubernetes labels into numeric identities. If your network policy specifies a namespace selector or pod selector that doesn't match the actual labels compiled into the identity map, traffic drops instantly.
We checked the identity assigned to the fraud engine pod:
kubectl exec -n kube-system cilium-agent-xyz4k -c cilium-agent -- cilium identity get 89312
```
Output:
Identity: 89312
Labels:
- io.kubernetes.pod.namespace=risk-assessment
- io.kubernetes.pod.template-hash=6f8c7b4
- tier=backend
- app=fraud-engine
Now we checked the NetworkPolicy YAML that the junior engineer applied:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: allow-billing-to-fraud
namespace: risk-assessment
spec:
endpointSelector:
matchLabels:
app: fraud-engine
ingress:
- fromEndpoints:
- matchLabels:
io.kubernetes.pod.namespace: billing
app: payment-processor
Spot the bug? Look closely at the payment processor deployment labels. The deployment used app.kubernetes.io/name: payment-processor, but the network policy specified app: payment-processor. Standard Kubernetes network policies are additive; Cilium Network Policies with strict enforcement default to deny-all. Because the label selector failed to find a match, the identity generation omitted the rule, and the default drop policy swallowed the SYN packet.
Kernel-Level Tracepoints with bpftool and perf
Sometimes Hubble isn't verbose enough, or you suspect a bug in the Cilium datapath itself (unlikely, but when you're desperate, you blame everything). When you need to get your hands dirty at the kernel layer, bpftool and custom tracing scripts are your only salvation.
We needed to intercept the exact execution path of the eBPF program attached to the network device to see where the drop verdict was decided.
Dumping the BPF Program Instructions
To inspect the compiled eBPF bytecode loaded into the kernel ring, we used bpftool from the host node shell. First, we got a root shell on the node via a privileged debug pod.
kubectl debug node/ip-10-0-42-15.ec2.internal -it --image=taoofmac/nettools:latest
```
Once inside the debug container running in the node's host namespace:
# Find the program ID for the cilium ingress datapath
bpftool prog show | grep "from-container"
```
Output:
3482: sched_cls tag 9b543787a2a0d1e5 gpl run_time_ns 482913104 run_cnt 1482910
We dumped the JIT-compiled instructions to inspect the branching logic around policy evaluation:
bpftool prog dump xlated id 3482 > /tmp/bpf_dump.txt
cat /tmp/bpf_dump.txt | grep -C 5 "drop"
```
This is where you realize that reading BPF pseudo-assembly at 4:00 AM is a special kind of self-inflicted torture. But it shows you the exact line of C code in the Cilium datapath source (`bpf/lib/policy.h`) where the return code evaluates to TC_ACT_SHOT.
Attaching Tracepoints for Real-Time Packet Drops
To confirm the kernel was indeed dropping packets at the Traffic Control layer without hitting the IP stack, we used perf to hook into the kfree_skb tracepoint.
perf record -e skb:kfree_skb -a -- sleep 10
perf script
```
This output is an unreadable wall of memory addresses unless you filter it through the symbol table of the loaded BPF object. Fortunately, Cilium includes built-in tooling to make this less painful via cilium monitor.
kubectl exec -n kube-system cilium-agent-xyz4k -c cilium-agent -- cilium monitor --type drop
```
Output:
-> 10.244.3.88:8443 bypass 10.244.1.20:54322 flow 0x0 3821 (ID: 42011) -> 5102 (ID: 89312), prot 6 TCP, Table: policy, Drop: IP Policy verdict: Denied
```
The proof was absolute. The IP Policy verdict returned "Denied" because the endpoint identities failed to resolve through the mismatching labels.
Fixing the Datapath State and Forcing Map Reconciliation
Identifying the broken label selector was only half the battle. After applying the corrected CiliumNetworkPolicy with app.kubernetes.io/name: payment-processor, the traffic *still* failed. Why? Because Cilium's local policy maps on the worker nodes hadn't invalidated their caches.
This is the dark side of connection tracking in eBPF. Conntrack entries and policy state maps can get stuck if stale states are held in memory during rapid policy iterations.
Flushing the Conntrack and Policy Maps Manually
When the control plane refuses to update fast enough, you have to force it. Do not wait for reconciliation intervals when production is down.
# Flush the Cilium connection tracking table on the affected node
kubectl exec -n kube-system cilium-agent-xyz4k -c cilium-agent -- cilium bpf ct flush
```
If flushing the conntrack table isn't enough, you must force a complete regeneration of the endpoint states.
# Force endpoint regeneration across the agent
kubectl exec -n kube-system cilium-agent-xyz4k -c cilium-agent -- cilium endpoint regenerate 3821
```
Watching the logs during endpoint regeneration gives you immediate feedback:
level=info msg="Compiling BPF datapath..." endpointID=3821 subsys=endpoint
level=info msg="Successfully compiled and loaded BPF program" endpointID=3821 subsys=endpoint
As soon as the compilation completed, the connection from the payment processor to the fraud engine instantly transitioned from dropped to established.
Oct 20 03:22:15.891: billing/payment-processor-v4-86c8d7b7f-9xq2m (ID: 42011) -> risk-assessment/fraud-engine-6f8c7b4-2l9tp (ID: 89312) to-overlay forwarded (TCP Flags: ACK)
```
Building Hardened Guardrails to Prevent Repeat Failures
Post-mortems are useless if you don't encode the lessons learned into automated checks. Relying on humans to remember label selectors across different teams in a microservices architecture is a fool's errand. We implemented three strict engineering controls the following morning.
Automated Validation Steps
- Enforced strict Conftest / OPA (Open Policy Agent) policies in our CI/CD pipeline that validate label presence against the running cluster's registered Cilium identities before permitting merge to the main branch.
- Deployed Prometheus alerts targeting the
cilium_drop_count{reason="Policy denied"}metric, configured to page the owning team immediately if drop rates exceed zero for more than 60 seconds on production namespaces. - Created an internal runbook containing only the precise
cilium monitorandbpftoolcommands needed to diagnose datapath drops without requiring developers to memorize kernel flags at 3:00 AM.
eBPF is incredible technology. It removes layers of network virtualization overhead, gives you line-rate packet processing, and makes traditional iptables look like a relic from the stone age. But when it breaks, it breaks invisibly. It bypasses every tool you grew up trusting. If you refuse to learn how to query the BPF maps, inspect the tc filters, and trace identity allocations down at the kernel socket layer, you are flying blind in a supersonic jet. Stop trusting the control plane status. Go look at the maps.
Related Articles
Explore more insights and best practices: