Table of Contents
The 03:14 AM OOM Sweep
[Wed Oct 18 03:14:02 2023] oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=900
[Wed Oct 18 03:14:02 2023] CPU: 37 PID: 14892 Comm: envoy Kdump: nullptr Tainted: G OE 5.15.0-88-generic #98-Ubuntu
[Wed Oct 18 03:14:02 2023] Mem-Total: 32901416kB, AnonPages: 29841904kB, Shmem: 11232kB, Slab: 412091kB
[Wed Oct 18 03:14:02 2023] Active(anonymous): 18402912 Inactive: 11419024 Isolated:₀
[Wed Oct 18 03:14:02 2023] Node 0 Node 0 Memory: not zeroed: Total: 32901416kB, Payload, `envoy-ingress-edge-7c9b` invoked oom-killer: gfp_mask=0x140cca, oom_score_adj=900
[Wed Oct 18 03:14:02 2023] Task in-level=0 o-let: 14892 total: 0 anon: kernel RSS: 28.9GB of 31.3GB RAM.
[Wed Oct 18 03:14:02 2023] Out of memory: Kill process 14892 (envoy-ingress-edge-7c9b) score 900 or sacrifice code paths tracing subsystem state.
That stack trace is the greeting card from a dead edge router. For three consecutive nights, my alarm has ripped me out of sleep at 03:00 UTC because `envoy-ingress-edge-7c9b` swallowed thirty gigabytes of anonymous memory before the Linux kernel stepped in and euthanized it. No traffic spike. No upstream service degradation. Just a slow, relentless, linear climb in RSS until the cgroups limit triggered a hard boundary. The vendor promised that ebpfbased distributed tracing would eliminate sidecar overhead, bypass userspace interception penalties, and keep our P99 latencies pristine. They neglected to mention that kernel-space data structures left unmanaged will silently consume every megabyte of RAM your nodes can spare.
If you deploy kernel probes, attach kprobes to SSL decryption routines, and stream context propagation identifiers across process boundaries without accounting for ring buffer backpressure and hash map eviction policies, you are building a time bomb. We are going to rip open the mechanics of ebpfbased distributed tracing inside Envoy proxies, examine the byte-level memory leaks caused by abandoned hash entries, fix the kernel-to-userspace communication channels, and configure proper eviction policies so your edge infrastructure survives the morning rush.
Anatomy of the Leak: BPF Maps and Uncollected Trace Contexts
The core issue lives at the intersection of network packet processing and long-running eBPF maps. When you attach an eBPF program to intercept HTTP/2 or gRPC frames processed by Envoy via socket filters or kprobes on `SSL_read` and `SSL_write`, you rely on BPF maps to correlate request IDs with trace spans. Every incoming request hitting port `10000` generates a trace context key consisting of source IP, source port, destination IP, and thread ID. This key indexes into a `BPF_MAP_TYPE_HASH` or `BPF_MAP_TYPE_LRU_HASH` to store transient state.
The Hash Map Retention Trap
Standard `BPF_MAP_TYPE_HASH` allocations do not automatically purge entries when an HTTP connection terminates or an Envoy worker thread drops a socket. If your eBPF program writes a span start timestamp into the map upon receiving headers on interface `eth0`, but the corresponding termination hook (`SSL_read` returning zero or an RST packet hitting the TCP stack) fails to fire or gets missed due to packet reordering, the entry stays resident. Forever.
// Simplified view of a leaky trace context map definition
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, 1000000);
__type(key, struct trace_key_t);
__type(value, struct span_context_t);
} active_spans SEC(".maps");
When `max_entries` hits one million items, insertions fail, but existing dead pointers persist. More critically, if you allocate maps without setting `max_entries` conservatively, or if your user-space daemon reading the trace telemetry crashes or blocks on I/O, the kernel buffers stall. The ring buffers fill up, drops mount, and the map memory footprint remains locked inside the pinned virtual memory space of the Envoy process handle.
Ring Buffer Backpressure and Memory Pinning
Modern tracing implementations prefer `BPF_MAP_TYPE_RINGBUF` over perf events because ring buffers avoid per-CPU allocations and drop less data under heavy load. However, a ring buffer is fundamentally a shared memory region mapped into both kernel and user space. Envoy loads the eBPF object, maps the ring buffer, and reads serialized trace payloads.
When upstream traffic targeting cluster `internal-auth-grpc` surges from 10,000 to 120,000 requests per second, the user-space polling loop in the tracing plugin cannot drain the ring buffer fast enough. The kernel-side producer advances the head pointer while user space lags behind. Instead of shedding telemetry gracefully, poorly configured ring buffers cause kernel allocation stalls. The memory pages allocated via `bpf_ringbuf_reserve` pin physical RAM pages, preventing the kernel from reclaiming them via page cache eviction. This is why RSS climbs linearly: the pages are wired directly into kernel page tables.
Diagnosing Ring Buffer Starvation via bpftool and Perf
Stop guessing with Grafana dashboards. When memory usage climbs on `envoy-ingress-edge-7c9b`, dive straight into the kernel using `bpftool` to inspect map utilization and pin down exact allocation sizes.
Inspecting BPF Map Allocations
Run the following command to dump map identifiers and locate the memory consumed by your active tracing maps:
bpftool map show
Look for the map IDs associated with your tracing collector. You will typically see output resembling this:
48: hash name active_spans flags 0x0
key 32B value 64B max_entries 2097152 memlock 142193280B
49: ringbuf name trace_ringbuf flags 0x0
key 0B value 0B max_entries 67108864 memlock 67141632B
That `memlock` value of 142MB for the hash map and 67MB for the ring buffer is just the base allocation. When accounting for overhead, fragmentation, and per-CPU allocations in older kernels, the actual RSS footprint swells by a factor of three. If `max_entries` is set to two million for a hash map holding 64-byte values, the memory footprint scales brutally with system limits.
Tracing Drop Counters with perf
To verify if your ring buffer is dropping events due to reader starvation, hook into the kernel tracepoints for ring buffer operations:
perf record -e bpf:* -a -- sleep 10
perf script
If you see repeated occurrences of `bpf_ringbuf_output_error` or drops incrementing in the output of `bpftool map stats`, your user-space consumer thread inside Envoy is blocked. The collector cannot keep up with the instrumentation overhead, leading to silent accumulation of unread records.
Architectural Fixes: LRU Maps and Garbage Collection Loops
You cannot trust transient network streams to always close cleanly. Relying on TCP state machine transitions to clean up eBPF maps is a fool’s errand because intermediate firewalls, packet drops, and load balancer health checks leave half-open connections hanging indefinitely. You must alter how the kernel stores trace states.
Migrating to LRU Hash Maps
Replace standard hash maps with Least Recently Used (LRU) variants. An LRU map guarantees that when `max_entries` is reached, the kernel automatically evicts the oldest, unaccessed trace contexts. This caps memory usage deterministically, protecting your nodes from OOM kills at the cost of dropping trace telemetry for extremely long-lived, idle connections.
struct {
__uint(type, BPF_MAP_TYPE_LRU_HASH);
__uint(max_entries, 500000); // Strict cap on memory footprint
__type(key, struct trace_key_t);
__type(value, struct span_context_t);
} active_spans_lru SEC(".maps");
By enforcing a strict ceiling of 500,000 entries, you limit the memory consumed by the hash map to a predictable ceiling, regardless of how many orphaned connections linger on the wire.
Implementing Kernel-Side Expiration Timestamps
LRU eviction helps, but long-lived active streaming gRPC connections can keep an entry fresh indefinitely if data flows continuously. You need explicit time-to-live (TTL) validation inside the eBPF lookup routine. Check the timestamp on every map lookup and purge stale entries inline:
#define TTL_NS 30000000000ULL // 30 seconds
SEC("kprobe/ssl_read")
int handle_ssl_read(struct pt_regs *ctx) {
struct trace_key_t key = {};
// Populate key from socket context...
struct span_context_t *val = bpf_map_lookup_elem(&active_spans_lru, &key);
if (val) {
__u64 now = bpf_ktime_get_ns();
if (now - val->start_time > TTL_NS) {
// Stale entry detected, delete it inline
bpf_map_delete_elem(&active_spans_lru, &key);
return 0;
}
}
return 0;
}
This snippet intercepts lookups, measures elapsed time against a strict thirty-second window, and evicts dead contexts directly from the eBPF program before they accumulate into gigabytes of waste.
Configuring Envoy Tracing Extensions Safely
Even with optimal eBPF map configurations, the way Envoy loads, attaches, and communicates with your tracing subsystem matters. A misconfigured tracing filter will spawn runaway background threads that bypass cgroup limits entirely.
Refining the Envoy Tracing Bootstrap Configuration
When defining your tracing provider inside the Envoy bootstrap YAML, ensure that buffer sizes and flushing intervals are explicitly tuned to prevent internal queue bloat. Never use default configurations in production.
static_resources:
listeners:
- name: ingress_listener
address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
tracing:
provider:
name: envoy.tracers.ebpf
typed_config:
"@type": type.googleapis.com/envoy.config.trace.v3.EbpfTraceConfig
ringbuf_size_bytes: 8388608 # Capped strictly at 8MB
max_active_spans: 100000
flush_interval_ms: 500
route_config:
name: local_route
virtual_hosts:
- name: edge_service
domains: ["*"]
routes:
- match:
prefix: "/"
route:
cluster: internal-auth-grpc
Setting `ringbuf_size_bytes` to 8MB forces the telemetry pipeline to shed load rather than consuming infinite kernel memory when downstream ingestion endpoints saturate. If your tracing collector goes down, Envoy drops spans instead of dying in an OOM event.
Resource Control via Cgroups
Ensure your container runtime enforces strict memory limits not just on Envoy’s userspace RSS, but on the kernel memory accounting tied to the container cgroup. Modern Linux kernels track BPF memory allocations via `memory.kmem.current` or unified cgroup v2 accounting.
# Verify cgroup v2 memory accounting for BPF maps inside the container
cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.stat
If `memsw.limit_in_bytes` or equivalent cgroup parameters are not configured to account for pinned BPF maps, the host kernel will allow eBPF allocations to bypass container memory constraints until the node itself panics.
Verifying the Patch in Staging
Before rolling these fixes to `envoy-ingress-edge-7c9b` in production, validate the memory profile under synthetic load. Fire up `vegeta` or `wrk` against your staging ingress gateway while monitoring the slab allocator and BPF map memory consumption.
Load Testing the Telemetry Pipeline
vegeta attack -targets=targets.txt -rate=50000 -duration=30m | vegeta encode > results.bin
cat results.bin | vegeta report
While the load test runs, execute a continuous loop to check map memory usage:
while true; do bpftool map show | grep -E "name|memlock"; sleep 5; done
With the LRU map implementation and explicit TTL eviction logic active, the `memlock` value will plateau rapidly, stabilizing at the configured `max_entries` threshold. The continuous linear climb disappears. RSS on the Envoy process flatlines well below your cgroup threshold.
Summary of Production Rules for eBPF Tracing
Implementing ebpfbased distributed tracing without understanding kernel memory limits is an operational disaster waiting for a high-traffic window. To keep your edge proxies alive, follow these absolute constraints:
- Never deploy standard `BPF_MAP_TYPE_HASH` maps for unbounded connection tracking; enforce `BPF_MAP_TYPE_LRU_HASH` with explicit capacity limits.
- Cap ring buffer allocations in your Envoy bootstrap configuration to force telemetry shedding instead of kernel memory exhaustion.
- Validate that your container runtime accounts for BPF `memlock` overhead within your cgroup memory limits.
- Incorporate inline timestamp checks and TTL-based deletions inside your eBPF programs to purge orphaned states left by dropped TCP connections.
Configure your telemetry defensively. The kernel owes your service nothing.
Related Articles
Explore more insights and best practices: