{"id":5338,"date":"2026-11-13T03:18:06","date_gmt":"2026-11-12T21:48:06","guid":{"rendered":"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/"},"modified":"2026-11-13T03:18:06","modified_gmt":"2026-11-12T21:48:06","slug":"ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead","status":"publish","type":"post","link":"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/","title":{"rendered":"ebpfbased distributed tracing in Envoy: Fix Leaks &#038; Overhead"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_89 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<label for=\"ez-toc-cssicon-toggle-item-6af83bd2e0f9a\" class=\"ez-toc-cssicon-toggle-label\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/label><input type=\"checkbox\"  id=\"ez-toc-cssicon-toggle-item-6af83bd2e0f9a\"  aria-label=\"Toggle\" \/><nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#The_03_14_AM_OOM_Sweep\" >The 03:14 AM OOM Sweep<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#Anatomy_of_the_Leak_BPF_Maps_and_Uncollected_Trace_Contexts\" >Anatomy of the Leak: BPF Maps and Uncollected Trace Contexts<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#The_Hash_Map_Retention_Trap\" >The Hash Map Retention Trap<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#Ring_Buffer_Backpressure_and_Memory_Pinning\" >Ring Buffer Backpressure and Memory Pinning<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#Diagnosing_Ring_Buffer_Starvation_via_bpftool_and_Perf\" >Diagnosing Ring Buffer Starvation via bpftool and Perf<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#Inspecting_BPF_Map_Allocations\" >Inspecting BPF Map Allocations<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#Tracing_Drop_Counters_with_perf\" >Tracing Drop Counters with perf<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#Architectural_Fixes_LRU_Maps_and_Garbage_Collection_Loops\" >Architectural Fixes: LRU Maps and Garbage Collection Loops<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#Migrating_to_LRU_Hash_Maps\" >Migrating to LRU Hash Maps<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#Implementing_Kernel-Side_Expiration_Timestamps\" >Implementing Kernel-Side Expiration Timestamps<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#Configuring_Envoy_Tracing_Extensions_Safely\" >Configuring Envoy Tracing Extensions Safely<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#Refining_the_Envoy_Tracing_Bootstrap_Configuration\" >Refining the Envoy Tracing Bootstrap Configuration<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-13\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#Resource_Control_via_Cgroups\" >Resource Control via Cgroups<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-14\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#Verifying_the_Patch_in_Staging\" >Verifying the Patch in Staging<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-15\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#Load_Testing_the_Telemetry_Pipeline\" >Load Testing the Telemetry Pipeline<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-16\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#Summary_of_Production_Rules_for_eBPF_Tracing\" >Summary of Production Rules for eBPF Tracing<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-17\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#Related_Articles\" >Related Articles<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"The_03_14_AM_OOM_Sweep\"><\/span>The 03:14 AM OOM Sweep<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<pre><code class=\"language-text\">\n[Wed Oct 18 03:14:02 2023] oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=900\n[Wed Oct 18 03:14:02 2023] CPU: 37 PID: 14892 Comm: envoy Kdump: nullptr Tainted: G           OE     5.15.0-88-generic #98-Ubuntu\n[Wed Oct 18 03:14:02 2023] Mem-Total: 32901416kB, AnonPages: 29841904kB, Shmem: 11232kB, Slab: 412091kB\n[Wed Oct 18 03:14:02 2023] Active(anonymous): 18402912 Inactive: 11419024 Isolated:\u2080\n[Wed Oct 18 03:14:02 2023] Node 0 Node 0 Memory: not zeroed: Total: 32901416kB, Payload, `envoy-ingress-edge-7c9b` invoked oom-killer: gfp_mask=0x140cca, oom_score_adj=900\n[Wed Oct 18 03:14:02 2023] Task in-level=0 o-let: 14892 total: 0 anon: kernel RSS: 28.9GB of 31.3GB RAM.\n[Wed Oct 18 03:14:02 2023] Out of memory: Kill process 14892 (envoy-ingress-edge-7c9b) score 900 or sacrifice code paths tracing subsystem state.\n<\/code><\/pre>\n<p>That stack trace is the greeting card from a dead edge router. For three consecutive nights, my alarm has ripped me out of sleep at 03:00 UTC because `envoy-ingress-edge-7c9b` swallowed thirty gigabytes of anonymous memory before the Linux kernel stepped in and euthanized it. No traffic spike. No upstream service degradation. Just a slow, relentless, linear climb in RSS until the cgroups limit triggered a hard boundary. The vendor promised that ebpfbased distributed tracing would eliminate sidecar overhead, bypass userspace interception penalties, and keep our P99 latencies pristine. They neglected to mention that kernel-space data structures left unmanaged will silently consume every megabyte of RAM your nodes can spare.<\/p>\n<p>If you deploy kernel probes, attach kprobes to SSL decryption routines, and stream context propagation identifiers across process boundaries without accounting for ring buffer backpressure and hash map eviction policies, you are building a time bomb. We are going to rip open the mechanics of ebpfbased distributed tracing inside Envoy proxies, examine the byte-level memory leaks caused by abandoned hash entries, fix the kernel-to-userspace communication channels, and configure proper eviction policies so your edge infrastructure survives the morning rush.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Anatomy_of_the_Leak_BPF_Maps_and_Uncollected_Trace_Contexts\"><\/span>Anatomy of the Leak: BPF Maps and Uncollected Trace Contexts<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>The core issue lives at the intersection of network packet processing and long-running eBPF maps. When you attach an eBPF program to intercept HTTP\/2 or gRPC frames processed by Envoy via socket filters or kprobes on `SSL_read` and `SSL_write`, you rely on BPF maps to correlate request IDs with trace spans. Every incoming request hitting port `10000` generates a trace context key consisting of source IP, source port, destination IP, and thread ID. This key indexes into a `BPF_MAP_TYPE_HASH` or `BPF_MAP_TYPE_LRU_HASH` to store transient state.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"The_Hash_Map_Retention_Trap\"><\/span>The Hash Map Retention Trap<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Standard `BPF_MAP_TYPE_HASH` allocations do not automatically purge entries when an HTTP connection terminates or an Envoy worker thread drops a socket. If your eBPF program writes a span start timestamp into the map upon receiving headers on interface `eth0`, but the corresponding termination hook (`SSL_read` returning zero or an RST packet hitting the TCP stack) fails to fire or gets missed due to packet reordering, the entry stays resident. Forever.<\/p>\n<pre><code class=\"language-c\">\n\/\/ Simplified view of a leaky trace context map definition\nstruct {\n    __uint(type, BPF_MAP_TYPE_HASH);\n    __uint(max_entries, 1000000);\n    __type(key, struct trace_key_t);\n    __type(value, struct span_context_t);\n} active_spans SEC(\".maps\");\n<\/code><\/pre>\n<p>When `max_entries` hits one million items, insertions fail, but existing dead pointers persist. More critically, if you allocate maps without setting `max_entries` conservatively, or if your user-space daemon reading the trace telemetry crashes or blocks on I\/O, the kernel buffers stall. The ring buffers fill up, drops mount, and the map memory footprint remains locked inside the pinned virtual memory space of the Envoy process handle.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Ring_Buffer_Backpressure_and_Memory_Pinning\"><\/span>Ring Buffer Backpressure and Memory Pinning<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Modern tracing implementations prefer `BPF_MAP_TYPE_RINGBUF` over perf events because ring buffers avoid per-CPU allocations and drop less data under heavy load. However, a ring buffer is fundamentally a shared memory region mapped into both kernel and user space. Envoy loads the eBPF object, maps the ring buffer, and reads serialized trace payloads.<\/p>\n<p>When upstream traffic targeting cluster `internal-auth-grpc` surges from 10,000 to 120,000 requests per second, the user-space polling loop in the tracing plugin cannot drain the ring buffer fast enough. The kernel-side producer advances the head pointer while user space lags behind. Instead of shedding telemetry gracefully, poorly configured ring buffers cause kernel allocation stalls. The memory pages allocated via `bpf_ringbuf_reserve` pin physical RAM pages, preventing the kernel from reclaiming them via page cache eviction. This is why RSS climbs linearly: the pages are wired directly into kernel page tables.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Diagnosing_Ring_Buffer_Starvation_via_bpftool_and_Perf\"><\/span>Diagnosing Ring Buffer Starvation via bpftool and Perf<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Stop guessing with Grafana dashboards. When memory usage climbs on `envoy-ingress-edge-7c9b`, dive straight into the kernel using `bpftool` to inspect map utilization and pin down exact allocation sizes.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Inspecting_BPF_Map_Allocations\"><\/span>Inspecting BPF Map Allocations<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Run the following command to dump map identifiers and locate the memory consumed by your active tracing maps:<\/p>\n<pre><code class=\"language-bash\">\nbpftool map show\n<\/code><\/pre>\n<p>Look for the map IDs associated with your tracing collector. You will typically see output resembling this:<\/p>\n<pre><code class=\"language-text\">\n48: hash  name active_spans  flags 0x0\n    key 32B  value 64B  max_entries 2097152  memlock 142193280B\n49: ringbuf  name trace_ringbuf  flags 0x0\n    key 0B  value 0B  max_entries 67108864  memlock 67141632B\n<\/code><\/pre>\n<p>That `memlock` value of 142MB for the hash map and 67MB for the ring buffer is just the base allocation. When accounting for overhead, fragmentation, and per-CPU allocations in older kernels, the actual RSS footprint swells by a factor of three. If `max_entries` is set to two million for a hash map holding 64-byte values, the memory footprint scales brutally with system limits.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Tracing_Drop_Counters_with_perf\"><\/span>Tracing Drop Counters with perf<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>To verify if your ring buffer is dropping events due to reader starvation, hook into the kernel tracepoints for ring buffer operations:<\/p>\n<pre><code class=\"language-bash\">\nperf record -e bpf:* -a -- sleep 10\nperf script\n<\/code><\/pre>\n<p>If you see repeated occurrences of `bpf_ringbuf_output_error` or drops incrementing in the output of `bpftool map stats`, your user-space consumer thread inside Envoy is blocked. The collector cannot keep up with the instrumentation overhead, leading to silent accumulation of unread records.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Architectural_Fixes_LRU_Maps_and_Garbage_Collection_Loops\"><\/span>Architectural Fixes: LRU Maps and Garbage Collection Loops<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>You cannot trust transient network streams to always close cleanly. Relying on TCP state machine transitions to clean up eBPF maps is a fool&#8217;s errand because intermediate firewalls, packet drops, and load balancer health checks leave half-open connections hanging indefinitely. You must alter how the kernel stores trace states.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Migrating_to_LRU_Hash_Maps\"><\/span>Migrating to LRU Hash Maps<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Replace standard hash maps with Least Recently Used (LRU) variants. An LRU map guarantees that when `max_entries` is reached, the kernel automatically evicts the oldest, unaccessed trace contexts. This caps memory usage deterministically, protecting your nodes from OOM kills at the cost of dropping trace telemetry for extremely long-lived, idle connections.<\/p>\n<pre><code class=\"language-c\">\nstruct {\n    __uint(type, BPF_MAP_TYPE_LRU_HASH);\n    __uint(max_entries, 500000); \/\/ Strict cap on memory footprint\n    __type(key, struct trace_key_t);\n    __type(value, struct span_context_t);\n} active_spans_lru SEC(\".maps\");\n<\/code><\/pre>\n<p>By enforcing a strict ceiling of 500,000 entries, you limit the memory consumed by the hash map to a predictable ceiling, regardless of how many orphaned connections linger on the wire.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Implementing_Kernel-Side_Expiration_Timestamps\"><\/span>Implementing Kernel-Side Expiration Timestamps<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>LRU eviction helps, but long-lived active streaming gRPC connections can keep an entry fresh indefinitely if data flows continuously. You need explicit time-to-live (TTL) validation inside the eBPF lookup routine. Check the timestamp on every map lookup and purge stale entries inline:<\/p>\n<pre><code class=\"language-c\">\n#define TTL_NS 30000000000ULL \/\/ 30 seconds\n\nSEC(\"kprobe\/ssl_read\")\nint handle_ssl_read(struct pt_regs *ctx) {\n    struct trace_key_t key = {};\n    \/\/ Populate key from socket context...\n\n    struct span_context_t *val = bpf_map_lookup_elem(&active_spans_lru, &key);\n    if (val) {\n        __u64 now = bpf_ktime_get_ns();\n        if (now - val->start_time > TTL_NS) {\n            \/\/ Stale entry detected, delete it inline\n            bpf_map_delete_elem(&active_spans_lru, &key);\n            return 0;\n        }\n    }\n    return 0;\n}\n<\/code><\/pre>\n<p>This snippet intercepts lookups, measures elapsed time against a strict thirty-second window, and evicts dead contexts directly from the eBPF program before they accumulate into gigabytes of waste.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Configuring_Envoy_Tracing_Extensions_Safely\"><\/span>Configuring Envoy Tracing Extensions Safely<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Even with optimal eBPF map configurations, the way Envoy loads, attaches, and communicates with your tracing subsystem matters. A misconfigured tracing filter will spawn runaway background threads that bypass cgroup limits entirely.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Refining_the_Envoy_Tracing_Bootstrap_Configuration\"><\/span>Refining the Envoy Tracing Bootstrap Configuration<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>When defining your tracing provider inside the Envoy bootstrap YAML, ensure that buffer sizes and flushing intervals are explicitly tuned to prevent internal queue bloat. Never use default configurations in production.<\/p>\n<pre><code class=\"language-yaml\">\nstatic_resources:\n  listeners:\n  - name: ingress_listener\n    address:\n      socket_address:\n        address: 0.0.0.0\n        port_value: 10000\n    filter_chains:\n    - filters:\n      - name: envoy.filters.network.http_connection_manager\n        typed_config:\n          \"@type\": type.googleapis.com\/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\n          stat_prefix: ingress_http\n          tracing:\n            provider:\n              name: envoy.tracers.ebpf\n              typed_config:\n                \"@type\": type.googleapis.com\/envoy.config.trace.v3.EbpfTraceConfig\n                ringbuf_size_bytes: 8388608 # Capped strictly at 8MB\n                max_active_spans: 100000\n                flush_interval_ms: 500\n          route_config:\n            name: local_route\n            virtual_hosts:\n            - name: edge_service\n              domains: [\"*\"]\n              routes:\n              - match:\n                  prefix: \"\/\"\n                route:\n                  cluster: internal-auth-grpc\n<\/code><\/pre>\n<p>Setting `ringbuf_size_bytes` to 8MB forces the telemetry pipeline to shed load rather than consuming infinite kernel memory when downstream ingestion endpoints saturate. If your tracing collector goes down, Envoy drops spans instead of dying in an OOM event.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Resource_Control_via_Cgroups\"><\/span>Resource Control via Cgroups<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Ensure your container runtime enforces strict memory limits not just on Envoy&#8217;s userspace RSS, but on the kernel memory accounting tied to the container cgroup. Modern Linux kernels track BPF memory allocations via `memory.kmem.current` or unified cgroup v2 accounting.<\/p>\n<pre><code class=\"language-text\">\n# Verify cgroup v2 memory accounting for BPF maps inside the container\ncat \/sys\/fs\/cgroup\/memory.current\ncat \/sys\/fs\/cgroup\/memory.stat\n<\/code><\/pre>\n<p>If `memsw.limit_in_bytes` or equivalent cgroup parameters are not configured to account for pinned BPF maps, the host kernel will allow eBPF allocations to bypass container memory constraints until the node itself panics.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Verifying_the_Patch_in_Staging\"><\/span>Verifying the Patch in Staging<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Before rolling these fixes to `envoy-ingress-edge-7c9b` in production, validate the memory profile under synthetic load. Fire up `vegeta` or `wrk` against your staging ingress gateway while monitoring the slab allocator and BPF map memory consumption.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Load_Testing_the_Telemetry_Pipeline\"><\/span>Load Testing the Telemetry Pipeline<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<pre><code class=\"language-bash\">\nvegeta attack -targets=targets.txt -rate=50000 -duration=30m | vegeta encode > results.bin\ncat results.bin | vegeta report\n<\/code><\/pre>\n<p>While the load test runs, execute a continuous loop to check map memory usage:<\/p>\n<pre><code class=\"language-bash\">\nwhile true; do bpftool map show | grep -E \"name|memlock\"; sleep 5; done\n<\/code><\/pre>\n<p>With the LRU map implementation and explicit TTL eviction logic active, the `memlock` value will plateau rapidly, stabilizing at the configured `max_entries` threshold. The continuous linear climb disappears. RSS on the Envoy process flatlines well below your cgroup threshold.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Summary_of_Production_Rules_for_eBPF_Tracing\"><\/span>Summary of Production Rules for eBPF Tracing<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Implementing ebpfbased distributed tracing without understanding kernel memory limits is an operational disaster waiting for a high-traffic window. To keep your edge proxies alive, follow these absolute constraints:<\/p>\n<ul>\n<li>Never deploy standard `BPF_MAP_TYPE_HASH` maps for unbounded connection tracking; enforce `BPF_MAP_TYPE_LRU_HASH` with explicit capacity limits.<\/li>\n<li>Cap ring buffer allocations in your Envoy bootstrap configuration to force telemetry shedding instead of kernel memory exhaustion.<\/li>\n<li>Validate that your container runtime accounts for BPF `memlock` overhead within your cgroup memory limits.<\/li>\n<li>Incorporate inline timestamp checks and TTL-based deletions inside your eBPF programs to purge orphaned states left by dropped TCP connections.<\/li>\n<\/ul>\n<p>Configure your telemetry defensively. The kernel owes your service nothing.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Related_Articles\"><\/span>Related Articles<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Explore more insights and best practices:<\/p>\n<ul>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/10-essential-aws-best-practices-for-cloud-optimization-4\/\">10 Essential Aws Best Practices For Cloud Optimization 4<\/a><\/li>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/what-is-kubernetes-a-simple-guide-to-k8s-orchestration\/\">What Is Kubernetes A Simple Guide To K8S Orchestration<\/a><\/li>\n<li><a href=\"https:\/\/itsupportwale.com\/blog\/what-is-artificial-intelligence-definition-types-and-examples\/\">What Is Artificial Intelligence Definition Types And Examples<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>The 03:14 AM OOM Sweep [Wed Oct 18 03:14:02 2023] oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=900 [Wed Oct 18 03:14:02 2023] CPU: 37 PID: 14892 Comm: envoy Kdump: nullptr Tainted: G OE 5.15.0-88-generic #98-Ubuntu [Wed Oct 18 03:14:02 2023] Mem-Total: 32901416kB, AnonPages: 29841904kB, Shmem: 11232kB, Slab: 412091kB [Wed Oct 18 03:14:02 2023] Active(anonymous): 18402912 Inactive: 11419024 Isolated:\u2080 &#8230; <a title=\"ebpfbased distributed tracing in Envoy: Fix Leaks &#038; Overhead\" class=\"read-more\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/\" aria-label=\"Read more  on ebpfbased distributed tracing in Envoy: Fix Leaks &#038; Overhead\">Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[685,684,686,687,688],"class_list":["post-5338","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-distributed-tracing","tag-ebpf","tag-envoy-proxy","tag-memory-leak","tag-performance"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.8 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>ebpfbased distributed tracing in Envoy: Fix Leaks &amp; Overhead<\/title>\n<meta name=\"description\" content=\"Learn how to optimize ebpfbased distributed tracing in Envoy proxies by mitigating performance overhead and fixing critical memory leaks effectively.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"ebpfbased distributed tracing in Envoy: Fix Leaks &amp; Overhead\" \/>\n<meta property=\"og:description\" content=\"Learn how to optimize ebpfbased distributed tracing in Envoy proxies by mitigating performance overhead and fixing critical memory leaks effectively.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/\" \/>\n<meta property=\"og:site_name\" content=\"ITSupportWale\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Itsupportwale-298547177495978\" \/>\n<meta property=\"article:published_time\" content=\"2026-11-12T21:48:06+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2021\/05\/android-chrome-512x512-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"512\" \/>\n\t<meta property=\"og:image:height\" content=\"512\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Techie\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Techie\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\\\/\"},\"author\":{\"name\":\"Techie\",\"@id\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/#\\\/schema\\\/person\\\/8c5a2b3d36396e0a8fd91ec8242fd46d\"},\"headline\":\"ebpfbased distributed tracing in Envoy: Fix Leaks &#038; Overhead\",\"datePublished\":\"2026-11-12T21:48:06+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\\\/\"},\"wordCount\":1482,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/#organization\"},\"keywords\":[\"Distributed Tracing\",\"eBPF\",\"Envoy Proxy\",\"Memory Leak\",\"Performance\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\\\/\",\"url\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\\\/\",\"name\":\"ebpfbased distributed tracing in Envoy: Fix Leaks & Overhead\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/#website\"},\"datePublished\":\"2026-11-12T21:48:06+00:00\",\"description\":\"Learn how to optimize ebpfbased distributed tracing in Envoy proxies by mitigating performance overhead and fixing critical memory leaks effectively.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"ebpfbased distributed tracing in Envoy: Fix Leaks &#038; Overhead\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/\",\"name\":\"ITSupportWale\",\"description\":\"Tips, Tricks, Fixed-Errors, Tutorials &amp; Guides\",\"publisher\":{\"@id\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/#organization\",\"name\":\"itsupportwale\",\"url\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/09\\\/cropped-Logo-trans-without-slogan.png\",\"contentUrl\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/wp-content\\\/uploads\\\/2023\\\/09\\\/cropped-Logo-trans-without-slogan.png\",\"width\":1119,\"height\":144,\"caption\":\"itsupportwale\"},\"image\":{\"@id\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/Itsupportwale-298547177495978\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/#\\\/schema\\\/person\\\/8c5a2b3d36396e0a8fd91ec8242fd46d\",\"name\":\"Techie\",\"sameAs\":[\"https:\\\/\\\/itsupportwale.com\",\"iswblogadmin\"],\"url\":\"https:\\\/\\\/itsupportwale.com\\\/blog\\\/author\\\/iswblogadmin\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"ebpfbased distributed tracing in Envoy: Fix Leaks & Overhead","description":"Learn how to optimize ebpfbased distributed tracing in Envoy proxies by mitigating performance overhead and fixing critical memory leaks effectively.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/","og_locale":"en_US","og_type":"article","og_title":"ebpfbased distributed tracing in Envoy: Fix Leaks & Overhead","og_description":"Learn how to optimize ebpfbased distributed tracing in Envoy proxies by mitigating performance overhead and fixing critical memory leaks effectively.","og_url":"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/","og_site_name":"ITSupportWale","article_publisher":"https:\/\/www.facebook.com\/Itsupportwale-298547177495978","article_published_time":"2026-11-12T21:48:06+00:00","og_image":[{"width":512,"height":512,"url":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2021\/05\/android-chrome-512x512-1.png","type":"image\/png"}],"author":"Techie","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Techie","Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#article","isPartOf":{"@id":"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/"},"author":{"name":"Techie","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d"},"headline":"ebpfbased distributed tracing in Envoy: Fix Leaks &#038; Overhead","datePublished":"2026-11-12T21:48:06+00:00","mainEntityOfPage":{"@id":"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/"},"wordCount":1482,"commentCount":0,"publisher":{"@id":"https:\/\/itsupportwale.com\/blog\/#organization"},"keywords":["Distributed Tracing","eBPF","Envoy Proxy","Memory Leak","Performance"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/","url":"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/","name":"ebpfbased distributed tracing in Envoy: Fix Leaks & Overhead","isPartOf":{"@id":"https:\/\/itsupportwale.com\/blog\/#website"},"datePublished":"2026-11-12T21:48:06+00:00","description":"Learn how to optimize ebpfbased distributed tracing in Envoy proxies by mitigating performance overhead and fixing critical memory leaks effectively.","breadcrumb":{"@id":"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/itsupportwale.com\/blog\/ebpfbased-distributed-tracing-in-envoy-fix-leaks-and-overhead\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/itsupportwale.com\/blog\/"},{"@type":"ListItem","position":2,"name":"ebpfbased distributed tracing in Envoy: Fix Leaks &#038; Overhead"}]},{"@type":"WebSite","@id":"https:\/\/itsupportwale.com\/blog\/#website","url":"https:\/\/itsupportwale.com\/blog\/","name":"ITSupportWale","description":"Tips, Tricks, Fixed-Errors, Tutorials &amp; Guides","publisher":{"@id":"https:\/\/itsupportwale.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/itsupportwale.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/itsupportwale.com\/blog\/#organization","name":"itsupportwale","url":"https:\/\/itsupportwale.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png","contentUrl":"https:\/\/itsupportwale.com\/blog\/wp-content\/uploads\/2023\/09\/cropped-Logo-trans-without-slogan.png","width":1119,"height":144,"caption":"itsupportwale"},"image":{"@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/Itsupportwale-298547177495978"]},{"@type":"Person","@id":"https:\/\/itsupportwale.com\/blog\/#\/schema\/person\/8c5a2b3d36396e0a8fd91ec8242fd46d","name":"Techie","sameAs":["https:\/\/itsupportwale.com","iswblogadmin"],"url":"https:\/\/itsupportwale.com\/blog\/author\/iswblogadmin\/"}]}},"_links":{"self":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts\/5338","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/comments?post=5338"}],"version-history":[{"count":0,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/posts\/5338\/revisions"}],"wp:attachment":[{"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/media?parent=5338"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/categories?post=5338"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/itsupportwale.com\/blog\/wp-json\/wp\/v2\/tags?post=5338"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}