[10842.123456] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=\/,mems_allowed=0,oom_memcg=\/kubepods\/besteffort\/pod123,task_memcg=\/kubepods\/besteffort\/pod123,task=java,pid=12345,uid=0\n[10842.123500] Memory cgroup out of memory: Killed process 12345 (java) total-vm:8421024kB, anon-rss:4194304kB, file-rss:0kB, shmem-rss:0kB\n<\/code><\/pre>\nYou see that uid=0<\/code>? We\u2019ll get to that. But look at the anon-rss<\/code>. 4GB. The container had no limits set in the docker-compose.yml<\/code> or the manifest. It just kept eating. In the world of Linux namespaces and cgroups, a container without a limit is a suicide pact. The kernel doesn’t care about your “critical business logic.” It sees a process hogging pages, and it executes it.<\/p>\nThe “fix” isn’t just adding a mem_limit<\/code>. It\u2019s understanding how the JVM interacts with cgroups. If you don’t use -XX:+UseContainerSupport<\/code>, the JVM looks at the host’s total memory, not the container’s limit. You\u2019re lying to your app, and the kernel is the debt collector.<\/p>\n<\/span>2. The Root Cause: 4GB Images and the “It Works on My Machine” Fallacy<\/span><\/h2>\nI pulled the image that caused the saturation. docker inspect<\/code> revealed a horror show. <\/p>\n"RootFS": {\n "Type": "layers",\n "Layers": [\n "sha256:7297da16...", \n "sha256:8527027...",\n "sha256:..." \/\/ 42 more layers of pure incompetence\n ]\n}\n<\/code><\/pre>\nThe developer used Ubuntu:latest<\/code> as a base. Then they ran apt-get update<\/code> and apt-get install<\/code> in six different RUN<\/code> commands. They left the apt<\/code> cache in the image. They included the entire build-time toolchain\u2014GCC, Python, Go, and probably a copy of the Oxford English Dictionary\u2014in the production runtime image.<\/p>\nThis is the “Layer Cake of Lies.” Every RUN<\/code> command creates a new layer on the overlay2<\/code> filesystem. If you delete a file in a subsequent layer, it\u2019s not gone; it\u2019s just hidden by a “whiteout” file. The bits are still there, taking up space, slowing down docker pull<\/code>, and increasing the attack surface. <\/p>\nWhen the cluster tried to recover, it had to move 4.2GB across the wire for every single pod. Our internal registry’s disk I\/O spiked so hard the metadata database started throwing 500s. We weren’t just down; we were dead-locked.<\/p>\n
<\/span>3. Layer Optimization: Stop Rebuilding the World on Every Git Commit<\/span><\/h2>\nI looked at the Dockerfile<\/code>. It was a masterpiece of inefficiency.<\/p>\n<\/span>The “Broken” Way (The Junior Dev Special)<\/span><\/h3>\nFROM ubuntu:latest\n# No version pinning. Good luck in six months.\nRUN apt-get update\nRUN apt-get install -y nodejs npm\nCOPY . \/app\n# This COPY is the kiss of death. \n# Any change to a README.md invalidates the cache for everything below.\nRUN npm install\nWORKDIR \/app\nCMD ["node", "server.js"]\n<\/code><\/pre>\nEvery time a dev changed a single line of CSS, the RUN npm install<\/code> would trigger. That\u2019s 500MB of node_modules<\/code> being downloaded and compressed into a new layer. Every. Single. Time.<\/p>\n<\/span>The “SRE Way” (Hardened Multi-stage)<\/span><\/h3>\nTo ensure long-term stability and keep the “docker best” practices alive, we use multi-stage builds and specific base images. We use Alpine 3.19<\/code> or Debian Bookworm-slim<\/code>. We pin versions. We respect the cache.<\/p>\n# Stage 1: Build\nFROM node:20.11.0-bookworm-slim AS builder\nWORKDIR \/app\n# Only copy files needed for dependency resolution\nCOPY package.json package-lock.json .\/\nRUN npm ci --only=production\n\n# Stage 2: Runtime\nFROM node:20.11.0-bookworm-slim\nRUN apt-get update && apt-get install -y --no-install-recommends \\\n dumb-init=1.2.5-2 \\\n && rm -rf \/var\/lib\/apt\/lists\/*\n\nWORKDIR \/app\n# Copy only the artifacts from the builder\nCOPY --from=builder \/app\/node_modules .\/node_modules\nCOPY . .\n\n# Security: Don't run as root\nUSER node\nENTRYPOINT ["\/usr\/bin\/dumb-init", "--"]\nCMD ["node", "server.js"]\n<\/code><\/pre>\nBy separating the build environment from the runtime environment, we dropped the image size from 4.2GB to 180MB. We used npm ci<\/code> for deterministic builds. We cleaned up the apt<\/code> lists in the same RUN<\/code> command to prevent layer bloat. This isn’t just “optimization”; it’s survival.<\/p>\n<\/span>4. The Security Gap: Why You\u2019re Running as Root and Why I\u2019m Revoking Your SSH Access<\/span><\/h2>\nDuring the incident, I noticed something even more disturbing. One of the compromised containers had a shell history that wasn’t mine. Because the container was running as root<\/code> (the default in Docker if you\u2019re lazy), a simple remote code execution (RCE) vulnerability in the web app gave the attacker a root shell inside the container.<\/p>\nFrom there, they checked for the existence of \/var\/run\/docker.sock<\/code>. And guess what? Some “DevOps Engineer” had mounted it so the container could “manage other containers.”<\/p>\nMounting the Docker socket is equivalent to giving the container sudo<\/code> access to the host without a password. It\u2019s a container escape waiting to happen. An attacker can just run docker run -v \/:\/host -it ubuntu:latest chroot \/host<\/code> and they own the entire node. They own the kernel. They own the data.<\/p>\nWe use seccomp<\/code> profiles and AppArmor<\/code>. We restrict syscalls. Does your app really need mount()<\/code>, ptrace()<\/code>, or kexec_load()<\/code>? No? Then why are they available to your container? Docker Engine v24 ships with a default seccomp profile, but it\u2019s broad. A “docker best” approach involves profiling the app and dropping every capability except NET_BIND_SERVICE<\/code>.<\/p>\nAnd for the love of all that is holy, stop using latest<\/code>. latest<\/code> is not a version. It\u2019s a moving target. It\u2019s a rolling dice. When you pull latest<\/code>, you have no idea what code is actually running. You can\u2019t roll back because latest<\/code> today is different from latest<\/code> yesterday. Use SHA256 hashes if you\u2019re serious, or at least specific SemVer tags.<\/p>\n<\/span>5. Signal Handling: Why Your App Won’t Shut Down Gracefully (PID 1 Blues)<\/span><\/h2>\nDuring the meltdown, I tried to restart the services. docker stop<\/code> took 10 seconds for every container. Why? Because your app is running as PID 1 and it\u2019s deaf to signals.<\/p>\nWhen you use the “shell form” of CMD<\/code> or ENTRYPOINT<\/code>:
\nCMD node server.js<\/code><\/p>\nDocker executes it as \/bin\/sh -c \"node server.js\"<\/code>. The shell starts as PID 1. When Docker sends a SIGTERM<\/code> to the container, it goes to the shell. The shell, being a stubborn piece of 1970s technology, does not forward that signal to the child process (node). The app keeps running, unaware that the reaper is coming. After 10 seconds, Docker loses patience and sends SIGKILL<\/code>.<\/p>\nSIGKILL<\/code> is the nuclear option. It doesn’t allow the app to close database connections, flush buffers, or finish processing a request. It just stops the CPU in its tracks. This leads to database corruption and “zombie” records.<\/p>\nThe fix is the “exec form”:
\nCMD [\"node\", \"server.js\"]<\/code><\/p>\nOr better yet, use dumb-init<\/code> or tini<\/code>. These are tiny init systems designed to run as PID 1, reap zombie processes, and correctly forward signals. If your app doesn’t shut down in under a second, you\u2019ve failed.<\/p>\n<\/span>6. The “docker best” Manifesto: A Checklist for People Who Don’t Want to Get Fired<\/span><\/h2>\nI\u2019m going to go get another coffee. While I\u2019m gone, you\u2019re going to rewrite your docker-compose.yml<\/code> and your Dockerfile<\/code>s. If I see another 4GB image or a container running as root, I\u2019m revoking your production access and moving you to the documentation team.<\/p>\nHere is the docker-compose.yml<\/code> that I expect to see. It uses version: '3.8'<\/code> syntax (or the newer Compose Specification) and defines strict resource constraints.<\/p>\nservices:\n api-gateway:\n image: our-registry.io\/api-gateway:v2.4.1@sha256:a1b2c3d4...\n deploy:\n resources:\n limits:\n cpus: '0.50'\n memory: 512M\n reservations:\n cpus: '0.25'\n memory: 256M\n restart: on-failure:3\n security_opt:\n - no-new-privileges:true\n read_only: true\n tmpfs:\n - \/tmp\n - \/run\n logging:\n driver: "json-file"\n options:\n max-size: "10m"\n max-file: "3"\n healthcheck:\n test: ["CMD", "curl", "-f", "http:\/\/localhost:8080\/health"]\n interval: 30s\n timeout: 10s\n retries: 3\n start_period: 40s\n<\/code><\/pre>\n<\/span>The Checklist<\/span><\/h3>\n\n- No Root Users:<\/strong> Always define a
USER<\/code> in your Dockerfile. If you need to bind to a port below 1024, use CAP_NET_BIND_SERVICE<\/code>, not root.<\/li>\n- Explicit Resource Limits:<\/strong> If you don’t set a
memory_limit<\/code>, you are volunteering to be the first person I call at 3 AM.<\/li>\n- Multi-Stage Builds:<\/strong> If your production image contains a compiler, you\u2019ve done it wrong.<\/li>\n
- Small Base Images:<\/strong> Use
Alpine<\/code> or distroless<\/code>. If you use Ubuntu<\/code>, use the -slim<\/code> variant and clean up your mess.<\/li>\n- Immutable Tags:<\/strong>
latest<\/code> is a crime. Pin your versions. Pin your SHA hashes.<\/li>\n- Read-Only Filesystems:<\/strong> Use
read_only: true<\/code> in your compose file. If your app needs to write to a temp directory, use a tmpfs<\/code> mount. This kills 90% of automated exploit payloads.<\/li>\n- Signal Handling:<\/strong> Use
exec<\/code> form for CMD<\/code>. Use an init process like tini<\/code>.<\/li>\n- The .dockerignore File:<\/strong> Stop sending your
.git<\/code> folder and node_modules<\/code> to the Docker daemon. It slows down the build and leaks secrets.<\/li>\n<\/ol>\nI\u2019ve spent a decade in the Linux terminal. I\u2019ve seen ext4<\/code> filesystems dissolve like cotton candy in the rain. I\u2019ve seen iptables<\/code> rules that would make a cryptographer weep. Docker is a tool, not a magic wand. If you treat it like a “black box” where you can just throw your garbage code and expect it to run forever, you are the problem.<\/p>\nThe cluster didn’t scream because of a bug in Docker. It screamed because of you. It screamed because it was bloated, insecure, and unmanaged. <\/p>\n
Go fix your images. I\u2019m going to sleep. If my pager goes off again because of an OOM event, I\u2019m not fixing the cluster\u2014I\u2019m fixing the hiring process.<\/p>\n
<\/span>Deep Dive: The Overlay2 Storage Driver and Why It Hates You<\/span><\/h3>\nSince you’re still here, let’s talk about why your 4GB image actually breaks the disk. Docker uses the overlay2<\/code> driver. It works by stacking “lower” directories and a single “upper” directory. When you write a file, it uses “copy-on-write” (CoW). <\/p>\nIf you have a 1GB log file in a lower layer and you run chmod 777<\/code> on it in your Dockerfile, overlay2<\/code> has to copy that entire 1GB file to the new layer just to change the permission bits. You now have 2GB of disk usage for a 1GB file. This is why we chain commands:<\/p>\n# WRONG\nRUN wget http:\/\/huge-file.tar.gz\nRUN tar -xvf huge-file.tar.gz\nRUN rm huge-file.tar.gz\n\n# RIGHT\nRUN wget http:\/\/huge-file.tar.gz && \\\n tar -xvf huge-file.tar.gz && \\\n rm huge-file.tar.gz\n<\/code><\/pre>\nIn the “RIGHT” example, the file is downloaded, extracted, and deleted in a single layer<\/em>. The overlay2<\/code> driver never has to commit the .tar.gz<\/code> to disk permanently. This is “docker best” practice 101, and yet, I see people failing it every single day.<\/p>\nAnd don’t get me started on storage-opts<\/code>. We’re running overlay2.override_kernel_check=true<\/code> because we know our kernel (6.1 LTS) can handle it. We\u2019ve tuned the xfs<\/code> backing store for the \/var\/lib\/docker<\/code> partition with pquota<\/code> to prevent a single container from filling the entire host’s disk with logs. Did you know you could do that? No, you were too busy adding another RUN<\/code> command to your Dockerfile.<\/p>\n<\/span>The Final Word on Seccomp and Syscalls<\/span><\/h3>\nIf you really want to impress me, stop looking at Docker as a way to “package apps” and start looking at it as a way to “sandbox processes.” <\/p>\n
A container is just a process with a fancy hat. It still talks to the same kernel. Every time your app makes a syscall\u2014open()<\/code>, read()<\/code>, write()<\/code>, socket()<\/code>\u2014it\u2019s an opportunity for something to go wrong. By using a custom seccomp<\/code> profile, you can restrict the process so it can only<\/em> do what it\u2019s supposed to do. <\/p>\nIf your Node.js app starts trying to call execve()<\/code> to run a shell script it just downloaded into \/tmp<\/code>, a good seccomp<\/code> profile will kill the process instantly. That\u2019s the difference between a “security incident” and a “blocked syscall” log entry.<\/p>\nI\u2019m done. The sun is coming up. The cluster is stable, for now. Don’t make me come back here. Fix your habits. Respect the cgroups. And never, ever use latest<\/code> again.<\/p>\n<\/span>Related Articles<\/span><\/h2>\n