POST-MORTEM REPORT: INCIDENT #8842-B (REGISTRY OOM-KILL)
DATE: 03:14 AM
AUTHOR: Senior Systems Architect (Infrastructure/Core)
STATUS: Severely Caffeinated / Highly Irritated
SUBJECT: THE MANIFESTO OF EFFICIENCY: WHY YOUR 4GB IMAGE IS A CRIME AGAINST ENGINEERING
It is 3:00 AM. My pager didn’t just beep; it screamed because our private registry’s storage volume hit 100% capacity and the garbage collector choked to death trying to process a single push. I looked at the logs. I saw the culprit. A “microservice”—and I use that term with the same irony one might use to describe a lead-lined coffin—that clocked in at 4.2GB.
Four. Gigabytes. For a Node.js API that returns a JSON object.
I’ve spent the last hour purging the registry and manually cleaning up the overlay2 directories on the build nodes. To ensure I never have to do this again, I am pinning this manifesto. If you push an image over 200MB without a written justification signed in blood, I am revoking your sudo access and moving your desk to the basement.
Here is the “masterpiece” I found in the repository. Look at it. Study its failure.
# THE CRIME SCENE
FROM ubuntu:latest
RUN apt-get update
RUN apt-get install -y nodejs npm python3 build-essential git
COPY . /app
WORKDIR /app
RUN npm install
ENV API_KEY="secret_key_that_is_now_in_git_history"
RUN rm -rf /var/lib/apt/lists/*
EXPOSE 3000
CMD ["npm", "start"]
This isn’t a container; it’s a virtual machine with an identity crisis. Let’s dissect why this is an objective failure of professional standards.
Table of Contents
1. STOP USING UBUNTU FOR MICROSERVICES
Why are you using ubuntu:latest? Do you need a full suite of networking tools, a package manager with a massive cache, and a kernel-adjacent userland just to run a Javascript runtime? No. You don’t. When you pull ubuntu:latest, you are pulling roughly 72MB of compressed air that you don’t need. But it’s not just the size; it’s the attack surface.
Every utility in that base image is a tool for an attacker once they find the inevitable RCE in your poorly-sanitized input fields. If you actually gave a damn about docker best practices, you’d be using alpine or distroless.
Alpine 3.19 is 5MB. Five. Megabytes.
“But Arthur, Alpine uses musl instead of glibc and my native modules won’t compile!” Then use a -slim Debian-based image like python:3.11-slim-bookworm or node:20-slim. It’s not hard. It’s laziness. You are trading my sleep for your convenience.
# Terminal Log: Base Image Comparison
$ docker images
REPOSITORY TAG IMAGE ID SIZE
ubuntu latest ba627df2ed29 72.8MB
node 20-slim 63399009890d 182MB
node 20-alpine 42846937243c 54.1MB
alpine 3.19 054309236301 7.34MB
2. THE LAYER CAKE IS A LIE
Docker images are a stack of read-only layers. Every RUN, COPY, or ADD command creates a new layer. If you create a 1GB file in one RUN command and delete it in the next, your image is still 1GB larger. The file is still there, buried in the previous layer, haunting the storage driver.
In the “Crime Scene” Dockerfile above, the junior dev tried to be clever by running RUN rm -rf /var/lib/apt/lists/* at the end. Too late. The layer containing the apt-get update cache was already committed. You’ve just added a new layer that says “pretend these files aren’t here,” while the registry still has to store and transmit the original data.
Look at the history of the 4GB monstrosity:
# Terminal Log: docker history --human --format "{{.CreatedBy}}: {{.Size}}"
$ docker history bloated-service:latest
/bin/sh -c #(nop) CMD ["npm" "start"]: 0B
/bin/sh -c rm -rf /var/lib/apt/lists/*: 0B
/bin/sh -c npm install: 2.1GB
/bin/sh -c #(nop) WORKDIR /app: 0B
/bin/sh -c COPY . /app: 1.8GB
/bin/sh -c apt-get install -y nodejs npm...: 450MB
/bin/sh -c apt-get update: 28MB
Notice the COPY . /app taking up 1.8GB? That’s because the .git folder, the local node_modules, and the dist folder were all sucked into the image. Which leads me to my next point.
3. YOUR DOCKERIGNORE IS NON-EXISTENT AND IT SHOWS
If I see one more image containing a .git directory, I’m going to start charging the offending developer for the S3 egress fees out of their paycheck. Disk space isn’t free, despite what your cloud provider tells you.
The “build context” is the set of files sent from your local machine (or the CI runner) to the Docker daemon. If you don’t have a .dockerignore file, you are sending everything. In the incident tonight, the junior dev had a tmp/ folder full of local test database dumps.
# Terminal Log: The Build Context Bloat
$ docker build -t bad-service .
Sending build context to Docker daemon 2.45GB
Step 1/10 : FROM ubuntu:latest
...
Two and a half gigabytes sent over the network before the build even started. A proper .dockerignore should be your first line of defense. It should look like this:
.git
node_modules
npm-debug.log
Dockerfile
.dockerignore
.env
tmp/
dist/
4. MULTI-STAGE BUILDS ARE NOT OPTIONAL
This is the hill I will die on. You do not need gcc, make, or python in your production image just because one of your dependencies needed to compile a C++ binding during installation.
Multi-stage builds allow you to create a “build” container with all the heavy tools and then copy only the compiled artifacts into a “run” container. This is how you get an image from 4GB down to 40MB.
If you don’t use multi-stage builds, you are essentially shipping a factory along with the car you built. It’s inefficient, it’s bloated, and it’s a security nightmare.
# THE RIGHT WAY: Multi-Stage
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
FROM node:20-alpine
WORKDIR /app
# Copy only the necessary bits from the builder
COPY --from=builder /app/node_modules ./node_modules
COPY . .
USER node
CMD ["node", "index.js"]
5. ROOT IS FOR KERNELS, NOT FOR YOUR NODE APP
The “Crime Scene” Dockerfile runs as root. By default, everything in a container runs as root. If an attacker escapes your application via a vulnerability, they aren’t just a user; they are the king of the container. If you’ve been foolish enough to mount /var/run/docker.sock into the container (and I know some of you are doing it for “monitoring”), they now have root access to the host.
Stop it.
Most reputable base images provide a non-privileged user. For Node, it’s node. For others, you should create one. Use a high UID/GID (like 10001) to avoid collisions with host users.
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser
This isn’t just “best practice” fluff; it’s the difference between a minor patch and a company-wide data breach. When you run as a non-root user, you limit the syscalls an attacker can leverage. You make their life harder. That is your job.
6. THE ENTRYPOINT VS CMD HOLY WAR
I see people using CMD ["npm", "start"] and it makes my blood boil. Why? Because npm does not pass signals (like SIGTERM) to the underlying process.
When Kubernetes or the OOM killer tries to shut down your container, it sends a SIGTERM. If your app is wrapped in npm or a shell script that doesn’t exec, the app never sees the signal. It keeps running until the 10-second grace period expires and the kernel brutally murders it with a SIGKILL. This leads to corrupted database connections and lost state.
Use ENTRYPOINT for the executable and CMD for the arguments, or just use CMD to call the binary directly. And for the love of all that is holy, use the “exec form” (the JSON array ["executable", "param"]), not the “shell form” (executable param). The shell form spawns /bin/sh -c, which—surprise—doesn’t pass signals.
# Correct Signal Handling
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["node", "index.js"]
(Yes, use tini. It handles zombie processes. Look it up.)
7. CACHE MISSES ARE A MORAL FAILURE
The junior’s Dockerfile had COPY . /app before RUN npm install. This means every time a single character changes in a README file, the Docker cache for npm install is invalidated. The build system then has to re-download the entire internet.
This is why our CI pipeline took 15 minutes instead of 30 seconds.
If you actually gave a damn about docker best practices, you’d copy the dependency manifests first, install, and then copy the source code. The source code changes often; the dependencies don’t.
# Terminal Log: Cache Miss Evidence
$ docker build -t service:latest .
Step 4/10 : COPY . /app
---> Using cache
Step 5/10 : RUN npm install
---> Running in a1b2c3d4e5f6
# ... 5 minutes of downloading ...
By reordering the commands, you ensure that the npm install layer is only rebuilt when package.json changes. This is basic. This is Day 1 stuff.
8. THE OCI SPEC DOESN’T CARE ABOUT YOUR FEELINGS
We are moving to a world of OCI (Open Container Initiative) compliance. This means your containers need to be predictable, reproducible, and lean. The syscall overhead of a bloated container is non-trivial. When the kernel has to manage thousands of unnecessary file descriptors and memory mappings for a “microservice” that thinks it’s a full Debian install, performance degrades.
I ran a trivy scan on the image that crashed the registry. The results were a horror show.
# Terminal Log: Trivy Scan Results (The Bloated Image)
$ trivy image bloated-service:latest
Total: 482 (UNKNOWN: 5, LOW: 156, MEDIUM: 220, HIGH: 82, CRITICAL: 19)
CRITICAL: CVE-2023-XXXX (glibc), CVE-2023-YYYY (openssl)...
Nineteen critical vulnerabilities. Nineteen ways for a script kiddie to turn our cluster into a Monero miner. Now, look at the scan of the optimized image using alpine and multi-stage builds:
# Terminal Log: Trivy Scan Results (The Optimized Image)
$ trivy image optimized-service:latest
Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)
Zero vulnerabilities. Zero. And the size?
# Terminal Log: Final df -h Comparison
$ df -h /var/lib/docker
Filesystem Size Used Avail Use% Mounted on
# BEFORE OPTIMIZATION
/dev/nvme0n1 100G 98G 1.5G 99% /var/lib/docker
# AFTER PURGING AND OPTIMIZING
/dev/nvme0n1 100G 12G 82G 13% /var/lib/docker
We went from 99% utilization to 13%. The registry is breathing again. The SRE team is going back to sleep. I, however, am staying awake to write a script that will automatically reject any PR containing a Dockerfile that uses ubuntu:latest or lacks a .dockerignore.
THE CORRECTIVE ACTION PLAN
Effective immediately:
1. Mandatory Multi-Stage Builds: No exceptions. If your final image contains a compiler, the build fails.
2. Base Image Whitelist: Use alpine, distroless, or -slim variants.
3. No Root: All containers must define a USER.
4. Layer Optimization: Manifests (package.json, requirements.txt, go.mod) must be copied and installed before the application source.
5. Secrets: Any secret found in a Dockerfile layer (even if deleted in a later layer) will result in an immediate revocation of production access. Use a secret manager or mount them as build secrets.
I don’t care about “velocity” if that velocity is driving us off a cliff. Speed without stability is just a slow-motion car crash.
Now, if you’ll excuse me, I’m going to find another pot of coffee and delete the junior’s latest tags from the registry. If you have questions, read the documentation. Don’t Slack me. I’m “Away” until noon.
FIX YOUR DOCKERFILES.
Related Articles
Explore more insights and best practices: