It was 3:14 AM when the heap dump hit the fan.
I was three hours into a deep REM cycle when the PagerDuty “EHEHEHEHE” alert ripped through my skull like a rusted chainsaw. I didn’t even have to look at the dashboard to know what was happening. The payment-processor-v2 service, the one the “rockstar” developers pushed on Friday afternoon while I was busy fixing the staging ingress, had finally decided to commit ritual suicide.
By 3:20 AM, I was staring at a Grafana dashboard that looked like a crime scene. Latency was spiking into the tens of seconds. The CPU usage on the worker nodes was pegged at 100%, but the actual throughput was near zero. Then came the dreaded OOMKilled events. One by one, the pods were being executed by the kernel like prisoners in a dystopian novel.
$ kubectl get pods -n production
NAME READY STATUS RESTARTS AGE
payment-processor-v2-7f8d9b6-x4j2l 0/1 CrashLoopBackOff 12 45m
payment-processor-v2-7f8d9b6-z9k1p 0/1 OOMKilled 8 42m
payment-processor-v2-7f8d9b6-m2n5q 0/1 CrashLoopBackOff 15 50m
I pulled the logs. It was a wall of garbage. But then I saw it—the smoking gun.
2024-05-20T03:15:22.441Z ERROR: Failed to load configuration.
Error: Cannot find module '/app/config/secrets.json'
at Function.Module._resolveFilename (node:internal/modules/cjs/loader:1077:15)
...
2024-05-20T03:15:23.102Z INFO: Attempting to pull latest image...
The “latest” image. My blood ran cold. Some junior dev had configured the deployment to use image: payment-processor:latest. Overnight, a CI/CD job had finished, pushed a broken build to the registry, and because the image pull policy was set to Always, the cluster dutifully pulled the poison pill and distributed it to every node.
I spent the next 44 hours manually rolling back tags, purging bloated layers from our local registry that had hit 98% disk utilization, and rewriting Dockerfiles that were so poorly constructed they should have been considered a violation of the Geneva Convention.
If you think you’re following docker best practices while still using :latest or running your processes as root, you’re delusional. You’re not a developer; you’re a liability.
Table of Contents
1. THE AUTOPSY OF A 2.4GB MONSTROSITY
Once the bleeding stopped and the service was stabilized on a pinned version, I went back to see how the “latest” image had grown to 2.4GB for a simple Node.js API. I ran docker history on the offending image. It was a horror show.
$ docker history payment-processor:latest
IMAGE CREATED CREATED BY SIZE COMMENT
<missing> 2 hours ago /bin/sh -c #(nop) CMD ["npm" "start"] 0B
<missing> 2 hours ago /bin/sh -c apt-get update && apt-get install… 850MB
<missing> 2 hours ago /bin/sh -c #(nop) COPY dir:7e3... in /app 1.2GB
<missing> 3 hours ago /bin/sh -c npm install 300MB
<missing> 4 hours ago /bin/sh -c #(nop) FROM node:latest 1.1GB
Look at that. FROM node:latest. That’s the first mistake. On that particular day, node:latest resolved to a full Debian Bullseye image packed with build tools, compilers, and headers that have no business being in a production environment.
Then, look at the COPY command. 1.2GB. Why? Because the developer didn’t bother with a .dockerignore file. They copied the .git folder, the local node_modules, the tests folder, and even a few 200MB heap dumps they had lying around in their local workspace.
I ran a docker inspect to see the security posture. It was exactly what I expected: non-existent.
$ docker inspect payment-processor:latest | grep -i "user"
"User": "",
Empty string. That means it’s running as root. If an attacker finds a remote code execution (RCE) vulnerability in that Node.js app—and given the state of the package.json, they will—they aren’t just inside a container. They are root inside a container. From there, escaping to the host is just a matter of time and a few unpatched kernel vulnerabilities.
2. THE RULES OF ENGAGEMENT: NON-NEGOTIABLE MANDATES
If you want to survive in this industry without being the person who wakes me up at 3:00 AM, you will follow these rules. These aren’t suggestions. They are the baseline for professional-grade containerization.
I. THOU SHALT PIN THY VERSIONS
Stop using :latest. Stop using :major versions like :20. You pin to the specific, immutable tag. If you’re using Python, you use python:3.12.1-slim-bookworm. If you’re using Node, you use node:20.11.0-bookworm-slim.
Why? Because node:20 might be 20.11.0 today and 20.12.0 tomorrow. If 20.12.0 introduces a regression in how it handles OpenSSL, your production environment just became a laboratory for unforced errors.
II. THOU SHALT NOT RUN AS ROOT
There is zero excuse for this. None. Most official images provide a non-privileged user (like the node user in the Node image). If they don’t, you create one.
RUN groupadd -g 10001 appuser && \
useradd -u 10000 -g appuser appuser
USER 10000:10001
III. THOU SHALT USE MULTI-STAGE BUILDS
Your build tools (compilers, git, ssh-keys for private repos) do not belong in your final image. If I see gcc or python3-dev in a production image, I’m revoking your merge rights. Multi-stage builds allow you to compile your binaries in a heavy image and then copy only the artifacts into a minimal, hardened runtime image.
IV. THOU SHALT RESPECT THE LAYER CACHE
Docker builds are cached based on the order of operations. If you change a file, every subsequent layer must be rebuilt. If you COPY . . before you RUN npm install, you are invalidating your cache every time you change a comment in a README file. You are wasting CI minutes and my patience.
3. THE REFACTOR: TEARING DOWN THE AMATEUR HOUR
Let’s look at the “naive” Dockerfile that killed our cluster. It’s a classic example of “it works on my machine” syndrome.
The Failure (Dockerfile.bad):
FROM node:latest
WORKDIR /app
COPY . .
RUN apt-get update && apt-get install -y vim git curl
RUN npm install
EXPOSE 3000
CMD ["npm", "start"]
This is garbage. It’s heavy, it’s insecure, and it’s slow. Now, let’s apply docker best practices to rebuild this into something that won’t make me want to throw my laptop into a wood chipper.
The Professional Version (Dockerfile.good):
# Stage 1: Build
FROM node:20.11.0-bookworm-slim AS builder
# Set build-time environment variables
ENV NODE_ENV=production
WORKDIR /build
# Copy only the package files first to leverage layer caching
COPY package.json package-lock.json ./
# Install dependencies including devDependencies for build steps
# We use npm ci for deterministic builds
RUN npm ci
# Copy the rest of the source code
COPY . .
# Run build scripts (e.g., TypeScript compilation)
RUN npm run build
# Stage 2: Runtime
FROM node:20.11.0-bookworm-slim
# Labels for metadata - useful for automated cleanup and tracking
LABEL maintainer="[email protected]"
LABEL version="1.4.2"
# Set production environment
ENV NODE_ENV=production
ENV PORT=3000
WORKDIR /app
# Create a non-root user and group
RUN groupadd -r appgroup && useradd -r -g appgroup -s /sbin/nologin appuser
# Copy only the necessary artifacts from the builder stage
# We copy the node_modules and the compiled dist folder
COPY --from=builder --chown=appuser:appgroup /build/dist ./dist
COPY --from=builder --chown=appuser:appgroup /build/node_modules ./node_modules
COPY --from=builder --chown=appuser:appgroup /build/package.json ./package.json
# Use a specific signal for graceful shutdown
STOPSIGNAL SIGTERM
# Switch to the non-root user
USER appuser
# Expose the port
EXPOSE 3000
# Use the exec form of CMD to ensure signals are passed to the process
# We use 'node' directly, not 'npm start', to avoid zombie processes
CMD ["node", "dist/index.js"]
Let’s break down why this version is superior.
- Base Image: We used
node:20.11.0-bookworm-slim. It’s based on Debian 12, it’s patched, and it’s small. We didn’t use Alpine because Node.js on Alpine (musl) can have weird performance edge cases with memory allocation that I don’t want to debug at 4:00 AM. - Layer Caching: We copied
package.jsonand rannpm cibefore copying the rest of the code. This means if you change a line of code but don’t change your dependencies, Docker skips thenpm cistep entirely. - Multi-Stage: The final image doesn’t contain
git,vim, or any of the build-time junk. It only contains the compiled code and the production dependencies. - Security: We created
appuser. We used--chownduring theCOPYphase so the files are owned by the user running the process. We set a shell of/sbin/nologinso the user can’t even be used for an interactive session if someone breaks in. - Signal Handling: We used the “exec form”
["node", "dist/index.js"]. If you use the “shell form”CMD node dist/index.js, Docker runs your app as a sub-process of/bin/sh. When you try to stop the container,/bin/shreceives theSIGTERMbut doesn’t pass it to your app. Your app keeps running until Docker loses patience andSIGKILLs it after 10 seconds. That’s how you get corrupted databases and lost state.
4. THE HIDDEN KILLER: .DOCKERIGNORE
If you don’t have a .dockerignore file, you are essentially uploading your entire hard drive to the Docker daemon every time you build.
I checked the build logs from the incident. The “context” sent to the Docker daemon was 1.8GB.
$ docker build -t payment-processor:latest .
Sending build context to Docker daemon 1.842GB
That 1.8GB has to be compressed and sent over the socket. It slows down builds, bloats the layers, and leaks secrets. Your .dockerignore should be the first thing you write.
Example .dockerignore:
.git
.gitignore
node_modules
npm-debug.log
Dockerfile
.dockerignore
.env
.aws
.ssh
dist
tests
docs
*.md
*.pdf
heapdump.*.heapsnapshot
By excluding the .git directory alone, you often save hundreds of megabytes. By excluding .env and .ssh, you prevent yourself from becoming the next headline in a data breach report.
5. VULNERABILITY SCANNING: TRUST BUT VERIFY
After I refactored the image, I ran a scan using trivy. This is what the “latest” image looked like:
$ trivy image payment-processor:latest
Total: 482 (UNKNOWN: 5, LOW: 182, MEDIUM: 190, HIGH: 95, CRITICAL: 10)
Ten critical vulnerabilities. Ten open doors for any script kiddie with a Metasploit modules list. After the refactor to bookworm-slim and proper dependency management, the results were different:
$ trivy image payment-processor:pro-v1.4.2
Total: 12 (LOW: 10, MEDIUM: 2, HIGH: 0, CRITICAL: 0)
Zero criticals. Zero highs. That’s a production-ready image. If you aren’t running vulnerability scans in your CI pipeline, you are just waiting for the inevitable.
6. THE LONG-TERM SURVIVAL PLAN
Maintaining these standards is harder than setting them. Developers will complain that slim images are missing “useful tools” like ping or netstat. They will complain that multi-stage builds are “too complex.” They will try to sneak :latest back into the Helm charts because they are too lazy to update a version number.
Here is how you stop them:
- Admission Controllers: Use something like Kyverno or OPA Gatekeeper in your Kubernetes cluster. Set a policy that rejects any pod using an image with the
:latesttag. If the deployment doesn’t have a pinned version, it doesn’t get scheduled. Period. - Hadolint: Integrate
hadolintinto your CI pipeline. It’s a linter for Dockerfiles. If someone tries to useapt-get updatewithoutrm -rf /var/lib/apt/lists/*, the build fails. If they don’t specify a version for the base image, the build fails. - Automated Dependency Updates: Use tools like Renovate or Dependabot. They will automatically open Pull Requests to update your base image tags and your
package.jsonversions. This takes the “it’s too much work to stay updated” excuse off the table. - Ephemeral Debug Containers: When developers complain they can’t debug because the image is too small, teach them to use
kubectl debug. It allows them to spin up a “sidecar” container with all the tools they want (likenmap,tcpdump,curl) that shares the process namespace of the production pod. They get their tools; I get my secure, slim image.
I’ve spent the last 48 hours cleaning up a mess that could have been avoided with ten minutes of forethought. I’ve seen entire companies go offline because a single docker build command was run with too much ego and too little discipline.
The “docker best” path isn’t the easy path. It requires you to actually understand how Linux works, how networking works, and how the kernel handles processes. But the alternative is a 3:00 AM phone call from me, and trust me, you don’t want to hear my voice when I haven’t slept.
Now, if you’ll excuse me, I’m going to go find a dark room, a bottle of cheap bourbon, and a way to forget the sight of a 2.4GB container layer. Don’t break anything. If I see a :latest tag in the registry when I wake up, there will be blood.
The cluster is stable. For now. But the OOM killer is always watching, and it has no mercy for the lazy.
The Final Word on Image Sizes:
$ ls -lh images.txt
-rw-r--r-- 1 sre-god staff 2.4G May 20 03:15 payment-processor-latest.tar
-rw-r--r-- 1 sre-god staff 184M May 22 11:00 payment-processor-optimized.tar
From 2.4GB to 184MB. That’s not just “optimization.” That’s the difference between a fragile toy and a professional tool. Build better, or get out of the way.
Related Articles
Explore more insights and best practices: