Docker Best Practices: 10 Tips for Faster, Leaner Images

by

INCIDENT LOG: 2024-05-14T03:14:22Z

[03:14:22] Kubelet: Warning FailedScheduling - 0/12 nodes are available: 12 Insufficient memory.
[03:15:01] Node-04: Kernel: [124098.44] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/kubepods/besteffort/pod-abc,task=node,pid=14221,uid=0
[03:15:01] Node-04: Kernel: Out of memory: Killed process 14221 (node) total-vm:4.2GB, anon-rss:1.8GB, file-rss:0B, shmem-rss:0B
[03:16:45] PagerDuty: [CRITICAL] Production API - High Error Rate (98%)
[03:17:10] me@workstation:~$ docker pull registry.internal/app:latest
Error response from daemon: manifest for registry.internal/app:latest not found: manifest unknown: manifest unknown
[03:18:30] me@workstation:~$ docker inspect registry.internal/app:latest
[]
Error: No such object: registry.internal/app:latest
[03:20:00] me@workstation:~$ # Who the hell pushed to production at 3 AM?

I’ve been awake for 48 hours. My eyes feel like they’ve been scrubbed with steel wool, and my caffeine intake has reached levels that would make a cardiologist weep. While the rest of the “Engineering” team was dreaming about whatever “clever” new framework is trending on Hacker News, I was digging through the wreckage of our production cluster because someone decided that “it works on my machine” was a valid deployment strategy.

The following is not just a post-mortem. It is a mandatory remediation guide. If I see another Dockerfile that looks like it was written by a toddler with a copy of “Docker for Dummies” from 2014, I will personally revoke your SSH access and move your desk to the basement next to the backup tape drives.

1. The ‘Latest’ Tag is a Suicide Note: SHA256 Pinning or Bust

The outage started because a junior developer—who shall remain nameless but knows exactly who they are—pushed a “quick fix” to the latest tag. Our CI/CD pipeline, configured with the backbone of a jellyfish, pulled node:latest as the base image. Between the last successful build and this morning’s disaster, the upstream node image updated from a stable Debian Bullseye base to a version that broke our native C++ bindings.

Using latest is not a “docker best” practice; it is a declaration of technical bankruptcy. When you use latest, you are playing Russian Roulette with your infrastructure. You have no guarantee of what code is actually running. You have no reproducibility. You have no soul.

From this moment forward, every FROM instruction in this organization will use specific version tags and, more importantly, SHA256 digests.

The Wrong Way:

FROM node:latest
# This is a ticking time bomb.

The SRE-Approved Way:

# Node v20.11.1 on Debian Bullseye Slim
FROM node:20.11.1-bullseye-slim@sha256:7d1a3674681607567786687006886e3260840c67537330777176793666f27f0d

By pinning the SHA256 digest, we ensure that the bits we tested in staging are the exact same bits that hit production. Docker Engine v24.0.7 doesn’t care about your feelings; it cares about the hash. If the hash changes, the build fails. That’s how we sleep at night.

SRE Rant: I don’t care if it’s “inconvenient” to update the hash. You know what’s inconvenient? Explaining to the CTO why we lost $40k in revenue because you were too lazy to copy-paste a string of hex characters.

2. The 3GB Layer Cake: Multi-Stage Builds or Death

When I finally managed to pull the failing image to my local machine to debug, I had enough time to go make a sandwich, eat it, and contemplate my career choices. The image was 3.4GB. For a Node.js API.

I inspected the layers using docker history and found the culprit. The image contained the entire build toolchain: gcc, g++, make, python3, and the entire node_modules directory including devDependencies. This isn’t just bloat; it’s a performance killer. Large images increase pull times, which increases the “Mean Time To Recovery” (MTTR). During a scaling event, those extra gigabytes mean the difference between a healthy cluster and a cascading failure.

We are moving to multi-stage builds. This is a non-negotiable “docker best” requirement. You build your binaries in a “heavy” image and then copy only the necessary artifacts into a “slim” runtime image.

The Disaster Dockerfile:

FROM node:20
WORKDIR /app
COPY . .
RUN apt-get update && apt-get install -y build-essential python3
RUN npm install
CMD ["node", "server.js"]
# Result: 3.4GB of garbage.

The Remediation Dockerfile:

# Stage 1: Build
FROM node:20.11.1-bullseye-slim@sha256:7d1a3... AS builder
WORKDIR /app
COPY package*.json ./
RUN apt-get update && apt-get install -y --no-install-recommends build-essential python3 \
    && npm ci \
    && apt-get purge -y --auto-remove build-essential python3 \
    && rm -rf /var/lib/apt/lists/*

COPY . .
RUN npm run build

# Stage 2: Runtime
FROM node:20.11.1-bullseye-slim@sha256:7d1a3...
WORKDIR /app
# Only copy what is strictly necessary
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/package.json ./package.json

USER node
CMD ["node", "dist/server.js"]

By using multi-stage builds, we dropped the image size from 3.4GB to 180MB. We also reduced the attack surface by removing the compiler and other utilities that a hacker would love to find in a compromised container. Also, notice the rm -rf /var/lib/apt/lists/*. If you don’t clean up your apt cache in the same layer it was created, it stays in the image forever. Layer caching is a double-edged sword; learn how to use it or get out of the kitchen.

3. Root is for Idiots: Implementing Non-Privileged Users

I checked the running processes on the compromised pod. Everything was running as root. Do you realize what that means? If there is a container breakout vulnerability (and they happen more often than you think), the attacker has root access to the host kernel.

“But it’s easier to handle file permissions as root!”

I don’t care. Your laziness is a security liability. A “docker best” configuration always specifies a non-privileged user. Most official images (like Node, Python, and Alpine) already include a default non-root user, but you’re too busy “innovating” to use them.

If you are using a base image that doesn’t have a user, you create one. Here is the exact syntax you will use. No exceptions.

FROM debian:bullseye-slim@sha256:6248bc...

# Create a system group and user
RUN groupadd -g 10001 appgroup && \
    useradd -u 10000 -g appgroup -m -s /bin/bash appuser

WORKDIR /home/appuser/app
COPY --chown=appuser:appgroup . .

# Switch to the non-privileged user
USER 10000

# Now the application runs without root privileges
CMD ["./my-binary"]

Why use IDs instead of names? Because Kubernetes and other orchestrators handle runAsUser directives better with UIDs. It avoids ambiguity when the host’s /etc/passwd doesn’t match the container’s. If I see USER root in a Dockerfile again, I will trigger a manual OOMKill on your local machine.

4. Signal Handling: Why Your Container Won’t Die

During the outage, I tried to restart the pods. Kubernetes sent a SIGTERM. The pods ignored it. Kubernetes waited 30 seconds, got annoyed, and sent a SIGKILL. This resulted in corrupted database transactions and orphaned file locks because the application didn’t have time to shut down gracefully.

The reason? The junior used the “shell form” of CMD instead of the “exec form.”

When you write CMD node server.js, Docker wraps your command in /bin/sh -c. This makes /bin/sh PID 1. When the kernel sends a SIGTERM to the container, it goes to /bin/sh. And guess what? Shells don’t pass signals to their child processes unless you explicitly tell them to. Your Node.js app never even knew it was being asked to stop.

The Wrong Way (Shell Form):

CMD node server.js
# PID 1 is /bin/sh. Signals are swallowed.

The “Docker Best” Way (Exec Form):

ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["node", "server.js"]
# PID 1 is tini (an init helper), which correctly reaps zombies and forwards signals.

We are now standardizing on using tini or dumb-init for all containers. It handles the PID 1 problem, ensures that SIGTERM actually reaches your application, and prevents zombie processes from clogging up the process table. If your app takes 30 seconds to die, it better be because it’s flushing a massive buffer, not because it’s ignoring the kernel.

Internal Note: If you don’t understand the difference between a signal and a syscall, go back to CS 101. I’m not here to tutor you; I’m here to keep the site alive.

5. The “Secret Leak”: ENV Instructions are Not for Secrets

This was the highlight of my 48-hour nightmare. While inspecting the image layers to find out why it was so big, I ran docker history --no-trunc. And there it was, in plain text, for anyone with access to our internal registry to see:

ENV STRIPE_API_KEY=sk_test_51Mz...
ENV AWS_SECRET_ACCESS_KEY=AKIA...

I had to spend four hours rotating every single production secret because someone thought ENV was a secure way to pass credentials.

Let me be clear: Anything in a Dockerfile is public knowledge to anyone who has the image. Even if you unset the variable in a later layer, it is still there in the previous layer. Docker layers are immutable. You cannot “delete” a secret once it’s been baked in.

To follow “docker best” practices, we use build-time secrets or runtime environment variables provided by the orchestrator (Kubernetes Secrets), never baked into the image.

How to use Build-Time Secrets (Docker BuildKit):

# syntax=docker/dockerfile:1
FROM node:20.11.1-bullseye-slim
WORKDIR /app

# Mount the secret during build, it never touches the image layers
RUN --mount=type=secret,id=my_secret \
    export SECRET_VAL=$(cat /run/secrets/my_secret) && \
    npm run build -- --api-key=$SECRET_VAL

CMD ["node", "server.js"]

When building, you pass the secret like this:
docker build --secret id=my_secret,src=./secret.txt .

The secret is mounted in a temporary filesystem (tmpfs) and never hits the disk or the image layers. If I find a secret in a docker history log again, I’m not just revoking your access; I’m calling HR.

6. Healthchecks That Actually Mean Something

The junior added a HEALTHCHECK instruction. Great, right? Wrong.
HEALTHCHECK CMD curl -f http://localhost:8080/ || exit 1

During the outage, the Node.js event loop was completely blocked by a synchronous CPU-intensive task (another “clever” optimization). The port was “open,” so curl succeeded, but the application was effectively dead. It couldn’t process a single request. Kubernetes thought the pod was healthy, so it kept sending traffic to a black hole.

A “docker best” healthcheck must validate the internal state of the application, not just the network stack. It should check database connectivity, cache availability, and whether the event loop lag is within acceptable limits.

The “I Actually Care About My App” Healthcheck:

Create a healthcheck.js script:

const http = require('http');
const options = {
    host: 'localhost',
    port: 8080,
    path: '/health',
    timeout: 2000
};

const request = http.request(options, (res) => {
    if (res.statusCode === 200) {
        process.exit(0);
    } else {
        process.exit(1);
    }
});

request.on('error', (err) => {
    process.exit(1);
});

request.end();

Then in your Dockerfile:

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
  CMD [ "node", "healthcheck.js" ]

And for the love of all that is holy, make sure your /health endpoint doesn’t just return {"status": "ok"}. It should actually ping the database. If the database is down, the app is down. Tell the truth.

7. Ephemeral Storage and the /tmp Explosion

The final nail in the coffin for Node-04 was the ephemeral storage. The application was writing debug logs to /app/logs/debug.log inside the container. Since containers use an overlay filesystem (OverlayFS), every write operation creates a “copy-on-write” action.

The junior didn’t set a log rotation policy. The container’s writable layer grew until it consumed all available disk space on the host’s /var/lib/docker/overlay2 directory. This triggered a DiskPressure eviction, which killed not just the offending pod, but three other critical services on the same node.

Remediation:
1. Logs go to STDOUT/STDERR. Period. No log files inside the container. Let the container runtime and the logging driver (Fluentd, Loki, etc.) handle the persistence.
2. Use Tmpfs for temporary files. If you absolutely must write temporary files, use a tmpfs mount so they stay in RAM and don’t bloat the image layers.

Example Kubernetes Pod Spec (since you clearly can’t be trusted with Docker alone):

spec:
  containers:
  - name: app
    image: registry.internal/app:v1.2.3@sha256:...
    resources:
      limits:
        memory: "512Mi"
        cpu: "500m"
        ephemeral-storage: "1Gi" # Limit the damage
    volumeMounts:
    - name: cache-volume
      mountPath: /tmp
  volumes:
  - name: cache-volume
    emptyDir:
      medium: Memory

8. Layer Caching: Stop Invalidating the World

Every time you change a single line of code, our CI/CD takes 15 minutes to build. Why? Because you put COPY . . at the top of your Dockerfile.

Docker builds images in layers. If a layer changes, every subsequent layer must be rebuilt. By copying your entire source code before running npm install, you invalidate the npm install layer every time you change a comment in a README file.

The “I Value My Time” Layering:

WORKDIR /app

# Copy only the dependency files first
COPY package.json package-lock.json ./

# This layer is cached unless package.json changes
RUN npm ci

# Now copy the rest of the code
COPY . .

# This layer only runs if the code changes
RUN npm run build

This simple change reduced our build time from 15 minutes to 2 minutes. That’s 13 minutes of my life I get back every time you push a bug. Multiply that by 50 developers and 20 pushes a day, and you’ve just saved the company thousands of dollars in compute costs and developer frustration.

9. The Base Image Choice: Alpine vs. Debian Slim

I see a lot of you using alpine because it’s “small.” Alpine uses musl libc. Node.js and many Python libraries are built against glibc (standard on Debian/Ubuntu). When you run a glibc application on musl, you often run into weird performance issues, DNS resolution bugs, or “File not found” errors that make no sense.

Unless you are an expert in C-standard library compatibility, use debian-slim. It’s slightly larger (maybe 30MB more), but it’s much more stable and uses glibc. We are an enterprise, not a hobbyist blog. Stability beats a few megabytes every single time.

Current Standard: debian:bullseye-slim or node:20-bullseye-slim.

If I see an alpine image that hasn’t been thoroughly vetted for musl compatibility, I will reject the PR. I don’t have the energy to debug your segmentation faults at 4 AM.

Summary of Required Actions

  1. Audit all Dockerfiles: Replace latest with SHA256 pinned tags.
  2. Refactor for Multi-Stage: Remove build tools from runtime images.
  3. Drop Privileges: Add USER instructions to every single image.
  4. Fix Signal Handling: Use the exec form ["cmd", "arg"] and tini.
  5. Purge Secrets: Move all ENV secrets to Kubernetes Secrets or build-time mounts.
  6. Implement Real Healthchecks: Check the DB, not just the port.

I am going to sleep now. If my pager goes off because someone ignored this guide, may the kernel have mercy on your soul, because I won’t.

SRE Signed Off: 2024-05-16T09:00:00Z
Status: Irritable.

Leave a Comment