I found the body of the production server at 3:14 AM, smothered by a 3.5GB image that had no business existing in a civilized society.
The monitoring dashboard wasn’t just red; it was a screaming, digital hemorrhage. Disk pressure had triggered a death spiral. The kubelet, in its infinite, mechanical wisdom, tried to garbage collect images to save itself, but it couldn’t delete the one currently being pulled—a bloated, necrotic mass of a container that was failing to start because the node had run out of IOPS just trying to unpack the damn thing.
I’ve spent fifteen years digging through the strata of failed “agile” deployments. I’ve seen the rise and fall of a dozen “game-changing” orchestrators. And yet, here I was, staring at a Dockerfile written by someone who clearly thought a container was just a virtual machine that you didn’t have to pay for. It’s the same story every time: a junior developer gets a “Hello World” running on their MacBook Pro with 32GB of RAM and assumes the cloud is just a bigger version of their laptop. It isn’t. The cloud is a cold, expensive place where every byte you ship costs money and every layer you mismanage is a security hole waiting to be kicked in.
Table of Contents
The Crime Scene
The repository was a graveyard of bad decisions. I pulled the Dockerfile and the docker-compose.yml from the wreckage. It was forty lines of pure, unadulterated architectural negligence.
# THE CRIME SCENE: Dockerfile
FROM ubuntu:latest
# Install everything including the kitchen sink
RUN apt-get update
RUN apt-get install -y python3 python3-pip build-essential git curl wget vim
# Why is this here? Nobody knows.
RUN curl -sL https://deb.nodesource.com/setup_18.x | bash -
RUN apt-get install -y nodejs
# Copy everything, including .git, node_modules, and secrets.env
ADD . /app
WORKDIR /app
# Install dependencies as root
RUN pip3 install -r requirements.txt
RUN npm install
# Open all the ports
EXPOSE 1-65535
# Run the app
CMD ["python3", "app.py"]
And the docker-compose.yml was no better:
version: '3.8'
services:
web:
build: .
privileged: true
volumes:
- .:/app
environment:
- DEBUG=True
- DATABASE_URL=postgres://admin:password123@db:5432/prod_db
ports:
- "80:80"
I ran a quick docker images and docker history on the resulting monstrosity. The results were enough to make a grown sysadmin weep.
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
bloated-nightmare latest a1b2c3d4e5f6 10 minutes ago 3.52GB
$ docker history bloated-nightmare
IMAGE CREATED CREATED BY SIZE COMMENT
a1b2c3d4e5f6 10 minutes ago CMD ["python3" "app.py"] 0B
<missing> 10 minutes ago RUN /bin/sh -c npm install 840MB
<missing> 11 minutes ago RUN /bin/sh -c pip3 install -r requirements.txt 420MB
<missing> 12 minutes ago ADD . /app 1.2GB # Included a 1GB .git folder
<missing> 15 minutes ago RUN /bin/sh -c apt-get install -y nodejs 210MB
<missing> 16 minutes ago RUN /bin/sh -c curl -sL https://deb.nodesource… 5MB
<missing> 17 minutes ago RUN /bin/sh -c apt-get install -y python3 pyth… 650MB
<missing> 18 minutes ago RUN /bin/sh -c apt-get update 45MB
<missing> 20 minutes ago FROM ubuntu:latest 77.8MB
3.52 gigabytes. For a Python script that queries a database and returns a JSON object. We are living in the end times.
The Layer Cake of Lies
The first thing we have to address in this surgical reconstruction is the fundamental misunderstanding of how the Union File System (UnionFS) works. Every RUN, COPY, and ADD instruction in a Dockerfile creates a new layer. These layers are stacked. If you delete a file in layer 4 that was created in layer 3, the file is “hidden” in the final view, but it still exists in layer 3, taking up space and being pulled across the network.
The junior dev who wrote this used RUN apt-get update as its own layer. Then RUN apt-get install. This is a classic rookie mistake. When you run apt-get update, you’re downloading package indexes into /var/lib/apt/lists/. If that happens in its own layer, those indexes stay in your image forever.
In our reconstruction, we use the “Golden Rule of Layers”: combine commands and clean up in the same layer. But even before that, we stop using ubuntu:latest. “Latest” is a ticking time bomb. It’s non-deterministic. Today it’s Ubuntu 22.04; tomorrow it’s 24.04, and suddenly your Python C-extensions won’t compile because the glibc version changed.
We use python:3.11-slim-bookworm. It’s Debian-based, stable, and doesn’t include the 400MB of “standard” utilities you’ll never use in a container.
# Comparison of base images
$ docker images
REPOSITORY TAG IMAGE ID SIZE
ubuntu latest ba627c3a43f1 77.8MB
python 3.11-slim-bookworm 1234567890ab 121MB
# Note: The python image is larger because it includes the runtime,
# but it's smaller than ubuntu + manual python install.
When we install packages, we do it like this:
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
libpq-dev \
&& rm -rf /var/lib/apt/lists/*
By chaining the commands with && and ending with rm -rf /var/lib/apt/lists/*, we ensure the package indexes never make it into the final image layer. This is why following a docker best practice isn’t a suggestion; it’s a survival requirement for your disk space.
The Root User Sin
Look at that docker-compose.yml again. privileged: true. Running as root inside the container. This is the equivalent of leaving your front door wide open, putting a “Free Money” sign on the lawn, and going on vacation.
Containers use namespaces to isolate processes, but the UID 0 (root) inside the container is the same UID 0 as the host unless you’ve gone through the pain of configuring user namespaces (which most people don’t). If a process escapes the container—and they do—it has root access to your host.
Furthermore, the original Dockerfile didn’t define a USER. It defaulted to root. This means the application, the package manager, and every script had full reign over the container’s internals.
In the reconstruction, we create a non-privileged user. We don’t give them a shell. We don’t give them a home directory if we don’t have to.
RUN groupadd -g 10001 appuser && \
useradd -u 10000 -g appuser --create-home --shell /sbin/nologin appuser
We then ensure that only the necessary directories are owned by this user. We don’t CHOWN the whole /app directory if we don’t have to, because—guess what—CHOWN creates a whole new layer, doubling the size of the files being changed. Instead, we use the --chown flag in the COPY command.
The Ghost of Dependencies Past
The original image had build-essential, git, curl, and nodejs all sitting in the final production image. Why? Because the developer needed them to build the app. But the app doesn’t need gcc to run. It doesn’t need git to query a database.
This is where multi-stage builds come in. It’s the single most effective tool in the infrastructure archaeologist’s kit for removing the bloat of “move fast and break things” culture. We use a “builder” stage to compile our wheels and install our dependencies, and then we copy only the finished products into a “runner” stage.
# Stage 1: The Builder
FROM python:3.11-slim-bookworm AS builder
WORKDIR /build
RUN apt-get update && apt-get install -y --no-install-recommends build-essential libpq-dev
COPY requirements.txt .
RUN pip install --user --no-cache-dir -r requirements.txt
# Stage 2: The Runner
FROM python:3.11-slim-bookworm AS runner
WORKDIR /app
RUN groupadd -g 10001 appuser && useradd -u 10000 -g appuser appuser
COPY --from=builder --chown=appuser:appuser /root/.local /home/appuser/.local
COPY --chown=appuser:appuser . .
USER appuser
ENV PATH=/home/appuser/.local/bin:$PATH
By doing this, we leave the build-essential (hundreds of megabytes) in the discarded builder stage. The final image only contains the Python runtime and the compiled libraries. This is how you turn a 3.5GB disaster into a 200MB professional artifact. Adhering to this docker best practice regarding layer squashing and stage separation is what separates the engineers from the script kiddies.
The PID 1 Zombie Apocalypse
When you run CMD ["python3", "app.py"], Python becomes PID 1 inside the container. This is a problem. In Unix-like systems, PID 1 (init) has two special responsibilities: reaping “zombie” child processes and handling signals like SIGTERM or SIGINT.
Python, like most high-level runtimes, isn’t designed to be an init system. If your app spawns subprocesses and they die, they stay as zombies because Python doesn’t know it’s supposed to wait on them. More importantly, when you run docker stop, Docker sends SIGTERM to PID 1. If Python hasn’t explicitly registered a signal handler for SIGTERM, the kernel will ignore the signal. Docker waits 10 seconds, gets annoyed, and sends SIGKILL, which is like pulling the power plug. Your app doesn’t close database connections, it doesn’t flush logs, it just dies.
The solution is tini. It’s a tiny init binary that handles these chores for you.
RUN apt-get update && apt-get install -y tini
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["python3", "app.py"]
Now, tini is PID 1. It catches SIGTERM, passes it to Python, and Python shuts down gracefully. It also reaps any zombies. It’s a simple fix that prevents a world of corrupted data and orphaned processes.
The Cache Mount Salvation
One of the biggest complaints from the “move fast” crowd is that Docker builds are slow. So they skip the cleanup steps to “leverage” (God, I hate that word) the cache. But they’re doing it wrong.
Modern Docker (BuildKit) supports cache mounts. This allows you to persist a cache directory (like the pip cache or the npm cache) between builds without actually including that cache in the final image layers.
# Using BuildKit cache mounts
RUN --mount=type=cache,target=/root/.cache/pip \
pip install -r requirements.txt
This is the holy grail. You get the speed of a local pip install because the /root/.cache/pip directory is persisted on the build host, but the resulting image layer only contains the installed packages, not the cached .whl files. This is another instance where ignoring every docker best practice ever written by someone with more than two weeks of experience leads to the kind of 3:14 AM phone call I had to endure.
The .dockerignore Exclusion Zone
The “Crime Scene” Dockerfile used ADD . /app. This is a disaster. ADD is more powerful than COPY—it can extract tarballs and fetch URLs—which is exactly why you shouldn’t use it unless you need those specific features. COPY is predictable.
But the real sin was copying everything. The .git directory in the project I found was 1.2GB. It contains the entire history of the project, every deleted binary, every large asset ever committed. It has no place in a production container.
We need a .dockerignore file. It’s not optional.
# .dockerignore
.git
.gitignore
node_modules
*.pyc
__pycache__
secrets.env
tests/
docs/
.vscode/
By excluding these, the build context sent to the Docker daemon drops from gigabytes to kilobytes. The build starts faster, the layers are smaller, and you aren’t accidentally shipping your secrets.env to a public registry.
The Surgical Reconstruction: The Final Artifact
After hours of cleaning up the mess, I produced a Dockerfile that wouldn’t make the server choke and die. It was lean, it was secure, and it was documented.
# --- STAGE 1: BUILDER ---
# Use a specific, stable version. No 'latest' allowed in this house.
FROM python:3.11-slim-bookworm AS builder
# Set environment variables to make Python/Apt behave in a container
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
DEBIAN_FRONTEND=noninteractive
WORKDIR /build
# Install only what is needed for the build phase
# Chaining and cleaning up in one layer to keep the image slim
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
libpq-dev \
&& rm -rf /var/lib/apt/lists/*
# Use BuildKit cache mounts for faster, cleaner builds
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --user --no-cache-dir -r requirements.txt
# --- STAGE 2: RUNNER ---
FROM python:3.11-slim-bookworm AS runner
# Tini is the init process we deserve
RUN apt-get update && apt-get install -y --no-install-recommends \
tini \
libpq5 \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Create a non-root user with a specific UID/GID for predictable permissions
RUN groupadd -g 10001 appuser && \
useradd -u 10000 -g appuser --create-home --shell /sbin/nologin appuser
# Copy only the installed site-packages from the builder
COPY --from=builder --chown=appuser:appuser /root/.local /home/appuser/.local
# Copy the application code, ensuring the non-root user owns it
# The .dockerignore ensures we don't bring in the garbage
COPY --chown=appuser:appuser . .
# Switch to the non-root user
USER appuser
# Update PATH to include the local bin where pip installed things
ENV PATH=/home/appuser/.local/bin:$PATH \
PYTHONUNBUFFERED=1
# Expose only the necessary port
EXPOSE 8080
# Use Tini to manage signals and zombies
ENTRYPOINT ["/usr/bin/tini", "--"]
# Run the application
CMD ["python", "app.py"]
I ran the build. I watched the output.
$ docker build -t optimized-app:1.0.2 .
...
=> exporting to image 0.2s
=> => exporting layers 0.2s
=> => writing image sha256:e9c1... 0.0s
=> => naming to docker.io/library/optimized-app:1.0.2 0.0s
$ docker images
REPOSITORY TAG IMAGE ID SIZE
optimized-app 1.0.2 e9c1d2e3f4g5 184MB
bloated-old latest a1b2c3d4e5f6 3.52GB
From 3.52GB to 184MB. A 95% reduction in size. The deployment time dropped from six minutes to twelve seconds. The memory footprint on the node stabilized because we weren’t running a dozen unnecessary background processes and the Python runtime was properly configured.
The Aftermath
I updated the docker-compose.yml too. No more privileged: true. No more mounting the entire project directory in production. We use read-only filesystems where possible to prevent any compromised process from writing to the container’s root.
services:
web:
image: optimized-app:1.0.2
read_only: true
tmpfs:
- /tmp
- /home/appuser/.local/
security_opt:
- no-new-privileges:true
cap_drop:
- ALL
cap_add:
- NET_BIND_SERVICE
deploy:
resources:
limits:
cpus: '0.5'
memory: 512M
This is what infrastructure work actually looks like. It’s not about the “vibrant” new tools or “fostering” a culture of “seamless” delivery. It’s about the gritty, boring work of making sure your systems don’t fall over because someone was too lazy to write a .dockerignore file.
The junior developer came in the next morning, saw the PR, and asked why I “wasted” so much time on the Dockerfile when the old one “worked fine on his machine.” I didn’t answer. I just pointed at the monitoring logs from 3:14 AM and went to get more coffee.
I’ve been doing this for fifteen years. I’ve seen the “move fast and break things” crowd break everything from startups to Fortune 500s. They move fast because they aren’t carrying the weight of their own mistakes—people like me are. Containers aren’t magic. They are just fancy chroot jails. If you don’t treat them with the respect that Linux kernel primitives deserve, they will turn on you.
This is why following a docker best practice isn’t a suggestion; it’s a survival requirement. If you want to play with the big boys in production, stop shipping your .git folders and start thinking about PID 1. Now, if you’ll excuse me, I have another server to exhume. Someone just tried to deploy a 5GB Machine Learning model inside a Node.js container using FROM fedora:latest.
God help us all.
Related Articles
Explore more insights and best practices: