It is 3:14 AM. The PagerDuty alert didn’t just wake me up; it vibrated my soul out of my body. The alert says service-auth-gateway is throwing 500s. I check the logs. It’s a library mismatch. Again. Some junior developer—let’s call him Kyle—pushed a “hotfix” that relied on a specific version of libssl that exists only on his Fedora Rawhide workstation. Production is running Debian Bullseye. The binary linked to a ghost. The process crashed. The orchestrator tried to restart it. It crashed again. A death loop.
This is why we are here. This is why I am sitting in the dark, illuminated only by the cold blue light of three monitors, preparing to perform another docker install on a fresh node because the old one’s filesystem turned into a read-only brick of sludge. We don’t use Docker because it’s good. We use it because developers cannot be trusted to manage their own dependencies, and I am tired of being the janitor who mops up their spilled shared objects. Docker is the industrial-grade hazmat suit we force onto the code so it doesn’t infect the rest of the host.
Table of Contents
The Lie of Portability
The marketing tells you that containers are lightweight. They tell you it’s “build once, run anywhere.” That is a lie. It’s “build once, debug everywhere.” Before you even think about the docker install command, you have to realize you’re just trading one set of problems for a more complex, obscured set of problems. You’re not fixing the dependency hell; you’re just burying it in a layer of YAML and virtualized networking.
When I start a docker install, I’m not “enabling innovation.” I’m building a cage. I’m setting up a system of cgroups and namespaces to ensure that when Kyle’s code inevitably leaks memory like a rusted pipe in a basement, it only drowns his specific container and doesn’t take down the entire kernel.
First, we have to purge the old sins. Every Linux distribution has its own broken, outdated version of the Docker engine sitting in its default repositories like stagnant water. If you run apt-get install docker.io, you’re inviting a ghost from 2018 into your house. We have to clear the deck.
# Removing the debris of previous failed attempts
for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do
sudo apt-get remove -y $pkg;
done
This doesn’t actually clean anything. It just removes the binaries. The config files, the broken network interfaces, the orphaned volumes—those stay. They’re like asbestos in the walls. You just hope you don’t disturb them.
The GPG Keyring Purgatory
Now we get to the security theater. We can’t just download a package. We have to verify it. We have to dance the dance of the GPG keys. This is the part where the “magic” of the internet usually fails because of a corporate proxy or a transient DNS hiccup.
We need to create a directory for the keyring. Not just any directory. It has to have the right permissions, or the system will complain with the stoic indifference of a bureaucrat.
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
Look at that curl -fsSL command. We are piping a script from the internet directly into a root-owned keyring. This is what passes for “best practice” in the modern era. It’s like drinking water directly from a storm drain because the city told you it was filtered. If the Docker CDN gets compromised, my entire cluster is a playground for whatever state-sponsored actor wants to mine Monero on my dime.
Once the key is in place, we have to tell apt where to find the actual sludge. We have to echo a string into a .list file.
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
The $(. /etc/os-release && echo "$VERSION_CODENAME") part is particularly galling. We are relying on the OS to identify itself correctly so we can fetch the right version of a tool that is supposed to make the OS irrelevant. The irony is thick enough to clog a drain.
The Repository Swamp
After the repository is added, we run apt-get update. This is the moment of truth. This is where you find out if your MTU settings are slightly off, causing the connection to hang indefinitely, or if the repository mirror is having a bad day.
Hit:1 http://deb.debian.org/debian bullseye InRelease
Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:3 https://download.docker.com/linux/debian bullseye InRelease [43.3 kB]
Get:4 https://download.docker.com/linux/debian bullseye/stable amd64 Packages [24.9 kB]
Fetched 116 kB in 1s (112 kB/s)
Reading package lists... Done
It worked. This time. But don’t get comfortable. Now we actually perform the docker install. We aren’t just installing “Docker.” We are installing a suite of tools, each with its own potential for catastrophic failure. docker-ce, docker-ce-cli, containerd.io, docker-buildx-plugin, and docker-compose-plugin.
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
Why are there five different packages? Because “Docker” isn’t a single tool anymore. It’s a fragmented ecosystem of components that used to be one cohesive unit but were broken apart to satisfy the whims of architectural purists. It’s like taking a perfectly functional car and deciding that the steering wheel, the tires, and the engine should all be sold by different vendors and bolted together at the last second.
The Daemon: A Leaky Valve in the Engine Room
Once the binaries are on the disk, the dockerd daemon starts. This is the heart of the beast. It runs as root. It has total control over your network stack. It can create bridges, modify iptables, and mount filesystems. It is a monolithic god-process that we all just accept because the alternative is manual LXC configuration, which is a form of psychological torture I wouldn’t wish on my worst enemy.
I need to check if it’s actually alive.
# docker version
Client: Docker Engine - Community
Version: 24.0.7
API version: 1.43
Go version: go1.20.10
Git commit: afdd53b
Built: Thu Oct 26 09:07:41 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 24.0.7
API version: 1.43 (minimum version 1.12)
Go version: go1.20.10
Git commit: 311b9ff
Built: Thu Oct 26 09:07:41 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.24
GitCommit: 61f9fd28f25cb100dd59cc37123989791427b9d7
runc:
Version: 1.1.9
GitCommit: v1.1.9-0-gccaecfc
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Look at that API version: 1.43. Every time that number increments, something breaks. A flag is deprecated. A default behavior changes. A security patch introduces a regression in how volumes are mounted.
Now, we have to configure the daemon. By default, Docker is chatty and dangerous. It logs everything to JSON files that grow until they consume the entire disk. I’ve lost more production nodes to json-file log exhaustion than to actual traffic. You have to throttle it. You have to put a regulator on the valve.
I open /etc/docker/daemon.json. It’s usually empty or doesn’t exist. I have to manifest it.
{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "3"
},
"storage-driver": "overlay2",
"live-restore": true,
"ipv6": false
}
The live-restore flag is a desperate plea. It tells Docker: “If the daemon crashes or needs an update, please, for the love of all that is holy, do not kill the containers.” It’s like trying to change the oil in a car while it’s doing 70 mph on the freeway. Sometimes it works. Sometimes you lose a finger.
Namespaces: Cardboard Walls in a Burning Building
To understand why we do a docker install, you have to understand what’s happening in the kernel. We are using namespaces and cgroups.
Namespaces provide the illusion of isolation. The process thinks it’s alone. It thinks it has its own network stack, its own process tree, its own mount points. But it’s a lie. It’s all sharing the same kernel. If a process triggers a kernel panic, the whole host goes down. The “container” is just a fancy wrapper around a clone() syscall with specific flags like CLONE_NEWPID or CLONE_NEWNET. It’s like putting cardboard dividers in a hospital ward and pretending it prevents the spread of a virus.
Then there are cgroups (control groups). This is the plumbing that limits how much “sludge” a container can pump out. If I don’t set these, Kyle’s Python script will eat 100% of the CPU and 64GB of RAM in three seconds.
# Example of the limits we have to manually enforce because the defaults are "unlimited"
docker run -d --name kyles-disaster --memory="512m" --cpus="0.5" my-broken-app:latest
But even with cgroups, there are leaks. The kernel’s slab memory isn’t always properly accounted for. The overlay2 storage driver—the current industry standard—is a complex layering system that merges a lower read-only layer with an upper writable layer. It’s efficient until you have thousands of small files, at which point the inode usage skyrockets and the filesystem starts screaming. I’ve spent nights cleaning up /var/lib/docker/overlay2 like I’m scrubbing grease off a factory floor.
The Non-Root User Illusion
After the docker install, the next thing every tutorial tells you to do is add your user to the docker group.
sudo usermod -aG docker $USER
This is a security nightmare. Giving a user access to the Docker socket is equivalent to giving them root access. Anyone in the docker group can run docker run -v /:/host -it alpine chroot /host and they are now root on the host machine. We do it anyway because typing sudo every time is “annoying.” We trade the entire security posture of the server for three fewer keystrokes. It’s like leaving the vault door wide open because the key is heavy.
And then there’s the networking. Docker loves to mess with iptables. It inserts its own chains at the top of the list. It bypasses your carefully crafted firewall rules. You think your database port is closed to the world? Docker doesn’t care. If you map a port with -p 5432:5432, Docker will punch a hole straight through your firewall like a sledgehammer through drywall.
To fix this, you have to set {"iptables": false} in daemon.json, but then you have to manually manage the NAT rules for the containers, which is a special kind of hell involving MASQUERADE targets and routing tables that look like a bowl of spaghetti. Most SREs just give up and let Docker have its way with the firewall. We just hope the VPC security groups catch what the host firewall misses.
The Storage Driver Abyss
We need to talk about /var/lib/docker. This directory is the graveyard of a thousand failed deployments. Every time you run a docker install, you are creating a ticking time bomb in this directory.
Docker uses a layered filesystem. When you pull an image, you’re pulling layers. When you build an image, you’re adding layers. Each layer is a diff. If you have a 1GB base image and you change one file, Docker doesn’t just store the change; it manages the metadata for the entire stack.
Over time, the overlay2 directory fills up with orphaned layers. The docker system prune command is supposed to be the janitor, but it’s a timid one. It won’t touch anything it thinks might be in use.
# The only command that keeps me from losing my mind
docker system prune -a --volumes --force
I’ve seen this command reclaim 200GB of space on a “clean” system. Where did that space go? It was consumed by “dead” containers that the daemon forgot to reap. It was consumed by build caches for images that haven’t existed in six months. It’s industrial waste, and we’re just piling it up in the corner of the disk.
And don’t get me started on the “User Namespaces” feature. It’s supposed to map the root user inside the container to a non-privileged user on the host. It’s a great idea on paper. In practice, it breaks every volume mount and permission setting you’ve ever touched. You spend four hours trying to figure out why your app can’t read its own config file, only to realize that UID 0 in the container is actually UID 100000 on the host, and the file is owned by UID 1000. It’s a shell game where everyone loses.
The Inevitable Decay
So, the docker install is finished. The daemon is running. The daemon.json is configured to prevent immediate disk death. The user is in the group, ready to bypass all security. The cgroups are ready to throttle the inevitable memory leaks.
Is the system stable? No.
Tomorrow, the containerd runtime will have a race condition during a high-load event. The week after, a kernel update will change how iptables handles the DOCKER-USER chain, and suddenly the staging environment will be reachable from the public internet. Or maybe the overlay2 driver will hit an obscure bug where it fails to unmount a layer, preventing a container from stopping and leaving the system in a “zombie” state where you can’t kill the process even with kill -9.
I’ve seen it all. I’ve seen the veth pairs multiply like rabbits until the kernel’s networking table overflows. I’ve seen the docker-proxy process consume 100% CPU because it’s trying to handle 50,000 concurrent connections on a single port.
We don’t install Docker because we want to. We install it because we have to contain the chaos. It’s a heavy, clunky, leaky suit of armor, but it’s all we have between us and the raw, unmanaged madness of “it works on my machine.”
I’m going back to sleep. Or I’m going to try. But I know as soon as I close my eyes, the docker-compose plugin will decide that it doesn’t like the syntax of a file that hasn’t changed in three years, or the GPG key I just installed will be revoked because of some security breach at a company I’ve never heard of.
The containers are running. For now. But the pipes are already starting to creak. The next outage isn’t a matter of “if,” it’s a matter of “when.” And when it happens, it won’t be a simple fix. It will be a deep, structural failure in the very abstraction we built to save us.
Watch the logs. If you see level=error msg="Handler for POST /v1.43/containers/create failed", don’t bother debugging. Just wipe the node and start the docker install again. It’s the only way to be sure the sludge is gone. For a few hours, anyway.
Related Articles
Explore more insights and best practices: