I remember when servers had physical buttons you could press, and if something broke, you could smell the ozone. There was a dignity in the hum of a SCSI drive spinning up, a physical manifestation of data being moved by magnetism and sheer willpower. You knew exactly where your bits were. They were on that platter, in that rack, connected by a Category 5 cable you crimped yourself. But today? Today, my CTO—a man who thinks “latency” is something that only happens to his Zoom calls—has decided that our rock-solid legacy monolith, a piece of software that has outlasted three of his predecessors, needs to be “containerized.”
The sheer insanity of the modern stack is enough to make a grown man weep. We are now in an era where developers are too lazy to manage a library dependency, so their solution is to ship an entire damn operating system userland just to run a thirty-line Python script. You want to run a “Hello World” app? Better pull a 500MB Debian slim image, layer on some build-essential tools, and then wonder why your “docker container” is eating 2GB of RAM before it even hits the first instruction pointer. It’s a nesting doll of inefficiency. We’ve traded understanding for abstraction, and the cost is a catastrophic loss of mechanical sympathy. Every time I see a developer pull a random image from a public registry, I see a security nightmare wrapped in a performance degradation, smothered in a layer of “I don’t want to learn how ldconfig works.” We are building cathedrals out of sand, and the tide of technical debt is coming in fast.
Table of Contents
H2: The Lie of Portability
They told me a “docker container” would make our deployment “portable.” That’s the first lie they teach you in DevOps school. Portability is a myth sold to people who don’t understand the Linux ABI (Application Binary Interface). You aren’t “portable”; you are just shipping your mess to someone else’s kernel.
Back in my day, we didn’t have 5.15.0-76-generic to bail us out of every syscall mismatch. We had to ensure our binaries were statically linked or that our LD_LIBRARY_PATH was surgically precise. Now, people think that because they wrapped their garbage in a container, it’ll run the same on a Raspberry Pi as it does on a Dual-Socket Xeon. It won’t. You’re still tethered to the host kernel. If your “docker container” expects a specific kernel feature—say, a certain io_uring behavior—and the host is running an older LTS kernel, your “portable” app is going to dump core faster than a 1990s BBS on a bad phone line.
Let’s look at what this “portability” actually looks like under the hood. It’s not magic. It’s just a series of smoke and mirrors managed by runc. When you run docker inspect, you see the sheer volume of metadata required just to pretend that a process is isolated.
# docker inspect legacy-monolith-container
[
{
"Id": "a1b2c3d4e5f6...",
"Created": "2023-10-27T14:22:01.123456789Z",
"Path": "/usr/bin/python3",
"Args": [
"app.py"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 12345,
"ExitCode": 0,
"Error": "",
"StartedAt": "2023-10-27T14:22:02.987654321Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/abc/diff:/var/lib/docker/overlay2/def/diff",
"MergedDir": "/var/lib/docker/overlay2/ghi/merged",
"UpperDir": "/var/lib/docker/overlay2/ghi/diff",
"WorkDir": "/var/lib/docker/overlay2/ghi/work"
},
"Name": "overlay2"
}
}
]
Look at that GraphDriver. That’s OverlayFS. Every time your application tries to write a log file, the kernel has to traverse those layers, find the file, copy it to the UpperDir, and then finally commit the write. You’ve just turned a simple I/O operation into a scavenger hunt for the VFS (Virtual File System). The “context switching overhead” alone is enough to make me want to retire to a farm with no electricity.
H2: The Technical Autopsy: Namespaces and the Illusion of Isolation
A “docker container” is not a virtual machine. It’s a fancy word for a process that has been lied to by the kernel. We use Linux Namespaces to create this illusion. It’s a series of unshare() and clone() syscalls that tell a process it’s the only thing running on the box.
If I run strace on the creation of a container, I don’t see a “magic box” being built. I see the raw, gritty reality of the kernel being told to partition its resources.
# strace -f -e trace=clone,unshare docker run --rm alpine sleep 1
...
[pid 12400] unshare(CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWIPC|CLONE_NEWPID|CLONE_NEWNET|CLONE_NEWUSER) = 0
[pid 12400] clone(child_stack=NULL, flags=CLONE_VM|CLONE_VFORK|SIGCHLD) = 12401
...
There it is. CLONE_NEWPID. That’s the kernel telling the process, “Congratulations, you are now PID 1.” But it’s a lie. On the host, that process is just another entry in the task list, likely PID 12401. This leads to the “PID 1 zombie reaping” problem. In a real system, init (or systemd, if you must) handles orphaned child processes. In your “docker container,” if your entrypoint doesn’t know how to handle SIGCHLD, you end up with a process table full of ghosts. You’ve created a graveyard because you couldn’t be bothered to write a proper init script.
And don’t get me started on CLONE_NEWNET. Look at the output of ip addr inside one of these things:
# docker exec legacy-monolith-container ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
24: eth0@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
eth0@if25. That @if25 is a pointer to a veth pair on the host. Every packet your app sends has to traverse a virtual bridge (docker0), go through a NAT (Network Address Translation) layer, and hit the host’s iptables rules before it even smells a real network card. You’re adding microseconds of latency to every request so that your developers don’t have to learn how to bind to a specific IP address. It’s offensive.
H2: Cgroups: The Throttling of Reality
If Namespaces are about what you can see, Control Groups (cgroups) are about what you can touch. This is where the “docker container” gets its lunch money taken away by the kernel’s CFS (Completely Fair Scheduler).
The CTO wants “resource isolation.” He wants to ensure the legacy monolith doesn’t starve the new, shiny (and bloated) microservices. So he sets memory limits. But the JVM or the Python runtime doesn’t know it’s in a cgroup. It looks at /proc/meminfo (which isn’t namespaced by default) and thinks it has 128GB of RAM, then tries to allocate a massive heap, only to be brutally murdered by the OOM (Out of Memory) Killer because the cgroup limit was set to 2GB.
Back in the day, if a process ran out of memory, it was because you didn’t have enough physical DIMMs. Now, it’s because some YAML-wrangler forgot to account for the overhead of the container itself.
Let’s see what happens when we actually enforce these limits. I’ll set a hard limit of 10MB on a container and try to run something that needs 20MB.
# docker run --rm --memory="10m" alpine python3 -c "x = ' ' * 20 * 1024 * 1024"
Killed
# dmesg | tail -n 5
[34567.89] memory: usage 10240kB, limit 10240kB, failcnt 123
[34567.90] Memory cgroup out of memory: Killed process 12500 (python3) total-vm:45678kB, anon-rss:9876kB, file-rss:1234kB, shmem-rss:0kB
The kernel doesn’t care about your “graceful shutdown.” It sees a violation and it sends a SIGKILL. No cleanup, no log flushing, just instant death. And because you’re running in a “docker container,” you probably don’t even have persistent logs set up correctly, so the evidence of why it died vanishes into the ether.
H2: The Layered Lie: OverlayFS and the Death of Disk I/O
We need to talk about the filesystem. Docker uses a “copy-on-write” (CoW) strategy. It sounds efficient, but in practice, it’s a “copy-on-write filesystem fragmentation” nightmare. Every layer in your Dockerfile is a new directory in /var/lib/docker/overlay2. When you run apt-get update, you aren’t just updating a list; you are creating a massive delta that has to be managed by the kernel.
If you have a deep chain of layers, every open() syscall has to check every single layer to see if the file exists there. It’s a linear search through the directory stack.
# ls -l /var/lib/docker/overlay2/
total 0
drwx------ 4 root root 72 Oct 27 14:00 1a2b3c4d5e6f7g8h9i0j...
drwx------ 4 root root 72 Oct 27 14:00 2b3c4d5e6f7g8h9i0j1a...
drwx------ 4 root root 72 Oct 27 14:00 3c4d5e6f7g8h9i0j1a2b...
...
And God help you if you’re doing heavy disk I/O inside the container without using a volume. You’re writing to the “upper” layer of the OverlayFS, which means the kernel is doing double the work for every write. You’re not just writing to a block device; you’re writing to a virtualized filesystem that’s sitting on top of another filesystem. It’s like trying to run a marathon while wearing three layers of heavy winter coats.
H2: Breaking the Toy: Seccomp and the Security Theater
The industry loves to talk about how “secure” containers are. “It’s a sandbox!” they cry. No, it’s a sieve. By default, a “docker container” has access to a wide array of syscalls that it has no business touching. To “secure” it, we use seccomp (Secure Computing Mode) to filter syscalls. But nobody actually writes custom seccomp profiles because it’s hard and requires knowing how your application actually works—a skill that seems to be in short supply these days.
Let’s purposefully break a container by giving it a restrictive seccomp profile. I’ll create a profile that forbids the mkdir syscall.
// no-mkdir.json
{
"defaultAction": "SCMP_ACT_ALLOW",
"syscalls": [
{
"name": "mkdir",
"action": "SCMP_ACT_ERRNO",
"args": []
}
]
}
Now, let’s try to run a simple command:
# docker run --rm --security-opt seccomp=no-mkdir.json alpine mkdir /test
mkdir: can't create directory '/test': Operation not permitted
Error code EPERM. This is what happens when the kernel actually does its job. But in the real world, developers just run everything as root inside the container because “it’s easier,” and then they run the container with --privileged because they can’t figure out why their app can’t talk to the hardware. At that point, you haven’t built a sandbox; you’ve built a ladder for attackers to climb directly into your kernel.
H2: Why Your Layering Strategy is a Garbage Fire
I’ve seen Dockerfiles that look like they were written by someone trying to win an obfuscated C contest.
FROM ubuntu:latest
RUN apt-get update
RUN apt-get install -y python3
RUN apt-get install -y pip
COPY . /app
RUN pip install -r /app/requirements.txt
CMD ["python3", "/app/app.py"]
Every single RUN command here creates a new layer. You’ve just created a 1GB image for a 5MB application. You’re storing the apt cache in your image forever. You’re storing the build tools you only needed for the pip install phase.
Back when disk space was expensive, we cared about this. Now, we just throw S3 buckets at the problem and wonder why our CI/CD pipeline takes twenty minutes to pull an image. We’ve lost the art of the “multi-stage build,” or better yet, the art of just building a static binary and putting it in a scratch image. But no, we need the whole ubuntu:latest userland because “I might need to ls inside the container to debug it.” Here’s a tip: if you need to ls inside your production container to see what’s wrong, your observability stack is a failure, and so are you.
H2: The PID 1 Problem: Zombie Reaping in the Void
I touched on this earlier, but it deserves its own section because it’s the most common “docker container” failure I see in the wild. In Linux, when a process dies, it becomes a “zombie” (defunct) until its parent calls wait() or waitpid() to collect its exit code. If the parent dies, the process is orphaned and re-parented to PID 1.
In a normal system, PID 1 is init, which is designed to reap these zombies. In a container, your application is PID 1. If your Python script spawns a shell command and doesn’t explicitly wait for it, that shell process stays in the process table forever as a zombie.
# docker exec legacy-monolith-container ps aux
PID USER TIME COMMAND
1 root 0:00 python3 app.py
10 root 0:00 [sh] <defunct>
11 root 0:00 [sh] <defunct>
12 root 0:00 [sh] <defunct>
Eventually, you hit the max_pid limit of the host, and the entire server stops being able to fork new processes. The “docker container” didn’t crash, but it killed the host anyway. This is why tools like tini exist—to act as a tiny init for people who don’t know how to handle signals. We are adding another layer of abstraction just to fix the problems caused by the first layer of abstraction.
The Bitter End
So here I am, sitting in a room that is far too quiet because all the servers are in “the cloud” (which, let’s be honest, is just a warehouse in Virginia owned by a book salesman), trying to migrate a perfectly functional monolith into this “docker container” hellscape. I’ve spent the last six hours debugging why the veth pair is dropping packets because of an MTU mismatch that wouldn’t have existed if we were just running on bare metal.
We’ve traded the smell of ozone and the tactile satisfaction of a physical toggle switch for a “dashboard” that shows us “green bubbles” while the underlying kernel is screaming in agony. We’ve replaced systems engineering with YAML templating. We’ve decided that it’s better to ship 1GB of redundant garbage than to spend twenty minutes understanding our dependencies.
The CTO is happy. The developers are happy because they can “ship faster.” But I’m looking at the strace output, and I’m looking at the cgroup pressure stalls, and I know the truth. We haven’t solved anything. We’ve just hidden the complexity under so many layers of virtualization that when it finally breaks—and it will—nobody will have the slightest clue how to fix it. They’ll just try to “restart the container” and hope the magic returns.
I’m going home. I have a 1984 IBM Model M keyboard waiting for me. At least when I press a key on that, I know exactly which spring is pushing back. There’s no abstraction there. Just physics. And right now, physics is the only thing I still trust.
Related Articles
Explore more insights and best practices: