Docker Development

Docker Development

Patterns for building, testing, and deploying Docker containers.

Core Principles

1. Minimal -- Alpine/distroless, multi-stage 2. Secure -- Non-root USER, no layer secrets, pin versions 3. Testable -- CI-verifiable: entrypoint bypass, DNS mocking 4. Cache-efficient -- deps first, clean in same layer

Quick Reference

Multi-Stage Build (Node.js)

FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .

FROM node:20-alpine
RUN addgroup -g 1001 app && adduser -u 1001 -G app -D app
USER app
COPY --from=builder /app .
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD wget -qO- http://localhost:3000/health || exit 1
CMD ["node", "server.js"]

Multi-Stage Build (Go -- scratch/distroless)

FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY go.* ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /app/server .

FROM gcr.io/distroless/static:nonroot
COPY --from=builder /app/server /server
CMD ["/server"]

Layer Optimization

RUN apt-get update && \
    apt-get install -y --no-install-recommends curl && \
    rm -rf /var/lib/apt/lists/*

Build Cache: Copy Dependency Files First

COPY package*.json ./
RUN npm ci
COPY . .

Manifests before source keeps install layers cached on source-only changes.

BuildKit Secrets

RUN --mount=type=secret,id=ssh_key,dst=/root/.ssh/id_rsa git clone git@github.com:org/repo.git

ENV/ARG/COPY secrets persist in docker history. Use --mount=type=secret.

Docker Bake (Multi-Platform)

target "app" {
  platforms = ["linux/amd64", "linux/arm64"]
  cache-from = ["type=gha"]
  cache-to = ["type=gha,mode=max"]
}

Security Anti-Patterns

CI Testing Gotchas

1. Bypass entrypoint: docker run --rm --entrypoint php myimage -v 2. Mock upstream DNS: docker run --rm --add-host backend:127.0.0.1 nginx-image nginx -t 3. Compose validation: cp .env.example .env before docker compose config 4. Secret scanning: Exclude .env.example, README, docs from scanners 5. Root-owned artifacts: in-container installs leave root-owned bind-mount dirs (EACCES on host) -- references/bind-mount-ownership.md

.dockerignore

Exclude: .git, node_modules/vendor, .env*, *.pem, *.key

Compose Essentials

• startup ordering: depends_on.condition: service_healthy + healthcheck start_period
• networks.internal: true isolates databases from external access
• profiles: [debug]: services start only with --profile debug

References

• references/ci-testing.md -- CI testing patterns for Docker images
• references/dind-testing-patterns.md -- Docker-in-Docker (DinD) testing patterns
• references/bind-mount-ownership.md -- root-owned bind-mount artifacts

Bind-Mount Ownership: Root-Owned Artifacts on the Host

The Problem

Containers that run as root and write into a bind-mounted project directory leave root-owned files on the host. Typical producers:

docker compose run --rm app npm install     # node_modules/ now root-owned
docker compose run --rm app composer install
docker compose run --rm app npm run build  # dist/, public/build/ root-owned

The host user then hits failures that look unrelated:

npm error EACCES: permission denied, rename '.../node_modules/@babel/code-frame' -> ...
rm: cannot remove 'node_modules/...': Permission denied

Host-side npm install, build-tool cleanup steps (e.g. webpack/Encore cleanupOutputBeforeBuild), and even git clean -fdx fail on these files.

Diagnosis

find node_modules public/build -maxdepth 2 -user root | head

Any hit means a containerized process wrote there as root.

Cleanup (no sudo required)

Use a throwaway container — root inside the container can delete what root created, and the mount scopes it to the project:

docker run --rm -v "$PWD:/work" -w /work alpine \
  sh -c 'rm -rf node_modules public/build dist'

Then reinstall/rebuild as the host user.

Prevention

Rootless Docker / userns-remap avoids the issue entirely but changes semantics for the whole daemon.

Related Gotcha: Named Volumes Mask Image Content

A named volume mounted over a path (e.g. public/) is populated from the image only on first use. After deploying a new image, the volume still holds the old content — refresh it explicitly (temp container + docker cp/rsync) or recreate the volume as part of the deploy.

CI Testing Patterns for Docker Images

Comprehensive patterns for testing Docker images in CI/CD pipelines.

The Challenge

Docker images often have:

• Entrypoint scripts that run before any command
• Service dependencies (databases, caches) that don't exist in CI
• Network configurations referencing other containers
• Required environment variables that fail validation

These cause CI failures that don't occur in local development.

Pattern 1: Entrypoint Bypass

Problem

# Dockerfile
ENTRYPOINT ["/entrypoint.sh"]
CMD ["php-fpm"]

# CI - FAILS
- run: docker run --rm myimage php -v
# Output: entrypoint.sh runs, starts services, php -v never executes properly

Solution

# Override entrypoint for direct command execution
- run: docker run --rm --entrypoint php myimage -v
- run: docker run --rm --entrypoint php myimage -m  # List modules
- run: docker run --rm --entrypoint node myimage --version
- run: docker run --rm --entrypoint /bin/sh myimage -c "cat /etc/os-release"

When to Use

• Verifying installed software versions
• Checking available extensions/modules
• Testing configuration files
• Running diagnostic commands

Pattern 2: DNS Mocking for Upstream Services

Problem

# nginx.conf
upstream backend {
    server app:9000;
}

# CI - FAILS
- run: docker run --rm nginx-image nginx -t
# Error: host not found in upstream "app:9000"

Solution

# Provide fake DNS resolution for upstream hosts
- run: |
    docker run --rm \
      --add-host app:127.0.0.1 \
      --add-host database:127.0.0.1 \
      --add-host cache:127.0.0.1 \
      nginx-image nginx -t

Multiple Upstreams

- name: Test nginx config
  run: |
    docker run --rm \
      --add-host php-fpm:127.0.0.1 \
      --add-host mailpit:127.0.0.1 \
      --add-host redis:127.0.0.1 \
      myapp-nginx nginx -t

Pattern 3: Docker Compose Validation

Problem

# compose.yml
services:
  app:
    environment:
      - DB_PASSWORD=${DB_PASSWORD:?Required}

# CI - FAILS
- run: docker compose config
# Error: required variable DB_PASSWORD is missing

Solution

- name: Create test environment
  run: |
    cp .env.example .env
    # Replace all CHANGE_ME placeholders
    sed -i 's/CHANGE_ME_[A-Z_]*/test_password/g' .env

- name: Validate compose syntax
  run: docker compose config > /dev/null

Alternative: Inline Variables

- name: Validate compose
  env:
    DB_PASSWORD: test
    REDIS_PASSWORD: test
  run: docker compose config > /dev/null

Pattern 4: Health Check Verification

Test Health Check Command

- name: Build image
  run: docker build -t myapp:test .

- name: Test health check
  run: |
    # Start container
    docker run -d --name test-container myapp:test

    # Wait for health
    timeout 60 bash -c 'until docker inspect test-container --format="{{.State.Health.Status}}" | grep -q healthy; do sleep 2; done'

    # Verify
    docker inspect test-container --format="{{.State.Health.Status}}"

- name: Cleanup
  if: always()
  run: docker rm -f test-container

Worker/Sidecar Services That Reuse an App Image

A compose service that reuses the app image (e.g. a queue worker on the php-fpm+nginx web image) inherits the image's baked-in `HEALTHCHECK`. Three traps, in the order they typically bite in CI:

1. Inherited check probes a daemon the worker doesn't run (nginx, php-fpm) → the worker is permanently unhealthy and breaks docker compose up -d --wait — and anything else gating on health. 2. `healthcheck: { disable: true }` is not a fix when `--wait` is used — compose fails with container ... has no healthcheck configured (explicitly listed services without a check are un-waitable). Give the worker a real check instead. 3. *A naive `pgrep -f` check is always healthy* — the CMD-SHELL wrapper's own command line contains the search string, so pgrep matches the probe shell itself, even with a dead worker.

services:
  worker:
    image: myapp:latest   # inherits the web image's HEALTHCHECK
    command: php bin/console messenger:consume async
    healthcheck:
      # WRONG: matches the probe's own shell -- healthy forever
      # test: ["CMD-SHELL", "pgrep -f 'messenger:consume' || exit 1"]
      # RIGHT: [c]haracter-class guard prevents self-match
      test: ["CMD-SHELL", "pgrep -f '[m]essenger:consume' || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3

Verify any health probe both ways: process up → healthy AND process killed → unhealthy. The naive pgrep pattern passes the positive test and hides the bug.

Prefer letting a worker exit on its own limits (exec the daemon as PID 1, restart policy with backoff) over in-container while true ... || true loops that mask fatal errors from orchestration.

Pattern 5: Secret Scanning with Exclusions

Problem

Documentation references placeholder passwords:

<!-- README.md -->
Set `DB_PASSWORD=CHANGE_ME` in your .env file

# CI - FALSE POSITIVE
- run: git ls-files | xargs grep -l "CHANGE_ME" && exit 1
# Fails on README.md, QUICKSTART.md, etc.

Solution

- name: Check for leaked secrets
  run: |
    # Files that legitimately reference placeholders
    EXCLUDE_PATTERN=".env.example|README|QUICKSTART|docs/|Makefile|\.github/"

    # Find files with secrets, excluding legitimate references
    FOUND=$(git ls-files | xargs grep -l "CHANGE_ME" | grep -vE "$EXCLUDE_PATTERN" || true)

    if [ -n "$FOUND" ]; then
      echo "Found secrets in:"
      echo "$FOUND"
      exit 1
    fi
    echo "No leaked secrets detected"

Pattern 6: Multi-Platform Build Testing

- name: Set up QEMU
  uses: docker/setup-qemu-action@v3

- name: Set up Buildx
  uses: docker/setup-buildx-action@v3

- name: Build multi-platform
  uses: docker/build-push-action@v6
  with:
    platforms: linux/amd64,linux/arm64
    load: false  # Can't load multi-platform
    cache-from: type=gha
    cache-to: type=gha,mode=max

Pattern 7: Integration Testing with Service Containers

jobs:
  test:
    services:
      database:
        image: mariadb:11
        env:
          MYSQL_ROOT_PASSWORD: test
          MYSQL_DATABASE: testdb
        options: >-
          --health-cmd="healthcheck.sh --connect --innodb_initialized"
          --health-interval=10s
          --health-timeout=5s
          --health-retries=5

    steps:
      - name: Build app image
        run: docker build -t myapp:test .

      - name: Run integration tests
        run: |
          docker run --rm \
            --network ${{ job.container.network }} \
            -e DB_HOST=database \
            -e DB_PASSWORD=test \
            myapp:test npm test

Pattern 8: GitLab CI — image entrypoint must be a shell (or be overridden)

Unlike a test-time --entrypoint bypass (Pattern 1), GitLab runs every job's `script:` via `sh -c`. If the image used as a job image: has a non-shell ENTRYPOINT ["mytool"], the runner effectively runs mytool sh -c '…' → `No such command 'sh'`, and the job fails before the script runs.

job:
  image:
    name: registry.example.com/mytool:1.0
    entrypoint: [""]   # let the runner's shell execute the script
  script:
    - mytool --help

A CLI image also meant for docker run mytool … can keep ENTRYPOINT ["mytool"], but document that GitLab consumers must set entrypoint: [""]. If the image is primarily a CI image, prefer no tool entrypoint (use CMD).

Pattern 9: Restricted runner egress — bundle external assets at build time

CI runners (especially internal/self-hosted) often have no outbound internet. An image that fetches something at runtime (page.add_script_tag(url="https://cdn…/axe.min.js"), curl https://… in the entrypoint, a remote pip/npm install) works locally but fails in CI.

Download the asset at build time and load it from the image. Use a multi-stage build so the fetch tooling (curl, ca-certificates) stays out of the final runtime image:

# Stage 1: fetch external assets
FROM alpine:3.20 AS asset-builder
RUN apk add --no-cache curl
RUN mkdir -p /opt/axe-core \
 && curl -sSfL https://cdnjs.cloudflare.com/ajax/libs/axe-core/4.9.1/axe.min.js \
      -o /opt/axe-core/axe.min.js

# Stage 2: final image carries only the asset
FROM python:3.12-slim
COPY --from=asset-builder /opt/axe-core/axe.min.js /opt/axe-core/axe.min.js
ENV AXE_PATH=/opt/axe-core/axe.min.js

…and have the app prefer the local file (CDN as a dev-only fallback).

Pattern 10: Test the built image, not just the editable dev install

A non-editable install in the image (pip install ., npm install <tarball>) does not behave like the editable/dev checkout your tests ran against. Classic failure: data files resolved by walking from `__file__` (Path(__file__).resolve().parents[2]/"data"/…) don't exist under site-packages, so the tool can't find its catalog/config inside the container even though pytest was green.

• Ship data files as package data (Python wheel force-include/package_data; npm files), not via filesystem-relative paths.
• Smoke-test the built image, not just the source tree:

- run: docker build -t app:test .
- run: docker run --rm --entrypoint python app:test -c "import app; app.load_catalog()"
- run: docker run --rm app:test render fixture.json /tmp/out   # real command, end-to-end

Complete CI Workflow Example

name: Docker CI

on: [push, pull_request]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup test environment
        run: |
          cp .env.example .env
          sed -i 's/CHANGE_ME_[A-Z_]*/ci_test_value/g' .env

      - name: Validate compose
        run: docker compose config > /dev/null

  build-and-test:
    runs-on: ubuntu-latest
    needs: validate
    steps:
      - uses: actions/checkout@v4

      - uses: docker/setup-buildx-action@v3

      - name: Build images
        run: docker compose build

      - name: Test PHP version
        run: docker run --rm --entrypoint php myapp:latest -v | grep "8.4"

      - name: Test nginx config
        run: docker run --rm --add-host app:127.0.0.1 myapp-nginx nginx -t

      - name: Start stack
        run: |
          cp .env.example .env
          sed -i 's/CHANGE_ME_[A-Z_]*/ci_test/g' .env
          docker compose up -d

      - name: Wait for healthy
        run: |
          timeout 120 bash -c 'until docker compose ps | grep -q healthy; do sleep 5; done'

      - name: Test endpoint
        run: curl -f http://localhost/ || exit 1

      - name: Cleanup
        if: always()
        run: docker compose down -v

Local boot-test pitfalls

When smoke/boot-testing an image by hand (not in the CI matrix):

• Host-port collisions mislead. If the published port (-p HOST:CONTAINER) is already taken by another container, docker run -d leaves the new container unstarted (it stays in Created state; the CLI typically exits non-zero, often 125) while your curl localhost:HOST is answered by the other container — a false pass, or a baffling failure. Use a free/unique host port, or skip -p and probe from inside: docker exec <c> sh -c 'curl -sf localhost:<port>'.
• Foreground apps that log to a file leave `docker logs` empty. E.g. Tomcat started with -fg writes to logs/catalina.out, not stdout — an empty docker logs does not mean "nothing happened". Read the in-container log files (docker exec <c> sh -c 'tail -n 80 .../catalina.out'), and check the process and state (docker inspect -f '{{.State.Status}} {{.State.ExitCode}}' <c>).
• Minimal/distroless images have no shell. docker exec … sh/tail/pgrep won't exist on scratch/distroless runtimes — probe with host-side curl against a published port, docker inspect for state, or a debug sidecar (docker run --rm --pid container:<c> busybox …).
• Grep for real failure signals, not benign noise. After a bundled-dependency swap, scan logs for NoSuchMethodError|AbstractMethodError|LinkageError|IncompatibleClassChangeError (binary incompatibility) — not bare ClassNotFoundException, which OSGi/plugin frameworks emit normally.

Docker-in-Docker (DinD) Testing Patterns

Patterns for running Docker inside Docker in CI environments (Molecule, Testcontainers, nested builds).

The Overlay-on-Overlay Problem

GitHub Actions runners (and most CI platforms) use the overlay2 filesystem driver for Docker. When you run Docker inside Docker (e.g., Molecule testing Ansible roles, Testcontainers, nested builds), the inner Docker daemon also tries to use overlay2. The Linux kernel cannot stack overlay-on-overlay — this fails with:

mount source: "overlay", fstype: overlay, err: invalid argument

This affects any CI job that starts Docker containers from within a Docker container.

Solution: VFS Storage Driver

Configure the inner Docker daemon to use the vfs storage driver instead of overlay2. VFS is slower (it copies full layers instead of using overlays) but works reliably inside containers.

Molecule + geerlingguy.docker Role

# molecule/default/prepare.yml
- name: Prepare
  hosts: all
  tasks:
    - name: Install Docker with VFS driver
      ansible.builtin.include_role:
        name: geerlingguy.docker
      vars:
        docker_daemon_options:
          storage-driver: vfs

Direct daemon.json Configuration

{
  "storage-driver": "vfs"
}

Write this to /etc/docker/daemon.json inside the container before starting Docker.

GitHub Actions Service Container

# .github/workflows/test.yml
jobs:
  test:
    runs-on: ubuntu-latest
    services:
      dind:
        image: docker:dind
        env:
          DOCKER_OPTS: "--storage-driver=vfs"
        options: --privileged

GitLab CI

test:
  image: docker:latest
  services:
    - name: docker:dind
      variables:
        DOCKER_OPTS: "--storage-driver=vfs"
  variables:
    DOCKER_HOST: tcp://docker:2376

Systemd in Containers

When testing with containers that run systemd (e.g., Molecule testing on systemd-based OS images), additional configuration is required:

# molecule/default/molecule.yml
platforms:
  - name: instance
    image: geerlingguy/docker-debian12-ansible:latest
    command: /lib/systemd/systemd
    privileged: true
    cgroupns_mode: host
    tmpfs:
      - /run
      - /run/lock
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:rw

Why These Settings

Privileged Mode

When It Is Needed

• Docker-in-Docker: The inner Docker daemon needs to create network namespaces, mount filesystems, and manage cgroups
• Systemd containers: systemd requires cgroup access and device control
• iptables/networking: Containers that modify firewall rules or create network bridges

Security Implications

• --privileged disables all security confinements (AppArmor, seccomp, capabilities)
• The container can access all host devices and modify the host kernel
• In CI, this is generally acceptable because the runner is ephemeral
• In production, never use --privileged — use specific --cap-add flags instead

# Production alternative: grant only needed capabilities
docker run --cap-add SYS_ADMIN --cap-add NET_ADMIN --security-opt apparmor=unconfined myimage

Alternative Approaches

Docker Socket Mounting

Mount the host's Docker socket instead of running a full inner daemon:

# The container uses the HOST's Docker daemon
docker run -v /var/run/docker.sock:/var/run/docker.sock myimage

Pros: No overlay-on-overlay issue, faster, less resource usage Cons: Containers share the host daemon — no isolation, cleanup is shared, security risk (container can control host Docker)

Podman Rootless

Podman runs without a daemon and supports rootless nested containers:

# GitHub Actions
- name: Test with Podman
  run: |
    podman run --rm --privileged \
      -v ./:/workspace:Z \
      quay.io/podman/stable \
      podman build /workspace

Buildah for Image Builds

If you only need to build images (not run containers), Buildah avoids DinD entirely:

- name: Build with Buildah
  run: |
    buildah bud -t myimage:test .
    buildah push myimage:test docker-daemon:myimage:test

Complete CI Example: Molecule with DinD

name: Ansible Role CI

on: [push, pull_request]

jobs:
  molecule:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        distro:
          - debian12
          - ubuntu2404
          - rockylinux9

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install molecule molecule-plugins[docker] ansible

      - name: Run Molecule
        run: molecule test
        env:
          MOLECULE_DISTRO: ${{ matrix.distro }}

With the corresponding Molecule prepare step using VFS:

# molecule/default/prepare.yml
- name: Prepare
  hosts: all
  tasks:
    - name: Install Docker with VFS driver
      ansible.builtin.include_role:
        name: geerlingguy.docker
      vars:
        docker_daemon_options:
          storage-driver: vfs

Troubleshooting