Docker Errors Runtime

docker-errors-runtime

Quick Reference

Exit Code Reference

Critical Warnings

NEVER ignore exit code 137 — it ALWAYS indicates the container was forcefully killed. Check OOM events with docker inspect and dmesg before increasing memory limits blindly.

NEVER use --oom-kill-disable without setting a memory limit (-m) — the container can consume ALL host memory and crash the entire system.

NEVER assume a container that exits with code 0 is healthy — it may have completed a one-shot command instead of running as a long-lived service. ALWAYS verify the process runs in the foreground.

ALWAYS check docker logs before any other debugging step — 90% of runtime issues are explained in the application output.

ALWAYS use docker inspect --format='{{.State.ExitCode}}' to get the exact exit code — docker ps -a truncates status information.

---

Debugging Workflow

Step 1: Check Logs

# Last 100 lines
docker logs --tail 100 <container>

# Follow live output with timestamps
docker logs -f -t <container>

# Logs from last 5 minutes
docker logs --since 5m <container>

Step 2: Inspect Container State

# Exit code and error message
docker inspect --format='{{.State.ExitCode}}' <container>
docker inspect --format='{{.State.Error}}' <container>

# OOM killed?
docker inspect --format='{{.State.OOMKilled}}' <container>

# Full state as JSON
docker inspect --format='{{json .State}}' <container> | jq .

Step 3: Exec Into Running Container

# Interactive shell (if container is still running)
docker exec -it <container> sh
docker exec -it <container> bash

# Check filesystem, processes, network
docker exec <container> ps aux
docker exec <container> df -h
docker exec <container> cat /etc/resolv.conf

Step 4: Check System Events

# Events for specific container in last 10 minutes
docker events --since 10m --filter container=<container>

# OOM events specifically
docker events --filter event=oom --since 1h

# All die events
docker events --filter event=die --since 1h

Step 5: Resource Usage

# Live resource stats
docker stats <container>

# Single snapshot
docker stats --no-stream <container>

# System-wide disk usage
docker system df -v

---

Runtime Error Diagnostic Table

Container Exits Immediately (Exit Code 0 or 1)

OOM Killed (Exit Code 137)

Permission Denied

Port Already in Use

Exec Format Error

Read-Only Filesystem

PID Limit and Resource Exhaustion

---

docker inspect for Debugging

Essential Inspect Commands

# Full state overview
docker inspect --format='{{json .State}}' <container> | jq .

# Why did it stop?
docker inspect --format='ExitCode={{.State.ExitCode}} OOM={{.State.OOMKilled}} Error={{.State.Error}}' <container>

# What command is it running?
docker inspect --format='Entrypoint={{.Config.Entrypoint}} Cmd={{.Config.Cmd}}' <container>

# Environment variables
docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}' <container>

# Mount points
docker inspect --format='{{range .Mounts}}{{.Type}}: {{.Source}} -> {{.Destination}} ({{if .RW}}rw{{else}}ro{{end}}){{println}}{{end}}' <container>

# Network settings
docker inspect --format='{{range $net, $config := .NetworkSettings.Networks}}{{$net}}: {{$config.IPAddress}}{{println}}{{end}}' <container>

# Resource limits
docker inspect --format='Memory={{.HostConfig.Memory}} CPUs={{.HostConfig.NanoCpus}} PidsLimit={{.HostConfig.PidsLimit}}' <container>

# Health check status
docker inspect --format='{{.State.Health.Status}}' <container>
docker inspect --format='{{json .State.Health}}' <container> | jq .

# Restart count
docker inspect --format='RestartCount={{.RestartCount}}' <container>

---

Decision Trees

Container Won't Start

Container won't start
├─ Exit 125 → Docker daemon error
│  ├─ "invalid reference format" → Fix image name/tag
│  ├─ "no such image" → Pull image first: docker pull <image>
│  └─ "invalid mount config" → Fix volume/mount syntax
├─ Exit 126 → Command not executable
│  ├─ Check file permissions → chmod +x
│  └─ Check binary format → file <binary>
├─ Exit 127 → Command not found
│  ├─ Typo in CMD/ENTRYPOINT → Fix spelling
│  ├─ Binary not in PATH → Use absolute path
│  └─ Binary not installed → Add to Dockerfile
└─ Exit 0/1 instantly → See "Container Exits Immediately" table

Container Crashes After Running

Container was running, then died
├─ Exit 137 → Killed
│  ├─ OOMKilled=true → Memory limit too low (see OOM section)
│  ├─ OOMKilled=false, after docker stop → Stop timeout too short
│  └─ OOMKilled=false, unexpected → Check docker events + dmesg
├─ Exit 139 → Segfault
│  ├─ Native library issue → Check library compatibility
│  └─ Memory corruption → Debug with --cap-add SYS_PTRACE
├─ Exit 143 → Graceful SIGTERM
│  └─ Expected from docker stop → Normal shutdown
└─ Exit 1 → Application error
   └─ Check docker logs → Fix application bug

---

Reference Links

• references/diagnostics.md -- Complete error-to-solution mapping for all runtime errors
• references/examples.md -- Step-by-step debugging sessions with real commands
• references/anti-patterns.md -- Runtime configuration mistakes and how to avoid them

Official Sources

• https://docs.docker.com/engine/daemon/troubleshoot/
• https://docs.docker.com/reference/cli/docker/container/run/
• https://docs.docker.com/reference/cli/docker/container/logs/
• https://docs.docker.com/reference/cli/docker/inspect/
• https://docs.docker.com/reference/cli/docker/system/events/

Runtime Anti-Patterns

Configuration mistakes that cause runtime failures, with explanations and correct alternatives.

---

Memory & Resource Anti-Patterns

AP-1: No Memory Limit in Production

# WRONG — container can consume ALL host memory
docker run -d myapp:v1

# CORRECT — ALWAYS set memory limits in production
docker run -d -m 512m myapp:v1

Why: Without limits, a memory leak in one container can trigger the host OOM killer, taking down ALL containers on the host.

AP-2: OOM Kill Disable Without Memory Limit

# WRONG — container can consume infinite memory, crash the host
docker run -d --oom-kill-disable myapp:v1

# CORRECT — ALWAYS pair with a memory limit
docker run -d -m 512m --oom-kill-disable myapp:v1

Why: --oom-kill-disable without -m means the kernel cannot reclaim memory from this container. If the application leaks memory, the host kernel will eventually kill random processes (including other containers) to free memory.

AP-3: Setting Memory Too Low Without Profiling

# WRONG — arbitrary memory limit without profiling
docker run -d -m 64m java-app:v1

# CORRECT — profile first, then set limit with headroom
docker run -d --name test java-app:v1
docker stats test --no-stream
# NAME  MEM USAGE / LIMIT    MEM %
# test  487.2MiB / 7.775GiB  6.12%
# Now set limit with ~30% headroom
docker run -d -m 650m java-app:v1

Why: Setting memory limits too low causes OOM kills that look like application bugs. ALWAYS measure actual usage before setting limits.

AP-4: No PID Limit (Fork Bomb Vulnerability)

# WRONG — no PID limit allows fork bombs to crash the host
docker run -d myapp:v1

# CORRECT — set PID limit to prevent fork bombs
docker run -d --pids-limit 200 myapp:v1

Why: A malicious or buggy process can fork infinitely, consuming all PIDs on the host and making it unresponsive.

---

Process & Entrypoint Anti-Patterns

AP-5: Daemonizing the Main Process

# WRONG — nginx forks to background, PID 1 exits, container stops
CMD ["nginx"]

# CORRECT — run in foreground
CMD ["nginx", "-g", "daemon off;"]

Why: Docker monitors PID 1. If it exits (even with code 0), the container stops. Background daemons fork and let PID 1 exit.

AP-6: Shell Form CMD/ENTRYPOINT

# WRONG — wraps in /bin/sh -c, signals not forwarded to app
CMD node server.js

# CORRECT — exec form, app is PID 1, receives signals directly
CMD ["node", "server.js"]

Why: Shell form runs the process as a child of /bin/sh. SIGTERM goes to the shell, not the application. The app never gets a chance to shut down gracefully, and Docker force-kills it after the stop timeout.

AP-7: Not Using exec in Entrypoint Scripts

# WRONG — entrypoint.sh
#!/bin/sh
echo "Starting app..."
node server.js
# Shell remains as PID 1, node is PID 2

# CORRECT — entrypoint.sh
#!/bin/sh
echo "Starting app..."
exec node server.js
# node replaces shell as PID 1

Why: Without exec, the shell process stays as PID 1. Signals are not forwarded to the actual application. docker stop kills the shell, leaving the app as an orphan that gets SIGKILL after the timeout.

AP-8: CRLF Line Endings in Scripts

# WRONG — script has Windows line endings, causes "exec format error"
COPY entrypoint.sh /entrypoint.sh

# CORRECT — convert line endings in Dockerfile
COPY entrypoint.sh /entrypoint.sh
RUN sed -i 's/\r$//' /entrypoint.sh && chmod +x /entrypoint.sh

Better: Add .gitattributes to the repository:

*.sh text eol=lf
Dockerfile text eol=lf
*.yml text eol=lf
*.yaml text eol=lf

Why: The kernel reads #!/bin/sh\r as the interpreter path. The \r (carriage return) is part of the path, and /bin/sh\r does not exist.

AP-9: Missing Shebang in Entrypoint Scripts

# WRONG — no shebang, kernel doesn't know how to execute
echo "Starting..."
node server.js

# CORRECT — always include shebang
#!/bin/sh
echo "Starting..."
exec node server.js

Why: Without a shebang, the kernel cannot determine the interpreter. The exec system call fails with "exec format error."

---

Networking Anti-Patterns

AP-10: Using Default Bridge Network

# WRONG — containers can't resolve each other by name
docker run -d --name api myapi:v1
docker run -d --name web myweb:v1
docker exec web curl http://api:3000  # FAILS: name resolution error

# CORRECT — use a user-defined network
docker network create mynet
docker run -d --name api --network mynet myapi:v1
docker run -d --name web --network mynet myweb:v1
docker exec web curl http://api:3000  # WORKS

Why: The default bridge network does not provide DNS resolution between containers. User-defined bridge networks include an embedded DNS server.

AP-11: Exposing Ports on 0.0.0.0 in Production

# WRONG — binds to all interfaces, accessible from any network
docker run -d -p 5432:5432 postgres:16

# CORRECT — bind to localhost or specific interface
docker run -d -p 127.0.0.1:5432:5432 postgres:16

Why: Binding to 0.0.0.0 exposes the port on all network interfaces, including public-facing ones. Database and internal services should NEVER be directly accessible from the internet.

AP-12: Hardcoded Container IPs

# WRONG — IPs change on container restart
docker exec web curl http://172.18.0.3:3000

# CORRECT — use DNS names (container names or aliases)
docker exec web curl http://api:3000

Why: Container IP addresses are dynamic and change on restart, network reconnect, or redeployment. DNS names are stable.

---

Volume & Storage Anti-Patterns

AP-13: Writing Application Data to Container Layer

# WRONG — data stored in container layer, lost on container removal
CMD ["node", "server.js"]
# Application writes to /app/data/ inside the container

# CORRECT — use a volume for persistent data
# docker run -d --mount type=volume,src=appdata,dst=/app/data myapp:v1

Why: The container's writable layer is ephemeral. When the container is removed (docker rm), all data in the writable layer is permanently lost.

AP-14: Anonymous Volumes with --rm

# WRONG — anonymous volume deleted when container stops
docker run --rm -v /data mydb:v1

# CORRECT — use a named volume
docker run --rm --mount type=volume,src=dbdata,dst=/data mydb:v1

Why: The --rm flag removes anonymous volumes when the container exits. Named volumes persist regardless of --rm.

AP-15: Bind Mounts in Production Without Understanding -v Auto-Create

# WRONG — -v auto-creates missing host directory as root-owned empty dir
docker run -v /host/config:/app/config myapp:v1
# If /host/config doesn't exist, Docker creates it as empty root-owned directory

# CORRECT — use --mount which errors on missing source
docker run --mount type=bind,src=/host/config,dst=/app/config myapp:v1
# Error: bind source path does not exist: /host/config

Why: The -v flag silently creates missing host directories, leading to containers starting with empty config directories instead of failing fast. --mount provides explicit error handling.

---

Security Anti-Patterns

AP-16: Running as Root

# WRONG — process runs as root inside container
FROM node:20
COPY . /app
CMD ["node", "/app/server.js"]

# CORRECT — create and use non-root user
FROM node:20
RUN groupadd -r appuser && useradd --no-log-init -r -g appuser appuser
COPY --chown=appuser:appuser . /app
USER appuser
CMD ["node", "/app/server.js"]

Why: If an attacker exploits the application, they have root access inside the container. Combined with misconfigurations (privileged mode, host mounts), this can lead to host compromise.

AP-17: Using --privileged for Specific Capabilities

# WRONG — grants ALL capabilities + device access + disables security
docker run --privileged myapp:v1

# CORRECT — add only the specific capability needed
docker run --cap-add SYS_PTRACE myapp:v1

Why: --privileged disables ALL security isolation: capabilities, seccomp, AppArmor, and device restrictions. It is equivalent to running directly on the host.

AP-18: Ignoring no-new-privileges

# WRONG — process can escalate privileges via setuid binaries
docker run myapp:v1

# CORRECT — prevent privilege escalation
docker run --security-opt no-new-privileges=true myapp:v1

Why: Without no-new-privileges, a process can gain additional privileges through setuid/setgid binaries. This is a common container escape vector.

---

Restart & Lifecycle Anti-Patterns

AP-19: Using restart=always for Everything

# WRONG — restarts even on clean exit, masks bugs
docker run -d --restart always debug-task:v1

# CORRECT — use on-failure for tasks that should only restart on errors
docker run -d --restart on-failure:5 debug-task:v1

# CORRECT — use unless-stopped for long-running services
docker run -d --restart unless-stopped nginx

Why: restart=always restarts the container even when it exits cleanly (code 0). This hides bugs where the container should stay stopped and wastes resources in restart loops.

AP-20: No Health Check

# WRONG — Docker only knows if PID 1 is running, not if app is healthy
docker run -d myapp:v1

# CORRECT — add health check for actual application health
docker run -d \
  --health-cmd='curl -f http://localhost:8080/health || exit 1' \
  --health-interval=30s \
  --health-timeout=10s \
  --health-retries=3 \
  --health-start-period=40s \
  myapp:v1

Why: Without a health check, Docker considers a container "healthy" as long as PID 1 is running. The application could be deadlocked, out of connections, or returning errors — Docker won't know. Orchestrators (Compose, Swarm) use health status for dependency ordering and rolling updates.

AP-21: Relying on restart=always Instead of Fixing Root Cause

# WRONG — masking a recurring crash with restart policy
docker run -d --restart always myapp:v1
# Container keeps crashing and restarting every 30 seconds

# CORRECT — investigate the root cause
docker update --restart no myapp
docker stop myapp
docker logs --tail 100 myapp
# Fix the bug, then deploy with appropriate restart policy

Why: Restart policies are a safety net, not a substitute for fixing bugs. A container in a restart loop consumes CPU, generates logs, and may leave corrupt state.

---

Compose-Specific Anti-Patterns

AP-22: depends_on Without Health Checks

# WRONG — depends_on only waits for container start, not readiness
services:
  web:
    depends_on:
      - db
  db:
    image: postgres:16

# CORRECT — use depends_on with condition
services:
  web:
    depends_on:
      db:
        condition: service_healthy
  db:
    image: postgres:16
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 5

Why: depends_on without a health check condition only ensures the dependency container has started. The database may not be ready to accept connections yet, causing the web service to crash on startup.

AP-23: Using container_name in Compose

# WRONG — prevents scaling, causes name conflicts across projects
services:
  web:
    container_name: web
    image: nginx

# CORRECT — let Compose manage names (project_service_N)
services:
  web:
    image: nginx

Why: Fixed container names prevent docker compose up --scale web=3 and cause conflicts if multiple Compose projects use the same name.

Runtime Error Diagnostics Reference

Complete error → cause → solution mapping for Docker container runtime errors.

---

Exit Code Reference (Extended)

How to Read Exit Codes

# Get exit code from stopped container
docker inspect --format='{{.State.ExitCode}}' <container>

# Get exit code from docker wait (blocking)
EXIT_CODE=$(docker wait <container>)
echo "Exit code: $EXIT_CODE"

# List all stopped containers with exit codes
docker ps -a --filter status=exited --format "table {{.Names}}\t{{.Status}}"

---

OOM (Out of Memory) Diagnostics

Confirming OOM Kill

# Check OOMKilled flag
docker inspect --format='{{.State.OOMKilled}}' <container>

# Check system logs for OOM events
dmesg | grep -i "oom\|killed process"

# Docker events for OOM
docker events --filter event=oom --since 1h

# Full memory config
docker inspect --format='Memory={{.HostConfig.Memory}} MemorySwap={{.HostConfig.MemorySwap}} MemoryReservation={{.HostConfig.MemoryReservation}}' <container>

OOM Scenario Matrix

OOM Prevention Checklist

1. ALWAYS set memory limits in production: docker run -m 512m 2. Profile actual usage first: docker stats --no-stream 3. Set memory reservation for soft limit: --memory-reservation 256m 4. Monitor with: docker stats --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}" 5. NEVER use --oom-kill-disable without -m — the container can crash the host

---

Permission Denied Diagnostics

File Permission Issues

Capability Issues

---

Port Conflict Diagnostics

Finding What Uses a Port

# On Linux
lsof -i :8080
ss -tlnp | grep 8080
netstat -tlnp | grep 8080

# On macOS
lsof -i :8080

# Find Docker container using port
docker ps --format "{{.Names}}: {{.Ports}}" | grep 8080

# Check all port mappings for a container
docker port <container>

Port Conflict Resolution

---

Exec Format Error Diagnostics

Architecture Mismatch

# Check image architecture
docker inspect --format='{{.Architecture}}' <image>

# Check host architecture
uname -m

# Pull for specific platform
docker pull --platform linux/amd64 <image>

# Build for specific platform
docker buildx build --platform linux/amd64 -t <image> .

Script Issues

---

Read-Only Filesystem Diagnostics

Common Write Paths That Need tmpfs

Read-Only Configuration Pattern

# Read-only root with specific writable paths
docker run --read-only \
  --tmpfs /tmp:size=64m \
  --tmpfs /run:size=64m \
  --mount type=volume,src=appdata,dst=/app/data \
  <image>

---

PID Limit Diagnostics

Detecting PID Exhaustion

# Check PID limit setting
docker inspect --format='{{.HostConfig.PidsLimit}}' <container>

# Count processes in container
docker top <container> | wc -l

# Check from inside container
docker exec <container> cat /sys/fs/cgroup/pids.max
docker exec <container> cat /sys/fs/cgroup/pids.current

PID Limit Guidelines

---

Resource Exhaustion Diagnostics

Disk Space

# Docker disk usage overview
docker system df

# Detailed breakdown
docker system df -v

# Container writable layer size
docker ps -s --format "table {{.Names}}\t{{.Size}}"

# Find large files in container
docker exec <container> du -sh /* 2>/dev/null | sort -rh | head -10

# Clean unused resources
docker system prune          # Safe: stopped containers, unused networks, dangling images
docker builder prune          # Build cache
docker volume prune           # CAREFUL: removes unused volumes including data

CPU Throttling

# Check CPU limits
docker inspect --format='CPUs={{.HostConfig.NanoCpus}} CPUShares={{.HostConfig.CpuShares}}' <container>

# Monitor CPU usage
docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}" --no-stream

# Update CPU limit on running container
docker update --cpus 2 <container>

---

Container Restart Loop Diagnostics

Detecting Restart Loops

# Check restart count
docker inspect --format='RestartCount={{.RestartCount}} Policy={{.HostConfig.RestartPolicy.Name}}' <container>

# Watch for rapid restarts
docker events --filter container=<container> --filter event=start

# Check last exit info
docker inspect --format='ExitCode={{.State.ExitCode}} FinishedAt={{.State.FinishedAt}}' <container>

Restart Policy Reference

Breaking a Restart Loop

# Stop the container (overrides restart policy)
docker stop <container>

# Update restart policy to prevent restarts
docker update --restart no <container>

# Check logs from the last crash
docker logs --tail 50 <container>

# Fix the issue, then re-enable restart policy
docker update --restart unless-stopped <container>
docker start <container>

Runtime Debugging Examples

Step-by-step debugging sessions with real commands for common Docker runtime failures.

---

Example 1: Container Exits Immediately

Scenario

A web application container starts and immediately exits with code 0.

Debugging Session

# Step 1: Check the container status
docker ps -a --filter name=webapp
# CONTAINER ID  IMAGE      STATUS                    NAMES
# abc123        myapp:v1   Exited (0) 2 seconds ago  webapp

# Step 2: Check logs
docker logs webapp
# (empty output or daemon startup message)

# Step 3: Inspect the CMD
docker inspect --format='Entrypoint={{.Config.Entrypoint}} Cmd={{.Config.Cmd}}' webapp
# Entrypoint=[] Cmd=[nginx]

# Step 4: The problem — nginx daemonizes by default
# Fix: Run nginx in foreground mode
docker run -d --name webapp myapp:v1 nginx -g "daemon off;"

# Or fix in Dockerfile:
# CMD ["nginx", "-g", "daemon off;"]

Root Cause

nginx (and many other services like Apache, sshd) daemonize by default. The container's main process (PID 1) forks a background process and exits. Docker sees PID 1 exit with code 0 and stops the container.

Prevention

ALWAYS ensure the main process runs in the foreground:

• nginx: nginx -g "daemon off;"
• Apache: httpd -D FOREGROUND
• sshd: /usr/sbin/sshd -D
• Custom scripts: Use exec to replace shell with the actual process

---

Example 2: OOM Killed Container

Scenario

A Java application container randomly crashes after running for a while.

Debugging Session

# Step 1: Check exit code and OOM status
docker inspect --format='ExitCode={{.State.ExitCode}} OOM={{.State.OOMKilled}}' myapp
# ExitCode=137 OOM=true

# Step 2: Check current memory limit
docker inspect --format='Memory={{.HostConfig.Memory}}' myapp
# Memory=268435456  (256MB)

# Step 3: Check actual memory usage before it crashed
docker events --filter container=myapp --filter event=oom --since 1h
# 2026-03-19T14:22:33 container oom abc123 (name=myapp, image=myapp:v1)

# Step 4: Profile memory usage with a fresh container
docker run -d --name myapp-test -m 1g myapp:v1
docker stats myapp-test --no-stream
# NAME        CPU%  MEM USAGE / LIMIT    MEM %
# myapp-test  2.3%  487.2MiB / 1GiB      47.58%

# Step 5: The problem — 256MB is too low for a JVM application
# Fix: Set appropriate memory limit based on profiling
docker run -d --name myapp -m 1g \
  -e JAVA_OPTS="-Xmx512m -Xms256m" \
  myapp:v1

Root Cause

The Java application's default heap size exceeded the 256MB container memory limit. The Linux OOM killer terminated the process (SIGKILL = exit code 137).

Prevention

• ALWAYS profile actual memory usage with docker stats before setting limits
• For JVM apps: Set -Xmx to ~75% of the container memory limit
• Set --memory-reservation as a soft limit for orchestrator awareness

---

Example 3: Permission Denied on Volume

Scenario

An application cannot write to a mounted volume directory.

Debugging Session

# Step 1: Check logs for the error
docker logs myapp
# Error: EACCES: permission denied, open '/data/output.json'

# Step 2: Check the user the container runs as
docker inspect --format='{{.Config.User}}' myapp
# 1001

# Step 3: Check file ownership on the volume
docker exec myapp ls -la /data
# drwxr-xr-x 2 root root 4096 Mar 19 12:00 .
# -rw-r--r-- 1 root root  128 Mar 19 12:00 config.json

# Step 4: The problem — container runs as UID 1001 but /data is owned by root
# Fix option A: Run as root (NOT recommended for production)
docker run -u 0 -v mydata:/data myapp:v1

# Fix option B: Match UIDs (preferred)
docker run -u $(id -u):$(id -g) -v mydata:/data myapp:v1

# Fix option C: Fix ownership in Dockerfile (best)
# In Dockerfile:
# RUN mkdir -p /data && chown -R 1001:1001 /data
# USER 1001

Root Cause

The container process runs as UID 1001 (non-root) but the volume directory is owned by root (UID 0). Linux file permissions prevent the write.

Prevention

• ALWAYS set ownership of data directories in the Dockerfile before switching to non-root user
• Use named volumes (Docker manages permissions) instead of bind mounts where possible
• On Linux with bind mounts and SELinux: add :z or :Z suffix

---

Example 4: Port Already in Use

Scenario

Starting a container fails with a port conflict error.

Debugging Session

# Step 1: See the error
docker run -d -p 8080:80 --name web nginx
# Error: driver failed programming external connectivity:
# Bind for 0.0.0.0:8080 failed: port is already allocated

# Step 2: Check what Docker container uses that port
docker ps --format "{{.Names}}: {{.Ports}}" | grep 8080
# old-web: 0.0.0.0:8080->80/tcp

# Step 3a: If it's a Docker container — stop it
docker stop old-web && docker rm old-web

# Step 3b: If no Docker container found — check host processes
lsof -i :8080
# COMMAND  PID   USER  FD  TYPE DEVICE SIZE NODE NAME
# node     1234  user  12u IPv4 54321  TCP *:8080 (LISTEN)

# Step 3c: Use a different host port instead
docker run -d -p 8081:80 --name web nginx

# Step 4: Start the container
docker run -d -p 8080:80 --name web nginx

Prevention

• Use --rm flag so containers auto-remove on stop
• Use Docker Compose with unique project names for automatic port management
• Use docker compose down before docker compose up to clean previous state

---

Example 5: Exec Format Error

Scenario

A container fails to start with an exec format error.

Debugging Session

# Step 1: See the error
docker run myapp:v1
# exec /entrypoint.sh: exec format error

# Step 2: Check the entrypoint script
docker run --entrypoint cat myapp:v1 /entrypoint.sh | head -1
# #!/bin/bash^M

# Step 3: The problem — CRLF line endings (^M = \r)
# The kernel cannot find interpreter "#!/bin/bash\r"

# Fix option A: In Dockerfile, convert line endings
# RUN sed -i 's/\r$//' /entrypoint.sh

# Fix option B: Configure Git on the build machine
# git config core.autocrlf input

# Fix option C: Add .gitattributes to the repo
# *.sh text eol=lf
# Dockerfile text eol=lf

Alternative Scenario: Architecture Mismatch

# Step 1: See the error
docker run myapp:v1
# exec /app/server: exec format error

# Step 2: Check image and host architecture
docker inspect --format='{{.Architecture}}' myapp:v1
# arm64

uname -m
# x86_64

# Step 3: The problem — ARM image on x86 host
# Fix: Pull or build for the correct platform
docker pull --platform linux/amd64 myapp:v1
# Or build for the correct platform
docker buildx build --platform linux/amd64 -t myapp:v1 .

Prevention

• ALWAYS add .gitattributes with *.sh text eol=lf to repositories
• ALWAYS add shebang (#!/bin/sh) as the first line of entrypoint scripts
• ALWAYS verify target architecture matches build architecture
• Use multi-platform builds: docker buildx build --platform linux/amd64,linux/arm64

---

Example 6: Read-Only Filesystem Failures

Scenario

An application fails to write temporary files in a security-hardened container.

Debugging Session

# Step 1: Check logs
docker logs myapp
# Error: EROFS: read-only file system, open '/tmp/cache.json'

# Step 2: Confirm read-only mode
docker inspect --format='{{.HostConfig.ReadonlyRootfs}}' myapp
# true

# Step 3: Check what paths the app needs to write to
docker run --rm myapp:v1 sh -c "find / -writable 2>/dev/null"
# (no output — everything is read-only)

# Step 4: Fix — add tmpfs mounts for writable paths
docker run -d --read-only \
  --tmpfs /tmp:size=64m \
  --tmpfs /var/cache:size=32m \
  --mount type=volume,src=appdata,dst=/app/data \
  myapp:v1

# Step 5: Verify the fix
docker exec myapp touch /tmp/test-file && echo "OK"
# OK

Prevention

• When using --read-only, ALWAYS map writable paths with tmpfs or volumes
• Common writable paths: /tmp, /var/tmp, /run, /var/run, /var/cache
• Application data ALWAYS goes on a volume, NEVER in the container layer

---

Example 7: Container Restart Loop

Scenario

A container keeps restarting and never becomes stable.

Debugging Session

# Step 1: Check restart count
docker inspect --format='RestartCount={{.RestartCount}} Policy={{.HostConfig.RestartPolicy.Name}}' myapp
# RestartCount=47 Policy=always

# Step 2: Stop the restart loop
docker update --restart no myapp
docker stop myapp

# Step 3: Check what's going wrong
docker logs --tail 50 myapp
# Error: Connection refused to database at db:5432
# Retry 1/5... failed
# Retry 5/5... failed
# Exiting with error

# Step 4: The problem — database dependency not available
# Check if the database is running
docker ps --filter name=db
# (no results — db container is not running)

# Step 5: Fix — start the dependency first
docker start db
# Wait for it to be ready
docker exec db pg_isready
# /var/run/postgresql:5432 - accepting connections

# Step 6: Restart the application
docker update --restart unless-stopped myapp
docker start myapp

# Step 7: Verify
docker ps --filter name=myapp
# STATUS: Up 30 seconds

Prevention

• Use Docker Compose with depends_on and health checks for service ordering
• Implement proper health checks in application containers
• Use exponential backoff in application retry logic
• NEVER rely on container restart policy alone for dependency management

---

Example 8: Debugging Network Connectivity

Scenario

A container cannot reach another container by hostname.

Debugging Session

# Step 1: Check what network the containers are on
docker inspect --format='{{range $k, $v := .NetworkSettings.Networks}}{{$k}} {{end}}' webapp
# bridge

docker inspect --format='{{range $k, $v := .NetworkSettings.Networks}}{{$k}} {{end}}' api
# bridge

# Step 2: Test DNS resolution
docker exec webapp nslookup api
# ** server can't find api: NXDOMAIN

# Step 3: The problem — default bridge network has no DNS resolution
# Fix: Create a user-defined network
docker network create mynet
docker network connect mynet webapp
docker network connect mynet api

# Step 4: Test again
docker exec webapp nslookup api
# Name:    api
# Address: 172.18.0.3

docker exec webapp curl http://api:3000/health
# {"status":"ok"}

Prevention

• ALWAYS use user-defined bridge networks, NEVER the default bridge
• Use Docker Compose (automatically creates a user-defined network)
• Use --network-alias for additional DNS names