Docker Troubleshooting

Docker Troubleshooting — 问题排查与调试

Systematic approach to diagnosing and resolving common Docker issues.

When to Use

ALWAYS use this skill when the user mentions:

• "docker 排查", "docker debug", "容器问题"
• "容器启动失败", "exit code", "crash"
• "OOM", "磁盘满", "disk full"
• "网络不通", "network issues"
• "docker daemon", "daemon not running"

Diagnostic Framework

1. Check container state    → docker ps -a, docker inspect
2. Check logs               → docker logs --tail 100
3. Check resources          → docker stats, docker system df
4. Check exit code          → docker inspect --format='{{.State.ExitCode}}'
5. Interactive debug        → docker run --rm -it image sh

Container Startup Failure

Exit Code Reference

# Quick diagnosis
docker ps -a | grep Exited
docker inspect --format='{{.State.ExitCode}} {{.State.Error}}' container
docker logs container

Disk Space Exhaustion

# Check disk usage
docker system df

# What's using space?
docker system df -v   # Verbose breakdown

# Clean up
docker system prune -a     # Remove all unused: images, containers, volumes, networks
docker image prune -a      # Unused images only
docker volume prune        # Unused volumes only

# Log rotation (prevent future issues)
docker run --log-opt max-size=10m --log-opt max-file=3 myapp

OOM (Out of Memory)

# Check if container was OOM-killed
docker inspect --format='{{.State.OOMKilled}}' container

# Check memory usage
docker stats --no-stream

# Solution: increase memory limit
docker update --memory=1g --memory-swap=1g container

Network Connectivity

# Check network configuration
docker network inspect bridge
docker exec container ip addr show
docker exec container ping other-container

# DNS check
docker exec container nslookup other-container
docker exec container cat /etc/resolv.conf

# Host networking check
iptables -t nat -L DOCKER -n   # Port forwarding rules

Permission Issues

# Bind mount permission denied
# Solution 1: Match UID
docker run -u $(id -u):$(id -g) -v $PWD:/app myapp

# Solution 2: Fix permissions in entrypoint
# In entrypoint script: chown -R appuser:appgroup /data

# Docker socket permission
sudo usermod -aG docker $USER   # Add user to docker group
newgrp docker                    # Refresh group (or re-login)

Workflow — 排查流程

Step 1: 查看状态: docker ps -a → 确认退出码和 OOMKilled Step 2: 查看日志: docker logs --tail 100 <container> 获取错误信息 Step 3: 深入检查: docker inspect → 检查网络/挂载/资源限制 Step 4: 交互调试: docker run -it --rm --entrypoint sh <image> 进入排查 Step 5: 资源排查: docker stats + docker system df 检查 CPU/内存/磁盘

Gotchas — Common Pitfalls

• `docker system prune` is destructive: Removes ALL unused objects. Don't run blindly in production. → Recovery: Use docker system df first to preview; add --filter "until=24h" for safe cleanup; never run prune -a without confirmation.
• Logs fill disk silently: JSON-file driver has NO rotation by default. A noisy app can fill the disk in hours. → Recovery: docker run --log-opt max-size=10m --log-opt max-file=3 app; existing: truncate -s 0 $(docker inspect -f '{{.LogPath}}' myapp).
• OOMKilled=TRUE but no error in logs: SIGKILL can't be logged. → Recovery: Check docker inspect -f '{{.State.OOMKilled}}' myapp; increase memory: docker update --memory 512m myapp.
• Exited (0) but service not working: Application started but crashed after health check passed. → Recovery: docker logs --tail 100 myapp; check if app listens on 0.0.0.0 (not 127.0.0.1); verify with docker exec myapp netstat -tlnp.

Boundary — 能力边界（适用与不适用场景）

When NOT to Use This Skill

Security & Stability

• docker exec into production containers can expose sensitive data. Prefer log-based debugging.
• docker system prune -a removes ALL unused images and volumes — use with caution in production.
• Inspecting containers with --privileged debug tools can bypass security controls.
• Never expose Docker socket (/var/run/docker.sock) to containers for debugging — it grants root access.
• Rotate debug logs — they may contain sensitive information. Use log drivers with TTL.

📚 官方文档参考

🧭 Docker Skills Journey

📍 You are here: `docker-troubleshooting` — 问题排查

← Previous: docker-production

Debug a crashing container

# Scenario: Container exits immediately after starting

# 1. Check status
docker ps -a
# STATUS: Exited (1) 2 seconds ago

# 2. Get exit code
docker inspect --format='{{.State.ExitCode}}' myapp
# → 1 (application error)

# 3. View logs (even for stopped containers!)
docker logs myapp
# → Error: Cannot connect to database

# 4. Override entrypoint to debug
docker run --rm -it --entrypoint sh myapp
# Now you're inside a shell — debug interactively

# 5. Check resource issues
docker inspect --format='{{.State.OOMKilled}}' myapp
# → false (not OOM this time)

# 6. Try with more memory
docker run --memory=1g myapp

容器崩溃调试全过程

症状：容器启动后立即退出

docker run -d --name myapp myapp:latest
docker ps -a | grep myapp
# myapp   Exited (1) 2 seconds ago

诊断流程

# Step 1: 查看退出码
docker inspect --format='{{.State.ExitCode}}' myapp
# 退出码速查：
# 0    正常退出
# 1    应用错误
# 126  权限不足（CMD 不可执行）
# 127  命令找不到
# 137  SIGKILL（OOM）
# 139  SIGSEGV（段错误）
# 143  SIGTERM（优雅终止）

# Step 2: 查看日志（即使 Exited 也能看到）
docker logs myapp
docker logs --tail 50 myapp

# Step 3: 检查是否 OOM
docker inspect --format='{{.State.OOMKilled}}' myapp
# true → 内存不足，增加 --memory

# Step 4: 覆盖 CMD 进入调试
docker run -it --rm --entrypoint sh myapp:latest
# 进入后手动运行应用，观察输出

# Step 5: 检查文件权限
docker run -it --rm --entrypoint sh myapp:latest
ls -la /app/entrypoint.sh
# 权限不足 → Dockerfile 加 RUN chmod +x

# Step 6: 检查端口绑定
docker inspect --format='{{.NetworkSettings.IPAddress}}' myapp
# 确认应用监听 0.0.0.0 而不是 127.0.0.1

常见问题速查

Docker Exit Code Reference

磁盘与资源问题排查

磁盘满

# 1. 查看使用量
docker system df
# TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
# Images          25        5         12.5GB    8.2GB (65%)
# Containers      8         3         1.2GB     800MB (66%)
# Local Volumes   12        4         3.5GB     2.1GB (60%)

# 2. 详细分析
docker system df -v

# 3. 清理
docker container prune     # 删除已停止容器
docker image prune -a      # 删除未使用镜像
docker volume prune        # 删除未使用卷
docker builder prune       # 清理构建缓存

# 4. 安全清理（保留 24h 内的）
docker system prune --filter "until=24h"

# 5. 检查大文件
docker run --rm -v /var/lib/docker:/docker:ro alpine du -sh /docker/*

日志膨胀

# 检查日志大小
docker inspect --format='{{.LogPath}}' myapp | xargs ls -lh

# 限制日志大小
docker run -d --log-opt max-size=10m --log-opt max-file=3 myapp

# Compose 中限制
services:
  app:
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

# 清理日志
truncate -s 0 $(docker inspect --format='{{.LogPath}}' myapp)

容器 CPU 100%

# 查看资源使用
docker stats --no-stream
docker top myapp

# 限制 CPU
docker update --cpus 0.5 myapp

# 查看进程
docker exec myapp ps aux --sort=-%cpu | head -5

网络耗时排查

# DNS 解析
docker exec myapp nslookup api-service

# 延迟测试
docker exec myapp time curl -s http://api-service:8080/health

# 抓包
docker run --rm --net container:myapp nicolaka/netshoot tcpdump -i eth0 -w /tmp/capture.pcap

inode 耗尽

# 检查 inode
df -i /var/lib/docker

# docker 产生的 inode 大户：overlay2 层、小文件多
# 清理：docker system prune -a