Docs

Docker Documentation Reference Skill

Overview

This skill provides access to comprehensive Docker and Docker Compose documentation. Use this skill to look up exact configurations, command syntax, and best practices before generating any Docker-related files.

Documentation Location

All documentation is stored in: /home/mwguerra/projects/mwguerra/claude-code-plugins/docker-specialist/skills/docs/references/

Directory Structure

references/
├── 01-introduction.md       # Docker concepts, key terms
├── 02-dockerfile.md         # Dockerfile instructions, multi-stage builds
├── 03-compose-fundamentals.md  # Compose file structure, options
├── 04-networking.md         # Network types, DNS, external networks
├── 05-databases.md          # PostgreSQL, MySQL, MongoDB, Redis
├── 06-services.md           # Dependencies, scaling, patterns
├── 07-ports-ssl.md          # Port mapping, Traefik, Nginx SSL
├── 08-volumes.md            # Volume types, persistence, backups
├── 09-environment.md        # Env vars, secrets, .env files
├── 10-architecture.md       # Project structures, folder organization
├── 11-global-local.md       # Global vs project containers
├── 12-examples.md           # Complete working examples
├── 13-commands.md           # Essential Docker commands
├── 14-security.md           # Security best practices
├── 15-port-conflicts.md     # Port conflict resolution
├── 16-restart-strategies.md # Restart policies, data persistence
└── 17-troubleshooting.md    # Common issues and solutions

Usage

When to Use This Skill

1. Before generating any Dockerfile or compose configuration 2. When troubleshooting Docker errors 3. To verify correct command syntax 4. To find proper configuration patterns 5. To understand Docker networking 6. For security best practices

Search Workflow

1. Identify Topic: Determine what documentation is needed 2. Navigate to File: Go to relevant documentation file 3. Read Documentation: Extract exact patterns 4. Apply Knowledge: Use in configuration generation

Common Lookups

Documentation Reading Pattern

When reading documentation:

1. Find the right file: Match topic to documentation file 2. Read the overview: Understand the concept 3. Extract code examples: Copy exact patterns 4. Note configuration options: Review available settings 5. Check best practices: Apply security and performance tips

Example Usage

Looking up PostgreSQL Configuration

1. Navigate to 05-databases.md 2. Find PostgreSQL section 3. Extract:

• Image version
• Environment variables
• Health check configuration
• Volume setup
• Network configuration

Looking up Network Configuration

1. Navigate to 04-networking.md 2. Find relevant section (bridge, external, internal) 3. Extract:

• Network definition syntax
• Service network configuration
• DNS resolution patterns

Output

After reading documentation, provide:

1. Exact configuration pattern from docs 2. Required settings 3. Optional configurations 4. Best practices noted 5. Security considerations

Introduction to Docker & Docker Compose

Docker is a containerization platform that packages applications and their dependencies into isolated units called containers. Docker Compose extends this by orchestrating multi-container applications through declarative YAML configuration.

Key Concepts

When to Use Docker

• Development Environments: Consistent environments across team members
• Microservices: Isolate services with their dependencies
• CI/CD: Reproducible builds and deployments
• Legacy Apps: Containerize older applications
• Testing: Isolated testing environments

Docker vs Docker Compose

Architecture Overview

┌─────────────────────────────────────────────────────────┐
│                     Docker Host                          │
│  ┌─────────────────────────────────────────────────┐    │
│  │              Docker Engine                       │    │
│  │  ┌───────────┐  ┌───────────┐  ┌───────────┐   │    │
│  │  │ Container │  │ Container │  │ Container │   │    │
│  │  │   App 1   │  │   App 2   │  │    DB     │   │    │
│  │  └───────────┘  └───────────┘  └───────────┘   │    │
│  │        │              │              │          │    │
│  │  ┌─────┴──────────────┴──────────────┴─────┐   │    │
│  │  │              Docker Network             │   │    │
│  │  └───────────────────────────────────────┘    │    │
│  │  ┌───────────┐  ┌───────────┐                 │    │
│  │  │  Volume   │  │  Volume   │                 │    │
│  │  │   Data    │  │   Logs    │                 │    │
│  │  └───────────┘  └───────────┘                 │    │
│  └─────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────┘

Version Information

• Docker Compose Version: v2.x (Compose Specification)
• The version field is deprecated since Docker Compose v1.27.0 (2020)
• Modern compose files start directly with services

Dockerfile Structure & Instructions

A Dockerfile is a sequential script that defines how to build a Docker image. Each instruction creates a layer in the final image.

Basic Dockerfile Structure

# syntax=docker/dockerfile:1

# 1. Base Image
FROM node:20-alpine

# 2. Metadata
LABEL maintainer="your-email@example.com"
LABEL version="1.0"

# 3. Environment Variables
ENV NODE_ENV=production
ENV APP_PORT=3000

# 4. Working Directory
WORKDIR /app

# 5. Copy Dependencies First (for better caching)
COPY package*.json ./

# 6. Install Dependencies
RUN npm ci --only=production

# 7. Copy Application Code
COPY . .

# 8. Create Non-Root User
RUN addgroup -g 10001 appgroup && \
    adduser -u 10001 -G appgroup -D appuser
USER appuser

# 9. Expose Port
EXPOSE 3000

# 10. Health Check
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
    CMD curl -f http://localhost:3000/health || exit 1

# 11. Default Command
CMD ["node", "dist/index.js"]

Essential Dockerfile Instructions

FROM - Base Image Selection

# Use specific version tags (avoid :latest in production)
FROM python:3.12-slim

# Multi-stage builds for smaller images
FROM node:20 AS builder
WORKDIR /app
COPY . .
RUN npm ci && npm run build

FROM node:20-alpine AS runtime
COPY --from=builder /app/dist ./dist
CMD ["node", "dist/index.js"]

Best Practices:

• Use official images from Docker Hub
• Prefer slim/alpine variants for smaller images
• Pin to specific versions (e.g., python:3.12-slim not python:latest)

RUN - Execute Commands

# BAD: Multiple RUN instructions create multiple layers
RUN apt-get update
RUN apt-get install -y curl
RUN apt-get clean

# GOOD: Combine commands and clean up in one layer
RUN apt-get update && apt-get install -y --no-install-recommends \
    curl \
    git \
    && rm -rf /var/lib/apt/lists/*

COPY vs ADD

# COPY: Simple file copying (preferred)
COPY requirements.txt /app/
COPY src/ /app/src/

# ADD: Use only when you need:
# - Automatic tar extraction
# - Remote URL fetching (not recommended)
ADD archive.tar.gz /app/

Rule: Use COPY unless you specifically need ADD's features.

WORKDIR - Set Working Directory

# Always use WORKDIR instead of RUN cd
WORKDIR /app

# Can chain WORKDIR
WORKDIR /app
WORKDIR src  # Now at /app/src

USER - Run as Non-Root

# Create and switch to non-root user
RUN groupadd -r appgroup && useradd -r -g appgroup appuser
USER appuser

# For Alpine
RUN addgroup -g 10001 appgroup && \
    adduser -u 10001 -G appgroup -D appuser
USER appuser

EXPOSE - Document Ports

# Document which ports the container listens on
EXPOSE 3000
EXPOSE 443/tcp
EXPOSE 53/udp

Note: EXPOSE is documentation only; use -p flag or compose ports to publish.

CMD vs ENTRYPOINT

# CMD: Default command, easily overridden
CMD ["npm", "start"]

# ENTRYPOINT: Fixed command, args appended
ENTRYPOINT ["python", "app.py"]
CMD ["--port", "8000"]  # Default args

# Running: docker run myimage --port 9000
# Executes: python app.py --port 9000

HEALTHCHECK

HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:3000/health || exit 1

Multi-Stage Builds

Multi-stage builds create smaller, more secure production images:

# Stage 1: Build
FROM node:20 AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Stage 2: Production
FROM node:20-alpine AS production
WORKDIR /app

# Copy only necessary files from builder
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY package.json ./

# Security: Non-root user
RUN addgroup -g 1001 nodejs && \
    adduser -u 1001 -G nodejs -D nodejs
USER nodejs

EXPOSE 3000
CMD ["node", "dist/index.js"]

.dockerignore

Always create a .dockerignore file:

# Dependencies
node_modules/
vendor/
__pycache__/
*.pyc

# Build artifacts
dist/
build/
*.egg-info/

# Development files
.git/
.gitignore
.env
.env.*
*.md
README*
Dockerfile*
docker-compose*
.dockerignore

# IDE
.vscode/
.idea/
*.swp

# Testing
coverage/
.pytest_cache/
tests/

# Logs
*.log
logs/

Common Dockerfile Patterns

Node.js Application

FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
USER node
EXPOSE 3000
CMD ["node", "src/index.js"]

Python Application

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
USER nobody
EXPOSE 8000
CMD ["python", "app.py"]

PHP/Laravel Application

FROM php:8.3-fpm-alpine
WORKDIR /var/www/html
RUN apk add --no-cache postgresql-dev && \
    docker-php-ext-install pdo pdo_pgsql
COPY --from=composer:latest /usr/bin/composer /usr/bin/composer
COPY . .
RUN composer install --no-dev --optimize-autoloader
EXPOSE 9000
CMD ["php-fpm"]

Go Application

FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY go.* ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o main .

FROM alpine:latest
RUN apk --no-cache add ca-certificates
WORKDIR /root/
COPY --from=builder /app/main .
EXPOSE 8080
CMD ["./main"]

Docker Compose Fundamentals

Docker Compose uses YAML files to define multi-container applications.

Modern Compose File Structure

Note: The version field is deprecated since Docker Compose v1.27.0 (2020). Modern compose files start directly with services.

# Modern Docker Compose file (no version field needed)
services:
  app:
    build: .
    ports:
      - "3000:3000"

  db:
    image: postgres:16
    volumes:
      - pgdata:/var/lib/postgresql/data

volumes:
  pgdata:

networks:
  default:
    driver: bridge

Service Configuration Options

services:
  webapp:
    # Build Configuration
    build:
      context: ./app
      dockerfile: Dockerfile
      args:
        - NODE_ENV=production
      target: production  # For multi-stage builds

    # Or use pre-built image
    image: nginx:alpine

    # Container name (optional)
    container_name: my-webapp

    # Restart Policy
    restart: unless-stopped  # no | always | on-failure | unless-stopped

    # Port Mapping
    ports:
      - "80:80"           # host:container
      - "443:443/tcp"
      - "127.0.0.1:8080:8080"  # Bind to localhost only

    # Environment Variables
    environment:
      - NODE_ENV=production
      - DATABASE_URL=postgresql://user:pass@db:5432/app

    # Or use env file
    env_file:
      - .env
      - .env.local

    # Volume Mounts
    volumes:
      - ./src:/app/src          # Bind mount
      - app-data:/app/data      # Named volume
      - /app/node_modules       # Anonymous volume (preserve)

    # Dependencies
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_started

    # Networks
    networks:
      - frontend
      - backend

    # Resource Limits
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M

    # Health Check
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

    # Logging
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

Profiles for Conditional Services

services:
  app:
    image: myapp:latest
    ports:
      - "3000:3000"

  debug:
    image: myapp:debug
    profiles:
      - debug
    ports:
      - "9229:9229"

  test:
    image: myapp:test
    profiles:
      - testing

# Run without profiles (only app starts)
docker compose up

# Run with debug profile
docker compose --profile debug up

# Run with multiple profiles
docker compose --profile debug --profile testing up

File Naming Conventions

Override Files

Docker Compose automatically merges compose.yaml with compose.override.yaml:

# compose.yaml (base)
services:
  app:
    build: .
    environment:
      - NODE_ENV=production

# compose.override.yaml (auto-loaded in development)
services:
  app:
    volumes:
      - ./src:/app/src
    environment:
      - NODE_ENV=development
    ports:
      - "3000:3000"

Using Multiple Compose Files

# Development (auto-loads override)
docker compose up

# Production (explicit files)
docker compose -f compose.yaml -f compose.prod.yaml up -d

# Testing
docker compose -f compose.yaml -f compose.test.yaml up

Compose File Reference Structure

# Top-level elements
services:      # Container definitions (required)
volumes:       # Named volumes
networks:      # Custom networks
configs:       # Configuration files (Swarm)
secrets:       # Secret management

Variable Substitution

services:
  app:
    image: ${IMAGE_NAME:-myapp}:${TAG:-latest}
    ports:
      - "${HOST_PORT:-3000}:3000"
    environment:
      - DB_PASSWORD=${DB_PASSWORD:?Database password required}

Extends (Reusable Services)

# common.yaml
services:
  base:
    image: node:20-alpine
    environment:
      - NODE_ENV=production
    restart: unless-stopped

# compose.yaml
services:
  app:
    extends:
      file: common.yaml
      service: base
    ports:
      - "3000:3000"

Anchors and Aliases (YAML Feature)

# Define anchor
x-common: &common
  restart: unless-stopped
  logging:
    driver: json-file
    options:
      max-size: "10m"

services:
  app:
    <<: *common  # Use alias
    build: .
    ports:
      - "3000:3000"

  worker:
    <<: *common  # Reuse configuration
    build: ./worker

Container Networking

Network Types

Docker provides several network drivers:

Creating Networks in Compose

services:
  frontend:
    image: nginx:alpine
    networks:
      - frontend-net
    ports:
      - "80:80"

  api:
    build: ./api
    networks:
      - frontend-net
      - backend-net

  database:
    image: postgres:16
    networks:
      - backend-net
    # No ports exposed to host - only accessible via backend-net

networks:
  frontend-net:
    driver: bridge
  backend-net:
    driver: bridge
    internal: true  # No external access

Container DNS Resolution

Within a Docker Compose network, containers can reach each other by service name:

services:
  app:
    environment:
      # Use service name as hostname
      - DATABASE_HOST=db
      - REDIS_HOST=redis
      - API_URL=http://api:3000

  db:
    image: postgres:16

  redis:
    image: redis:7

  api:
    build: ./api

External Networks

Share networks between multiple Compose projects:

# Create external network
docker network create shared-network

# compose.yaml (Project A)
services:
  api:
    networks:
      - shared

networks:
  shared:
    external: true
    name: shared-network

# compose.yaml (Project B)
services:
  frontend:
    networks:
      - shared

networks:
  shared:
    external: true
    name: shared-network

Network Configuration Options

networks:
  # Simple network
  app-net:

  # Network with driver options
  backend:
    driver: bridge
    driver_opts:
      com.docker.network.bridge.name: backend-br

  # Internal network (no external access)
  database:
    internal: true

  # Custom IPAM configuration
  custom:
    ipam:
      driver: default
      config:
        - subnet: 172.28.0.0/16
          gateway: 172.28.0.1

  # External network
  proxy:
    external: true
    name: traefik_proxy

Service Network Options

services:
  app:
    networks:
      frontend:
        aliases:
          - webapp
          - web
        ipv4_address: 172.28.0.10
      backend:

Network Isolation Patterns

Three-Tier Architecture

services:
  # Public-facing
  nginx:
    image: nginx:alpine
    networks:
      - frontend
    ports:
      - "80:80"

  # Application layer
  api:
    build: .
    networks:
      - frontend
      - backend
    # No public ports

  # Database layer
  db:
    image: postgres:16
    networks:
      - backend
    # No public ports, isolated

networks:
  frontend:
    driver: bridge
  backend:
    driver: bridge
    internal: true  # Cannot reach internet

Debugging Network Issues

# List networks
docker network ls

# Inspect network
docker network inspect mynetwork

# Check container connectivity
docker compose exec app ping db
docker compose exec app nc -zv db 5432

# View container's network settings
docker inspect container_name | grep -A 50 "Networks"

# Test DNS resolution
docker compose exec app nslookup db

Host Networking Mode

services:
  app:
    network_mode: host
    # No port mapping needed - uses host ports directly

Use cases:

• Performance-critical applications
• Applications that need to bind to many ports
• When container must appear as host

Limitations:

• Only works on Linux
• Port conflicts with host services
• Less isolation

Common Network Patterns

Reverse Proxy Pattern

services:
  traefik:
    image: traefik:v3.0
    networks:
      - proxy
    ports:
      - "80:80"
      - "443:443"

  app1:
    build: ./app1
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.app1.rule=Host(`app1.example.com`)"

  app2:
    build: ./app2
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.app2.rule=Host(`app2.example.com`)"

networks:
  proxy:
    name: proxy

Service Mesh Pattern

services:
  api-gateway:
    networks:
      - public
      - services

  user-service:
    networks:
      - services
      - user-db

  order-service:
    networks:
      - services
      - order-db

  user-db:
    image: postgres:16
    networks:
      - user-db

  order-db:
    image: postgres:16
    networks:
      - order-db

networks:
  public:
  services:
    internal: true
  user-db:
    internal: true
  order-db:
    internal: true

Database Containers Best Practices

PostgreSQL Configuration

services:
  postgres:
    image: postgres:16-alpine
    container_name: postgres
    restart: unless-stopped

    # Shared memory for PostgreSQL
    shm_size: 256mb

    environment:
      POSTGRES_USER: ${DB_USER:-appuser}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_DB: ${DB_NAME:-appdb}
      # Performance tuning
      POSTGRES_INITDB_ARGS: "--encoding=UTF-8 --lc-collate=C --lc-ctype=C"

    volumes:
      # Data persistence
      - postgres_data:/var/lib/postgresql/data
      # Custom configuration
      - ./config/postgresql.conf:/etc/postgresql/postgresql.conf
      # Initialization scripts (run once on first start)
      - ./init-scripts:/docker-entrypoint-initdb.d

    ports:
      - "127.0.0.1:5432:5432"  # Localhost only

    networks:
      - database

    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-appuser} -d ${DB_NAME:-appdb}"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s

volumes:
  postgres_data:

networks:
  database:
    internal: true  # No external access

MySQL/MariaDB Configuration

services:
  mysql:
    image: mysql:8.0
    container_name: mysql
    restart: unless-stopped

    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
      MYSQL_DATABASE: ${DB_NAME}
      MYSQL_USER: ${DB_USER}
      MYSQL_PASSWORD: ${DB_PASSWORD}

    volumes:
      - mysql_data:/var/lib/mysql
      - ./config/my.cnf:/etc/mysql/conf.d/custom.cnf
      - ./init-scripts:/docker-entrypoint-initdb.d

    ports:
      - "127.0.0.1:3306:3306"

    command: >
      --default-authentication-plugin=mysql_native_password
      --character-set-server=utf8mb4
      --collation-server=utf8mb4_unicode_ci

    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-u", "root", "-p${MYSQL_ROOT_PASSWORD}"]
      interval: 10s
      timeout: 5s
      retries: 5

volumes:
  mysql_data:

MongoDB Configuration

services:
  mongodb:
    image: mongo:7
    container_name: mongodb
    restart: unless-stopped

    environment:
      MONGO_INITDB_ROOT_USERNAME: ${MONGO_USER}
      MONGO_INITDB_ROOT_PASSWORD: ${MONGO_PASSWORD}
      MONGO_INITDB_DATABASE: ${MONGO_DB}

    volumes:
      - mongo_data:/data/db
      - mongo_config:/data/configdb
      - ./init-scripts:/docker-entrypoint-initdb.d

    ports:
      - "127.0.0.1:27017:27017"

    healthcheck:
      test: ["CMD", "mongosh", "--eval", "db.adminCommand('ping')"]
      interval: 10s
      timeout: 5s
      retries: 5

volumes:
  mongo_data:
  mongo_config:

Redis Configuration

services:
  redis:
    image: redis:7-alpine
    container_name: redis
    restart: unless-stopped

    command: >
      redis-server
      --appendonly yes
      --maxmemory 256mb
      --maxmemory-policy allkeys-lru
      --requirepass ${REDIS_PASSWORD}

    volumes:
      - redis_data:/data

    ports:
      - "127.0.0.1:6379:6379"

    healthcheck:
      test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD}", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5

volumes:
  redis_data:

Database Best Practices Summary

1. Always use named volumes for data persistence 2. Never expose database ports publicly unless absolutely necessary 3. Use health checks to ensure database is ready before dependent services start 4. Store credentials in environment variables or Docker secrets 5. Use initialization scripts for schema setup 6. Regular backups using docker exec with database dump commands 7. Use internal networks to isolate database traffic

Backup Commands

# Backup PostgreSQL
docker exec -t postgres pg_dumpall -c -U appuser > backup.sql

# Backup MySQL
docker exec mysql mysqldump -u root -p${MYSQL_ROOT_PASSWORD} --all-databases > backup.sql

# Backup MongoDB
docker exec mongodb mongodump --archive --gzip > backup.gz

Restore Commands

# Restore PostgreSQL
docker exec -i postgres psql -U appuser -d appdb < backup.sql

# Restore MySQL
docker exec -i mysql mysql -u root -p${MYSQL_ROOT_PASSWORD} < backup.sql

# Restore MongoDB
docker exec -i mongodb mongorestore --archive --gzip < backup.gz

Connection Strings

PostgreSQL

DATABASE_URL=postgresql://user:password@db:5432/dbname

MySQL

DATABASE_URL=mysql://user:password@db:3306/dbname

MongoDB

MONGO_URI=mongodb://user:password@mongodb:27017/dbname?authSource=admin

Redis

REDIS_URL=redis://:password@redis:6379

Database Initialization Scripts

Scripts in /docker-entrypoint-initdb.d/ run on first container start:

-- init-scripts/01-schema.sql
CREATE TABLE IF NOT EXISTS users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) UNIQUE NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Create indexes
CREATE INDEX idx_users_email ON users(email);

# init-scripts/02-seed.sh
#!/bin/bash
psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" <<-EOSQL
    INSERT INTO users (email) VALUES ('admin@example.com');
EOSQL

Multi-Database Setup

services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: admin
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_MULTIPLE_DATABASES: app,analytics,logs
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./create-multiple-databases.sh:/docker-entrypoint-initdb.d/create-multiple-databases.sh

volumes:
  postgres_data:

# create-multiple-databases.sh
#!/bin/bash

set -e
set -u

function create_database() {
    local database=$1
    echo "Creating database '$database'"
    psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL
        CREATE DATABASE $database;
        GRANT ALL PRIVILEGES ON DATABASE $database TO $POSTGRES_USER;
EOSQL
}

if [ -n "$POSTGRES_MULTIPLE_DATABASES" ]; then
    for db in $(echo $POSTGRES_MULTIPLE_DATABASES | tr ',' ' '); do
        create_database $db
    done
fi

Services & Multi-Container Applications

Service Dependencies and Startup Order

services:
  app:
    build: .
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy
      migrations:
        condition: service_completed_successfully

  migrations:
    build: .
    command: npm run migrate
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:16
    healthcheck:
      test: ["CMD-SHELL", "pg_isready"]
      interval: 5s
      timeout: 5s
      retries: 5

  redis:
    image: redis:7
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 5s
      retries: 5

Dependency Conditions

Scaling Services

services:
  worker:
    build: ./worker
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: '0.5'
          memory: 256M

# Scale a service
docker compose up -d --scale worker=5

# Check running instances
docker compose ps

Inter-Service Communication Pattern

services:
  # API Gateway / Load Balancer
  nginx:
    image: nginx:alpine
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "80:80"
    depends_on:
      - api

  # Application API
  api:
    build: ./api
    environment:
      - DATABASE_URL=postgresql://user:pass@db:5432/app
      - REDIS_URL=redis://redis:6379
      - QUEUE_URL=amqp://rabbitmq:5672
    depends_on:
      - db
      - redis
      - rabbitmq
    expose:
      - "3000"  # Internal only

  # Background Worker
  worker:
    build: ./worker
    environment:
      - DATABASE_URL=postgresql://user:pass@db:5432/app
      - QUEUE_URL=amqp://rabbitmq:5672
    depends_on:
      - db
      - rabbitmq

  # Services
  db:
    image: postgres:16
    volumes:
      - pgdata:/var/lib/postgresql/data

  redis:
    image: redis:7
    volumes:
      - redis_data:/data

  rabbitmq:
    image: rabbitmq:3-management
    volumes:
      - rabbitmq_data:/var/lib/rabbitmq

volumes:
  pgdata:
  redis_data:
  rabbitmq_data:

Service Restart Policies

services:
  app:
    restart: unless-stopped  # Recommended for production

One-Off Services

services:
  app:
    build: .

  migrate:
    build: .
    command: npm run migrate
    depends_on:
      db:
        condition: service_healthy
    profiles:
      - tools

  seed:
    build: .
    command: npm run seed
    depends_on:
      db:
        condition: service_healthy
    profiles:
      - tools

# Run migration
docker compose run --rm migrate

# Or with profile
docker compose --profile tools run migrate

Init Containers Pattern

services:
  init-permissions:
    image: busybox
    command: chown -R 1000:1000 /data
    volumes:
      - app_data:/data
    restart: "no"

  app:
    build: .
    depends_on:
      init-permissions:
        condition: service_completed_successfully
    volumes:
      - app_data:/app/data

volumes:
  app_data:

Sidecar Pattern

services:
  app:
    build: .
    volumes:
      - logs:/app/logs

  log-shipper:
    image: fluent/fluent-bit
    volumes:
      - logs:/logs:ro
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    depends_on:
      - app

volumes:
  logs:

Service Discovery

Services can discover each other by name within the same network:

services:
  frontend:
    environment:
      - API_URL=http://api:3000
      - WS_URL=ws://websocket:8080

  api:
    environment:
      - DB_HOST=postgres
      - CACHE_HOST=redis

  websocket:
    environment:
      - REDIS_HOST=redis

  postgres:
    image: postgres:16

  redis:
    image: redis:7

Service Configuration Patterns

Environment-Based Configuration

services:
  app:
    build: .
    environment:
      - NODE_ENV=${NODE_ENV:-development}
      - LOG_LEVEL=${LOG_LEVEL:-info}
    env_file:
      - .env
      - .env.${NODE_ENV:-development}

Config Files via Volumes

services:
  nginx:
    image: nginx:alpine
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./nginx/conf.d:/etc/nginx/conf.d:ro

Docker Configs (Swarm Mode)

services:
  app:
    configs:
      - source: app_config
        target: /app/config.json

configs:
  app_config:
    file: ./config.json

Health Check Patterns

HTTP Health Check

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 40s

TCP Health Check

healthcheck:
  test: ["CMD-SHELL", "nc -z localhost 3000"]
  interval: 10s
  timeout: 5s
  retries: 5

Custom Script Health Check

healthcheck:
  test: ["CMD", "/healthcheck.sh"]
  interval: 30s
  timeout: 10s
  retries: 3

Graceful Shutdown

services:
  app:
    stop_grace_period: 30s
    stop_signal: SIGTERM

Handle in application:

process.on('SIGTERM', async () => {
  console.log('SIGTERM received, shutting down gracefully');
  await server.close();
  await db.disconnect();
  process.exit(0);
});

Host Interaction: Ports, URLs & SSL

Port Mapping

services:
  web:
    ports:
      # Standard mapping
      - "8080:80"                    # HOST:CONTAINER

      # Localhost only (security)
      - "127.0.0.1:3000:3000"

      # Random host port
      - "80"                         # Container port 80 → random host port

      # UDP protocol
      - "53:53/udp"

      # Port range
      - "6000-6010:6000-6010"

Port vs Expose

services:
  frontend:
    ports:
      - "80:80"  # Accessible from host

  api:
    expose:
      - "3000"  # Only accessible within Docker network

SSL/TLS with Traefik (Recommended)

Traefik is a modern reverse proxy with automatic SSL certificate management:

services:
  traefik:
    image: traefik:v3.0
    container_name: traefik
    restart: unless-stopped
    command:
      # API and Dashboard
      - "--api.dashboard=true"
      - "--api.insecure=false"

      # Docker provider
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=proxy"

      # Entrypoints
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"

      # HTTP to HTTPS redirect
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entryPoint.scheme=https"

      # Let's Encrypt
      - "--certificatesresolvers.letsencrypt.acme.email=your-email@example.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.letsencrypt.acme.tlschallenge=true"

    ports:
      - "80:80"
      - "443:443"

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - traefik_letsencrypt:/letsencrypt

    networks:
      - proxy

    labels:
      # Dashboard
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.rule=Host(`traefik.yourdomain.com`)"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.tls.certresolver=letsencrypt"
      - "traefik.http.routers.dashboard.middlewares=auth"
      - "traefik.http.middlewares.auth.basicauth.users=admin:$$apr1$$xyz..."

  # Your application
  webapp:
    build: ./app
    restart: unless-stopped
    networks:
      - proxy
      - internal
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.webapp.rule=Host(`app.yourdomain.com`)"
      - "traefik.http.routers.webapp.tls.certresolver=letsencrypt"
      - "traefik.http.services.webapp.loadbalancer.server.port=3000"

networks:
  proxy:
    external: true
  internal:
    internal: true

volumes:
  traefik_letsencrypt:

SSL with Nginx + Certbot

services:
  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./nginx/ssl:/etc/nginx/ssl:ro
      - ./certbot/conf:/etc/letsencrypt:ro
      - ./certbot/www:/var/www/certbot:ro
    depends_on:
      - app

  certbot:
    image: certbot/certbot
    volumes:
      - ./certbot/conf:/etc/letsencrypt
      - ./certbot/www:/var/www/certbot
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"

  app:
    build: .
    expose:
      - "3000"

nginx.conf:

server {
    listen 80;
    server_name yourdomain.com;

    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl http2;
    server_name yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;

    location / {
        proxy_pass http://app:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Self-Signed Certificates (Development)

# Generate self-signed certificate
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout ./ssl/privkey.pem \
  -out ./ssl/fullchain.pem \
  -subj "/CN=localhost"

services:
  nginx:
    image: nginx:alpine
    ports:
      - "443:443"
    volumes:
      - ./ssl:/etc/nginx/ssl:ro
      - ./nginx.conf:/etc/nginx/nginx.conf:ro

Traefik Labels Reference

Basic Routing

labels:
  - "traefik.enable=true"
  - "traefik.http.routers.myapp.rule=Host(`myapp.com`)"
  - "traefik.http.services.myapp.loadbalancer.server.port=3000"

Path-Based Routing

labels:
  - "traefik.http.routers.api.rule=Host(`example.com`) && PathPrefix(`/api`)"
  - "traefik.http.routers.api.middlewares=strip-api"
  - "traefik.http.middlewares.strip-api.stripprefix.prefixes=/api"

Multiple Domains

labels:
  - "traefik.http.routers.myapp.rule=Host(`app.com`) || Host(`www.app.com`)"

Wildcard SSL

labels:
  - "traefik.http.routers.myapp.tls.domains[0].main=example.com"
  - "traefik.http.routers.myapp.tls.domains[0].sans=*.example.com"

Local Development with SSL

Using mkcert

# Install mkcert
brew install mkcert  # macOS
# or
sudo apt install mkcert  # Linux

# Install local CA
mkcert -install

# Generate certificates
mkcert -cert-file ./ssl/cert.pem -key-file ./ssl/key.pem localhost 127.0.0.1 ::1

Using Traefik for Local Dev

services:
  traefik:
    image: traefik:v3.0
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./ssl:/etc/traefik/ssl:ro
    labels:
      - "traefik.http.routers.traefik.tls=true"

  app:
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.app.rule=Host(`app.localhost`)"
      - "traefik.http.routers.app.tls=true"

WebSocket Support

labels:
  - "traefik.http.routers.ws.rule=Host(`ws.example.com`)"
  - "traefik.http.services.ws.loadbalancer.server.port=8080"

# Nginx WebSocket proxy
location /ws {
    proxy_pass http://app:8080;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
}

HTTP/2 Configuration

server {
    listen 443 ssl http2;
    # ... SSL config
}

Rate Limiting with Traefik

labels:
  - "traefik.http.middlewares.ratelimit.ratelimit.average=100"
  - "traefik.http.middlewares.ratelimit.ratelimit.burst=50"
  - "traefik.http.routers.api.middlewares=ratelimit"

CORS Headers

labels:
  - "traefik.http.middlewares.cors.headers.accesscontrolallowmethods=GET,OPTIONS,PUT"
  - "traefik.http.middlewares.cors.headers.accesscontrolallowheaders=*"
  - "traefik.http.middlewares.cors.headers.accesscontrolalloworiginlist=https://app.com"
  - "traefik.http.middlewares.cors.headers.accesscontrolmaxage=100"

Volumes & Data Persistence

Volume Types

services:
  app:
    volumes:
      # Named Volume (recommended for data)
      - app_data:/app/data

      # Bind Mount (for development)
      - ./src:/app/src

      # Anonymous Volume (temporary, for node_modules etc.)
      - /app/node_modules

      # Read-only mount
      - ./config:/app/config:ro

      # tmpfs (in-memory, for sensitive data)
      # Defined separately in tmpfs section

    tmpfs:
      - /app/tmp
      - /app/cache:size=100M

volumes:
  app_data:
    driver: local

Volume Comparison

Volume Configuration Options

volumes:
  # Simple named volume
  data:

  # Volume with driver options
  postgres_data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /path/on/host

  # External volume (created outside compose)
  external_data:
    external: true
    name: my-external-volume

Development vs Production Volumes

# compose.yaml (base)
services:
  app:
    build: .
    volumes:
      - app_data:/app/data

volumes:
  app_data:

# compose.override.yaml (development - auto-loaded)
services:
  app:
    volumes:
      - ./src:/app/src  # Live reload
      - /app/node_modules  # Preserve node_modules

# compose.prod.yaml (production)
services:
  app:
    # No source code mounts
    volumes:
      - app_data:/app/data

Volume Permissions

Fix Ownership Issues

services:
  app:
    user: "1000:1000"  # Match host user
    volumes:
      - ./data:/app/data

Init Container for Permissions

services:
  init:
    image: busybox
    command: chown -R 1000:1000 /data
    volumes:
      - app_data:/data
    restart: "no"

  app:
    depends_on:
      init:
        condition: service_completed_successfully
    volumes:
      - app_data:/app/data

volumes:
  app_data:

Bind Mount Options

volumes:
  # Read-only
  - ./config:/app/config:ro

  # Cached (macOS performance)
  - ./src:/app/src:cached

  # Delegated (macOS performance)
  - ./logs:/app/logs:delegated

  # SELinux label
  - ./data:/app/data:z       # Private
  - ./shared:/shared:Z       # Shared

Volume Backup and Restore

Backup

# Backup a volume to tar file
docker run --rm \
  -v myproject_data:/data \
  -v $(pwd):/backup \
  alpine tar czf /backup/data-backup.tar.gz -C /data .

# Using docker compose
docker compose exec -T db pg_dump -U postgres > backup.sql

Restore

# Restore from tar file
docker run --rm \
  -v myproject_data:/data \
  -v $(pwd):/backup \
  alpine tar xzf /backup/data-backup.tar.gz -C /data

# Restore database
docker compose exec -T db psql -U postgres < backup.sql

Volume Management Commands

# List volumes
docker volume ls

# Inspect volume
docker volume inspect myproject_postgres_data

# Create volume
docker volume create my-volume

# Remove volume
docker volume rm my-volume

# Remove unused volumes
docker volume prune

# Remove ALL volumes (dangerous!)
docker volume prune -a

Data Persistence Matrix

Common Volume Patterns

Node.js with node_modules

services:
  app:
    build: .
    volumes:
      - ./src:/app/src
      - ./package.json:/app/package.json
      - /app/node_modules  # Preserve container's node_modules

PHP with Vendor

services:
  app:
    build: .
    volumes:
      - ./:/var/www/html
      - /var/www/html/vendor  # Preserve vendor

Multiple Apps Sharing Data

services:
  app:
    volumes:
      - shared_data:/data

  worker:
    volumes:
      - shared_data:/data

volumes:
  shared_data:

tmpfs for Sensitive Data

services:
  app:
    tmpfs:
      - /app/tmp:size=100M,mode=1777
      - /app/secrets:size=10M,uid=1000,gid=1000

Volume Drivers

Local Driver Options

volumes:
  nfs_data:
    driver: local
    driver_opts:
      type: nfs
      o: addr=192.168.1.100,rw
      device: ":/path/to/share"

  cifs_data:
    driver: local
    driver_opts:
      type: cifs
      device: "//server/share"
      o: "addr=server,username=user,password=pass"

Cloud Storage Drivers

volumes:
  s3_data:
    driver: rexray/s3fs
    driver_opts:
      size: 10

  gcs_data:
    driver: gcsfuse

Handling External Files

services:
  app:
    volumes:
      # SOURCE CODE - Always on host
      - ./src:/app/src

      # UPLOADS - Bind mount for persistence + easy access
      - ./data/uploads:/app/uploads

      # LOGS - Bind mount for easy access
      - ./logs:/app/logs

      # CONFIG - Read-only
      - ./config/app.json:/app/config.json:ro

      # NODE_MODULES - Anonymous (recreated on rebuild)
      - /app/node_modules

      # CACHE - Named volume (persists across restarts)
      - app_cache:/app/.cache

volumes:
  app_cache:

Environment Variables & Secrets

Environment Variable Sources

services:
  app:
    # Direct values
    environment:
      - NODE_ENV=production
      - API_KEY=${API_KEY}  # From .env or shell

    # From file
    env_file:
      - .env
      - .env.local

.env File

# .env
NODE_ENV=development
DB_HOST=db
DB_PORT=5432
DB_USER=appuser
DB_PASSWORD=secretpassword
DB_NAME=appdb

# Computed values
DATABASE_URL=postgresql://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:${DB_PORT}/${DB_NAME}

Environment Variable Precedence

1. Compose file environment values 2. Shell environment variables 3. Environment file (.env) 4. Dockerfile ENV values 5. Variable not defined

Variable Substitution Syntax

services:
  app:
    image: myapp:${TAG:-latest}
    environment:
      - DB_HOST=${DB_HOST:-localhost}
      - DB_PASSWORD=${DB_PASSWORD:?DB password is required}
      - DEBUG=${DEBUG:+true}

Docker Secrets (Sensitive Data)

services:
  app:
    secrets:
      - db_password
      - api_key
    environment:
      - DB_PASSWORD_FILE=/run/secrets/db_password
      - API_KEY_FILE=/run/secrets/api_key

secrets:
  db_password:
    file: ./secrets/db_password.txt
  api_key:
    file: ./secrets/api_key.txt

Reading Secrets in Application

// Node.js
const fs = require('fs');
const dbPassword = process.env.DB_PASSWORD_FILE
  ? fs.readFileSync(process.env.DB_PASSWORD_FILE, 'utf8').trim()
  : process.env.DB_PASSWORD;

# Python
import os

def get_secret(name):
    file_path = os.environ.get(f'{name}_FILE')
    if file_path:
        with open(file_path, 'r') as f:
            return f.read().strip()
    return os.environ.get(name)

db_password = get_secret('DB_PASSWORD')

Best Practices

.env.example (Commit this)

# .env.example
NODE_ENV=development
DB_HOST=db
DB_PASSWORD=  # Set in .env
API_KEY=      # Set in .env

.gitignore

.env
.env.local
.env.*.local
secrets/
*.pem
*.key

Environment by Stage

# compose.yaml (base)
services:
  app:
    environment:
      - NODE_ENV=${NODE_ENV:-development}
    env_file:
      - .env

# compose.override.yaml (development)
services:
  app:
    environment:
      - DEBUG=true
      - LOG_LEVEL=debug

# compose.prod.yaml (production)
services:
  app:
    environment:
      - DEBUG=false
      - LOG_LEVEL=warn

Multi-Environment Setup

project/
├── .env                    # Local development
├── .env.example            # Template (committed)
├── .env.staging            # Staging environment
├── .env.production         # Production (not committed)
├── compose.yaml
├── compose.override.yaml   # Dev overrides
├── compose.staging.yaml
└── compose.prod.yaml

# Development (uses .env and compose.override.yaml automatically)
docker compose up

# Staging
docker compose --env-file .env.staging -f compose.yaml -f compose.staging.yaml up

# Production
docker compose --env-file .env.production -f compose.yaml -f compose.prod.yaml up

Environment Variable Patterns

Database Connection

# .env
DB_HOST=postgres
DB_PORT=5432
DB_USER=appuser
DB_PASSWORD=secretpass
DB_NAME=appdb
DATABASE_URL=postgresql://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:${DB_PORT}/${DB_NAME}

API Keys

# .env
STRIPE_SECRET_KEY=sk_test_xxx
STRIPE_WEBHOOK_SECRET=whsec_xxx
SENDGRID_API_KEY=SG.xxx

Feature Flags

# .env
FEATURE_NEW_DASHBOARD=true
FEATURE_BETA_API=false
ENABLE_ANALYTICS=true

Service URLs

# .env
API_URL=http://api:3000
FRONTEND_URL=http://localhost:8080
REDIS_URL=redis://redis:6379
ELASTICSEARCH_URL=http://elasticsearch:9200

Runtime Environment Inspection

# View container environment
docker compose exec app env

# View specific variable
docker compose exec app printenv DATABASE_URL

# Pass environment to command
docker compose exec -e DEBUG=true app npm run test

Interpolation in Config Files

Using environment variables in configuration files:

# docker-compose.yaml can interpolate
services:
  app:
    image: ${REGISTRY:-docker.io}/myapp:${TAG:-latest}
    ports:
      - "${APP_PORT:-3000}:3000"

# Nginx requires envsubst
# nginx.conf.template
server {
    listen ${NGINX_PORT};
    server_name ${NGINX_HOST};
}

services:
  nginx:
    image: nginx:alpine
    volumes:
      - ./nginx.conf.template:/etc/nginx/templates/default.conf.template
    environment:
      - NGINX_PORT=80
      - NGINX_HOST=localhost

External Secrets Management

HashiCorp Vault

services:
  vault:
    image: vault:latest
    cap_add:
      - IPC_LOCK
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    volumes:
      - vault_data:/vault/data

  app:
    environment:
      - VAULT_ADDR=http://vault:8200
      - VAULT_TOKEN=${VAULT_TOKEN}

volumes:
  vault_data:

AWS Secrets Manager

services:
  app:
    environment:
      - AWS_REGION=us-east-1
      - AWS_SECRET_NAME=my-app/production
    # Application fetches secrets at runtime

Debugging Environment Issues

# Check what variables compose sees
docker compose config

# Check .env parsing
docker compose config --format json | jq '.services.app.environment'

# Verify variable expansion
echo "DB_HOST is: ${DB_HOST}"

Project Architecture & Folder Structure

Single Project Structure

my-project/
├── .env                      # Environment variables
├── .env.example              # Template (commit this)
├── .gitignore
├── compose.yaml              # Main compose file
├── compose.override.yaml     # Development overrides (auto-loaded)
├── compose.prod.yaml         # Production overrides
├── Dockerfile
├── .dockerignore
│
├── src/                      # Application source code
│   └── ...
│
├── config/                   # Configuration files
│   ├── nginx/
│   │   └── nginx.conf
│   └── postgres/
│       └── postgresql.conf
│
├── scripts/                  # Utility scripts
│   ├── backup.sh
│   └── deploy.sh
│
├── init-scripts/             # Database initialization
│   └── 01-schema.sql
│
└── secrets/                  # Sensitive files (gitignored)
    ├── db_password.txt
    └── api_key.txt

Multi-Service Project Structure

my-platform/
├── .env
├── compose.yaml              # Orchestrates all services
├── compose.override.yaml
├── compose.prod.yaml
│
├── services/
│   ├── api/
│   │   ├── Dockerfile
│   │   ├── .dockerignore
│   │   ├── package.json
│   │   └── src/
│   │
│   ├── frontend/
│   │   ├── Dockerfile
│   │   ├── .dockerignore
│   │   └── src/
│   │
│   └── worker/
│       ├── Dockerfile
│       ├── .dockerignore
│       └── src/
│
├── infrastructure/
│   ├── nginx/
│   │   └── nginx.conf
│   ├── traefik/
│   │   └── traefik.yaml
│   └── postgres/
│       └── init.sql
│
└── scripts/
    └── deploy.sh

compose.yaml for Multi-Service:

services:
  api:
    build:
      context: ./services/api
      dockerfile: Dockerfile
    depends_on:
      - db

  frontend:
    build:
      context: ./services/frontend
      dockerfile: Dockerfile
    depends_on:
      - api

  worker:
    build:
      context: ./services/worker
      dockerfile: Dockerfile
    depends_on:
      - db
      - redis

  db:
    image: postgres:16
    volumes:
      - ./infrastructure/postgres/init.sql:/docker-entrypoint-initdb.d/init.sql

  redis:
    image: redis:7-alpine

Monorepo Structure

monorepo/
├── .env
├── compose.yaml
│
├── packages/
│   ├── shared/              # Shared libraries
│   │   ├── package.json
│   │   └── src/
│   │
│   ├── api/
│   │   ├── Dockerfile
│   │   ├── package.json
│   │   └── src/
│   │
│   └── web/
│       ├── Dockerfile
│       ├── package.json
│       └── src/
│
├── infrastructure/
│   └── docker/
│       ├── api.Dockerfile
│       └── web.Dockerfile
│
└── scripts/

Microservices Structure

microservices/
├── docker/
│   └── compose.yaml          # All services
│
├── services/
│   ├── user-service/
│   │   ├── compose.yaml      # Service-specific
│   │   ├── Dockerfile
│   │   └── src/
│   │
│   ├── order-service/
│   │   ├── compose.yaml
│   │   ├── Dockerfile
│   │   └── src/
│   │
│   └── payment-service/
│       ├── compose.yaml
│       ├── Dockerfile
│       └── src/
│
├── infrastructure/
│   ├── api-gateway/
│   ├── service-mesh/
│   └── monitoring/
│
└── scripts/
    ├── start-all.sh
    └── stop-all.sh

Laravel Project Structure

laravel-project/
├── .env
├── compose.yaml
├── Dockerfile
├── .dockerignore
│
├── app/
├── bootstrap/
├── config/
├── database/
├── public/
├── resources/
├── routes/
├── storage/
├── tests/
│
├── docker/
│   ├── nginx/
│   │   └── default.conf
│   ├── php/
│   │   └── php.ini
│   └── supervisor/
│       └── supervisord.conf
│
└── scripts/
    └── entrypoint.sh

Node.js Project Structure

nodejs-project/
├── .env
├── compose.yaml
├── Dockerfile
├── .dockerignore
│
├── src/
│   ├── controllers/
│   ├── models/
│   ├── routes/
│   ├── services/
│   └── index.js
│
├── config/
│   └── default.json
│
├── tests/
│
└── docker/
    └── nginx/

File Naming Conventions

Configuration Management

Per-Environment Configuration

config/
├── default.json          # Base config
├── development.json      # Dev overrides
├── staging.json          # Staging overrides
├── production.json       # Prod overrides
└── custom-environment-variables.json  # Env var mapping

Docker-Specific Config

docker/
├── development/
│   ├── Dockerfile
│   └── nginx.conf
│
├── production/
│   ├── Dockerfile
│   └── nginx.conf
│
└── scripts/
    ├── entrypoint.sh
    └── healthcheck.sh

Gitignore for Docker Projects

# Environment
.env
.env.local
.env.*.local
!.env.example

# Secrets
secrets/
*.pem
*.key

# Docker volumes data
data/
volumes/

# Logs
logs/
*.log

# Node
node_modules/

# Build artifacts
dist/
build/

# IDE
.vscode/
.idea/

# OS
.DS_Store
Thumbs.db

Makefile for Docker Operations

.PHONY: help build up down restart logs shell

help:
	@echo "Available commands:"
	@echo "  make build    - Build images"
	@echo "  make up       - Start services"
	@echo "  make down     - Stop services"
	@echo "  make restart  - Restart services"
	@echo "  make logs     - View logs"
	@echo "  make shell    - Shell into app"

build:
	docker compose build

up:
	docker compose up -d

down:
	docker compose down

restart:
	docker compose restart

logs:
	docker compose logs -f

shell:
	docker compose exec app sh

Global vs Local Containers

Global Containers (Machine-Wide Services)

Located in a dedicated directory, running shared services:

/opt/docker/                  # or ~/docker/
├── global/
│   ├── compose.yaml
│   ├── .env
│   │
│   ├── traefik/
│   │   ├── traefik.yaml
│   │   └── dynamic/
│   │
│   ├── portainer/
│   │   └── data/
│   │
│   └── monitoring/
│       ├── prometheus/
│       └── grafana/

global/compose.yaml:

services:
  traefik:
    image: traefik:v3.0
    container_name: traefik
    restart: always
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik:/etc/traefik
      - traefik_certs:/letsencrypt
    networks:
      - proxy

  portainer:
    image: portainer/portainer-ce:latest
    container_name: portainer
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - portainer_data:/data
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.portainer.rule=Host(`portainer.yourdomain.com`)"
      - "traefik.http.routers.portainer.tls.certresolver=letsencrypt"

networks:
  proxy:
    name: proxy
    driver: bridge

volumes:
  traefik_certs:
  portainer_data:

Setup Commands:

# Create the proxy network first
docker network create proxy

# Start global services
cd /opt/docker/global
docker compose up -d

# Enable on boot (systemd)
sudo systemctl enable docker

Local Containers (Project-Specific)

Each project manages its own containers:

~/projects/
├── project-a/
│   ├── compose.yaml
│   └── ...
│
├── project-b/
│   ├── compose.yaml
│   └── ...

project-a/compose.yaml:

services:
  app:
    build: .
    networks:
      - proxy  # Connect to global Traefik
      - internal
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.project-a.rule=Host(`project-a.localhost`)"
      - "traefik.docker.network=proxy"

  db:
    image: postgres:16
    networks:
      - internal
    volumes:
      - db_data:/var/lib/postgresql/data

networks:
  proxy:
    external: true  # Use global network
  internal:
    driver: bridge

volumes:
  db_data:

Workflow Summary

# GLOBAL SERVICES (run once, always available)
cd /opt/docker/global
docker compose up -d

# PROJECT A (start when needed)
cd ~/projects/project-a
docker compose up -d

# PROJECT B (start when needed)
cd ~/projects/project-b
docker compose up -d

# Stop a project
cd ~/projects/project-a
docker compose down

# Global services remain running

Common Global Services

Traefik (Reverse Proxy)

services:
  traefik:
    image: traefik:v3.0
    restart: always
    command:
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"  # Dashboard
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - proxy

Portainer (Docker Management UI)

services:
  portainer:
    image: portainer/portainer-ce:latest
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - portainer_data:/data
    ports:
      - "9443:9443"
    networks:
      - proxy

volumes:
  portainer_data:

Mailhog (Email Testing)

services:
  mailhog:
    image: mailhog/mailhog
    restart: unless-stopped
    ports:
      - "1025:1025"  # SMTP
      - "8025:8025"  # Web UI
    networks:
      - proxy

Redis (Shared Cache)

services:
  redis:
    image: redis:7-alpine
    restart: always
    command: redis-server --appendonly yes
    volumes:
      - redis_data:/data
    ports:
      - "127.0.0.1:6379:6379"
    networks:
      - proxy

volumes:
  redis_data:

Connecting Local Projects to Global Services

Using Global Traefik

# Local project compose.yaml
services:
  app:
    build: .
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.myapp.rule=Host(`myapp.localhost`)"
      - "traefik.http.services.myapp.loadbalancer.server.port=3000"
      - "traefik.docker.network=proxy"

networks:
  proxy:
    external: true

Using Global Redis

# Local project compose.yaml
services:
  app:
    build: .
    environment:
      - REDIS_URL=redis://redis:6379
    networks:
      - proxy  # Same network as global redis

networks:
  proxy:
    external: true

Host Entries for Local Development

Add to /etc/hosts:

127.0.0.1 project-a.localhost
127.0.0.1 project-b.localhost
127.0.0.1 api.project-a.localhost
127.0.0.1 traefik.localhost
127.0.0.1 portainer.localhost

Or use a wildcard DNS service like nip.io or sslip.io:

• myapp.127.0.0.1.nip.io resolves to 127.0.0.1

Starting Global Services on Boot

Using systemd

# /etc/systemd/system/docker-global.service
[Unit]
Description=Global Docker Services
Requires=docker.service
After=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/docker/global
ExecStart=/usr/bin/docker compose up -d
ExecStop=/usr/bin/docker compose down

[Install]
WantedBy=multi-user.target

sudo systemctl enable docker-global
sudo systemctl start docker-global

Using cron

# Add to crontab
@reboot cd /opt/docker/global && docker compose up -d

Project Isolation Strategies

Fully Isolated (Default)

# Each project has its own network
services:
  app:
    networks:
      - default

  db:
    networks:
      - default

# Networks are project-scoped by default

Shared Database Server

# Global database server
services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: admin_password
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:
      - databases

networks:
  databases:
    name: databases

volumes:
  postgres_data:

# Local project uses global database
services:
  app:
    environment:
      - DATABASE_URL=postgresql://app_user:password@postgres:5432/app_db
    networks:
      - proxy
      - databases

networks:
  proxy:
    external: true
  databases:
    external: true

Complete Examples

Example 1: Full-Stack Web Application

# compose.yaml
services:
  # Reverse Proxy
  traefik:
    image: traefik:v3.0
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - frontend

  # Frontend (React/Vue/etc.)
  frontend:
    build: ./frontend
    networks:
      - frontend
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.frontend.rule=Host(`localhost`)"
      - "traefik.http.services.frontend.loadbalancer.server.port=80"
    depends_on:
      - api

  # Backend API
  api:
    build: ./api
    environment:
      - DATABASE_URL=postgresql://${DB_USER}:${DB_PASSWORD}@db:5432/${DB_NAME}
      - REDIS_URL=redis://redis:6379
    networks:
      - frontend
      - backend
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.api.rule=Host(`localhost`) && PathPrefix(`/api`)"
      - "traefik.http.services.api.loadbalancer.server.port=3000"
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_started

  # Background Worker
  worker:
    build: ./api
    command: npm run worker
    environment:
      - DATABASE_URL=postgresql://${DB_USER}:${DB_PASSWORD}@db:5432/${DB_NAME}
      - REDIS_URL=redis://redis:6379
    networks:
      - backend
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_started

  # PostgreSQL Database
  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_DB: ${DB_NAME}
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./init-scripts:/docker-entrypoint-initdb.d
    networks:
      - backend
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${DB_USER}"]
      interval: 5s
      timeout: 5s
      retries: 5

  # Redis Cache/Queue
  redis:
    image: redis:7-alpine
    command: redis-server --appendonly yes
    volumes:
      - redis_data:/data
    networks:
      - backend

networks:
  frontend:
  backend:
    internal: true

volumes:
  postgres_data:
  redis_data:

.env:

DB_USER=appuser
DB_PASSWORD=supersecretpassword
DB_NAME=myapp

Example 2: Development Environment with Hot Reload

# compose.yaml
services:
  app:
    build:
      context: .
      target: development
    volumes:
      - ./src:/app/src
      - /app/node_modules
    ports:
      - "3000:3000"
      - "9229:9229"  # Debug port
    environment:
      - NODE_ENV=development
    command: npm run dev

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_PASSWORD: devpassword
      POSTGRES_DB: devdb
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

  adminer:
    image: adminer
    ports:
      - "8080:8080"
    depends_on:
      - db

volumes:
  postgres_data:

Example 3: Laravel Application

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
    volumes:
      - .:/var/www/html
      - /var/www/html/vendor
    depends_on:
      - db
      - redis
    networks:
      - laravel

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - .:/var/www/html
      - ./docker/nginx/default.conf:/etc/nginx/conf.d/default.conf
    depends_on:
      - app
    networks:
      - laravel

  db:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: ${DB_PASSWORD}
      MYSQL_DATABASE: ${DB_DATABASE}
    volumes:
      - mysql_data:/var/lib/mysql
    networks:
      - laravel

  redis:
    image: redis:7-alpine
    networks:
      - laravel

  queue:
    build:
      context: .
      dockerfile: Dockerfile
    command: php artisan queue:work
    volumes:
      - .:/var/www/html
    depends_on:
      - db
      - redis
    networks:
      - laravel

  scheduler:
    build:
      context: .
      dockerfile: Dockerfile
    command: php artisan schedule:work
    volumes:
      - .:/var/www/html
    depends_on:
      - db
    networks:
      - laravel

networks:
  laravel:

volumes:
  mysql_data:

Example 4: Microservices with API Gateway

services:
  gateway:
    image: kong:latest
    environment:
      KONG_DATABASE: "off"
      KONG_PROXY_ACCESS_LOG: /dev/stdout
      KONG_ADMIN_ACCESS_LOG: /dev/stdout
      KONG_PROXY_ERROR_LOG: /dev/stderr
      KONG_ADMIN_ERROR_LOG: /dev/stderr
      KONG_ADMIN_LISTEN: 0.0.0.0:8001
    ports:
      - "8000:8000"
      - "8001:8001"
    networks:
      - gateway

  user-service:
    build: ./services/user
    environment:
      - DATABASE_URL=postgresql://user:pass@user-db:5432/users
    networks:
      - gateway
      - user-db-net

  order-service:
    build: ./services/order
    environment:
      - DATABASE_URL=postgresql://user:pass@order-db:5432/orders
      - USER_SERVICE_URL=http://user-service:3000
    networks:
      - gateway
      - order-db-net

  user-db:
    image: postgres:16-alpine
    environment:
      POSTGRES_PASSWORD: pass
      POSTGRES_DB: users
    volumes:
      - user_db_data:/var/lib/postgresql/data
    networks:
      - user-db-net

  order-db:
    image: postgres:16-alpine
    environment:
      POSTGRES_PASSWORD: pass
      POSTGRES_DB: orders
    volumes:
      - order_db_data:/var/lib/postgresql/data
    networks:
      - order-db-net

networks:
  gateway:
  user-db-net:
    internal: true
  order-db-net:
    internal: true

volumes:
  user_db_data:
  order_db_data:

Example 5: ELK Stack for Logging

services:
  elasticsearch:
    image: elasticsearch:8.11.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - elasticsearch_data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    networks:
      - elk

  logstash:
    image: logstash:8.11.0
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
    ports:
      - "5044:5044"
    depends_on:
      - elasticsearch
    networks:
      - elk

  kibana:
    image: kibana:8.11.0
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
    networks:
      - elk

networks:
  elk:

volumes:
  elasticsearch_data:

Example 6: WordPress with Traefik SSL

services:
  traefik:
    image: traefik:v3.0
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.le.acme.tlschallenge=true"
      - "--certificatesresolvers.le.acme.email=admin@example.com"
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - letsencrypt:/letsencrypt
    networks:
      - proxy

  wordpress:
    image: wordpress:latest
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: ${DB_USER}
      WORDPRESS_DB_PASSWORD: ${DB_PASSWORD}
      WORDPRESS_DB_NAME: ${DB_NAME}
    volumes:
      - wordpress_data:/var/www/html
    networks:
      - proxy
      - backend
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.wordpress.rule=Host(`blog.example.com`)"
      - "traefik.http.routers.wordpress.tls.certresolver=le"

  db:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}
      MYSQL_DATABASE: ${DB_NAME}
      MYSQL_USER: ${DB_USER}
      MYSQL_PASSWORD: ${DB_PASSWORD}
    volumes:
      - db_data:/var/lib/mysql
    networks:
      - backend

networks:
  proxy:
  backend:
    internal: true

volumes:
  wordpress_data:
  db_data:
  letsencrypt:

Example 7: Python ML Pipeline

services:
  jupyter:
    build:
      context: .
      dockerfile: Dockerfile.jupyter
    ports:
      - "8888:8888"
    volumes:
      - ./notebooks:/home/jovyan/work
      - ./data:/home/jovyan/data
    environment:
      - JUPYTER_ENABLE_LAB=yes
    networks:
      - ml

  mlflow:
    image: ghcr.io/mlflow/mlflow:latest
    ports:
      - "5000:5000"
    command: mlflow server --host 0.0.0.0 --backend-store-uri sqlite:///mlflow.db
    volumes:
      - mlflow_data:/mlflow
    networks:
      - ml

  minio:
    image: minio/minio
    command: server /data --console-address ":9001"
    ports:
      - "9000:9000"
      - "9001:9001"
    volumes:
      - minio_data:/data
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    networks:
      - ml

networks:
  ml:

volumes:
  mlflow_data:
  minio_data:

Essential Commands Reference

Container Management

# Start services
docker compose up                    # Foreground
docker compose up -d                 # Detached (background)
docker compose up --build            # Rebuild images first
docker compose up -d --force-recreate  # Recreate containers

# Stop services
docker compose down                  # Stop and remove containers
docker compose down -v               # Also remove volumes
docker compose down --rmi all        # Also remove images

# Restart services
docker compose restart               # Restart all
docker compose restart api           # Restart specific service

# View status
docker compose ps                    # List containers
docker compose ps -a                 # Include stopped

# View logs
docker compose logs                  # All services
docker compose logs -f               # Follow
docker compose logs -f api           # Specific service
docker compose logs --tail=100 api   # Last 100 lines

# Execute commands
docker compose exec api bash         # Interactive shell
docker compose exec db psql -U user  # Database CLI
docker compose run api npm test      # Run one-off command

# Scale services
docker compose up -d --scale worker=3

Image Management

# Build images
docker compose build                 # Build all
docker compose build --no-cache      # Without cache
docker compose build api             # Specific service

# Pull images
docker compose pull                  # Pull all images

# List images
docker images

# Remove images
docker rmi image_name
docker image prune                   # Remove unused
docker image prune -a                # Remove all unused

Volume Management

# List volumes
docker volume ls

# Inspect volume
docker volume inspect myproject_postgres_data

# Create volume
docker volume create my-volume

# Remove volume
docker volume rm my-volume

# Remove unused volumes
docker volume prune

# Backup volume
docker run --rm -v myproject_data:/data -v $(pwd):/backup \
    alpine tar czf /backup/data-backup.tar.gz -C /data .

# Restore volume
docker run --rm -v myproject_data:/data -v $(pwd):/backup \
    alpine tar xzf /backup/data-backup.tar.gz -C /data

Network Management

# List networks
docker network ls

# Create network
docker network create proxy

# Inspect network
docker network inspect proxy

# Remove network
docker network rm network_name

# Remove unused networks
docker network prune

# Connect container to network
docker network connect proxy container_name

# Disconnect container from network
docker network disconnect proxy container_name

Cleanup Commands

# Remove stopped containers
docker container prune

# Remove unused images
docker image prune
docker image prune -a    # Including tagged images

# Remove unused volumes
docker volume prune

# Remove unused networks
docker network prune

# Remove everything unused
docker system prune
docker system prune -a --volumes  # Including volumes

# Check disk usage
docker system df
docker system df -v  # Verbose

Container Inspection

# View container details
docker inspect container_name

# View container logs
docker logs container_name
docker logs -f container_name           # Follow
docker logs --since 10m container_name  # Last 10 minutes
docker logs --tail 100 container_name   # Last 100 lines

# View container processes
docker top container_name

# View container stats
docker stats
docker stats container_name

# View container changes
docker diff container_name

Executing Commands

# Interactive shell
docker compose exec app bash
docker compose exec app sh          # For Alpine

# Run command
docker compose exec app npm test
docker compose exec db psql -U postgres

# Run as root
docker compose exec -u root app bash

# Run with environment variable
docker compose exec -e DEBUG=true app npm test

# One-off container
docker compose run --rm app npm test
docker compose run --rm -v $(pwd):/app app npm install

Copying Files

# Copy from container
docker cp container_name:/app/file.txt ./file.txt

# Copy to container
docker cp ./file.txt container_name:/app/file.txt

# Using compose
docker compose cp app:/app/logs ./logs
docker compose cp ./config app:/app/config

Debugging Commands

# Shell into running container
docker compose exec app sh

# Shell into failed container
docker compose run --entrypoint sh app

# View container resource usage
docker stats

# View container processes
docker compose top

# View real-time events
docker events

# Check container health
docker inspect --format='{{json .State.Health}}' container_name | jq

Docker Compose Config

# Validate compose file
docker compose config

# View parsed config
docker compose config --format json

# Check specific service
docker compose config --services

# View volumes
docker compose config --volumes

# View networks
docker compose config --networks

Quick Reference Card

# ==========================================
# STARTING SERVICES
# ==========================================
docker compose up -d              # Start all (detached)
docker compose up -d --build      # Rebuild then start
docker compose up -d service      # Start specific service

# ==========================================
# STOPPING/RESTARTING (DATA SAFE)
# ==========================================
docker compose stop               # Stop (keeps containers)
docker compose start              # Start stopped containers
docker compose restart            # Restart all
docker compose restart api        # Restart specific service

# ==========================================
# RECREATING (KEEPS VOLUMES/DATA)
# ==========================================
docker compose up -d --force-recreate    # Fresh containers, keep data
docker compose up -d --build             # Rebuild images, keep data

# ==========================================
# RESET (DATA LOSS WARNING)
# ==========================================
docker compose down               # Remove containers (KEEPS volumes)
docker compose down -v            # Remove containers AND volumes (DELETES DATA)
docker compose down -v --rmi all  # Remove everything

# ==========================================
# LOGS & DEBUGGING
# ==========================================
docker compose logs -f            # Follow all logs
docker compose logs -f api        # Follow specific service
docker compose exec app bash      # Shell access
docker compose exec db psql -U postgres  # Database shell

# ==========================================
# DATABASE BACKUP/RESTORE
# ==========================================
# PostgreSQL
docker compose exec -T db pg_dump -U user dbname > backup.sql
docker compose exec -T db psql -U user dbname < backup.sql

# MySQL
docker compose exec -T db mysqldump -u root -p"$PASS" db > backup.sql
docker compose exec -T db mysql -u root -p"$PASS" db < backup.sql

# ==========================================
# STATUS & INFO
# ==========================================
docker compose ps                 # Container status
docker compose ps -a              # Include stopped
docker compose config             # Validate compose file
docker stats                      # Resource usage

Security Best Practices

Container Security

1. Run as Non-Root User

# Dockerfile
RUN addgroup -g 1001 appgroup && \
    adduser -u 1001 -G appgroup -D appuser
USER appuser

# compose.yaml
services:
  app:
    user: "1000:1000"

2. Use Read-Only Filesystem

services:
  app:
    read_only: true
    tmpfs:
      - /tmp
      - /app/cache

3. Drop Unnecessary Capabilities

services:
  app:
    cap_drop:
      - ALL
    cap_add:
      - NET_BIND_SERVICE  # Only if needed

4. Set Resource Limits

services:
  app:
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M

5. Scan Images for Vulnerabilities

# Docker Scout
docker scout quickview myimage:latest
docker scout cves myimage:latest

# Trivy
trivy image myimage:latest

# Grype
grype myimage:latest

Network Security

1. Use Internal Networks for Databases

networks:
  backend:
    internal: true

services:
  db:
    networks:
      - backend
    # No ports exposed to host

2. Bind to Localhost for Development

ports:
  - "127.0.0.1:5432:5432"

3. Use TLS for Production

# Use Traefik with Let's Encrypt
labels:
  - "traefik.http.routers.app.tls.certresolver=letsencrypt"

4. Network Segmentation

services:
  frontend:
    networks:
      - public

  api:
    networks:
      - public
      - internal

  db:
    networks:
      - internal

networks:
  public:
  internal:
    internal: true

Secret Management

1. Never Commit Secrets

# .gitignore
.env
secrets/
*.pem
*.key

2. Use Docker Secrets

services:
  app:
    secrets:
      - db_password
      - api_key
    environment:
      - DB_PASSWORD_FILE=/run/secrets/db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt
  api_key:
    file: ./secrets/api_key.txt

3. Use Environment Variables from Secure Sources

environment:
  - DB_PASSWORD  # From shell, not compose file

4. External Secret Managers

# HashiCorp Vault integration
services:
  app:
    environment:
      - VAULT_ADDR=http://vault:8200
      - VAULT_TOKEN=${VAULT_TOKEN}

Image Security

1. Use Minimal Base Images

# Prefer Alpine or distroless
FROM node:20-alpine
FROM gcr.io/distroless/nodejs

2. Pin Image Versions

# BAD
FROM node:latest

# GOOD
FROM node:20.10.0-alpine3.19

3. Multi-Stage Builds

# Build stage
FROM node:20 AS builder
WORKDIR /app
COPY . .
RUN npm ci && npm run build

# Production stage - minimal image
FROM node:20-alpine
COPY --from=builder /app/dist ./dist
USER node
CMD ["node", "dist/index.js"]

4. Verify Image Signatures

# Docker Content Trust
export DOCKER_CONTENT_TRUST=1
docker pull myimage:latest

Dockerfile Security

1. Don't Store Secrets in Images

# BAD
ENV API_KEY=secret123

# GOOD - Pass at runtime
# docker run -e API_KEY=secret123 myimage

2. Use COPY Instead of ADD

# Prefer COPY (more explicit)
COPY requirements.txt .

# ADD only for tar extraction
ADD archive.tar.gz /app/

3. Minimize Layers and Clean Up

RUN apt-get update && apt-get install -y \
    curl \
    git \
    && rm -rf /var/lib/apt/lists/* \
    && apt-get clean

4. Use .dockerignore

.git
.env
secrets/
*.pem
node_modules/
tests/

Runtime Security

1. Limit Container Resources

services:
  app:
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 1G
          pids: 100

2. Use Security Options

services:
  app:
    security_opt:
      - no-new-privileges:true
      - seccomp:unconfined  # Only if needed

3. Disable Privilege Escalation

services:
  app:
    privileged: false
    security_opt:
      - no-new-privileges:true

4. Use AppArmor/SELinux

services:
  app:
    security_opt:
      - apparmor:docker-default
      - label:type:container_runtime_t

Access Control

1. Protect Docker Socket

services:
  app:
    volumes:
      # Read-only access to docker socket (if needed)
      - /var/run/docker.sock:/var/run/docker.sock:ro

2. Use Docker Context for Remote Access

docker context create remote --docker "host=ssh://user@remote-host"
docker context use remote

3. Enable TLS for Docker Daemon

# Generate certificates
# Configure dockerd with --tlsverify

Logging and Monitoring

1. Configure Log Rotation

services:
  app:
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

2. Use Centralized Logging

services:
  app:
    logging:
      driver: syslog
      options:
        syslog-address: "tcp://logserver:514"

3. Monitor Container Health

services:
  app:
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3

Security Checklist

• [ ] Run containers as non-root
• [ ] Use read-only filesystem where possible
• [ ] Drop all capabilities, add only needed ones
• [ ] Set resource limits
• [ ] Use internal networks for backend services
• [ ] Never expose database ports publicly
• [ ] Use TLS in production
• [ ] Never commit secrets to version control
• [ ] Use minimal base images
• [ ] Pin image versions
• [ ] Scan images for vulnerabilities
• [ ] Use multi-stage builds
• [ ] Enable log rotation
• [ ] Implement health checks
• [ ] Disable privilege escalation

Port Conflict Resolution

Detecting Port Conflicts

When you run docker compose up and a port is already in use, you'll see an error like:

Error response from daemon: driver failed programming external connectivity:
Bind for 0.0.0.0:3000 failed: port is already allocated

Finding What's Using a Port

# Linux/macOS
lsof -i :3000
lsof -Pi :3000 -sTCP:LISTEN -t

# Linux alternative
ss -tulpn | grep 3000
netstat -tulpn | grep 3000

# Check if Docker container is using the port
docker ps --filter "publish=3000"

Killing Processes on a Port

# Kill process using port
kill $(lsof -t -i:3000)

# Force kill
kill -9 $(lsof -t -i:3000)

# If it's a Docker container
docker stop $(docker ps -q --filter "publish=3000")

Dynamic Port Assignment

Configure compose file to handle port conflicts:

services:
  app:
    build: .
    ports:
      # Use environment variable with fallback
      - "${APP_PORT:-3000}:3000"
    environment:
      - PORT=3000

  db:
    image: postgres:16
    ports:
      - "${DB_PORT:-5432}:5432"

  redis:
    image: redis:7
    ports:
      - "${REDIS_PORT:-6379}:6379"

.env:

# Change these if defaults are in use
APP_PORT=3000
DB_PORT=5432
REDIS_PORT=6379

Port Conflict Resolution Strategies

Automatic Port Conflict Detection Script

scripts/check-ports.sh:

#!/bin/bash

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

# Ports to check (customize for your project)
PORTS=(3000 5432 6379 80 443)

echo "Checking port availability..."

CONFLICTS=()

for PORT in "${PORTS[@]}"; do
    if lsof -Pi :$PORT -sTCP:LISTEN -t >/dev/null 2>&1; then
        PROCESS=$(lsof -Pi :$PORT -sTCP:LISTEN | tail -1 | awk '{print $1, $2}')
        echo -e "${RED}Port $PORT is in use by: $PROCESS${NC}"
        CONFLICTS+=($PORT)
    else
        echo -e "${GREEN}Port $PORT is available${NC}"
    fi
done

if [ ${#CONFLICTS[@]} -gt 0 ]; then
    echo ""
    echo -e "${YELLOW}Port conflicts detected!${NC}"
    echo ""
    echo "Options:"
    echo "  1. Stop the conflicting processes"
    echo "  2. Change ports in compose.yaml or .env"
    echo "  3. Use dynamic port assignment"
    exit 1
fi

echo -e "\n${GREEN}All ports available! Starting containers...${NC}"
docker compose up -d

Find Free Port Script

scripts/find-free-port.sh:

#!/bin/bash

# Find a free port starting from a given port
find_free_port() {
    local port=$1
    while lsof -Pi :$port -sTCP:LISTEN -t >/dev/null 2>&1; do
        ((port++))
    done
    echo $port
}

# Auto-configure ports
APP_PORT=$(find_free_port 3000)
DB_PORT=$(find_free_port 5432)
REDIS_PORT=$(find_free_port 6379)

echo "APP_PORT=$APP_PORT"
echo "DB_PORT=$DB_PORT"
echo "REDIS_PORT=$REDIS_PORT"

# Export for docker compose
export APP_PORT DB_PORT REDIS_PORT

# Start with auto-assigned ports
docker compose up -d

echo ""
echo "Services started on:"
echo "  App:   http://localhost:$APP_PORT"
echo "  DB:    localhost:$DB_PORT"
echo "  Redis: localhost:$REDIS_PORT"

Common Port Conflicts

Using Random Host Ports

For development when you don't need a specific port:

services:
  app:
    ports:
      - "3000"  # No host port = random assignment

  api:
    ports:
      - "3000"  # Each service gets different random port

Find the assigned port:

docker compose port app 3000

Localhost-Only Binding

For services that shouldn't be accessible from network:

services:
  db:
    ports:
      - "127.0.0.1:5432:5432"  # Only accessible from localhost

Complete Port Management Script

scripts/docker-start.sh:

#!/bin/bash
set -e

PROJECT_NAME=${1:-$(basename $(pwd))}

echo "Docker Compose Smart Starter"
echo "============================"

# Function to check if a port is in use
port_in_use() {
    lsof -Pi :$1 -sTCP:LISTEN -t >/dev/null 2>&1
}

# Function to get process using a port
get_port_process() {
    lsof -Pi :$1 -sTCP:LISTEN | tail -1 | awk '{print $1 " (PID: " $2 ")"}'
}

# Extract ports from compose file
PORTS=$(grep -E '^\s+-\s*"?[0-9]+:[0-9]+"?' compose.yaml | grep -oE '[0-9]+:' | tr -d ':' | sort -u)

echo "Checking ports: $PORTS"

CONFLICTS=()

for PORT in $PORTS; do
    if port_in_use $PORT; then
        echo "Port $PORT: CONFLICT - $(get_port_process $PORT)"
        CONFLICTS+=($PORT)
    else
        echo "Port $PORT: Available"
    fi
done

if [ ${#CONFLICTS[@]} -gt 0 ]; then
    echo ""
    echo "Found ${#CONFLICTS[@]} port conflict(s)"
    echo ""
    echo "Choose an action:"
    echo "  1) Kill conflicting processes"
    echo "  2) Exit and fix manually"
    read -p "Enter choice [1-2]: " choice

    case $choice in
        1)
            for PORT in "${CONFLICTS[@]}"; do
                echo "Killing process on port $PORT..."
                kill $(lsof -t -i:$PORT) 2>/dev/null || true
                sleep 1
            done
            ;;
        *)
            exit 1
            ;;
    esac
fi

echo ""
echo "Starting containers..."
docker compose up -d

echo ""
echo "Done! Containers are running."
docker compose ps

Restart Strategies & Data Persistence

Understanding What Gets Preserved

Restart Commands Reference

Scenario 1: Quick Restart (Keep Everything)

Use when: Service is unresponsive, need to apply env changes

docker compose restart                    # Restart all
docker compose restart api               # Restart one service

Scenario 2: Recreate Containers (Keep Volumes/Data)

Use when: Changed compose.yaml, need fresh container state

docker compose up -d --force-recreate    # Recreate all
docker compose up -d --force-recreate api # Recreate one service

Scenario 3: Rebuild and Recreate (Keep Data)

Use when: Changed Dockerfile or source code

docker compose up -d --build             # Rebuild changed images
docker compose up -d --build --force-recreate  # Rebuild all

Scenario 4: Stop Without Removing (Pause Work)

Use when: Taking a break, need ports freed temporarily

docker compose stop                      # Stop all (can start later)
docker compose start                     # Start stopped containers

Scenario 5: Full Reset Keeping Data (Clean Container State)

Use when: Troubleshooting, major config changes

docker compose down                      # Remove containers
docker compose up -d                     # Fresh containers, same data

Scenario 6: Full Reset Removing Data (Fresh Start)

Use when: Corrupted data, schema changes, starting over

docker compose down -v                   # Remove everything including volumes
docker compose up -d                     # Completely fresh

Scenario 7: Nuclear Option (Remove Everything)

Use when: Complete cleanup, switching projects

docker compose down -v --rmi all --remove-orphans

Restart Decision Tree

WHAT DO YOU NEED TO DO?
         │
    ┌────┴────┬─────────────┐
    ▼         ▼             ▼
Quick      Update        Fresh
restart    config        start
    │         │             │
    ▼         ▼             ▼
restart   up -d         down -v
          --force       up -d
          -recreate

When to Use What

Restart Policies in Compose

services:
  app:
    restart: unless-stopped  # Recommended for production

Health Checks for Auto-Recovery

services:
  app:
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

Graceful Shutdown

services:
  app:
    stop_grace_period: 30s
    stop_signal: SIGTERM

Handle in application:

process.on('SIGTERM', async () => {
  console.log('SIGTERM received, shutting down gracefully');
  await server.close();
  await db.disconnect();
  process.exit(0);
});

Smart Restart Script

scripts/docker-restart.sh:

#!/bin/bash

echo "Docker Compose Restart Manager"
echo "=============================="

echo "Current status:"
docker compose ps --format "table {{.Name}}\t{{.Status}}\t{{.Ports}}" 2>/dev/null || echo "No containers running"

echo ""
echo "Data-Safe Options:"
echo "  1) Restart services (keeps everything, fastest)"
echo "  2) Recreate containers (keeps volumes/data)"
echo "  3) Rebuild and recreate (keeps volumes/data)"
echo "  4) Stop services (free ports, keep data)"
echo ""
echo "Data-Risk Options:"
echo "  5) Reset all data (DELETES DATABASE)"
echo ""
echo "  0) Exit"

read -p "Enter choice [0-5]: " choice

case $choice in
    1)
        echo "Restarting services..."
        docker compose restart
        ;;
    2)
        echo "Recreating containers..."
        docker compose up -d --force-recreate
        ;;
    3)
        echo "Rebuilding and recreating..."
        docker compose up -d --build --force-recreate
        ;;
    4)
        echo "Stopping services..."
        docker compose stop
        ;;
    5)
        read -p "This will DELETE all data. Type 'DELETE' to confirm: " confirm
        if [ "$confirm" = "DELETE" ]; then
            docker compose down -v
            docker compose up -d
        else
            echo "Aborted."
        fi
        ;;
    0)
        exit 0
        ;;
    *)
        echo "Invalid option"
        ;;
esac

echo ""
docker compose ps

Database Backup Before Restart

# Always backup before risky operations
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

# PostgreSQL
docker compose exec -T db pg_dumpall -U postgres > "backup_$TIMESTAMP.sql"

# MySQL
docker compose exec -T db mysqldump -u root -p"$PASS" --all-databases > "backup_$TIMESTAMP.sql"

# MongoDB
docker compose exec -T db mongodump --archive > "backup_$TIMESTAMP.archive"

Data Persistence Best Practices

What to Use Named Volumes For

• Database data
• Cache data that should persist
• Application data (uploads, files)

What to Use Bind Mounts For

• Source code (development)
• Configuration files
• Log files (for easy access)

What to Use Anonymous Volumes For

• node_modules (preserve between rebuilds)
• Temporary build artifacts

volumes:
  # Named volume - persists across down/up
  - db_data:/var/lib/postgresql/data

  # Bind mount - always on host
  - ./src:/app/src

  # Anonymous volume - lost on down
  - /app/node_modules

Troubleshooting

Container Won't Start

Check Logs

docker compose logs servicename
docker compose logs --tail=100 servicename

Check Container Status

docker compose ps -a
docker inspect containername

Run Container Interactively

docker compose run --rm servicename sh
docker compose run --entrypoint sh servicename

Check Exit Code

docker inspect --format='{{.State.ExitCode}}' containername

Port Already in Use

Find What's Using the Port

lsof -i :3000
# or
netstat -tulpn | grep 3000
# or
ss -tulpn | grep 3000

Kill the Process

kill $(lsof -t -i:3000)

Check for Docker Containers

docker ps --filter "publish=3000"
docker stop $(docker ps -q --filter "publish=3000")

Container Keeps Restarting (Crash Loop)

Check Logs for Errors

docker compose logs --tail=100 servicename

Check Health Status

docker inspect --format='{{json .State.Health}}' containername | jq

Run Interactively to Debug

docker compose run --rm servicename sh

Check Resource Limits

docker stats containername

Volume Permission Issues

Check Volume Permissions

docker compose exec app ls -la /app/data

Fix Ownership Inside Container

docker compose exec -u root app chown -R appuser:appgroup /app/data

Fix Host Permissions

sudo chown -R $(id -u):$(id -g) ./mounted-folder

Use User Mapping

services:
  app:
    user: "1000:1000"  # Match your host user

Data Disappearing After Restart

Check Volume Configuration

docker compose config | grep -A5 "volumes:"

Verify Volume Exists

docker volume ls
docker volume inspect projectname_dbdata

Common Mistake

# BAD: Anonymous volume (gets deleted)
volumes:
  - /var/lib/postgresql/data

# GOOD: Named volume (persists)
volumes:
  - postgres_data:/var/lib/postgresql/data

Container Can't Reach Another Container

Check Network Configuration

docker network inspect networkname

Test Connectivity

docker compose exec app ping db
docker compose exec app nc -zv db 5432

Test DNS Resolution

docker compose exec app nslookup db

Verify Same Network

services:
  app:
    networks:
      - backend
  db:
    networks:
      - backend  # Must be same network

networks:
  backend:

Out of Disk Space

Check Docker Disk Usage

docker system df
docker system df -v

Clean Up

# Remove unused containers
docker container prune

# Remove unused images
docker image prune -a

# Remove unused volumes
docker volume prune

# Remove everything unused
docker system prune -a --volumes

Build Cache Issues

Build Without Cache

docker compose build --no-cache
docker compose build --no-cache servicename

Clear All Build Cache

docker builder prune -a

Service Won't Start After Config Change

Validate Compose File

docker compose config
docker compose config --quiet && echo "Valid!" || echo "Invalid!"

Force Recreate

docker compose up -d --force-recreate

Database Won't Start (Data Corruption)

Check Database Logs

docker compose logs db

Backup and Reset

# Try to backup first
docker compose exec db pg_dumpall -U postgres > emergency_backup.sql 2>/dev/null || true

# Reset database
docker compose down -v
docker compose up -d

# Restore from backup if available
docker compose exec -T db psql -U postgres < backup.sql

Environment Variables Not Working

Check Variable Values

docker compose config
docker compose exec app env
docker compose exec app printenv DATABASE_URL

Check .env File Parsing

docker compose config --format json | jq '.services.app.environment'

Variable Expansion Issues

# Make sure to use correct syntax
environment:
  - VAR=${VAR:-default}  # Works
  - VAR=$VAR             # May not work

Health Check Failing

Check Health Status

docker inspect --format='{{json .State.Health}}' containername | jq

Run Health Check Manually

docker compose exec app curl -f http://localhost:3000/health

Check Health Check Config

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 40s  # Time before first check

Memory/CPU Issues

View Resource Usage

docker stats
docker stats --no-stream

Set Limits

services:
  app:
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 512M

Check for Memory Leaks

docker compose exec app top
docker compose exec app ps aux

Debugging Commands Summary

# Shell into running container
docker compose exec app sh

# Shell into failed container
docker compose run --entrypoint sh app

# View resource usage
docker stats

# View container processes
docker compose top

# Copy files from container
docker compose cp app:/app/logs ./logs

# View real-time events
docker events

# Inspect container
docker inspect containername

# View container logs with timestamps
docker compose logs -t servicename

Quick Diagnostic Checklist

1. Check if container is running: docker compose ps 2. Check logs: docker compose logs servicename 3. Check network: docker network ls 4. Check volumes: docker volume ls 5. Check resources: docker stats 6. Validate config: docker compose config 7. Check disk space: docker system df

Common Error Messages