Docker Ai Ml

Docker AI/ML — AI 工作负载

Guidance for running AI models and agents on Docker.

When to Use

ALWAYS use this skill when the user mentions:

• "docker ai", "docker model", "model runner"
• "本地 LLM", "local model", "offline model"
• "docker agent", "AI agent"
• "GPU docker", "nvidia docker"
• "sandbox", "沙箱"
• "LLM 部署", "模型部署"

Docker Model Runner

# List available models
docker model ls

# Pull a model
docker model pull ai/gemma3
docker model pull ai/qwen2.5

# Run inference
docker model run ai/gemma3 "Explain Docker in one sentence"

# Interactive chat
docker model run -i ai/gemma3

# API server mode
docker model serve ai/gemma3 --port 11434
# → access at http://localhost:11434/v1

Install Skills for AI Coding Assistants

# Install skills for Claude Code
docker model skills --claude

# For Codex CLI
docker model skills --codex

GPU Configuration

# Install NVIDIA Container Toolkit (once)
# Ubuntu/Debian:
sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

# Run with GPU access
docker run --gpus all -it nvidia/cuda:12.6-base nvidia-smi

# Specific GPU
docker run --gpus device=0 -it nvidia/cuda:12.6-base

# GPU + Model Runner
docker model run --gpus all ai/llama3.2

Docker Agent

# agent.yaml — Define a multi-agent system
name: my-coding-agent
model:
  provider: openai
  model: gpt-4o
tools:
  - write_file
  - execute_command
  - web_search
sandbox:
  image: node:22-alpine
  workdir: /workspace

# Run agent with Docker
docker agent run -f agent.yaml "Fix the bug in src/app.js"

Docker Sandbox

# Isolated execution environment for AI agents
sandbox:
  image: python:3.12-slim
  network: none          # No network access
  read_only: true        # Read-only filesystem
  memory: 512m
  cpus: 1

Local LLM Stack

# Ollama (open-source) as alternative
docker run -d --name ollama \
  -p 11434:11434 \
  -v ollama_data:/root/.ollama \
  --gpus all \
  ollama/ollama

# Download and run model
docker exec ollama ollama pull llama3.2
docker exec ollama ollama run llama3.2 "Hello"

Workflow — 推荐使用流程

Step 1: 选择推理方案: Docker Model Runner (docker model) 优先，Ollama 备选 Step 2: 检查硬件: GPU 需 Linux + nvidia-container-toolkit；CPU 推理需 8-32GB RAM Step 3: 拉取模型: docker model pull <model> 或 ollama pull <model> Step 4: 启动服务: docker model run 或 ollama serve + 客户端 Step 5: 验证: 发送测试 prompt 确认模型正常响应

Gotchas — Common Pitfalls

• GPU not available: Docker Desktop Mac/Windows has limited GPU passthrough. → Recovery: Full GPU support requires Linux host + nvidia-container-toolkit; for Mac, use docker model which can offload inference.
• Model download size: LLMs are large (2-70 GB). → Recovery: docker model pull caches models in ~/.docker/models; check df -h before pulling; use quantized models (Q4/Q8) for smaller footprint.
• Memory limits: LLM inference needs significant RAM (8-32 GB typical). → Recovery: docker run --memory=16g model-runner; check RAM: free -h; use smaller models (7B instead of 70B) if limited.
• Model Runner vs Ollama: Docker Model Runner is Docker-native. Ollama has larger ecosystem. → Recovery: Use docker model for Docker integration; use Ollama for broader model selection; both can coexist.

Boundary — 能力边界（适用与不适用场景）

When NOT to Use This Skill

Security & Stability

• Run model containers with --read-only when possible — models are read-only weights.
• Limit network access for model containers. Local inference doesn't need internet.
• GPU passthrough gives the container direct hardware access. Combine with --security-opt for isolation.
• Monitor GPU memory usage: nvidia-smi for utilization.

📚 官方文档参考

🧭 Docker Skills Journey

📍 You are here: `docker-ai-ml` — AI/ML 工作负载

← Prev: docker-testcontainers — Integration testing

FAQ

Q1: 如何快速上手此技能？ A: 参考上方的快速开始章节，按步骤操作即可。

Q2: 遇到版本不兼容问题怎么办？ A: 检查依赖版本，使用 lock 文件锁定，参考常见陷阱章节。

Q3: 如何在生产环境使用？ A: 参考最佳实践章节，确保配置正确，做好监控和日志。

Q4: 性能如何优化？ A: 参考性能优化相关文档，使用缓存、索引等手段。

Q5: 如何贡献或反馈问题？ A: 在 GitHub 仓库提交 Issue 或 Pull Request。

Q6: 是否支持中文？ A: 支持中文文档和中文注释，详见国内适配章节。

Run Ollama with Docker + GPU

# Start Ollama with GPU support
docker run -d --name ollama   --gpus all   -p 11434:11434   -v ollama_data:/root/.ollama   ollama/ollama

# Pull and run a model
docker exec ollama ollama pull llama3.2
docker exec -it ollama ollama run llama3.2

# API access
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Explain Docker in one sentence"
}'

Docker Model Runner

# List available models
docker model ls

# Pull common models
docker model pull ai/gemma3
docker model pull ai/qwen2.5

# Interactive chat
docker model run -i ai/gemma3

# One-shot
docker model run ai/gemma3 "What is containerization?"

# Serve as API
docker model serve ai/gemma3 --port 11434
# → http://localhost:11434/v1/chat/completions

# Install skills for AI coding tools
docker model skills --claude    # Claude Code
docker model skills --codex     # OpenAI Codex CLI

GPU Setup for Docker

NVIDIA Container Toolkit

# Ubuntu/Debian
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list |   sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' |   sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

# Verify
docker run --rm --gpus all nvidia/cuda:12.6-base nvidia-smi

Platform Support