Developing With Docker

Developing With Docker

Overview

Provide deterministic, debugging-first guidance for Docker CLI and Compose, focusing on root causes that vary by platform and runtime. Prefer concise, actionable checks and commands, and reference the corpus for deeper explanations.

Quick Start

• Load the split references for deep dives:
• references/guide-foundations.md
• references/guide-installation-connectivity.md
• references/guide-cli-debugging.md
• references/guide-advanced-debugging.md
• references/guide-networking-compose-ops.md
• Use the workflow below for most troubleshooting requests; branch into the relevant section.

Example Requests

• "Containers can't reach my host database on Linux."
• "Docker Desktop on Mac can't access container IPs."
• "My bind mount is slow on Windows with WSL2."
• "Compose service keeps restarting after depends_on."

Debugging Workflow (Default)

1. Identify platform and runtime

• Ask for OS, Docker Desktop vs Rancher Desktop, and runtime backend (dockerd vs containerd/nerdctl).
• Clarify where the daemon runs (native Linux vs VM/WSL2) before suggesting network or file fixes.

2. Validate daemon and context

• Check docker context ls and current context.
• If CLI hangs, suspect dockerd API availability vs containerd still running.

3. Inspect container state

• Use targeted docker inspect --format queries for exit codes, mounts, log path, and PID.

4. Check logs and signals

• Confirm PID 1 behavior and signal handling; suggest exec in entrypoint scripts.
• Address stdout buffering or logs written to files instead of stdout/stderr.

5. Branch by symptom:

• Connectivity: port bindings, host.docker.internal, localhost vs 0.0.0.0, Desktop proxying.
• Volumes/permissions: UID/GID mismatch, rootless constraints, bind mount performance.
• Compose: depends_on readiness, env precedence, orphaned services.

Validation Checklist

• [ ] Verified active context with docker context ls
• [ ] Confirmed container state via docker inspect --format (exit code, mounts, PID)
• [ ] Checked logs for stdout/stderr buffering issues
• [ ] Verified port bindings and host reachability for the platform
• [ ] Confirmed bind mount performance path (VirtioFS/WSL2 path)

Findings Template

• Platform/runtime:
• Active context:
• Container state (exit code, PID):
• Networking diagnosis:
• Volume/permissions diagnosis:
• Recommended next command:

Example Output:

• Platform/runtime: macOS, Docker Desktop 4.27
• Active context: default
• Container state (exit code, PID): exit 137, PID 4021
• Networking diagnosis: port bound to 127.0.0.1, not reachable externally
• Volume/permissions diagnosis: VirtioFS enabled, no UID mismatch
• Recommended next command: docker compose logs --tail 50 api

Core Capabilities

• Explain Docker architecture (CLI, dockerd, containerd, runc, shim) and why it matters for debugging.
• Distinguish Linux-native behavior from macOS/Windows VM and WSL2 boundaries.
• Provide reliable CLI/Compose command recipes for state inspection and debugging.
• Diagnose performance issues tied to file sharing (VirtioFS, 9P) and build context size.
• Apply pragmatic networking guidance for host-to-container and container-to-host access.

Usage Notes

• Favor deterministic checks over trial-and-error. Explain "why" briefly when it helps avoid repeat mistakes.
• If the user mentions Docker Desktop versions, Rancher Desktop settings, or WSL2 paths, align advice to those specifics.
• When giving commands, prefer minimal, copy-pasteable sequences; avoid long scripts unless necessary.

When Not to Use

• Do not use for general Kubernetes orchestration guidance unless the issue is specifically Docker Desktop/Rancher Desktop related.
• Do not use for container security hardening beyond local development troubleshooting.

Reference Material

• Use the split reference set for deep dives and platform-specific behavior:
• references/guide-foundations.md
• references/guide-installation-connectivity.md
• references/guide-cli-debugging.md
• references/guide-advanced-debugging.md
• references/guide-networking-compose-ops.md
• For migration from Docker Desktop to Rancher Desktop:
• references/guide-rancher-migration.md

Advanced Container Debugging

Contents

• 5. Debugging Containers Like a Pro
• 5.5 Debug Image Pattern

5. Debugging Containers Like a Pro

Summary: Distroless debugging, docker debug, sidecar pattern, and kubectl debug.

When standard logs fail, developers need to get inside the container.

5.1 Interactive Shells and the "Distroless" Challenge

The standard debugging approach is docker exec -it <container> /bin/sh. This spawns a secondary process inside the running container, allowing exploration of the filesystem.

However, modern security best practices advocate for "distroless" images - minimal images that contain the application binary but no shell, no package manager, and no debug tools. docker exec fails on these images because there is no /bin/sh to execute.

5.2 Docker Debug (Desktop Feature)

Introduced in Docker Desktop 4.27+, the docker debug command solves the distroless problem. It functions by attaching a "toolbox" container to the target container's namespaces.

Mechanism: It mounts a set of statically linked tools (curl, vim, htop, netstat) into the target container.

Usage:

docker debug <container_name>

This drops the user into a shell with these tools available, even if the underlying image is effectively empty.

5.3 The Sidecar Pattern (Universal/Linux)

For users without Docker Desktop (e.g., on Linux CI servers or using Rancher Desktop's nerdctl), the "Sidecar Pattern" replicates the functionality of docker debug using native primitives.

The concept: Launch a temporary container equipped with tools (like nicolaka/netshoot or alpine) and instruct it to share the PID and Network namespaces of the distressed container.

The command:

docker run -it --rm \
  --pid=container:<target_container_id> \
  --net=container:<target_container_id> \
  --cap-add=SYS_ADMIN \
  nicolaka/netshoot

What this enables:

• Process debugging: Running ps aux in the sidecar shows the processes of the target container. You can run strace -p <pid> to trace system calls of a process in the target.
• Network debugging: Since they share the network stack, localhost in the sidecar is localhost in the target. You can use tcpdump or curl localhost:8080 to diagnose if the app is listening, bypassing any external bridge issues.

5.4 Ephemeral Containers in Kubernetes

For developers using Rancher Desktop's Kubernetes features, kubectl debug is the equivalent of the sidecar pattern. It injects an ephemeral container into a running Pod.

Command:

kubectl debug -it <pod_name> --image=busybox --target=<container_name>

Shared namespaces: By default, it might not share the PID namespace unless shareProcessNamespace: true is set in the Pod spec, which is a key difference from the docker run --pid approach.

5.5 Debug Image Pattern

Summary: Add tooling without changing the production image.

# Dockerfile.debug
FROM myapp:latest
RUN apk add --no-cache curl bind-tools netcat-openbsd tcpdump strace
ENTRYPOINT ["/bin/sh"]

docker build -f Dockerfile.debug -t myapp:debug .
docker run -it --rm myapp:debug

Core Docker CLI Debugging

Contents

• 4. Core Docker CLI Workflow (With a Debugger's Mindset)
• 4.4 Exit Codes Quick Map
• 4.5 Build Diagnostics Essentials

4. Core Docker CLI Workflow (With a Debugger's Mindset)

Summary: Container lifecycle, inspect templates, and log diagnostics.

To debug effectively, one must understand the state machine of a container and the signals that drive its lifecycle.

4.1 The Container Lifecycle: Signals and Exit Codes

A container is simply a process wrapper. When you stop a container, you are sending a UNIX signal to the process with PID 1 inside that namespace.

The shutdown sequence:

• docker stop: Sends SIGTERM. The application receives this and should begin a graceful shutdown (closing DB connections, flushing logs).
• Grace period: Docker waits (default 10 seconds).
• docker kill: If the process is still running, Docker sends SIGKILL, terminating it immediately without cleanup.

Debugging insight: If a container always takes exactly 10 seconds to stop, the application is likely ignoring SIGTERM. This often happens when the application is not PID 1.

Scenario: An entrypoint script like:

#!/bin/sh
./start-app.sh

runs the app as a subprocess. The shell receives the signal but does not forward it.

Fix: Use exec in shell scripts:

exec ./start-app.sh

This replaces the shell process with the application process, ensuring it becomes PID 1 and receives signals correctly.

4.2 Inspecting State with Go Templates

docker inspect is the ultimate source of truth, returning a massive JSON object with the container's configuration and runtime state. Browsing this raw JSON is inefficient. The --format flag, utilizing Go templates, allows for surgical data extraction.

Table 1: Essential Docker Inspect Formats for Debugging

• Check exit code: docker inspect --format='{{.State.ExitCode}}' <id>
• Use to confirm crash vs clean exit.
• Find IP address: docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <id>
• Use for inter-container connectivity checks.
• Verify log path: docker inspect --format='{{.LogPath}}' <id>
• Use to locate JSON logs on the host and check rotation or corruption issues.
• Check mounts: docker inspect -f '{{json.Mounts}}' <id>
• Use to confirm host-to-container mappings and spot typos.
• Get PID on host: docker inspect --format '{{.State.Pid}}' <id>
• Use host tools like strace or jstack against the containerized process (Linux only).

4.3 Logging: Where Data Goes to Die

docker logs captures STDOUT and STDERR. A common issue is "The container crashed, but the logs are empty."

Root causes:

• Buffering: Languages like Python and Node.js buffer STDOUT by default. If the app crashes before the buffer flushes, the logs are lost.
• Fix: Set PYTHONUNBUFFERED=1 in the Dockerfile or environment.
• Wrong output stream: The application writes to a file (e.g., /var/log/nginx/access.log) instead of STDOUT.
• Fix: Symlink the internal log files to /dev/stdout and /dev/stderr. This is a standard pattern in official images like Nginx.

4.4 Exit Codes Quick Map

• 0: Clean exit
• 1: Application error
• 126: Command not executable
• 127: Command not found
• 137: OOMKilled (often SIGKILL)

4.5 Build Diagnostics Essentials

docker build --progress=plain --no-cache -t myapp:debug .
docker build --target builder -t myapp-builder .
docker buildx build --platform linux/amd64,linux/arm64 -t myapp:multi .

Docker Foundations and Desktop Landscape

Contents

• 0. Introduction: The Debugger's Perspective
• 0.1 Who This Guide Is For
• 0.2 Quick Glossary
• 1. The Platform Landscape: Abstractions and Leaks
• 1.3 Platform-Specific Dev Gotchas
• 2. Desktop Tools: Docker Desktop vs. Rancher Desktop
• 2.3 Cross-Platform Cheatsheet

0. Introduction: The Debugger's Perspective

Summary: Debugging-first framing and why platform details matter.

This guide is designed for the professional software developer who has moved beyond the "Hello World" phase of containerization and is now navigating the complex, often frustrating reality of developing, testing, and debugging distributed applications in Docker. It assumes familiarity with basic concepts - images, containers, volumes - but recognizes that the "happy path" described in introductory tutorials rarely survives contact with enterprise networking, legacy codebases, and cross-platform inconsistencies.

The following analysis adopts a "debugging-first" methodology. Rather than merely listing commands, we deconstruct the underlying architectures of Docker Desktop and Rancher Desktop on Linux, macOS, and Windows to explain why failures occur. Whether it is a "connection refused" error on localhost, a permission denial on a bind-mounted volume, or a silent crash in a CI pipeline, the root cause invariably lies in the specific interaction between the container runtime and the host operating system's kernel abstractions. By mastering the command-line interface (CLI) and the internal mechanics of Docker Compose, developers can transition from trial-and-error troubleshooting to deterministic problem solving.

0.1 Who This Guide Is For

Summary: Developer and DevOps troubleshooting focus.

• Application developers building containerized apps locally
• Platform/DevOps engineers debugging stacks and CI
• Teams switching between Docker Desktop and Rancher Desktop

0.2 Quick Glossary

Summary: Common Docker terms in one place.

• Image vs container: Image is a read-only template; container is a running instance.
• Volume vs bind mount: Volume is Docker-managed storage; bind mount maps a host path into a container.
• Bridge vs host network: Bridge isolates containers with NAT; host shares the host network stack.
• Compose project: A set of services, networks, and volumes grouped under one project name.

1. The Platform Landscape: Abstractions and Leaks

Summary: Docker is a stack of layers; platform differences drive most debugging issues.

To debug Docker effectively, one must first understand that "Docker" is not a single technology but a stack of abstractions. The behavior of a container is dictated by the platform on which it runs. The notion that "containers run everywhere" is a useful simplification for deployment, but a dangerous fallacy for debugging. The host operating system introduces architectural boundaries that determine network performance, file system latency, and permission structures.

1.1 The Architecture of Abstraction: Client, Daemon, and Runtime

The Docker system follows a strict client-server model, a distinction that is critical when debugging connectivity issues. The docker CLI is merely a REST API client that transmits instructions to the Docker daemon (dockerd), the persistent process responsible for managing container objects.

In modern architectures, the daemon itself is an orchestrator rather than a monolithic executor. It delegates the heavy lifting to lower-level components:

• containerd: The industry-standard container runtime that manages the lifecycle of the container, including image transfer, storage, and execution. When a developer issues a docker pull, it is containerd that interacts with the registry.
• runc: A lightweight CLI tool for spawning and running containers according to the OCI (Open Container Initiative) specification.
• The shim: A process that sits between containerd and runc. It allows the runtime to remain active even if the daemon restarts, enabling "daemonless" containers and preserving state during updates.

For the developer, this separation implies that a frozen CLI does not necessarily mean the containers are dead; it may simply mean the API endpoint of dockerd is unresponsive, while containerd continues to manage the workloads.

1.2 The Operating System Divide

The most significant variable in the Docker equation is the host operating system. The "native" environment for Docker is Linux, where the daemon interacts directly with the kernel to create namespaces (for isolation) and cgroups (for resource limitation). On macOS and Windows, these features are absent, requiring virtualization layers that introduce specific debugging challenges.

Linux: The Native Host

On a Linux workstation, Docker is a process running on the host kernel.

• Networking: The docker0 bridge is a real network interface on the host. Containers can be accessed directly via their bridge IP addresses (e.g., 172.17.0.2) from the host.
• Storage: Bind mounts map a host directory to a container directory using native kernel features. Performance is practically indistinguishable from local disk access.
• Permissions: This is the primary pain point. Because the container shares the host kernel, a process running as root (UID 0) inside the container is effectively root on the host filesystem (within the mount). Conversely, files created by a containerized process usually inherit the UID of that process, often leading to files on the host that the developer (running as UID 1000) cannot modify or delete.

macOS: The Virtualized Host

macOS is UNIX-based but lacks the Linux kernel primitives required for containers. Consequently, Docker Desktop and Rancher Desktop spin up a lightweight Linux VM to host the daemon.

• The abstraction leak: When a user runs docker run on macOS, the CLI talks to the daemon inside this hidden VM. The containers live inside the VM, not on the Mac itself.
• Networking: There is no direct route from the macOS host to the container network. You cannot ping a container's IP address from the Terminal. Docker Desktop uses a user-space proxy (VPNKit) to forward traffic from localhost ports on the Mac to the container ports in the VM. This explains why standard network debugging tools like nmap or ping behave differently on Mac versus Linux.
• Filesystem penalty: Bind-mounting a folder involves crossing the VM boundary. Historically, this used osxfs or gRPC FUSE, mechanisms that introduced significant latency for I/O-heavy workloads (like npm install or massive PHP codebases). The modern standard, VirtioFS, leverages the Apple Virtualization Framework to map memory directly, improving file operation speeds by up to 98 percent compared to legacy solutions.

Windows: WSL 2 Integration

Legacy Docker on Windows used a Hyper-V VM, which suffered from similar I/O issues as macOS. Modern setups utilize the Windows Subsystem for Linux version 2 (WSL 2).

• Architecture: WSL 2 is a lightweight utility VM running a real Linux kernel. Docker Desktop integrates deeply with it, placing the daemon inside the WSL 2 context.
• The "9P" problem: Files stored inside the WSL 2 filesystem (e.g., \\wsl$\Ubuntu\home\project) are accessed at native speeds. However, files mounted from the Windows NTFS host (e.g., C:\Users\Project) must cross the virtualization boundary using the 9P protocol. This acts as a massive bottleneck. The debugging implication is clear: if an application is slow on Windows, verify whether the source code resides in the Linux filesystem or the Windows filesystem.

1.3 Platform-Specific Dev Gotchas

Summary: File watching, path performance, and host networking limits.

• macOS: inotify events do not propagate reliably from host to VM; use polling-based watchers or keep hot-reload-sensitive paths inside the VM. VirtioFS improves bind mount performance.
• Windows: store projects inside the WSL filesystem for speed and reliable file events; /mnt/c mounts are slower. Configure git to use LF line endings for Linux containers.
• Linux: rootless mode improves security but can limit host networking and some features on older kernels.
• macOS/Windows: host networking is not supported because containers run inside a VM.

2. Desktop Tools: Docker Desktop vs. Rancher Desktop

Summary: Desktop tooling affects runtime choice, DNS behavior, and filesystem performance.

The choice of desktop tool defines the developer experience (DX), influencing everything from Kubernetes integration to specific network quirks.

2.1 Docker Desktop

Docker Desktop remains the default for many due to its polished ergonomics and integrated tooling.

• Runtime: It strictly uses the Moby (dockerd) engine.
• File sharing: It offers the most mature implementation of VirtioFS on macOS and seamless WSL 2 integration on Windows.
• Networking magic: It provides automatic DNS resolution for host.docker.internal across all platforms, simplifying container-to-host communication.
• Licensing: It requires a paid subscription for commercial use in larger organizations, which has driven the adoption of alternatives.

2.2 Rancher Desktop

Rancher Desktop is an open-source alternative that prioritizes Kubernetes management but also provides full Docker CLI compatibility.

• Runtime flexibility: A key differentiator is the ability to choose the container runtime. Developers can select dockerd (Moby), which allows the use of the standard docker CLI, or containerd, which uses nerdctl.
• nerdctl vs docker: If using the containerd backend, the nerdctl tool is CLI-compatible with Docker but supports advanced features like lazy-pulling (starting containers before the full image is downloaded) and IPFS-based image distribution.
• Networking constraints: Historically, Rancher Desktop struggled with the seamless DNS resolution provided by Docker Desktop. While host.docker.internal is supported in newer versions (v1.1.0+), it relies on specific configurations (like the host-gateway mapping) and may require manual firewall rules on Windows to allow traffic from the WSL interface.
• Virtualization: On macOS, Rancher allows users to choose between QEMU (legacy, slower) and VZ (Apple Virtualization Framework, faster). Selecting VZ is mandatory to enable VirtioFS for performant bind mounts.

2.3 Cross-Platform Cheatsheet

Summary: High-level differences for debugging.

Installation, Permissions, and Connectivity

Contents

• 3. Installation, Permissions, and Connectivity
• Socket Locations (Desktop and Rancher)
• Context and Environment Overrides
• Credential Helpers
• 3.4 Rootless Docker Setup (Linux)
• 3.5 Sanity Checks Before Debugging

3. Installation, Permissions, and Connectivity

Summary: Socket permissions, contexts, and daemon config conflicts.

A broken environment is the first hurdle. Issues typically manifest as "permission denied" on the socket or startup failures due to configuration conflicts.

Socket Locations (Desktop and Rancher)

• Linux: /var/run/docker.sock
• macOS (Docker Desktop): ~/.docker/run/docker.sock (often symlinked to /var/run/docker.sock)
• macOS (Rancher Desktop, moby): ~/.rd/docker.sock
• Windows (Docker Desktop): npipe:////./pipe/docker_engine or WSL socket
• Windows (Rancher Desktop): npipe:////./pipe/rancher_desktop or WSL socket

Context and Environment Overrides

Priority order (highest to lowest): 1. --host flag 2. DOCKER_HOST 3. DOCKER_CONTEXT 4. Active context (docker context use)

Credential Helpers

• macOS: osxkeychain
• Windows: wincred
• Linux: secretservice or pass

3.1 The Docker Socket and Security Groups

On Linux (and Linux-based CI/CD environments), the Docker socket (/var/run/docker.sock) is the gatekeeper. Access to this socket is functionally equivalent to root access on the host, as it allows a user to mount the host's root filesystem into a privileged container.

Symptom: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock

Diagnosis: The socket file is owned by the root user and the docker group. The user attempting the command is not a member of this group.

Resolution:

• Group membership: Add the current user to the docker group: sudo usermod -aG docker $USER.
• Session refresh: This is the most missed step. The user must log out and back in, or run newgrp docker, for the group membership to apply to the current shell session.
• Security warning: Membership in the docker group is effectively root access because the daemon runs as root.
• Rootless Docker: For environments requiring strict security (e.g., shared dev servers), "Rootless Docker" runs the daemon inside a user namespace. This isolates the daemon entirely from the host's root user. While secure, it complicates bind mounts because the daemon cannot read files owned by other users on the host without intricate UID mapping.

Systemd checks:

sudo systemctl status docker
sudo journalctl -u docker.service -f

3.2 Managing Environments with Docker Contexts

Developers often juggle multiple environments: a local instance, a remote staging server, and perhaps a cloud-based build server. The DOCKER_HOST environment variable was the traditional way to switch targets, but Docker Contexts provide a superior, stateful mechanism.

The debugging advantage: Contexts prevent the "phantom container" problem, where a developer mistakenly debugs the local environment while thinking they are working on remote staging.

Workflow:

# Define a context for a remote server via SSH
docker context create staging \
  --docker "host=ssh://deploy-user@192.168.1.50" \
  --description "Staging Server - Do not touch DB"

# Switch context
docker context use staging

# Verify active context
docker context ls
# OUTPUT:
# NAME      TYPE    DESCRIPTION                     DOCKER ENDPOINT
# default   moby    Current DOCKER_HOST based...    unix:///var/run/docker.sock
# staging * moby    Staging Server...               ssh://deploy-user@192.168.1.50

When a developer reports that "Docker is showing containers that shouldn't be there," the first debugging step is docker context ls to verify which daemon is actually receiving the commands.

3.3 Daemon Configuration Conflicts

Both Docker Desktop and Rancher Desktop manage the daemon's configuration dynamically. However, power users often modify /etc/docker/daemon.json (Linux) or ~/.docker/daemon.json (Mac/Windows) to add insecure registries or mirror settings.

Critical error: "Unable to configure the Docker daemon... directives specified both as a flag and in the configuration file".

Cause: This occurs when the desktop application launches dockerd with command-line flags (e.g., --hosts) that conflict with keys defined in daemon.json.

Fix: On Desktop platforms, avoid editing daemon.json directly if the UI provides a setting for it (e.g., "Docker Engine" tab in settings). If manual editing is necessary, ensure the keys do not overlap with the arguments the desktop application passes to the daemon during its initialization sequence.

3.4 Rootless Docker Setup (Linux)

Summary: Safer daemon, with feature trade-offs.

dockerd-rootless-setuptool.sh install
export DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock

Prereqs: subordinate UID/GID ranges and newuidmap/newgidmap.

Limitations: no privileged mode, reduced networking features on older kernels, and stricter bind-mount access.

3.5 Sanity Checks Before Debugging

docker context show
docker context inspect $(docker context show) | grep -i endpoint
docker version
docker info
env | grep DOCKER

Networking, Compose, Volumes, and Operations

Contents

• 6. Ports and Connectivity: The Number One Dev Pain Point
• 7. Volumes, Bind Mounts, and Permissions
• 8. Docker Networking Deep Dive
• 9. Docker Compose: Building Reliable Local Stacks
• 10. Debugging Compose Stacks
• 11. Image Management for Dev Velocity
• 12. Troubleshooting Cookbook
• 13. Handy Reference
• 14. Appendix: Permissions Deep Dive

6. Ports and Connectivity: The Number One Dev Pain Point

Summary: Port binding, host access patterns, and LAN routing rules.

Networking issues are the most frequent source of developer friction.

6.1 The Binding Paradox: 127.0.0.1 vs 0.0.0.0

A misunderstanding of interfaces leads to "I can't access my app" or "I accidentally exposed my database to the internet."

• 0.0.0.0 (all interfaces): When you run docker run -p 8080:80, Docker binds port 8080 on all available interfaces on the host. This means the service is accessible via localhost, the LAN IP (e.g., 192.168.1.5), and potentially the public WAN IP.
• 127.0.0.1 (loopback only): Running docker run -p 127.0.0.1:8080:80 restricts access strictly to the host machine. This is a security best practice for local development databases that should not be visible to other devices on the WiFi network.

6.2 Host-to-Container and Container-to-Host

Scenario: A containerized backend needs to connect to a non-containerized database running on the host machine.

• Linux: The container cannot reach localhost of the host because localhost inside the container refers to itself. The developer must use --add-host=host.docker.internal:host-gateway. This adds an entry to /etc/hosts inside the container, resolving host.docker.internal to the gateway IP of the Docker bridge (usually 172.17.0.1).
• Docker Desktop (Mac/Windows): The DNS name host.docker.internal is configured automatically and resolves correctly to the host's internal IP.
• Rancher Desktop: Support for host.docker.internal is available but relies on the host-gateway mechanism. On Windows, Windows Defender Firewall often blocks traffic arriving from the WSL 2 network adapter. Users may need to run a PowerShell command to create an "Allow" rule for the vEthernet (WSL) interface.

6.3 LAN Connectivity and Bridging

If a developer needs to access a container from a mobile device on the same LAN:

• Bind to 0.0.0.0: Ensure the port mapping is not restricted to localhost.
• Bridge limitations (Mac): On macOS, you cannot route traffic directly to the container's internal IP (e.g., 172.17.0.x). The VM isolation prevents this. You must access it via the host's IP and the mapped port.
• Macvlan: For advanced use cases where a container needs to appear as a physical device on the network with its own MAC address, the macvlan driver can be used. However, a kernel limitation prevents the host from communicating with its own macvlan containers directly. A secondary "shim" bridge is often required to bypass this restriction.

7. Volumes, Bind Mounts, and Permissions

Summary: File sharing performance and UID/GID mismatches.

Data persistence brings us to the most complex intersection of Docker and the OS: filesystem permissions.

7.1 Performance: Synchronization Mechanics

VirtioFS (macOS): The introduction of VirtioFS has been a game-changer for Docker on macOS. Previously, osxfs had to translate every file system call between macOS (HFS+/APFS) and Linux (ext4), causing massive overhead for metadata operations (like git status or npm install). VirtioFS allows the Linux VM to access the macOS file descriptors more directly.

Troubleshooting: If disk performance drops or "dubious ownership" errors appear in git, verify that VirtioFS is enabled in Docker Desktop settings. In some edge cases, switching back to gRPC FUSE resolves specific permission locking issues, though at a performance cost.

7.2 The UID/GID Mismatch (Linux)

On Linux, there is no VM to mask permission issues.

The problem: A container running as root writes a file to a bind mount. On the host, that file is owned by root. The developer (UID 1000) cannot edit or delete it.

The inverse: A container running as a non-root user (e.g., node, UID 1000) tries to write to a host directory. If the host directory is owned by root, the container gets Permission Denied.

Table 2: Strategies for Handling Permissions on Linux

• Runtime user mapping: docker run -u $(id -u):$(id -g) ...
• Pros: Simplest fix. Matches container user to host user.
• Cons: Requires the container image to support running as an arbitrary UID (some apps crash if they cannot write to /home).
• Entrypoint chown: Script runs as root, chowns data dir, then drops privileges (gosu).
• Pros: Guarantees correct permissions inside container.
• Cons: Can be very slow on large volumes (recursively changing permissions on startup).
• User namespaces: userns-remap in daemon config.
• Pros: Secure. Maps container root to a non-privileged host user.
• Cons: Complex to configure; makes sharing bind mounts with the host user difficult.
• Dockerfile user: RUN useradd -u 1000... USER 1000
• Pros: Hardcodes the ID into the image.
• Cons: Brittle; assumes every developer on the team uses UID 1000.

7.3 Windows Permissions

On Windows, the filesystem permission model (ACLs) is fundamentally different from Linux (chmod). Docker Desktop handles this translation automatically for mounts from the C: drive, generally making all files executable and owned by root inside the container.

Rancher Desktop caveat: Rancher Desktop's handling of permissions on Windows can be stricter. If using the WSL 2 backend, it is highly recommended to store project code inside the WSL 2 filesystem rather than on the Windows C: drive. This bypasses the permission translation layer entirely and offers native Linux performance.

8. Docker Networking Deep Dive

Summary: Embedded DNS behavior and VPN-related DNS failures.

While basic port mapping covers 90 percent of use cases, debugging requires understanding the internal DNS and packet flow.

8.1 The Embedded DNS Server (127.0.0.11)

Every Docker container has a resolv.conf that points to 127.0.0.11. This is Docker's embedded DNS server.

Function: It intercepts DNS queries. If the query matches a container name in the same network (e.g., db), it resolves it to the container's internal IP. If not, it forwards the query to the host's configured DNS resolvers.

Debugging: When service discovery fails (container A cannot ping container B), the first step is to docker exec into container A and check /etc/resolv.conf.

VPN issues: Corporate VPNs often push DNS settings that are only valid within the VPN tunnel. If Docker fails to inherit these, or if the VPN client blocks split tunneling, containers effectively lose internet access. Docker Desktop attempts to mitigate this with custom networking implementations, but manual DNS overrides (setting "dns": ["10.x.x.x"] in daemon.json) are a common workaround.

9. Docker Compose: Building Reliable Local Stacks

Summary: Readiness checks and environment precedence.

Docker Compose transforms individual container commands into a coherent infrastructure definition.

Project naming: Compose prefixes containers, networks, and volumes with the project name (directory name by default), and labels resources with com.docker.compose.project.

9.1 Service Dependency and Healthchecks

The depends_on directive in docker-compose.yml controls startup order, but by default, it only waits for the container to be "running," not "ready."

The problem: The web server starts, tries to connect to the database, and crashes because the database is still initializing its files.

The solution: Use the service_healthy condition.

services:
  db:
    image: postgres:15
    healthcheck:
      test:
      interval: 5s
      timeout: 5s
      retries: 5
  api:
    build: .
    depends_on:
      db:
        condition: service_healthy

This configuration forces the api service to wait until the db service passes its healthcheck.

9.2 Environment Variable Precedence

Few things cause more confusion than environment variables in Compose. The precedence order determines which value "wins" when a variable is defined in multiple places.

Precedence hierarchy (highest to lowest):

1. Command line: docker compose run -e DEBUG=1 2. Shell environment: Variables exported in the terminal (export DEBUG=1) run before docker compose up. 3. .env file: Variables defined in the .env file in the project root. These are used to substitute ${VARIABLES} inside the YAML file itself. 4. environment attribute: Variables defined explicitly in docker-compose.yml. 5. env_file attribute: Variables loaded from a file referenced in the YAML. 6. Dockerfile ENV: Default values baked into the image.

Debugging tip: Use docker compose config to print the final, resolved configuration. This reveals exactly which values are being injected into the containers.

10. Debugging Compose Stacks

Summary: Restart loops, orphan cleanup, and event streams.

When a stack behaves badly, docker compose logs is just the start.

10.1 Real-World Scenarios

• The "zombie" service: A service keeps restarting. docker compose ps shows status "Restarting". Use docker compose logs --tail 50 <service_name> to catch the immediate crash error.
• Orphaned containers: If you rename a service in the YAML, the old container might still be running. docker compose up --remove-orphans cleans up these ghostly remnants.
• Events stream: docker compose events --json provides a real-time stream of container events (start, stop, die, oom). This is invaluable for detecting if a container is being killed by the OOM (Out of Memory) killer silently.

10.2 Services Can't Talk to Each Other

Summary: Hostnames, ports, and network mismatches.

• Use service names on container ports (not host ports).
• Ensure both services are on the same network.

services:
  web:
    networks: [frontend, backend]
  db:
    networks: [backend]

10.3 Hot Reload and File Watching Failures

Summary: macOS/Windows VM file events often require polling.

• Enable polling in dev servers (webpack/vite/nodemon).
• Keep source code inside the WSL filesystem on Windows for reliable file events.
• Use named volumes for heavy-write directories (node_modules, .venv).

10.4 "It Worked Yesterday" Failures

Summary: Stale images, volumes, or networks.

docker compose build --no-cache
docker compose down -v --remove-orphans
docker network prune

11. Image Management for Dev Velocity

Summary: Build context control and multi-arch strategies.

11.1 Build Context and .dockerignore

A slow build often starts with "Sending build context to Docker daemon." If this takes seconds (or minutes), you are likely sending the entire node_modules or .git folder to the daemon context.

Fix: Create a .dockerignore file. Excluding node_modules, .git, and build artifacts significantly speeds up the build start time.

11.2 Multi-Architecture Builds

With the rise of Apple Silicon (ARM64), building images that run on both local MacBooks and x86_64 production servers is standard.

Buildx: The docker buildx command enables multi-platform builds.

docker buildx build --platform linux/amd64,linux/arm64 -t myimage .

Performance warning: Building an AMD64 image on an ARM64 Mac uses QEMU emulation, which is extremely slow for CPU-intensive tasks (like compilation).

Optimization: Use "native nodes." You can connect a remote AMD64 server to your local Docker instance via SSH and use it as a builder node in Buildx. This routes the AMD64 build steps to the native hardware while keeping the workflow local.

12. Troubleshooting Cookbook

Summary: Quick symptom-to-check mappings for common issues.

Symptom: Cannot connect to the Docker daemon

• Check: daemon running (systemd or Desktop)
• Check: socket exists and permissions
• Check: DOCKER_HOST/DOCKER_CONTEXT overrides

Symptom: Connection refused on localhost

• Check: Is the container running? (docker ps)
• Check: Is the port mapped to 127.0.0.1 or 0.0.0.0?
• Check: On Mac, are you using the mapped port? (Container IP is not reachable).

Symptom: Container exits immediately

• Check: docker logs, exit code, and entrypoint/CMD
• Check: missing binaries or wrong working directory

Symptom: Slow file operations

• Check: Are you using VirtioFS (Mac)?
• Check: Are you mounting from the Windows C: drive instead of the WSL 2 filesystem?

Symptom: DNS resolution fails in containers

• Check: /etc/resolv.conf in the container
• Check: container network membership

Symptom: Permission denied in volume

• Check: Are you on Linux?
• Fix: Use docker run -u $(id -u) or check the entrypoint script logic.

Symptom: Docker command hangs

• Check: Is the context set correctly? (docker context ls)
• Check: Is the daemon responsive? (Restart Docker Desktop).

Symptom: "No space left on device"

• Check: docker system df and prune unused images/volumes

Symptom: Compose service unhealthy or flapping

• Check: docker inspect health status/logs
• Fix: increase healthcheck start_period/timeout

13. Handy Reference

Summary: High-impact CLI commands for cleanup and inspection.

• Context and daemon checks:
• docker context show
• docker version
• docker info | head -20

• docker system prune -a --volumes: Nuclear option: Deletes all stopped containers, unused images, and volumes.
• docker stats --no-stream: Snapshot of CPU/RAM usage for all running containers.
• docker compose up -d --build --force-recreate: Forces a complete rebuild and restart of the stack.
• docker run --rm -it --entrypoint /bin/sh <image>: Overrides the default command to get a shell in an image that crashes immediately.
• docker buildx prune: Cleans up the BuildKit build cache (distinct from image pruning).

Safe vs aggressive cleanup:

docker container prune
docker image prune
docker volume prune
docker network prune

14. Appendix: Permissions Deep Dive

Summary: UID mapping and virtualization edge cases.

Linux Bind Mounts

The kernel maps the UID directly.

Scenario: Host user 1000 runs docker run -v $(pwd):/app... Process inside runs as root (UID 0).

Result: Files created in /app are owned by root. Host user cannot delete them.

Solution: Configure the container process to run as UID 1000.

macOS/Windows Bind Mounts

The file sharing system (VirtioFS/9P) acts as a proxy.

Scenario: Same command.

Result: Docker Desktop effectively "lies" to both sides. The container sees the files as owned by root. The host sees the files as owned by the user.

Edge case: If you need to chmod a file inside the container, this metadata change might not propagate to the host file system correctly, or might be ignored. This is a known limitation of the virtualization layer.

This report synthesizes official documentation and community troubleshooting patterns to provide a robust guide for developer workflows in 2025.

Rancher Desktop Migration Guide

Contents

• 1. Executive Summary
• 2. Runtime Choice: dockerd vs containerd
• 3. Pre-Migration Checklist
• 4. Migration Workflow
• 5. Context Management
• 6. Compose Migration Checks
• 7. Post-Migration Verification
• 8. Common Migration Issues
• 9. Rollback Strategy
• 10. Best Practices

1. Executive Summary

Summary: Rancher Desktop is a drop-in replacement if you choose dockerd (moby).

• Choose dockerd (moby) for zero workflow changes.
• containerd requires nerdctl and can change behavior.
• You can run Docker Desktop and Rancher Desktop side-by-side and switch contexts.

2. Runtime Choice: dockerd vs containerd

• dockerd (moby): use docker CLI, Compose works as-is.
• containerd: use nerdctl CLI, some flags differ.
• Switching runtimes does not share images/containers.

3. Pre-Migration Checklist

Inventory and backups:

docker ps -a --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"
docker images --format "{{.Repository}}:{{.Tag}}" > images-inventory.txt
docker volume ls
docker network ls --filter type=custom
docker compose ls

Backup volumes (example):

docker volume ls --format "{{.Name}}" | while read volume; do
  docker run --rm -v "$volume":/data -v "$PWD":/backup \
    alpine tar czf "/backup/${volume}.tar.gz" -C /data .
done

4. Migration Workflow

Clean migration (recommended):

1. Stop Docker Desktop. 2. Install Rancher Desktop. 3. Select dockerd (moby). 4. Restore images/volumes if needed. 5. Validate Compose projects.

Side-by-side:

1. Keep Docker Desktop installed. 2. Install Rancher Desktop. 3. Switch contexts as needed.

5. Context Management

docker context ls
docker context use rancher-desktop
docker context use desktop-linux

Check overrides:

env | grep DOCKER
unset DOCKER_HOST
unset DOCKER_CONTEXT

6. Compose Migration Checks

docker compose up -d
docker compose ps
docker compose logs -f
docker compose down

Verify bind mounts under shared directories and adjust host ports if conflicts exist.

7. Post-Migration Verification

docker version
docker info
docker run --rm hello-world

Test volume mounts and port mappings:

echo "test" > test.txt
docker run --rm -v "$PWD":/data alpine cat /data/test.txt
docker run -d -p 8080:80 --name test-nginx nginx
curl -I http://localhost:8080
docker rm -f test-nginx

8. Common Migration Issues

• Images/containers missing: export from Docker Desktop and load into Rancher Desktop.
• Volume data missing: backup and restore volumes per runtime.
• host.docker.internal: use host.rancher-desktop.internal if needed.
• WSL integration (Windows): enable in Rancher Desktop settings.

9. Rollback Strategy

docker context use desktop-linux
docker version

Restore images/volumes from backups if needed.

10. Best Practices

• Start with dockerd (moby).
• Keep a backup of volumes and images before switching.
• Use contexts to switch safely.
• Test a non-critical Compose project first.