Kubectl Debugging

kubectl debug - Interactive Kubernetes Debugging

Expert knowledge for debugging Kubernetes resources using kubectl debug - ephemeral containers, pod copies, and node access.

When to Use This Skill

Core Capabilities

kubectl debug automates common debugging tasks:

• Ephemeral Containers: Add debug containers to running pods without restart
• Pod Copying: Create modified copies for debugging (different images, commands)
• Node Debugging: Access node host namespaces and filesystem

Context Safety (CRITICAL)

Always specify `--context` explicitly in every kubectl command:

# CORRECT: Explicit context
kubectl --context=prod-cluster debug mypod -it --image=busybox

# WRONG: Relying on current context
kubectl debug mypod -it --image=busybox  # Which cluster?

Quick Reference

Add Ephemeral Debug Container

# Interactive debugging with busybox
kubectl --context=my-context debug mypod -it --image=busybox

# Target specific container's process namespace
kubectl --context=my-context debug mypod -it --image=busybox --target=mycontainer

# Use a specific debug profile
kubectl --context=my-context debug mypod -it --image=busybox --profile=netadmin

Copy Pod for Debugging

# Create debug copy
kubectl --context=my-context debug mypod -it --copy-to=mypod-debug --image=busybox

# Copy and change container image
kubectl --context=my-context debug mypod --copy-to=mypod-debug --set-image=app=busybox

# Copy and modify command
kubectl --context=my-context debug mypod -it --copy-to=mypod-debug --container=myapp -- sh

# Copy on same node
kubectl --context=my-context debug mypod -it --copy-to=mypod-debug --same-node --image=busybox

Debug Node

# Interactive node debugging (host namespaces, filesystem at /host)
kubectl --context=my-context debug node/mynode -it --image=busybox

# With sysadmin profile for full capabilities
kubectl --context=my-context debug node/mynode -it --image=ubuntu --profile=sysadmin

Debug Profiles

# Network debugging (tcpdump, netstat, ss)
kubectl --context=my-context debug mypod -it --image=nicolaka/netshoot --profile=netadmin

# System debugging (strace, perf)
kubectl --context=my-context debug mypod -it --image=ubuntu --profile=sysadmin

Common Debug Images

Debugging Patterns

Network Connectivity Issues

# Add netshoot container for network debugging
kubectl --context=my-context debug mypod -it \
  --image=nicolaka/netshoot \
  --profile=netadmin

# Inside container:
# - tcpdump -i any port 80
# - dig kubernetes.default
# - curl -v http://service:port
# - ss -tlnp
# - netstat -an

Application Crashes

# Copy pod with different entrypoint to inspect
kubectl --context=my-context debug mypod -it \
  --copy-to=mypod-debug \
  --container=app \
  -- sh

# Inside: check filesystem, env vars, config files

Process Inspection

# Target container's process namespace
kubectl --context=my-context debug mypod -it \
  --image=busybox \
  --target=mycontainer

# Inside: ps aux, /proc inspection

Node-Level Issues

# Debug node with host access
kubectl --context=my-context debug node/worker-1 -it \
  --image=ubuntu \
  --profile=sysadmin

# Inside:
# - Host filesystem at /host
# - chroot /host for full access
# - journalctl, systemctl, dmesg

Non-Destructive Debugging

# Create copy, keeping original running
kubectl --context=my-context debug mypod -it \
  --copy-to=mypod-debug \
  --same-node \
  --share-processes \
  --image=busybox

# Original pod continues serving traffic
# Debug copy shares storage if on same node

Key Options

Best Practices

1. Use appropriate profiles - Match capabilities to debugging needs 2. Prefer ephemeral containers - Less disruptive than pod copies 3. Use `--copy-to` for invasive debugging - Preserve original pod 4. Clean up debug pods - Delete copies after debugging 5. Use `--same-node` - For accessing shared storage/network conditions

Cleanup

# List debug pod copies
kubectl --context=my-context get pods | grep -E "debug|copy"

# Delete debug pods
kubectl --context=my-context delete pod mypod-debug

Requirements

• Kubernetes 1.23+ for ephemeral containers (stable)
• Kubernetes 1.25+ for debug profiles
• RBAC permissions for pods/ephemeralcontainers

For detailed option reference, examples, and troubleshooting patterns, see REFERENCE.md.

kubectl debug Reference

Comprehensive reference for debugging Kubernetes resources with kubectl debug.

Table of Contents

• Synopsis
• Complete Options Reference
• Ephemeral Containers
• Pod Copying
• Node Debugging
• Debug Profiles Deep Dive
• Debug Images
• Advanced Patterns
• Troubleshooting

---

Synopsis

kubectl debug (POD | TYPE[[.VERSION].GROUP]/NAME) [ -- COMMAND [args...] ]

The kubectl debug command provides automation for common debugging tasks:

• Pods: Add ephemeral containers or create modified copies
• Nodes: Create pods in host namespaces with filesystem access

---

Complete Options Reference

Debug-Specific Options

Global Options (Inherited)

---

Ephemeral Containers

Ephemeral containers are temporary containers added to running pods for debugging purposes.

How They Work

1. Added to running pod's spec without restart 2. Share pod's network, IPC, and optionally PID namespace 3. Cannot be removed once added (pod must be recreated) 4. Ideal for debugging without disrupting the application

Basic Usage

# Add ephemeral container with busybox
kubectl --context=my-context debug mypod -it --image=busybox

# Named debug container
kubectl --context=my-context debug mypod -it --image=busybox -c debugger

# With environment variables
kubectl --context=my-context debug mypod -it --image=busybox --env DEBUG=1

# Target specific container's processes
kubectl --context=my-context debug mypod -it --image=busybox --target=app

Process Namespace Sharing

With --target, the debug container shares the process namespace:

# Share process namespace with 'app' container
kubectl --context=my-context debug mypod -it --image=busybox --target=app

# Inside debug container:
# - ps aux shows processes from 'app' container
# - /proc/<pid>/* accessible for process inspection
# - strace, gdb can attach to processes

Viewing Ephemeral Containers

# Check ephemeral containers on a pod
kubectl --context=my-context get pod mypod -o jsonpath='{.spec.ephemeralContainers[*].name}'

# Describe pod to see ephemeral container status
kubectl --context=my-context describe pod mypod

# Get full pod spec
kubectl --context=my-context get pod mypod -o yaml | grep -A 20 ephemeralContainers

Limitations

• Cannot be removed (pod must be deleted/recreated)
• Cannot modify existing containers
• Non-root users in target may have limited capabilities
• Requires EphemeralContainers feature (stable in 1.23+)

---

Pod Copying

Create a copy of a pod with modifications for debugging.

Basic Copy

# Copy pod with debug container
kubectl --context=my-context debug mypod -it --copy-to=mypod-debug --image=busybox

# Copy keeps original running (non-destructive)
kubectl --context=my-context get pods
# NAME          READY   STATUS    RESTARTS   AGE
# mypod         1/1     Running   0          1h
# mypod-debug   2/2     Running   0          5s

Modify Container Images

# Change single container image
kubectl --context=my-context debug mypod --copy-to=mypod-debug \
  --set-image=app=myapp:debug

# Change multiple container images
kubectl --context=my-context debug mypod --copy-to=mypod-debug \
  --set-image=app=myapp:debug,sidecar=sidecar:debug

# Change all container images
kubectl --context=my-context debug mypod --copy-to=mypod-debug \
  --set-image='*=busybox'

Modify Container Command

# Override command for specific container
kubectl --context=my-context debug mypod -it --copy-to=mypod-debug \
  --container=app \
  -- sh

# Override with arguments only
kubectl --context=my-context debug mypod --copy-to=mypod-debug \
  --container=app \
  --arguments-only \
  -- --debug --verbose

Scheduling Options

# Schedule on same node (access same storage, network conditions)
kubectl --context=my-context debug mypod -it --copy-to=mypod-debug \
  --same-node \
  --image=busybox

# Replace original pod (destructive)
kubectl --context=my-context debug mypod -it --copy-to=mypod-debug \
  --replace \
  --image=busybox

Preserve or Remove Features

# Keep labels (for service inclusion)
kubectl --context=my-context debug mypod --copy-to=mypod-debug \
  --keep-labels \
  --image=busybox

# Skip init containers (faster startup)
kubectl --context=my-context debug mypod --copy-to=mypod-debug \
  --keep-init-containers=false \
  --image=busybox

# Keep probes (test health check behavior)
kubectl --context=my-context debug mypod --copy-to=mypod-debug \
  --keep-liveness \
  --keep-readiness \
  --image=busybox

# Disable process sharing
kubectl --context=my-context debug mypod --copy-to=mypod-debug \
  --share-processes=false \
  --image=busybox

---

Node Debugging

Create a pod in the node's host namespaces for node-level debugging.

Basic Node Debug

# Interactive node debugging
kubectl --context=my-context debug node/mynode -it --image=busybox

# With Ubuntu for more tools
kubectl --context=my-context debug node/mynode -it --image=ubuntu

What You Get

• Host Network Namespace: See host network interfaces
• Host PID Namespace: See host processes
• Host IPC Namespace: Access host IPC
• Host Filesystem: Mounted at /host

Common Node Debugging Tasks

# Inside node debug pod:

# Access host filesystem
ls /host
cat /host/etc/os-release
cat /host/var/log/syslog

# Full host access with chroot
chroot /host

# View host processes
ps aux

# Check node resources
cat /host/proc/meminfo
cat /host/proc/cpuinfo

# View system logs (after chroot)
journalctl -u kubelet
dmesg | tail -50

# Check systemd services
systemctl status kubelet
systemctl status containerd

Node Debug Pod Cleanup

# List node debug pods
kubectl --context=my-context get pods -A | grep node-debugger

# Delete node debug pod
kubectl --context=my-context delete pod node-debugger-mynode-xxxxx -n default

---

Debug Profiles Deep Dive

Debug profiles control security settings for debug containers.

Profile Comparison

Profile Use Cases

netadmin - Network Debugging

kubectl --context=my-context debug mypod -it \
  --image=nicolaka/netshoot \
  --profile=netadmin

# Enabled capabilities:
# - tcpdump, packet capture
# - iptables inspection
# - network namespace operations

sysadmin - System Debugging

kubectl --context=my-context debug mypod -it \
  --image=ubuntu \
  --profile=sysadmin

# Enabled capabilities:
# - strace, ptrace operations
# - system administration
# - full process inspection

restricted - High Security

kubectl --context=my-context debug mypod -it \
  --image=alpine \
  --profile=restricted

# Enforces:
# - Non-root user
# - Read-only root filesystem
# - Dropped capabilities

Custom Profile (--custom)

# Create custom container spec
cat > debug-container.yaml << 'EOF'
securityContext:
  capabilities:
    add:
    - NET_ADMIN
    - SYS_PTRACE
  runAsUser: 0
  runAsGroup: 0
resources:
  limits:
    memory: "256Mi"
    cpu: "500m"
EOF

# Use custom spec
kubectl --context=my-context debug mypod -it \
  --image=ubuntu \
  --custom=debug-container.yaml

---

Debug Images

Lightweight Images

Full-Featured Images

Specialized Images

Building Custom Debug Image

FROM alpine:3.19

# Common debugging tools
RUN apk add --no-cache \
    bash \
    curl \
    wget \
    jq \
    bind-tools \
    tcpdump \
    strace \
    netcat-openbsd \
    procps \
    htop \
    vim

CMD ["sleep", "infinity"]

---

Advanced Patterns

Debug Multiple Containers

# Add multiple debug containers
kubectl --context=my-context debug mypod -it --image=busybox -c debug1
kubectl --context=my-context debug mypod -it --image=nicolaka/netshoot -c debug2

# Each has access to pod's network

Automated Debug Scripts

#!/bin/bash
# debug-pod.sh - Automated pod debugging

CONTEXT="${1:?Usage: debug-pod.sh CONTEXT POD [NAMESPACE]}"
POD="${2:?Usage: debug-pod.sh CONTEXT POD [NAMESPACE]}"
NAMESPACE="${3:-default}"

# Create debug pod copy
kubectl --context="$CONTEXT" debug "$POD" \
  -n "$NAMESPACE" \
  --copy-to="${POD}-debug" \
  --same-node \
  --image=nicolaka/netshoot \
  --profile=netadmin \
  -- sleep infinity

echo "Debug pod created: ${POD}-debug"
echo "Connect with: kubectl --context=$CONTEXT exec -it ${POD}-debug -n $NAMESPACE -- bash"
echo "Cleanup with: kubectl --context=$CONTEXT delete pod ${POD}-debug -n $NAMESPACE"

Debug from File

# Debug pod defined in file
kubectl --context=my-context debug -f pod.yaml -it --image=busybox

# Useful for:
# - Debugging manifests before applying
# - Testing configurations
# - CI/CD pipeline debugging

Debug with Specific User

# Create custom spec for user
cat > debug-user.yaml << 'EOF'
securityContext:
  runAsUser: 1000
  runAsGroup: 1000
EOF

kubectl --context=my-context debug mypod -it \
  --image=alpine \
  --custom=debug-user.yaml

Capture Network Traffic

# Start debug container with netadmin profile
kubectl --context=my-context debug mypod -it \
  --image=nicolaka/netshoot \
  --profile=netadmin \
  -c tcpdump

# Inside container:
tcpdump -i any -w /tmp/capture.pcap port 80

# Copy capture out (from another terminal)
kubectl --context=my-context cp mypod:/tmp/capture.pcap ./capture.pcap -c tcpdump

Strace a Process

# Start debug with sysadmin profile, targeting container
kubectl --context=my-context debug mypod -it \
  --image=ubuntu \
  --profile=sysadmin \
  --target=app

# Inside container:
apt update && apt install -y strace
strace -p 1  # Trace main process

---

Troubleshooting

Error: Unable to Create Ephemeral Container

Symptom:

error: ephemeralcontainers are disabled for this cluster

Cause: EphemeralContainers feature not enabled or old Kubernetes version.

Solution:

• Upgrade to Kubernetes 1.23+ (ephemeral containers stable)
• Use --copy-to instead for pod copying approach

Error: Debug Container Does Not Have Required Permissions

Symptom:

Error: operation not permitted

Cause: Profile doesn't have required capabilities.

Solution:

# Use appropriate profile
kubectl --context=my-context debug mypod -it \
  --image=ubuntu \
  --profile=sysadmin  # or netadmin

Error: Target Container Not Found

Symptom:

error: container "xyz" not found in pod

Solution:

# List containers in pod
kubectl --context=my-context get pod mypod -o jsonpath='{.spec.containers[*].name}'

# Use correct container name
kubectl --context=my-context debug mypod -it --image=busybox --target=correct-name

Debug Pod Copy Not Starting

Symptom: Debug pod stays in Pending state.

Causes:

• Insufficient resources
• Node affinity/tolerations
• Image pull issues

Solution:

# Check pod events
kubectl --context=my-context describe pod mypod-debug

# Try without same-node constraint
kubectl --context=my-context debug mypod -it \
  --copy-to=mypod-debug \
  --image=busybox  # Remove --same-node

# Use smaller image
kubectl --context=my-context debug mypod -it \
  --copy-to=mypod-debug \
  --image=busybox  # Instead of ubuntu

Cannot See Processes from Target Container

Symptom: ps aux only shows debug container processes.

Cause: Process namespace sharing not working or container exited.

Solution:

# Ensure target container is running
kubectl --context=my-context get pod mypod -o jsonpath='{.status.containerStatuses[*].ready}'

# Verify process sharing is enabled
kubectl --context=my-context get pod mypod -o yaml | grep shareProcessNamespace

# For pod copy, ensure sharing enabled
kubectl --context=my-context debug mypod -it \
  --copy-to=mypod-debug \
  --share-processes=true \
  --image=busybox

Node Debug Pod Cleanup

Symptom: Node debug pods accumulate.

Solution:

# Find and delete all node debug pods
kubectl --context=my-context get pods -A -o name | grep node-debugger | \
  xargs -I {} kubectl --context=my-context delete {} -n default

# Or by label
kubectl --context=my-context delete pods -l app=node-debugger

---

Related Skills

• Kubernetes Operations - General kubectl usage and troubleshooting
• Helm Debugging - Debugging Helm deployments
• ArgoCD CLI Login - GitOps debugging with ArgoCD

References

• kubectl debug documentation
• Debug Running Pods
• Debug Nodes
• Ephemeral Containers
• Debug Profiles KEP