K8s Troubleshoot

Kubernetes Troubleshooting

Expert debugging and diagnostics for Kubernetes clusters using kubectl-mcp-server tools.

When to Apply

Use this skill when:

• User mentions: "debug", "troubleshoot", "diagnose", "failing", "crash", "not starting", "broken"
• Pod states: Pending, CrashLoopBackOff, ImagePullBackOff, OOMKilled, Error, Unknown
• Node issues: NotReady, MemoryPressure, DiskPressure, NetworkUnavailable, PIDPressure
• Keywords: "logs", "events", "describe", "why isn't working", "stuck", "not responding"

Priority Rules

Quick Reference

Diagnostic Workflows

Pod Not Starting

1. get_pods(namespace, label_selector) - Get pod status
2. describe_pod(name, namespace) - See events and conditions
3. get_events(namespace, field_selector="involvedObject.name=<pod>") - Check events
4. get_pod_logs(name, namespace, previous=True) - For crash loops

Common Pod States

Node Issues

1. get_nodes() - List nodes and status
2. describe_node(name) - See conditions and capacity
3. Check: Ready, MemoryPressure, DiskPressure, PIDPressure
4. node_logs_tool(name, "kubelet") - Kubelet logs

Deep Debugging Workflows

CrashLoopBackOff Investigation

1. get_pod_logs(name, namespace, previous=True) - See why it crashed
2. describe_pod(name, namespace) - Check resource limits, probes
3. get_pod_metrics(name, namespace) - Memory/CPU at crash time
4. If OOM: compare requests/limits to actual usage
5. If app error: check logs for stack trace

Networking Issues

1. get_services(namespace) - Verify service exists
2. get_endpoints(namespace) - Check endpoint backends
3. If empty endpoints: pods don't match selector
4. get_network_policies(namespace) - Check traffic rules
5. For Cilium: cilium_endpoints_list_tool(), hubble_flows_query_tool()

Storage Problems

1. get_pvc(namespace) - Check PVC status
2. describe_pvc(name, namespace) - See binding issues
3. get_storage_classes() - Verify provisioner exists
4. If Pending: check storage class, access modes

DNS Resolution

1. kubectl_exec(pod, namespace, "nslookup kubernetes.default") - Test DNS
2. If fails: check coredns pods in kube-system
3. get_pods(namespace="kube-system", label_selector="k8s-app=kube-dns")
4. get_pod_logs(name="coredns-*", namespace="kube-system")

Multi-Cluster Debugging

All tools support context parameter for targeting different clusters:

get_pods(namespace="kube-system", context="production-cluster")
get_events(namespace="default", context="staging-cluster")
describe_pod(name="myapp-xyz", namespace="prod", context="prod-east")

Diagnostic Scripts

For comprehensive diagnostics, run the bundled scripts:

• See scripts/diagnose-pod.py for automated pod analysis
• See scripts/health-check.sh for cluster health checks

Decision Tree

See references/DECISION-TREE.md for visual troubleshooting flowcharts.

Common Errors Reference

See references/COMMON-ERRORS.md for error message explanations and fixes.

Related Tools

Core Diagnostics

• get_pods, describe_pod, get_pod_logs, get_pod_metrics
• get_events, get_nodes, describe_node
• get_resource_usage, compare_namespaces

Advanced (Ecosystem)

• Cilium: cilium_endpoints_list_tool, hubble_flows_query_tool
• Istio: istio_proxy_status_tool, istio_analyze_tool

Related Skills

• k8s-diagnostics - Metrics and health checks
• k8s-incident - Emergency runbooks
• k8s-networking - Network troubleshooting

Common Kubernetes Error Messages

Quick reference for error messages and their solutions.

Pod Errors

CrashLoopBackOff

Meaning: Container keeps crashing and restarting.

Causes:

• Application crash on startup
• Missing configuration or secrets
• Resource exhaustion (OOM)
• Failing health checks

Fix:

get_pod_logs(name, namespace, previous=True)
describe_pod(name, namespace)

ImagePullBackOff / ErrImagePull

Meaning: Cannot pull container image.

Causes:

• Image doesn't exist
• Private registry without credentials
• Network issues
• Rate limiting (Docker Hub)

Fix:

describe_pod(name, namespace)

OOMKilled (Exit Code 137)

Meaning: Container exceeded memory limit.

Causes:

• Memory limit too low
• Memory leak in application
• Large data processing

Fix:

get_pod_metrics(name, namespace)

Exit Code 1

Meaning: Application error.

Fix: Check application logs for stack trace.

Exit Code 127

Meaning: Command not found.

Fix: Check container image has required binaries.

Exit Code 128+N

Meaning: Container killed by signal N.

Scheduling Errors

0/N nodes are available

Causes:

• Insufficient resources
• Node selector/affinity not matching
• Taints without tolerations
• All nodes cordoned

Fix:

describe_pod(name, namespace)
get_nodes()

PodExceedsFreeCPU / PodExceedsFreeMemory

Fix: Reduce resource requests or add capacity.

NodeNotReady

Fix:

describe_node(name)

Storage Errors

PersistentVolumeClaim not found

Fix: Create the PVC before the pod.

Unable to attach or mount volumes

Causes:

• PV already attached elsewhere
• Storage class misconfigured
• Node has no access to storage

Fix:

describe_pvc(name, namespace)
get_storage_classes()

Network Errors

Connection refused

Causes:

• Service not exposed on expected port
• Application not listening
• Firewall/NetworkPolicy blocking

Fix:

get_endpoints(namespace)
get_network_policies(namespace)

No route to host

Causes:

• Node network issues
• CNI problems

Fix: Check CNI pods in kube-system.

DNS resolution failed

Fix:

get_pods(namespace="kube-system", label_selector="k8s-app=kube-dns")

RBAC Errors

Forbidden: User cannot...

Causes:

• Missing Role/ClusterRole
• Missing RoleBinding/ClusterRoleBinding
• ServiceAccount not assigned

Fix:

get_cluster_roles()
get_role_bindings(namespace)

Admission Controller Errors

Admission webhook denied

Causes:

• Policy violation (Kyverno/Gatekeeper)
• Webhook unavailable

Fix:

kyverno_clusterpolicies_list_tool()
gatekeeper_constraints_list_tool()

Resource Limit Errors

Forbidden: exceeded quota

Fix:

get_resource_quotas(namespace)

LimitRange rejection

Fix: Adjust resource requests/limits to match LimitRange.

Probe Failures

Liveness probe failed

Meaning: Container will be restarted.

Readiness probe failed

Meaning: Pod removed from service endpoints.

Common causes:

• Wrong port in probe config
• Application slow to start
• Dependencies unavailable

Fix:

describe_pod(name, namespace)
get_pod_logs(name, namespace)

Quick Lookup

Troubleshooting Decision Trees

Visual flowcharts for diagnosing Kubernetes issues.

Pod Not Running

Pod Status?
├── Pending
│   ├── Events show "Insufficient cpu/memory"
│   │   └── Scale cluster or reduce requests
│   ├── Events show "no nodes available"
│   │   └── Check node taints, affinity rules
│   ├── Events show "PersistentVolumeClaim not found"
│   │   └── Create PVC or check storage class
│   └── No events
│       └── Check scheduler pods in kube-system
│
├── CrashLoopBackOff
│   ├── get_pod_logs(previous=True)
│   ├── Exit Code 137 (OOMKilled)
│   │   └── Increase memory limits
│   ├── Exit Code 1 (App Error)
│   │   └── Check application logs, config
│   └── Exit Code 127 (Command Not Found)
│       └── Check entrypoint/command in spec
│
├── ImagePullBackOff
│   ├── "unauthorized"
│   │   └── Create/update imagePullSecrets
│   ├── "not found"
│   │   └── Verify image name and tag
│   └── "timeout"
│       └── Check network, registry availability
│
├── ContainerCreating (stuck)
│   ├── Volume issues
│   │   └── Check PVC status, storage class
│   ├── ConfigMap/Secret not found
│   │   └── Create missing resources
│   └── Network issues
│       └── Check CNI pods
│
└── Running but not ready
    ├── Readiness probe failing
    │   └── Check probe config, app health
    └── Init containers not complete
        └── Check init container logs

Service Not Accessible

Service unreachable?
├── get_endpoints(namespace) empty?
│   ├── Yes
│   │   ├── Pods exist with matching labels?
│   │   │   ├── No → Fix selector labels
│   │   │   └── Yes → Pods not ready
│   │   │       └── Fix pod readiness
│   │   └── Service selector correct?
│   │       └── Update service spec
│   └── No (endpoints exist)
│       ├── Check NetworkPolicy
│       │   └── get_network_policies(namespace)
│       ├── Check Service type
│       │   ├── ClusterIP → Only internal access
│       │   ├── NodePort → Access via node:port
│       │   └── LoadBalancer → Check cloud LB
│       └── DNS resolution working?
│           └── Test from inside pod

Node Issues

Node Not Ready?
├── describe_node(name)
├── Conditions show:
│   ├── MemoryPressure
│   │   └── Eviction happening, free memory
│   ├── DiskPressure
│   │   └── Clean up images, logs
│   ├── PIDPressure
│   │   └── Kill zombie processes
│   └── NetworkUnavailable
│       └── Check CNI, kubelet
├── Kubelet not running?
│   └── Check systemctl status kubelet
└── Node cordoned?
    └── kubectl uncordon node

Storage Issues

PVC Pending?
├── describe_pvc(name, namespace)
├── Events show:
│   ├── "no persistent volumes available"
│   │   ├── Dynamic provisioning enabled?
│   │   │   └── Check StorageClass exists
│   │   └── Static PV exists with matching spec?
│   │       └── Check access modes, capacity
│   ├── "waiting for first consumer"
│   │   └── Normal with WaitForFirstConsumer
│   └── "provisioning failed"
│       └── Check storage backend, quotas

Deployment Not Progressing

Deployment stuck?
├── rollout_status(name, namespace)
├── Shows "waiting for rollout to finish"
│   ├── New pods starting?
│   │   ├── No → Check pod issues above
│   │   └── Yes but failing
│   │       └── Check pod logs
│   ├── Old pods not terminating?
│   │   ├── Check finalizers
│   │   └── Check PDBs (PodDisruptionBudget)
│   └── Deadline exceeded?
│       └── Increase progressDeadlineSeconds

Quick Commands Reference