Kubernetes Operations

Kubernetes Operations

Comprehensive kubectl assistance for debugging, resource management, and cluster operations with token-efficient scripts.

BEFORE YOU START

This skill prevents 5 common errors and saves ~70% tokens.

Known Issues This Skill Prevents

1. Running kubectl commands in wrong namespace/context 2. Verbose output flooding context with unnecessary data 3. Missing critical debugging steps (events, previous logs) 4. Exposing secrets in plain text output 5. Destructive operations without dry-run verification

Quick Start

Step 1: Verify Context

kubectl config current-context
kubectl config get-contexts

Why this matters: Running commands in the wrong cluster can cause production incidents.

Step 2: Debug a Pod

uv run scripts/debug_pod.py <pod-name> [-n namespace]

Why this matters: The script combines describe, logs, and events into a condensed summary, saving ~800 tokens.

Step 3: Check Cluster Health

uv run scripts/cluster_health.py

Why this matters: Quick overview of node status and unhealthy pods without verbose output.

Critical Rules

Always Do

• Always verify kubectl config current-context before operations
• Always use -n namespace to be explicit about target
• Always use --dry-run=client -o yaml before applying changes
• Always check events when debugging: kubectl get events --sort-by='.lastTimestamp'
• Always use --previous flag when pod is in CrashLoopBackOff

Never Do

• Never run kubectl delete without --dry-run first in production
• Never output secrets without filtering: avoid kubectl get secret -o yaml
• Never assume default namespace - always specify -n
• Never ignore resource limits when debugging OOMKilled pods
• Never skip describe when logs show no errors

Common Mistakes

Wrong:

kubectl logs my-pod

Correct:

kubectl logs my-pod -n my-namespace --tail=100 --timestamps

Why: Default namespace may not be correct, unlimited logs flood context, timestamps help correlate with events.

Known Issues Prevention

Debugging Workflows

Pod Not Starting

# 1. Get pod status and events
kubectl describe pod <name> -n <namespace>

# 2. Check logs (current or previous)
kubectl logs <name> -n <namespace> --tail=100
kubectl logs <name> -n <namespace> --previous  # If restarting

# 3. Check events for scheduling issues
kubectl get events -n <namespace> --sort-by='.lastTimestamp' | grep <name>

# 4. Interactive debugging
kubectl exec -it <name> -n <namespace> -- /bin/sh

Service Connectivity

# 1. Verify service exists and has endpoints
kubectl get svc <name> -n <namespace>
kubectl get endpoints <name> -n <namespace>

# 2. Check pod labels match service selector
kubectl get pods -n <namespace> --show-labels

# 3. Test from within cluster
kubectl run debug --rm -it --image=busybox -- wget -qO- http://<service>:<port>

# 4. Port-forward for local testing
kubectl port-forward svc/<name> 8080:80 -n <namespace>

Resource Management

Deployments

# List deployments
kubectl get deployments -n <namespace>

# Scale
kubectl scale deployment <name> --replicas=3 -n <namespace>

# Rollout status
kubectl rollout status deployment/<name> -n <namespace>

# Rollback
kubectl rollout undo deployment/<name> -n <namespace>

# History
kubectl rollout history deployment/<name> -n <namespace>

ConfigMaps and Secrets

# List
kubectl get configmaps -n <namespace>
kubectl get secrets -n <namespace>

# View ConfigMap data
kubectl get configmap <name> -n <namespace> -o jsonpath='{.data}'

# View Secret keys (NOT values)
kubectl get secret <name> -n <namespace> -o jsonpath='{.data}' | jq 'keys'

# Create from file
kubectl create configmap <name> --from-file=<path> -n <namespace> --dry-run=client -o yaml

Cluster Operations

Node Management

# List nodes with status
kubectl get nodes -o wide

# Node details
kubectl describe node <name>

# Cordon (prevent scheduling)
kubectl cordon <node>

# Drain (evict pods)
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

# Uncordon
kubectl uncordon <node>

Resource Usage

# Node resources
kubectl top nodes

# Pod resources
kubectl top pods -n <namespace>

# Sort by memory
kubectl top pods -n <namespace> --sort-by=memory

Bundled Resources

Scripts

Located in scripts/:

• debug_pod.py - Comprehensive pod debugging with condensed output
• get_resources.py - Resource summary using jsonpath for minimal tokens
• cluster_health.py - Quick cluster status overview

References

Located in references/:

• `kubectl-cheatsheet.md` - Condensed command reference
• `jsonpath-patterns.md` - Common JSONPath expressions
• `debugging-flowchart.md` - Decision tree for pod issues

Note: For deep dives on specific topics, see the reference files above.

Dependencies

Required

Optional

Official Documentation

• kubectl Quick Reference
• JSONPath Support
• kubectl Cheat Sheet
• Debug Running Pods

Troubleshooting

kubectl command not found

Symptoms: command not found: kubectl

Solution:

# macOS
brew install kubectl

# Verify
kubectl version --client

Context not set

Symptoms: error: no context is currently set

Solution:

# List available contexts
kubectl config get-contexts

# Set context
kubectl config use-context <context-name>

Permission denied

Symptoms: Error from server (Forbidden)

Solution:

# Check current user
kubectl auth whoami

# Check permissions
kubectl auth can-i get pods -n <namespace>
kubectl auth can-i --list -n <namespace>

Timeout connecting to cluster

Symptoms: Unable to connect to the server: dial tcp: i/o timeout

Solution:

# Check cluster endpoint
kubectl cluster-info

# Verify network connectivity
curl -k https://<cluster-api-endpoint>/healthz

# Check kubeconfig
cat ~/.kube/config

Setup Checklist

Before using this skill, verify:

• [ ] kubectl installed (kubectl version --client)
• [ ] Kubeconfig configured (~/.kube/config exists)
• [ ] Context set to correct cluster (kubectl config current-context)
• [ ] Permissions verified (kubectl auth can-i get pods)
• [ ] jq installed for JSON parsing (jq --version)

Kubernetes Operations

Comprehensive kubectl assistance for debugging, resource management, and cluster operations with token-efficient scripts.

What This Skill Does

Provides intelligent assistance for Kubernetes operations using kubectl. Includes token-efficient Python scripts that condense verbose kubectl output into actionable summaries.

Core Capabilities

• Debug pods with condensed status, events, and logs
• Manage resources (deployments, services, configmaps, secrets)
• Monitor cluster health and resource usage
• Troubleshoot common Kubernetes issues

Auto-Trigger Keywords

Primary Keywords

Exact terms that strongly trigger this skill:

• kubectl
• kubernetes
• k8s
• pods
• deployments

Secondary Keywords

Related terms that may trigger in combination:

• cluster
• namespace
• service
• configmap
• secret
• nodes
• replicas
• rollout

Error-Based Keywords

Common error messages that should trigger this skill:

• "CrashLoopBackOff"
• "ImagePullBackOff"
• "Pending"
• "OOMKilled"
• "connection refused"
• "no endpoints available"
• "forbidden"

Known Issues Prevention

When to Use

Use This Skill For

• Debugging pod startup issues
• Checking cluster health
• Managing Kubernetes resources
• Troubleshooting service connectivity
• Viewing logs and events

Don't Use This Skill For

• Creating Helm charts (use helm-chart-scaffolding skill)
• ArgoCD/GitOps workflows (use argocd agents)
• Terraform/IaC for cluster provisioning
• Custom Resource Definition development

Quick Usage

# Debug a pod
uv run scripts/debug_pod.py my-pod -n my-namespace

# List resources compactly
uv run scripts/get_resources.py pods -n my-namespace

# Check cluster health
uv run scripts/cluster_health.py

Token Efficiency

File Structure

kubernetes-operations/
├── SKILL.md        # Detailed instructions and patterns
├── README.md       # This file - discovery and quick reference
├── scripts/        # Token-efficient automation scripts
│   ├── debug_pod.py
│   ├── get_resources.py
│   └── cluster_health.py
├── references/     # Supporting documentation
│   ├── kubectl-cheatsheet.md
│   ├── jsonpath-patterns.md
│   └── debugging-flowchart.md
└── assets/         # Templates and resources

Dependencies

Official Documentation

• kubectl Quick Reference
• JSONPath Support
• Debug Running Pods

Related Skills

• helm-chart-scaffolding - Helm chart creation and management
• k8s-manifest-generator - Generate Kubernetes YAML manifests
• gitops-workflow - ArgoCD/Flux GitOps patterns

---

License: MIT

Kubernetes Debugging Flowchart

Decision tree for common pod issues.

Pod Not Starting

Pod Status?
├── Pending
│   ├── Check: kubectl describe pod <name>
│   ├── Look for: "Events" section
│   └── Common causes:
│       ├── Insufficient resources → Check node capacity, adjust requests/limits
│       ├── No matching nodes → Check nodeSelector, affinity, tolerations
│       ├── PVC not bound → Check PVC status and StorageClass
│       └── Image pull issues → Check imagePullSecrets
│
├── ContainerCreating
│   ├── Check: kubectl describe pod <name>
│   └── Common causes:
│       ├── Image pull error → Verify image name and registry auth
│       ├── ConfigMap/Secret missing → Verify referenced resources exist
│       └── Volume mount issues → Check PVC and volume configuration
│
├── CrashLoopBackOff
│   ├── Check: kubectl logs <name> --previous
│   ├── Check: kubectl describe pod <name> (exit code)
│   └── Common causes:
│       ├── Exit code 1 → Application error, check logs
│       ├── Exit code 137 → OOMKilled, increase memory limit
│       ├── Exit code 143 → SIGTERM, check liveness probe
│       └── Missing config → Check env vars and mounts
│
├── ImagePullBackOff
│   ├── Check: kubectl describe pod <name>
│   └── Common causes:
│       ├── Image not found → Verify image:tag exists
│       ├── Auth error → Check imagePullSecrets
│       └── Network error → Check registry connectivity
│
└── Running but not working
    ├── Check: kubectl logs <name>
    ├── Check: kubectl exec -it <name> -- /bin/sh
    └── Common causes:
        ├── Readiness probe failing → Check probe config and endpoint
        ├── Wrong command/args → Verify entrypoint
        └── Missing dependencies → Check service connectivity

Service Not Accessible

Can't reach service?
│
├── Check service exists
│   kubectl get svc <name> -n <ns>
│
├── Check endpoints
│   kubectl get endpoints <name> -n <ns>
│   └── Empty? → Pod labels don't match service selector
│
├── Check pod labels match selector
│   kubectl get svc <name> -o jsonpath='{.spec.selector}'
│   kubectl get pods -l <selector> -n <ns>
│
├── Test from within cluster
│   kubectl run debug --rm -it --image=busybox -- wget -qO- http://<svc>:<port>
│
└── Check NetworkPolicy
    kubectl get networkpolicy -n <ns>
    └── Blocking traffic? → Update policy or test without it

High Resource Usage

Resource issues?
│
├── Check current usage
│   kubectl top pods -n <ns>
│   kubectl top nodes
│
├── OOMKilled containers
│   ├── Check: kubectl describe pod <name> (look for OOMKilled)
│   ├── Solution: Increase memory limits
│   └── Investigate: Profile application memory usage
│
├── CPU throttling
│   ├── Check: kubectl describe pod <name> (look for CPU limits)
│   └── Solution: Increase CPU limits or optimize application
│
└── Node pressure
    ├── Check: kubectl describe node <name>
    ├── Look for: MemoryPressure, DiskPressure, PIDPressure
    └── Solution: Add nodes, evict pods, or clean up disk

Quick Diagnosis Commands

# 1. Get overview
kubectl get pods -n <ns> -o wide

# 2. Check specific pod
kubectl describe pod <name> -n <ns>

# 3. Get logs
kubectl logs <name> -n <ns> --tail=100

# 4. Get previous logs (if restarting)
kubectl logs <name> -n <ns> --previous

# 5. Check events
kubectl get events -n <ns> --sort-by='.lastTimestamp' | tail -20

# 6. Shell into pod
kubectl exec -it <name> -n <ns> -- /bin/sh

# 7. Check resource usage
kubectl top pod <name> -n <ns>

Exit Code Reference

JSONPath Patterns for kubectl

Reference for kubectl JSONPath support.

Basic Syntax

kubectl get <resource> -o jsonpath='{.field.subfield}'

Common Patterns

Pod Status

# Pod phase
kubectl get pod <name> -o jsonpath='{.status.phase}'

# Container status
kubectl get pod <name> -o jsonpath='{.status.containerStatuses[0].state}'

# Restart count
kubectl get pod <name> -o jsonpath='{.status.containerStatuses[0].restartCount}'

# Pod IP
kubectl get pod <name> -o jsonpath='{.status.podIP}'

# Node name
kubectl get pod <name> -o jsonpath='{.spec.nodeName}'

Multiple Values

# Name and status for all pods
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\n"}{end}'

# With custom separator
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}|{.status.phase}\n{end}'

Filtering

# Pods with specific label
kubectl get pods -o jsonpath='{.items[?(@.metadata.labels.app=="nginx")].metadata.name}'

# Ready nodes only
kubectl get nodes -o jsonpath='{.items[?(@.status.conditions[?(@.type=="Ready")].status=="True")].metadata.name}'

# Non-running pods
kubectl get pods -o jsonpath='{.items[?(@.status.phase!="Running")].metadata.name}'

Node Information

# Node capacity
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.capacity.cpu}{"\t"}{.status.capacity.memory}{"\n"}{end}'

# Node conditions
kubectl get nodes -o jsonpath='{.items[*].status.conditions[?(@.type=="Ready")].status}'

# Allocatable resources
kubectl get node <name> -o jsonpath='{.status.allocatable}'

Service & Endpoints

# Service ClusterIP
kubectl get svc <name> -o jsonpath='{.spec.clusterIP}'

# Service ports
kubectl get svc <name> -o jsonpath='{.spec.ports[*].port}'

# Endpoint addresses
kubectl get endpoints <name> -o jsonpath='{.subsets[*].addresses[*].ip}'

ConfigMaps & Secrets

# ConfigMap data keys
kubectl get configmap <name> -o jsonpath='{.data}' | jq 'keys'

# Specific ConfigMap value
kubectl get configmap <name> -o jsonpath='{.data.key-name}'

# Secret keys (not values)
kubectl get secret <name> -o jsonpath='{.data}' | jq 'keys'

# Decode secret value
kubectl get secret <name> -o jsonpath='{.data.password}' | base64 -d

Deployments

# Replica count
kubectl get deployment <name> -o jsonpath='{.spec.replicas}'

# Ready replicas
kubectl get deployment <name> -o jsonpath='{.status.readyReplicas}'

# Container images
kubectl get deployment <name> -o jsonpath='{.spec.template.spec.containers[*].image}'

# Deployment conditions
kubectl get deployment <name> -o jsonpath='{.status.conditions[?(@.type=="Available")].status}'

Events

# Recent event messages
kubectl get events -o jsonpath='{range .items[*]}{.reason}: {.message}{"\n"}{end}'

# Events for specific object
kubectl get events --field-selector involvedObject.name=<pod> -o jsonpath='{range .items[*]}{.reason}: {.message}{"\n"}{end}'

Output Formatting

Custom Columns

kubectl get pods -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,IP:.status.podIP

Sorted Output

# Sort by creation time
kubectl get pods --sort-by='.metadata.creationTimestamp'

# Sort by restart count
kubectl get pods --sort-by='.status.containerStatuses[0].restartCount'

Tips

1. Test incrementally: Start with '{.metadata}' then drill down 2. Use jq for complex parsing: Pipe JSON output to jq 3. Escape special characters: Use \" inside jsonpath strings 4. Range for iteration: {range .items[*]}...{end} for lists 5. Filters use ?(@): {.items[?(@.field=="value")]}

kubectl Cheatsheet

Condensed from official quick reference.

Context & Config

kubectl config current-context          # Show current context
kubectl config get-contexts             # List all contexts
kubectl config use-context <name>       # Switch context
kubectl config set-context --current --namespace=<ns>  # Set default namespace

Get Resources

kubectl get <resource> -n <ns>          # List resources
kubectl get <resource> -o wide          # More columns
kubectl get <resource> -o yaml          # YAML output
kubectl get <resource> -o json          # JSON output
kubectl get <resource> --show-labels    # Show labels
kubectl get <resource> -l key=value     # Filter by label
kubectl get <resource> --all-namespaces # All namespaces

Describe & Logs

kubectl describe <resource> <name> -n <ns>  # Detailed info
kubectl logs <pod> -n <ns>                  # Pod logs
kubectl logs <pod> -c <container>           # Specific container
kubectl logs <pod> --tail=100               # Last 100 lines
kubectl logs <pod> -f                       # Follow logs
kubectl logs <pod> --previous               # Previous container
kubectl logs <pod> --timestamps             # With timestamps

Create & Apply

kubectl create -f <file.yaml>           # Create from file
kubectl apply -f <file.yaml>            # Apply (create or update)
kubectl apply -f <dir>/                 # Apply all in directory
kubectl apply -k <dir>/                 # Apply with kustomize
kubectl create configmap <name> --from-file=<path>
kubectl create secret generic <name> --from-literal=key=value

Edit & Patch

kubectl edit <resource> <name> -n <ns>  # Edit in editor
kubectl patch <resource> <name> -p '{"spec":{"replicas":3}}'
kubectl set image deployment/<name> <container>=<image>

Delete

kubectl delete <resource> <name> -n <ns>
kubectl delete -f <file.yaml>
kubectl delete <resource> --all -n <ns>
kubectl delete pod <name> --force --grace-period=0  # Force delete

Exec & Debug

kubectl exec -it <pod> -- /bin/sh           # Shell into pod
kubectl exec -it <pod> -c <container> -- /bin/sh  # Specific container
kubectl run debug --rm -it --image=busybox -- /bin/sh  # Debug pod
kubectl cp <pod>:<path> <local-path>        # Copy from pod
kubectl cp <local-path> <pod>:<path>        # Copy to pod

Port Forward

kubectl port-forward <pod> 8080:80          # Pod
kubectl port-forward svc/<name> 8080:80     # Service
kubectl port-forward deploy/<name> 8080:80  # Deployment

Scaling & Rollouts

kubectl scale deployment <name> --replicas=3
kubectl autoscale deployment <name> --min=2 --max=10 --cpu-percent=80
kubectl rollout status deployment/<name>
kubectl rollout history deployment/<name>
kubectl rollout undo deployment/<name>
kubectl rollout restart deployment/<name>

Labels & Annotations

kubectl label <resource> <name> key=value
kubectl label <resource> <name> key-           # Remove label
kubectl annotate <resource> <name> key=value

Resource Usage

kubectl top nodes
kubectl top pods -n <ns>
kubectl top pods --sort-by=memory
kubectl top pods --sort-by=cpu

Events

kubectl get events -n <ns>
kubectl get events --sort-by='.lastTimestamp'
kubectl get events --field-selector reason=Failed

Auth & RBAC

kubectl auth whoami
kubectl auth can-i get pods -n <ns>
kubectl auth can-i --list -n <ns>
kubectl get roles,rolebindings -n <ns>
kubectl get clusterroles,clusterrolebindings

Dry Run

kubectl apply -f file.yaml --dry-run=client -o yaml
kubectl create deployment <name> --image=<img> --dry-run=client -o yaml