Kubernetes

Kubernetes

Quick Start (kubectl)

kubectl describe pod/<pod> -n <ns>
kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -n 30
kubectl logs pod/<pod> -n <ns> --previous --tail=200

Production Minimums

• Health: readinessProbe and startupProbe for safe rollouts
• Resources: set requests/limits to prevent noisy-neighbor failures
• Security: run as non-root and grant least privilege

Load Next (References)

• references/core-objects.md — choose the right workload/controller and service type
• references/rollouts-and-probes.md — probes, rollouts, graceful shutdown, rollback
• references/debugging-runbook.md — common failure modes and a fast triage flow
• references/security-hardening.md — pod security, RBAC, network policy, supply chain

Kubernetes Core Objects (Cheat Sheet)

Mental Model

• Control plane stores desired state; controllers reconcile actual state toward it.
• Pods are ephemeral; prefer controllers (Deployment/StatefulSet/DaemonSet/Job).
• Selectors bind things together (Service → Pods, Deployment → ReplicaSet → Pods).

Workload Controllers

Networking

Service types

• ClusterIP: internal service discovery (default)
• NodePort: exposes a port on every node (often avoided in production)
• LoadBalancer: cloud LB integration (when available)
• Ingress: HTTP routing (path/host TLS termination) to Services

Selector sanity check

kubectl get svc/<svc> -n <ns>
kubectl get endpoints/<svc> -n <ns> -o wide
kubectl get pods -n <ns> -l app=<label>

If endpoints is empty, check:

• Service selector matches Pod labels
• Pods pass readiness checks (not ready ⇒ not in endpoints)

Configuration

• ConfigMap: non-secret configuration (mounted files or env vars)
• Secret: sensitive configuration (protect etcd + RBAC; treat cluster access as secret access)

Prefer mounted files for large configs; prefer env vars for small values.

Scheduling and Scaling

• Resources: set requests and limits for CPU/memory
• Placement: nodeSelector, affinity, taints/tolerations
• Autoscaling: HPA based on CPU/memory/custom metrics (avoid scaling on high-cardinality metrics)

Minimal Deployment Template

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
        - name: app
          image: example/app:1.2.3
          ports:
            - containerPort: 8080

Labels (Recommended)

Use the standard label keys to keep tooling compatible:

• app.kubernetes.io/name
• app.kubernetes.io/instance
• app.kubernetes.io/version
• app.kubernetes.io/component
• app.kubernetes.io/part-of
• app.kubernetes.io/managed-by

Kubernetes Debugging Runbook

Triage Flow (Fast)

1) Identify what is failing (Pod, Service routing, Ingress, dependency, node capacity). 2) Inspect events and current state before changing manifests. 3) Verify labels/selectors and readiness gating.

Snapshot the state

kubectl get deploy,rs,po,svc,ing -n <ns> -o wide
kubectl get events -n <ns> --sort-by=.lastTimestamp | tail -n 50

Drill into a single Pod

kubectl describe pod/<pod> -n <ns>
kubectl logs pod/<pod> -n <ns> --tail=200
kubectl logs pod/<pod> -n <ns> --previous --tail=200
kubectl exec -it pod/<pod> -n <ns> -- sh

Common Failure Modes

Pending

Signals:

• kubectl describe pod shows scheduling errors.

Likely causes:

• Insufficient CPU/memory on nodes
• Node selectors/affinity too strict
• Missing tolerations for taints
• PVC not bound

Next actions:

• Inspect Events: in describe
• Check cluster capacity: kubectl top nodes (if metrics-server exists)
• Validate PVC: kubectl get pvc -n <ns>

ImagePullBackOff / ErrImagePull

Likely causes:

• Wrong image name/tag
• Registry auth missing (imagePullSecrets)
• Rate limits or network egress blocked

Next actions:

• Check events for registry error details
• Confirm pull secret exists and is referenced in the Pod spec

CrashLoopBackOff

Likely causes:

• App exits on startup (config/secrets/migrations)
• Liveness probe too aggressive
• OOM kills or file permission issues

Next actions:

• Read --previous logs first
• Check container exit code and Reason: in describe
• Look for OOMKilled and memory limits

Service returns 503/504

Likely causes:

• No ready endpoints (readiness failing)
• Service selector mismatch
• Ingress routes to wrong Service/port

Next actions:

• Verify endpoints:

kubectl get endpoints/<svc> -n <ns> -o wide
kubectl get pods -n <ns> -l app=<label> -o wide

Ingress not routing

Likely causes:

• Ingress controller missing or misconfigured
• TLS secret missing/invalid
• Path/host rules mismatch

Next actions:

• Inspect Ingress events and controller logs
• Validate DNS/host rule and backend Service port mapping

Debugging Without Changing the Image

If the image lacks tooling, use ephemeral containers (requires cluster support):

kubectl debug -it pod/<pod> -n <ns> --image=busybox:1.36 --target=<container-name>

Use this for DNS checks, curl, and filesystem inspection without rebuilding.

Rollouts, Probes, and Graceful Shutdown

Health Probes

Readiness

• Gate traffic.
• Fail readiness when dependencies are unavailable (DB down, migrations running) if serving requests would fail.

Liveness

• Restart stuck/crashed processes.
• Avoid liveness probes that depend on external services; prefer process health only.

Startup

• Prevent liveness/readiness from failing during slow boot.

Minimal Probe Example

containers:
  - name: app
    image: example/app:1.2.3
    ports:
      - containerPort: 8080
    startupProbe:
      httpGet: { path: /healthz, port: 8080 }
      failureThreshold: 30
      periodSeconds: 2
    readinessProbe:
      httpGet: { path: /readyz, port: 8080 }
      periodSeconds: 5
      timeoutSeconds: 2
    livenessProbe:
      httpGet: { path: /healthz, port: 8080 }
      periodSeconds: 10
      timeoutSeconds: 2

Rolling Updates

Prefer RollingUpdate for stateless services.

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1

Rollout Commands

kubectl rollout status deploy/<name> -n <ns>
kubectl rollout history deploy/<name> -n <ns>
kubectl rollout undo deploy/<name> -n <ns>

Graceful Shutdown

Kubernetes sends SIGTERM and waits terminationGracePeriodSeconds.

Checklist:

• Handle SIGTERM in the app (stop accepting new work, drain connections, flush buffers).
• Keep readiness failing during shutdown so traffic drains.
• Use preStop only when an explicit delay is required (prefer app-level drains).

terminationGracePeriodSeconds: 30
containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "sleep 5"]

Disruption and Availability

Use a Pod Disruption Budget (PDB) to keep at least N replicas available during voluntary disruptions:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: app

Kubernetes Security Hardening

Pod Security (Baseline)

Prefer restrictive defaults:

• Run as non-root
• Drop Linux capabilities
• Use a read-only root filesystem where possible
• Set a seccomp profile

securityContext:
  runAsNonRoot: true
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop: ["ALL"]
  seccompProfile:
    type: RuntimeDefault

RBAC (Least Privilege)

Checklist:

• Grant namespace-scoped roles where possible.
• Avoid cluster-admin and broad wildcard rules.
• Separate “read” roles from “write” roles.

Network Policy (Default Deny)

If the CNI supports NetworkPolicy, adopt a default-deny stance and allow only required traffic between namespaces and workloads.

Key patterns:

• Allow ingress only from the ingress controller namespace.
• Allow egress only to required dependencies (DB/cache) and DNS.

Secrets Handling

Assume the cluster can read Secrets; treat cluster access as secret access.

Checklist:

• Restrict Secrets with RBAC and namespace boundaries
• Prefer external secret managers (external-secrets, CSI drivers) for high-value secrets
• Avoid logging environment values and full config dumps

Supply Chain

Checklist:

• Pin images by digest for critical workloads (image@sha256:...)
• Scan images in CI and block known critical CVEs
• Use minimal base images and drop unused packages