Operating Kubernetes

Kubernetes Operations

Purpose

Operating Kubernetes clusters in production requires mastery of resource management, scheduling patterns, networking architecture, storage strategies, security hardening, and autoscaling. This skill provides operations-first frameworks for right-sizing workloads, implementing high-availability patterns, securing clusters with RBAC and Pod Security Standards, and systematically troubleshooting common failures.

Use this skill when deploying applications to Kubernetes, configuring cluster resources, implementing NetworkPolicies for zero-trust security, setting up autoscaling (HPA, VPA, KEDA), managing persistent storage, or diagnosing operational issues like CrashLoopBackOff or resource exhaustion.

When to Use This Skill

Common Triggers:

• "Deploy my application to Kubernetes"
• "Configure resource requests and limits"
• "Set up autoscaling for my pods"
• "Implement NetworkPolicies for security"
• "My pod is stuck in Pending/CrashLoopBackOff"
• "Configure RBAC with least privilege"
• "Set up persistent storage for my database"
• "Spread pods across availability zones"

Operations Covered:

• Resource management (CPU/memory, QoS classes, quotas)
• Advanced scheduling (affinity, taints, topology spread)
• Networking (NetworkPolicies, Ingress, Gateway API)
• Storage operations (StorageClasses, PVCs, CSI)
• Security hardening (RBAC, Pod Security Standards, policies)
• Autoscaling (HPA, VPA, KEDA, cluster autoscaler)
• Troubleshooting (systematic debugging playbooks)

Resource Management

Quality of Service (QoS) Classes

Kubernetes assigns QoS classes based on resource requests and limits:

Guaranteed (Highest Priority):

• Requests equal limits for CPU and memory
• Never evicted unless exceeding limits
• Use for critical production services

resources:
  requests:
    memory: "512Mi"
    cpu: "500m"
  limits:
    memory: "512Mi"  # Same as request
    cpu: "500m"

Burstable (Medium Priority):

• Requests less than limits (or only requests set)
• Can burst above requests
• Evicted under node pressure
• Use for web servers, most applications

resources:
  requests:
    memory: "256Mi"
    cpu: "250m"
  limits:
    memory: "512Mi"  # 2x request
    cpu: "500m"

BestEffort (Lowest Priority):

• No requests or limits set
• First to be evicted under pressure
• Use only for development/testing

Decision Framework: Which QoS Class?

Resource Quotas and LimitRanges

Enforce multi-tenancy with ResourceQuotas (namespace limits) and LimitRanges (per-container defaults):

# ResourceQuota: Namespace-level limits
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-alpha
spec:
  hard:
    requests.cpu: "10"
    requests.memory: "20Gi"
    limits.cpu: "20"
    limits.memory: "40Gi"
    pods: "50"

For detailed resource management patterns including Vertical Pod Autoscaler (VPA), see references/resource-management.md.

Advanced Scheduling

Node Affinity

Control which nodes pods schedule on with required (hard) or preferred (soft) constraints:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: node.kubernetes.io/instance-type
          operator: In
          values:
          - g4dn.xlarge  # GPU instance

Taints and Tolerations

Reserve nodes for specific workloads (inverse of affinity):

# Taint GPU nodes to prevent non-GPU workloads
kubectl taint nodes gpu-node-1 workload=gpu:NoSchedule

# Pod tolerates GPU taint
tolerations:
- key: "workload"
  operator: "Equal"
  value: "gpu"
  effect: "NoSchedule"

Topology Spread Constraints

Distribute pods evenly across failure domains (zones, nodes):

topologySpreadConstraints:
- maxSkew: 1  # Max difference in pod count
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app: critical-app

For advanced scheduling patterns including pod priority and preemption, see references/scheduling-patterns.md.

Networking

NetworkPolicies (Zero-Trust Security)

Implement default-deny security with NetworkPolicies:

# Default deny all traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

# Allow specific ingress (frontend → backend)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-allow-frontend
spec:
  podSelector:
    matchLabels:
      app: backend
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

Ingress vs. Gateway API

Ingress (Legacy):

• Widely supported, mature ecosystem
• Limited expressiveness
• Use for existing applications

Gateway API (Modern):

• Role-oriented design (cluster ops vs. app devs)
• More expressive (HTTPRoute, TCPRoute, TLSRoute)
• Recommended for new applications (GA in Kubernetes 1.29+)

# Gateway API example
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-routes
spec:
  parentRefs:
  - name: production-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - name: backend
      port: 8080

For detailed networking patterns including service mesh integration, see references/networking.md.

Storage

StorageClasses (Define Performance Tiers)

StorageClasses define storage tiers for different workload needs:

# AWS EBS SSD (high performance)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iopsPerGB: "50"
  encrypted: "true"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete

Storage Decision Matrix

Access Modes:

• ReadWriteOnce (RWO): Single node read-write (most common)
• ReadOnlyMany (ROX): Multiple nodes read-only
• ReadWriteMany (RWX): Multiple nodes read-write (requires network storage)

For detailed storage operations including volume snapshots and CSI drivers, see references/storage.md.

Security

RBAC (Role-Based Access Control)

Implement least-privilege access with RBAC:

# Role (namespace-scoped)
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: production
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]
---
# RoleBinding (assign role to user)
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: production
subjects:
- kind: User
  name: jane@example.com
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Pod Security Standards

Enforce secure pod configurations at the namespace level:

# Namespace with Restricted PSS (most secure)
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Pod Security Levels:

• Restricted: Most secure, removes all privilege escalations (use for applications)
• Baseline: Minimally restrictive, prevents known escalations
• Privileged: Unrestricted (only for system workloads)

For detailed security patterns including policy enforcement (Kyverno/OPA) and secrets management, see references/security.md.

Autoscaling

Horizontal Pod Autoscaler (HPA)

Scale pod replicas based on CPU, memory, or custom metrics:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # Wait 5min before scaling down

KEDA (Event-Driven Autoscaling)

Scale based on events beyond CPU/memory (queues, cron schedules, Prometheus metrics):

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: rabbitmq-scaler
spec:
  scaleTargetRef:
    name: message-processor
  minReplicaCount: 0   # Scale to zero when queue empty
  maxReplicaCount: 30
  triggers:
  - type: rabbitmq
    metadata:
      queueName: tasks
      queueLength: "10"  # Scale up when >10 messages

Autoscaling Decision Matrix

For detailed autoscaling patterns including VPA and cluster autoscaler configuration, see references/autoscaling.md.

Troubleshooting

Common Pod Issues

Pod Stuck in Pending:

kubectl describe pod <pod-name>

# Common causes:
# - Insufficient CPU/memory: Reduce requests or add nodes
# - Node selector mismatch: Fix nodeSelector or add labels
# - PVC not bound: Create PVC or fix name
# - Taint intolerance: Add toleration or remove taint

CrashLoopBackOff:

kubectl logs <pod-name>
kubectl logs <pod-name> --previous  # Check previous crash

# Common causes:
# - Application crash: Fix code or configuration
# - Missing environment variables: Add to deployment
# - Liveness probe failing: Increase initialDelaySeconds
# - OOMKilled: Increase memory limit or fix leak

ImagePullBackOff:

kubectl describe pod <pod-name>

# Common causes:
# - Image doesn't exist: Fix image name/tag
# - Authentication required: Create imagePullSecrets
# - Network issues: Check NetworkPolicies, firewall rules

Service Not Accessible:

kubectl get endpoints <service-name>  # Should list pod IPs

# If endpoints empty:
# - Service selector doesn't match pod labels
# - Pods aren't ready (readiness probe failing)
# - Check NetworkPolicies blocking traffic

For systematic troubleshooting playbooks including networking and storage issues, see references/troubleshooting.md.

Reference Documentation

Deep Dives

• references/resource-management.md - Resource requests/limits, QoS classes, ResourceQuotas, VPA
• references/scheduling-patterns.md - Node affinity, taints/tolerations, topology spread, priority
• references/networking.md - NetworkPolicies, Ingress, Gateway API, service mesh integration
• references/storage.md - StorageClasses, PVCs, CSI drivers, volume snapshots
• references/security.md - RBAC, Pod Security Standards, policy enforcement, secrets
• references/autoscaling.md - HPA, VPA, KEDA, cluster autoscaler configuration
• references/troubleshooting.md - Systematic debugging playbooks for common failures

Examples

• examples/manifests/ - Copy-paste ready YAML manifests
• examples/python/ - Automation scripts (audit, cost analysis, validation)
• examples/go/ - Operator development examples

Tools

• scripts/validate-resources.sh - Audit pods without resource limits
• scripts/audit-networkpolicies.sh - Find namespaces without NetworkPolicies
• scripts/cost-analysis.sh - Resource cost breakdown by namespace

Related Skills

• building-ci-pipelines - Deploy to Kubernetes from CI/CD (kubectl apply, Helm, GitOps)
• observability - Monitor clusters and workloads (Prometheus, Grafana, tracing)
• secret-management - Secure secrets in Kubernetes (External Secrets, Sealed Secrets)
• testing-strategies - Test manifests and deployments (Kubeval, Conftest, Kind)
• infrastructure-as-code - Provision Kubernetes clusters (Terraform, Cluster API)
• gitops-workflows - Declarative cluster management (Flux, ArgoCD)

Best Practices Summary

Resource Management:

• Always set CPU/memory requests and limits
• Use VPA for automated rightsizing
• Implement resource quotas per namespace
• Monitor actual usage vs. requests

Scheduling:

• Use topology spread constraints for high availability
• Apply taints for workload isolation (GPU, spot instances)
• Set pod priority for critical workloads

Networking:

• Implement NetworkPolicies with default-deny
• Use Gateway API for new applications
• Apply rate limiting at ingress layer

Storage:

• Use CSI drivers (not legacy provisioners)
• Define StorageClasses per performance tier
• Enable volume snapshots for stateful apps

Security:

• Enforce Pod Security Standards (Restricted for apps)
• Implement RBAC with least privilege
• Use policy engines for guardrails (Kyverno/OPA)
• Scan images for vulnerabilities

Autoscaling:

• Use HPA for stateless workloads
• Use KEDA for event-driven workloads
• Enable cluster autoscaler with limits
• Set PodDisruptionBudgets to prevent over-disruption

Autoscaling

Table of Contents

1. Horizontal Pod Autoscaler (HPA) 2. Vertical Pod Autoscaler (VPA) 3. KEDA (Event-Driven Autoscaling) 4. Cluster Autoscaler 5. Autoscaling Decision Framework

Horizontal Pod Autoscaler (HPA)

HPA v2 (Recommended)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-app-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: AverageValue
        averageValue: 500Mi
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # Wait 5min before scaling down
      policies:
      - type: Percent
        value: 50        # Scale down max 50% of current pods
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100       # Can double pods in 1 minute
        periodSeconds: 60
      - type: Pods
        value: 4         # Or add 4 pods, whichever is higher
        periodSeconds: 60
      selectPolicy: Max

Target Types

Utilization (Percentage):

metrics:
- type: Resource
  resource:
    name: cpu
    target:
      type: Utilization
      averageUtilization: 70  # 70% of requested CPU

AverageValue (Absolute):

metrics:
- type: Resource
  resource:
    name: memory
    target:
      type: AverageValue
      averageValue: 500Mi  # 500MB per pod

Value (Total):

metrics:
- type: Resource
  resource:
    name: cpu
    target:
      type: Value
      value: "2"  # 2 CPU cores total across all pods

Custom Metrics (Prometheus)

Prerequisites:

# Install Prometheus Adapter
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus-adapter prometheus-community/prometheus-adapter \
  --namespace monitoring

HPA with Custom Metric:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "100"  # Scale when >100 req/s per pod

Prometheus Adapter ConfigMap:

rules:
- seriesQuery: 'http_requests_total{namespace="production"}'
  resources:
    overrides:
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}
  name:
    matches: "^(.*)_total$"
    as: "${1}_per_second"
  metricsQuery: 'rate(<<.Series>>{<<.LabelMatchers>>}[2m])'

External Metrics (Datadog, New Relic)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: External
    external:
      metric:
        name: datadog.metric.name
        selector:
          matchLabels:
            app: api
      target:
        type: Value
        value: "100"

Scaling Behavior

Aggressive Scale-Up, Conservative Scale-Down:

behavior:
  scaleUp:
    stabilizationWindowSeconds: 0  # Scale up immediately
    policies:
    - type: Percent
      value: 100       # Double pods
      periodSeconds: 15
    - type: Pods
      value: 4         # Or add 4 pods
      periodSeconds: 15
    selectPolicy: Max
  scaleDown:
    stabilizationWindowSeconds: 300  # Wait 5 minutes
    policies:
    - type: Percent
      value: 25        # Remove max 25% of pods
      periodSeconds: 60

Prevent Flapping:

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300  # Look at 5min window
    selectPolicy: Max               # Take max recommendation
    policies:
    - type: Pods
      value: 1
      periodSeconds: 60

HPA Troubleshooting

# Check HPA status
kubectl get hpa -n production

# Describe HPA (see events)
kubectl describe hpa web-app-hpa -n production

# Common issues:
# 1. "unable to get metrics": Metrics server not installed
# 2. "failed to get cpu utilization": No resource requests set
# 3. "invalid metrics": Wrong metric name or query

Fix:

# Install metrics server
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# Ensure pods have resource requests
kubectl get pods -n production -o json | \
  jq '.items[].spec.containers[].resources.requests'

Vertical Pod Autoscaler (VPA)

Installation

git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler
./hack/vpa-up.sh

VPA Configuration

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app

  updatePolicy:
    updateMode: "Auto"  # Off, Initial, Recreate, Auto

  resourcePolicy:
    containerPolicies:
    - containerName: app
      minAllowed:
        cpu: "100m"
        memory: "128Mi"
      maxAllowed:
        cpu: "2"
        memory: "2Gi"
      controlledResources: ["cpu", "memory"]
      mode: "Auto"
    - containerName: sidecar
      mode: "Off"  # Don't VPA this container

Update Modes

Off (Recommendations Only):

updatePolicy:
  updateMode: "Off"

• VPA calculates recommendations
• No automatic updates
• View: kubectl describe vpa my-app-vpa

Initial:

updatePolicy:
  updateMode: "Initial"

• Apply recommendations on pod creation
• Existing pods unchanged
• Good for StatefulSets

Recreate:

updatePolicy:
  updateMode: "Recreate"

• Update running pods (causes restart)
• Evicts and recreates with new resources
• Not for single-replica

Auto (Future):

updatePolicy:
  updateMode: "Auto"

• In-place updates without restart
• Not yet production-ready

View VPA Recommendations

kubectl describe vpa my-app-vpa

# Output:
# Recommendation:
#   Container Recommendations:
#     Container Name:  app
#     Lower Bound:
#       Cpu:     250m
#       Memory:  256Mi
#     Target:
#       Cpu:     500m
#       Memory:  512Mi
#     Upper Bound:
#       Cpu:     1
#       Memory:  1Gi
#     Uncapped Target:
#       Cpu:     750m
#       Memory:  768Mi

Interpretation:

• Lower Bound: Minimum for cost efficiency
• Target: Recommended values
• Upper Bound: Maximum safe values
• Uncapped Target: What VPA would recommend without limits

VPA + HPA Compatibility

Compatible:

# VPA on CPU/memory
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: app
      controlledResources: ["memory"]  # Only memory
---
# HPA on CPU
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Not Compatible:

• VPA Auto/Recreate + HPA on same metric (CPU or memory)
• Results in thrashing and instability

KEDA (Event-Driven Autoscaling)

Installation

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda --namespace keda --create-namespace

RabbitMQ Scaler

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: rabbitmq-scaler
  namespace: production
spec:
  scaleTargetRef:
    name: message-processor
  minReplicaCount: 0   # Scale to zero when queue empty
  maxReplicaCount: 30
  triggers:
  - type: rabbitmq
    metadata:
      host: amqp://user:password@rabbitmq.default.svc.cluster.local:5672
      queueName: tasks
      queueLength: "10"  # Scale up when >10 messages

Kafka Scaler

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kafka-scaler
spec:
  scaleTargetRef:
    name: kafka-consumer
  minReplicaCount: 1
  maxReplicaCount: 50
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: kafka-broker.kafka:9092
      consumerGroup: my-consumer-group
      topic: events
      lagThreshold: "100"  # Scale when lag >100

AWS SQS Scaler

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sqs-scaler
spec:
  scaleTargetRef:
    name: sqs-processor
  minReplicaCount: 0
  maxReplicaCount: 20
  triggers:
  - type: aws-sqs-queue
    metadata:
      queueURL: https://sqs.us-east-1.amazonaws.com/123456789/my-queue
      queueLength: "5"
      awsRegion: us-east-1
      identityOwner: operator  # Use IRSA

Prometheus Scaler

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: prometheus-scaler
spec:
  scaleTargetRef:
    name: api-server
  minReplicaCount: 2
  maxReplicaCount: 20
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring.svc:9090
      metricName: http_requests_per_second
      query: sum(rate(http_requests_total{service="api"}[2m]))
      threshold: "100"  # Scale when >100 req/sec total

Cron Scaler (Schedule-Based)

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: cron-scaler
spec:
  scaleTargetRef:
    name: batch-processor
  minReplicaCount: 0
  maxReplicaCount: 10
  triggers:
  - type: cron
    metadata:
      timezone: America/New_York
      start: 0 8 * * 1-5     # Scale up at 8am Mon-Fri
      end: 0 18 * * 1-5      # Scale down at 6pm Mon-Fri
      desiredReplicas: "5"

CPU/Memory Scaler (KEDA Alternative to HPA)

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: cpu-scaler
spec:
  scaleTargetRef:
    name: web-app
  minReplicaCount: 2
  maxReplicaCount: 10
  triggers:
  - type: cpu
    metricType: Utilization
    metadata:
      value: "70"
  - type: memory
    metricType: Utilization
    metadata:
      value: "80"

KEDA Fallback

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: fallback-scaler
spec:
  scaleTargetRef:
    name: processor
  fallback:
    failureThreshold: 3
    replicas: 5  # Scale to 5 if scaler fails
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring.svc:9090
      query: queue_depth
      threshold: "10"

Cluster Autoscaler

AWS Installation

helm repo add autoscaler https://kubernetes.github.io/autoscaler
helm install cluster-autoscaler autoscaler/cluster-autoscaler \
  --namespace kube-system \
  --set autoDiscovery.clusterName=my-cluster \
  --set awsRegion=us-east-1 \
  --set rbac.create=true

Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-autoscaler-status
  namespace: kube-system
data:
  scale-down-enabled: "true"
  scale-down-delay-after-add: "10m"
  scale-down-unneeded-time: "10m"
  scale-down-utilization-threshold: "0.5"
  max-node-provision-time: "15m"
  skip-nodes-with-local-storage: "false"
  skip-nodes-with-system-pods: "true"

Node Groups (AWS)

# Auto Scaling Group tags
k8s.io/cluster-autoscaler/<cluster-name>=owned
k8s.io/cluster-autoscaler/enabled=true
k8s.io/cluster-autoscaler/node-template/label/workload=general

Cluster Autoscaler Behavior

Scale Up:

• Pods are Pending due to insufficient resources
• Provision new nodes from auto-scaling group
• Takes 2-5 minutes (cloud provider dependent)

Scale Down:

• Node utilization < threshold (default 50%)
• All pods can be rescheduled elsewhere
• Node not running system pods (unless configured)
• Wait 10 minutes (configurable) before scale down

Prevent Scale Down:

# Annotate pod to prevent node scale-down
apiVersion: v1
kind: Pod
metadata:
  annotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

PodDisruptionBudget (PDB)

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
  namespace: production
spec:
  minAvailable: 2  # Keep at least 2 replicas
  selector:
    matchLabels:
      app: api

Alternative (percentage):

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  maxUnavailable: 1  # Max 1 pod disrupted
  selector:
    matchLabels:
      app: api

Autoscaling Decision Framework

Decision Matrix

Pattern Combinations

Web Application (Full Stack):

# 1. HPA for horizontal scaling
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-hpa
spec:
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
---
# 2. VPA for right-sizing (recommendations only)
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: web-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  updatePolicy:
    updateMode: "Off"  # Recommendations only
---
# 3. PDB to protect during scale-down
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: web
---
# 4. Cluster Autoscaler (configured at cluster level)

Queue Processor:

# KEDA for event-driven scaling
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: processor-scaler
spec:
  scaleTargetRef:
    name: processor
  minReplicaCount: 0
  maxReplicaCount: 50
  triggers:
  - type: rabbitmq
    metadata:
      queueName: tasks
      queueLength: "10"
---
# VPA for right-sizing (Off mode)
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: processor-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: processor
  updatePolicy:
    updateMode: "Off"

Summary

Autoscaling Pattern Selection:

Best Practices: 1. Start with HPA for stateless workloads 2. Use KEDA for event-driven scaling (better than HPA for queues) 3. Use VPA in Off mode first (review recommendations) 4. Don't combine VPA Auto + HPA on same metric 5. Set PodDisruptionBudgets to protect critical workloads 6. Enable Cluster Autoscaler with appropriate limits 7. Monitor autoscaling events and adjust thresholds 8. Test autoscaling behavior under load before production

Networking

Table of Contents

1. NetworkPolicies 2. Services 3. Ingress 4. Gateway API 5. Service Mesh Integration 6. DNS and Service Discovery

NetworkPolicies

Default Deny Pattern

Implement zero-trust networking with default-deny policies:

# Step 1: Deny all ingress and egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}  # Applies to all pods
  policyTypes:
  - Ingress
  - Egress

# Step 2: Allow specific traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-allow-frontend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

Ingress Policies

Allow from specific namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-frontend-ns
  namespace: backend
spec:
  podSelector:
    matchLabels:
      app: api
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: frontend
    ports:
    - protocol: TCP
      port: 8080

Allow from external IP range:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-external-lb
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: web
  ingress:
  - from:
    - ipBlock:
        cidr: 10.0.0.0/16      # Load balancer CIDR
        except:
        - 10.0.5.0/24          # Exclude monitoring subnet
    ports:
    - protocol: TCP
      port: 80

Egress Policies

Allow DNS only:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    - podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53

Allow external API:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-external-api
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Egress
  egress:
  # Allow DNS
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    ports:
    - protocol: UDP
      port: 53
  # Allow external API
  - to:
    - ipBlock:
        cidr: 203.0.113.0/24  # External API CIDR
    ports:
    - protocol: TCP
      port: 443

Allow all egress (common pattern):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-egress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Egress
  egress:
  - {}  # Allow all egress

Database Access Pattern

# Backend → Database
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: postgres-allow-backend
  namespace: database
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: backend
      podSelector:
        matchLabels:
          tier: api
    ports:
    - protocol: TCP
      port: 5432

Multi-Tier Application Pattern

# Tier 1: Frontend → only from ingress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      tier: frontend
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: ingress-nginx
    ports:
    - protocol: TCP
      port: 80
  egress:
  - to:
    - podSelector:
        matchLabels:
          tier: backend
    ports:
    - protocol: TCP
      port: 8080
  - to:  # DNS
    - namespaceSelector:
        matchLabels:
          name: kube-system
    ports:
    - protocol: UDP
      port: 53
---
# Tier 2: Backend → only from frontend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      tier: backend
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          tier: frontend
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          tier: database
    ports:
    - protocol: TCP
      port: 5432
  - to:  # DNS
    - namespaceSelector:
        matchLabels:
          name: kube-system
    ports:
    - protocol: UDP
      port: 53
---
# Tier 3: Database → only from backend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: database-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      tier: database
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          tier: backend
    ports:
    - protocol: TCP
      port: 5432

Testing NetworkPolicies

# Test connectivity from pod
kubectl run test --rm -it --image=busybox -- wget -O- http://backend:8080

# If connection refused:
# 1. Check NetworkPolicy exists
kubectl get networkpolicies -n production

# 2. Describe policy
kubectl describe networkpolicy backend-allow-frontend -n production

# 3. Check pod labels match selectors
kubectl get pods -n production --show-labels

# 4. Test DNS resolution
kubectl run test --rm -it --image=busybox -- nslookup backend

# 5. Check CNI plugin supports NetworkPolicies
kubectl get nodes -o wide
# (Calico, Cilium, Weave support NetworkPolicies; Flannel does not)

Services

Service Types

ClusterIP (Default):

• Internal cluster access only
• Use for internal microservices

apiVersion: v1
kind: Service
metadata:
  name: backend
spec:
  type: ClusterIP
  selector:
    app: backend
  ports:
  - protocol: TCP
    port: 80        # Service port
    targetPort: 8080  # Container port

NodePort:

• Expose on each node's IP at static port
• Use for testing or edge cases

apiVersion: v1
kind: Service
metadata:
  name: nodeport-svc
spec:
  type: NodePort
  selector:
    app: web
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
    nodePort: 30080  # Optional (auto-assigned if omitted)

LoadBalancer:

• Provision cloud load balancer
• Use for external HTTP/gRPC services

apiVersion: v1
kind: Service
metadata:
  name: web-lb
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb  # AWS NLB
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

ExternalName:

• DNS CNAME to external service
• Use for migrating to Kubernetes

apiVersion: v1
kind: Service
metadata:
  name: legacy-db
spec:
  type: ExternalName
  externalName: db.example.com

Headless Services

For StatefulSets requiring stable network IDs:

apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  clusterIP: None  # Headless
  selector:
    app: postgres
  ports:
  - protocol: TCP
    port: 5432

DNS entries:

• postgres-0.postgres.default.svc.cluster.local
• postgres-1.postgres.default.svc.cluster.local

Session Affinity

apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800  # 3 hours
  ports:
  - protocol: TCP
    port: 80

Service Topology

Route traffic to same zone/node for cost savings:

apiVersion: v1
kind: Service
metadata:
  name: backend
spec:
  selector:
    app: backend
  topologyKeys:
  - "kubernetes.io/hostname"           # Prefer same node
  - "topology.kubernetes.io/zone"      # Then same zone
  - "*"                                # Then any node
  ports:
  - protocol: TCP
    port: 8080

Ingress

Basic Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: backend
            port:
              number: 8080
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend
            port:
              number: 80

TLS/HTTPS

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tls-ingress
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - myapp.example.com
    secretName: myapp-tls  # Created by cert-manager
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: frontend
            port:
              number: 80

Path Types

Prefix:

path: /api
pathType: Prefix
# Matches: /api, /api/, /api/v1, /api/v1/users

Exact:

path: /api
pathType: Exact
# Matches: /api only (not /api/ or /api/v1)

ImplementationSpecific:

path: /api
pathType: ImplementationSpecific
# Depends on ingress controller

Common Annotations (Nginx)

metadata:
  annotations:
    # Rate limiting
    nginx.ingress.kubernetes.io/limit-rps: "10"

    # Timeouts
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "30"

    # CORS
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "https://example.com"

    # Authentication
    nginx.ingress.kubernetes.io/auth-url: "https://auth.example.com"
    nginx.ingress.kubernetes.io/auth-response-headers: "X-User-ID"

    # Rewrite
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    # Matches: /something(/|$)(.*)

Gateway API

Gateway Resource

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: production-gateway
spec:
  gatewayClassName: nginx
  listeners:
  - name: http
    protocol: HTTP
    port: 80
  - name: https
    protocol: HTTPS
    port: 443
    tls:
      mode: Terminate
      certificateRefs:
      - name: myapp-tls
    allowedRoutes:
      namespaces:
        from: Selector
        selector:
          matchLabels:
            gateway: production

HTTPRoute

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app-routes
  namespace: production
spec:
  parentRefs:
  - name: production-gateway
    namespace: default
  hostnames:
  - myapp.example.com
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - name: backend
      port: 8080
      weight: 100
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: frontend
      port: 80

Advanced Routing

Header-based routing:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: header-route
spec:
  parentRefs:
  - name: production-gateway
  rules:
  - matches:
    - headers:
      - name: X-Version
        value: v2
    backendRefs:
    - name: backend-v2
      port: 8080
  - matches:
    - headers:
      - name: X-Version
        value: v1
    backendRefs:
    - name: backend-v1
      port: 8080

Traffic splitting (canary):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: canary-route
spec:
  parentRefs:
  - name: production-gateway
  rules:
  - backendRefs:
    - name: backend-v2
      port: 8080
      weight: 10     # 10% traffic
    - name: backend-v1
      port: 8080
      weight: 90     # 90% traffic

Query parameter routing:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: query-route
spec:
  parentRefs:
  - name: production-gateway
  rules:
  - matches:
    - queryParams:
      - name: debug
        value: "true"
    backendRefs:
    - name: backend-debug
      port: 8080

Gateway vs. Ingress Comparison

Service Mesh Integration

Istio Integration

Enable sidecar injection:

kubectl label namespace production istio-injection=enabled

VirtualService (L7 routing):

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: backend
spec:
  hosts:
  - backend
  http:
  - match:
    - headers:
        x-version:
          exact: v2
    route:
    - destination:
        host: backend
        subset: v2
  - route:
    - destination:
        host: backend
        subset: v1

DestinationRule (load balancing, circuit breaking):

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: backend
spec:
  host: backend
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 50
        http2MaxRequests: 100
    outlierDetection:
      consecutiveErrors: 5
      interval: 30s
      baseEjectionTime: 30s

Linkerd Integration

Inject proxy:

kubectl annotate namespace production linkerd.io/inject=enabled

ServiceProfile (retries, timeouts):

apiVersion: linkerd.io/v1alpha2
kind: ServiceProfile
metadata:
  name: backend.production.svc.cluster.local
  namespace: production
spec:
  routes:
  - name: GET /api/users
    condition:
      method: GET
      pathRegex: /api/users
    responseClasses:
    - condition:
        status:
          min: 500
          max: 599
      isFailure: true
    retries:
      maxAttempts: 3
      timeout: 10s

DNS and Service Discovery

DNS Names

Within same namespace:

• backend → backend.production.svc.cluster.local

Cross-namespace:

• backend.production → backend.production.svc.cluster.local

Full FQDN:

• backend.production.svc.cluster.local

Custom DNS

apiVersion: v1
kind: Pod
metadata:
  name: custom-dns
spec:
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
    - 8.8.8.8
    - 8.8.4.4
    searches:
    - production.svc.cluster.local
    - svc.cluster.local
    - cluster.local
    options:
    - name: ndots
      value: "2"
  containers:
  - name: app
    image: myapp:latest

DNS Caching

# NodeLocal DNSCache (DaemonSet)
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-local-dns
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: node-local-dns
  template:
    spec:
      containers:
      - name: node-cache
        image: registry.k8s.io/dns/k8s-dns-node-cache:1.22.20

Summary

Networking Pattern Selection:

Best Practices: 1. Always implement NetworkPolicies (start with default-deny) 2. Use Gateway API for new applications 3. Enable TLS for external traffic (cert-manager) 4. Consider service mesh for advanced routing and mTLS 5. Test NetworkPolicies thoroughly before production 6. Monitor DNS resolution latency 7. Use headless services for StatefulSets

Resource Management

Table of Contents

1. Resource Requests and Limits 2. Quality of Service Classes 3. Resource Quotas 4. LimitRanges 5. Vertical Pod Autoscaler 6. Cost Optimization Patterns

Resource Requests and Limits

CPU Resources

CPU Units:

• 1 CPU = 1 vCPU/core on cloud provider
• 1000m (millicores) = 1 CPU
• 100m = 0.1 CPU (10% of one core)

CPU Requests:

• Minimum guaranteed CPU for scheduling
• Scheduler finds nodes with available CPU
• Pod can use more than request (up to limit)

CPU Limits:

• Maximum CPU pod can use
• Enforced by throttling (CFS quota)
• Pod slowed down, not killed

Example:

resources:
  requests:
    cpu: "500m"     # Guaranteed 0.5 cores
  limits:
    cpu: "1"        # Max 1 core (can burst 2x)

Memory Resources

Memory Units:

• Ki/Mi/Gi (binary, 1024-based): 1Ki = 1024 bytes, 1Mi = 1048576 bytes
• k/M/G (decimal, 1000-based): 1k = 1000 bytes
• Use binary units (Ki/Mi/Gi) for consistency

Memory Requests:

• Minimum guaranteed memory for scheduling
• Pod evicted if exceeds request under node pressure

Memory Limits:

• Maximum memory pod can allocate
• Pod killed (OOMKilled) if exceeds limit
• No throttling like CPU

Example:

resources:
  requests:
    memory: "512Mi"   # Guaranteed 512MB
  limits:
    memory: "1Gi"     # Max 1GB (OOMKilled if exceeded)

Compressible vs. Incompressible Resources

Compressible (CPU):

• Can be throttled without killing pod
• Degraded performance, but pod survives
• Safer to overcommit

Incompressible (Memory):

• Cannot be reclaimed once allocated
• Pod killed if exceeds limit or under pressure
• Must be carefully sized

Quality of Service Classes

Kubernetes assigns QoS classes automatically based on requests and limits:

Guaranteed QoS

Criteria:

• Every container has CPU and memory requests
• Every container has CPU and memory limits
• Requests equal limits for both CPU and memory

Behavior:

• Highest priority
• Never evicted unless exceeding limits
• Predictable performance
• Most expensive (can't overcommit)

Example:

apiVersion: v1
kind: Pod
metadata:
  name: guaranteed-pod
spec:
  containers:
  - name: app
    image: myapp:latest
    resources:
      requests:
        memory: "512Mi"
        cpu: "500m"
      limits:
        memory: "512Mi"  # Same as request
        cpu: "500m"      # Same as request

Use Cases:

• Critical production services (payments, auth)
• Databases requiring stable performance
• Real-time processing systems
• Compliance-required workloads

Burstable QoS

Criteria:

• At least one container has CPU or memory request
• Requests less than limits (or only requests set)

Behavior:

• Medium priority
• Can burst above requests (use spare capacity)
• Evicted under node pressure (after BestEffort)
• Cost-effective balance

Example:

apiVersion: v1
kind: Pod
metadata:
  name: burstable-pod
spec:
  containers:
  - name: app
    image: myapp:latest
    resources:
      requests:
        memory: "256Mi"
        cpu: "250m"
      limits:
        memory: "512Mi"  # 2x request (can burst)
        cpu: "500m"      # 2x request

Use Cases:

• Web servers (handle traffic spikes)
• API services (most common pattern)
• Batch jobs (use spare capacity)
• Non-critical background workers

BestEffort QoS

Criteria:

• No requests or limits set for any container

Behavior:

• Lowest priority
• First to be evicted under node pressure
• No resource guarantees
• Most cost-effective

Example:

apiVersion: v1
kind: Pod
metadata:
  name: besteffort-pod
spec:
  containers:
  - name: app
    image: myapp:latest
    # No resources specified

Use Cases:

• Development environments
• Testing workloads
• Experimental services
• Low-priority batch jobs

QoS Selection Decision Tree

START: Which QoS class for my workload?

Q1: Is this a critical production service?
  ├─ YES → Can you tolerate ANY performance variability?
  │   ├─ YES → Burstable (cost-effective)
  │   └─ NO  → Guaranteed (predictable)
  └─ NO  → Is this production at all?
      ├─ YES → Burstable (default for most apps)
      └─ NO  → BestEffort (dev/test only)

Resource Quotas

Namespace-Level Limits

ResourceQuotas enforce limits at the namespace level for multi-tenancy:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-alpha
spec:
  hard:
    # Compute resources
    requests.cpu: "10"        # Max 10 CPU cores requested
    requests.memory: "20Gi"   # Max 20GB memory requested
    limits.cpu: "20"          # Max 20 CPU cores limit
    limits.memory: "40Gi"     # Max 40GB memory limit

    # Storage
    persistentvolumeclaims: "10"
    requests.storage: "100Gi"

    # Object counts
    pods: "50"
    services: "10"
    secrets: "20"
    configmaps: "20"

Quota Scopes

Apply quotas to specific pod classes:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: best-effort-quota
  namespace: team-alpha
spec:
  hard:
    pods: "5"  # Max 5 BestEffort pods
  scopes:
  - BestEffort
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: high-priority-quota
  namespace: team-alpha
spec:
  hard:
    requests.cpu: "20"
    requests.memory: "40Gi"
  scopeSelector:
    matchExpressions:
    - operator: In
      scopeName: PriorityClass
      values: ["high"]

Checking Quota Usage

# View quota details
kubectl describe resourcequota -n team-alpha

# Example output:
# Name:            team-quota
# Namespace:       team-alpha
# Resource         Used   Hard
# --------         ----   ----
# requests.cpu     8      10
# requests.memory  15Gi   20Gi
# pods             35     50

LimitRanges

Default Constraints per Container

LimitRanges set defaults and boundaries for individual containers:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: team-alpha
spec:
  limits:
  - max:
      memory: "2Gi"    # No container can exceed 2GB
      cpu: "2"         # No container can exceed 2 cores
    min:
      memory: "128Mi"  # All containers must request at least 128MB
      cpu: "100m"      # All containers must request at least 0.1 cores
    default:
      memory: "512Mi"  # Applied if no limit specified
      cpu: "500m"
    defaultRequest:
      memory: "256Mi"  # Applied if no request specified
      cpu: "250m"
    maxLimitRequestRatio:
      memory: "2"      # Limit can't exceed 2x request
      cpu: "2"
    type: Container

LimitRange for PVCs

apiVersion: v1
kind: LimitRange
metadata:
  name: storage-limits
  namespace: team-alpha
spec:
  limits:
  - max:
      storage: "50Gi"
    min:
      storage: "1Gi"
    type: PersistentVolumeClaim

LimitRange Behavior

When pod is created: 1. If no request set → apply defaultRequest 2. If no limit set → apply default 3. Validate against min and max 4. Validate maxLimitRequestRatio 5. Reject pod if violations found

Vertical Pod Autoscaler

VPA Modes

Off (Recommendations Only):

• Safest starting point
• VPA calculates recommendations
• No automatic changes
• View recommendations: kubectl describe vpa

Initial:

• Apply recommendations only on pod creation
• Existing pods unchanged
• Useful for StatefulSets

Recreate:

• Update running pods (causes restart)
• Evicts and recreates pods with new resources
• Not suitable for single-replica workloads

Auto (Future):

• Update requests in-place without restart
• Not yet available in production

VPA Configuration

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app

  updatePolicy:
    updateMode: "Off"  # Start with recommendations only

  resourcePolicy:
    containerPolicies:
    - containerName: app
      minAllowed:
        cpu: "100m"
        memory: "128Mi"
      maxAllowed:
        cpu: "2"
        memory: "2Gi"
      controlledResources: ["cpu", "memory"]
      mode: "Auto"  # Per-container override

VPA Recommendations

View VPA recommendations:

kubectl describe vpa my-app-vpa

# Output includes:
# Recommendation:
#   Container Recommendations:
#     Container Name:  app
#     Lower Bound:     (minimum for cost efficiency)
#       Cpu:     250m
#       Memory:  256Mi
#     Target:          (recommended values)
#       Cpu:     500m
#       Memory:  512Mi
#     Upper Bound:     (maximum safe values)
#       Cpu:     1
#       Memory:  1Gi

VPA Best Practices

When to Use VPA:

• ✅ Stateless applications with predictable patterns
• ✅ Long-running services (collect 7+ days of metrics)
• ✅ Applications where restarts are acceptable
• ✅ Unknown resource requirements (let VPA learn)

When to Avoid VPA:

• ❌ Running with HPA on same metric (conflicts)
• ❌ Stateful apps requiring stable resources
• ❌ Single-replica critical workloads
• ❌ Short-lived jobs (insufficient data)

VPA + HPA Compatibility:

• ✅ VPA on CPU/memory + HPA on custom metric
• ✅ VPA off mode (recommendations) + HPA on any metric
• ❌ VPA auto mode + HPA on same metric (CPU/memory)

VPA Installation

# Install VPA (Helm)
helm repo add fairwinds-stable https://charts.fairwinds.com/stable
helm install vpa fairwinds-stable/vpa --namespace vpa --create-namespace

# Or via manifest
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler
./hack/vpa-up.sh

Cost Optimization Patterns

Right-Sizing Workflow

Phase 1: Baseline (Week 1) 1. Deploy with conservative requests (overestimate) 2. Set higher limits (allow bursting) 3. Monitor actual usage

# Initial deployment (conservative)
resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "2"
    memory: "2Gi"

Phase 2: Observation (Week 2-4) 1. Deploy VPA in "Off" mode 2. Collect 2-4 weeks of usage data 3. Review VPA recommendations 4. Check Prometheus/metrics for actual usage

# View actual usage
kubectl top pods -n production

# View VPA recommendations
kubectl describe vpa my-app-vpa

Phase 3: Optimization (Week 5+) 1. Apply VPA recommendations 2. Monitor for OOMKilled or CPU throttling 3. Adjust based on real behavior 4. Iterate quarterly

# Optimized resources (based on VPA)
resources:
  requests:
    cpu: "250m"      # Reduced from 500m
    memory: "256Mi"  # Reduced from 512Mi
  limits:
    cpu: "500m"      # Tighter than initial 2
    memory: "512Mi"  # Tighter than initial 2Gi

Cost Monitoring

# Audit pods without resource limits
kubectl get pods --all-namespaces -o json | \
  jq -r '.items[] | select(.spec.containers[].resources.limits == null) | .metadata.name'

# Calculate namespace cost (requests)
kubectl get pods -n production -o json | \
  jq -r '.items[].spec.containers[] |
    (.resources.requests.cpu // "0") + " " +
    (.resources.requests.memory // "0")'

Cost Allocation Tags

Label resources for cost allocation:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  labels:
    app: my-app
    team: backend          # Cost center
    env: production        # Environment
    cost-center: "12345"   # Finance tracking
spec:
  containers:
  - name: app
    image: myapp:latest

Savings Estimates

Typical Cost Reductions:

• Overprovisioned CPU: 40-60% reduction
• Overprovisioned Memory: 30-50% reduction
• BestEffort → Burstable: No cost change (same requests)
• Burstable → Guaranteed: Cost increase (tighter limits)

Example Calculation:

• Before: 100 pods × 2 CPU × 4 GB = 200 cores, 400GB
• After VPA: 100 pods × 0.5 CPU × 2 GB = 50 cores, 200GB
• Savings: 75% CPU, 50% memory = ~60% total cost

Resource Waste Detection

# Python script: detect overprovisioned pods
from kubernetes import client, config

config.load_kube_config()
metrics = client.CustomObjectsApi()
v1 = client.CoreV1Api()

# Get pod metrics
metrics_data = metrics.list_cluster_custom_object(
    "metrics.k8s.io", "v1beta1", "pods"
)

for pod in metrics_data["items"]:
    for container in pod["containers"]:
        usage_cpu = container["usage"]["cpu"]
        usage_mem = container["usage"]["memory"]

        # Compare to requests
        pod_obj = v1.read_namespaced_pod(
            pod["metadata"]["name"],
            pod["metadata"]["namespace"]
        )

        for c in pod_obj.spec.containers:
            if c.resources.requests:
                req_cpu = c.resources.requests.get("cpu")
                req_mem = c.resources.requests.get("memory")

                # Flag if usage < 20% of request
                # (potential waste)

Monitoring and Alerts

Key Metrics to Track

Per-Pod:

• container_cpu_usage_seconds_total - Actual CPU usage
• container_memory_working_set_bytes - Actual memory usage
• kube_pod_container_resource_requests - Configured requests
• kube_pod_container_resource_limits - Configured limits

Per-Node:

• kube_node_status_allocatable - Available resources
• kube_node_status_capacity - Total capacity
• Node CPU/memory utilization

Prometheus Queries

# CPU throttling (pods hitting limits)
rate(container_cpu_cfs_throttled_seconds_total[5m]) > 0.1

# Memory usage vs. limit
container_memory_working_set_bytes / container_spec_memory_limit_bytes > 0.9

# Pods without requests
kube_pod_container_resource_requests{resource="memory"} == 0

# Node CPU pressure
(1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m]))) > 0.8

Recommended Alerts

# Alert: Pod OOMKilled
- alert: PodOOMKilled
  expr: increase(kube_pod_container_status_terminated_reason{reason="OOMKilled"}[5m]) > 0
  annotations:
    summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} was OOMKilled"

# Alert: High CPU throttling
- alert: HighCPUThrottling
  expr: rate(container_cpu_cfs_throttled_seconds_total[5m]) > 0.2
  annotations:
    summary: "Pod {{ $labels.pod }} is CPU throttled >20%"

# Alert: No resource limits
- alert: MissingResourceLimits
  expr: kube_pod_container_resource_limits{resource="memory"} == 0
  annotations:
    summary: "Pod {{ $labels.pod }} has no memory limit"

Summary

Key Takeaways: 1. Always set requests and limits to enable QoS 2. Use Guaranteed QoS for critical services 3. Use Burstable QoS for most applications 4. Implement ResourceQuotas for multi-tenancy 5. Use VPA for automated rightsizing 6. Monitor actual usage vs. requests 7. Iterate on resource sizing quarterly 8. Label resources for cost allocation

Scheduling Patterns

Table of Contents

1. Node Affinity 2. Taints and Tolerations 3. Topology Spread Constraints 4. Pod Priority and Preemption 5. Pod Affinity and Anti-Affinity 6. Scheduling Decision Framework

Node Affinity

Required Node Affinity (Hard Constraint)

Pod will not schedule unless node matches criteria:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-workload
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node.kubernetes.io/instance-type
            operator: In
            values:
            - g4dn.xlarge   # AWS GPU instance
            - g4dn.2xlarge
          - key: topology.kubernetes.io/zone
            operator: In
            values:
            - us-west-2a
            - us-west-2b
  containers:
  - name: ml-training
    image: pytorch:latest
    resources:
      limits:
        nvidia.com/gpu: 1

Preferred Node Affinity (Soft Constraint)

Scheduler tries to match but will schedule anyway if no match:

apiVersion: v1
kind: Pod
metadata:
  name: cache-service
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100  # Highest preference
        preference:
          matchExpressions:
          - key: node.kubernetes.io/instance-type
            operator: In
            values:
            - r5.xlarge  # Prefer memory-optimized
      - weight: 50   # Lower preference
        preference:
          matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values:
            - us-west-2a  # Prefer this zone (cost savings)
  containers:
  - name: redis
    image: redis:7

Node Affinity Operators

Common Node Labels

# Built-in labels
kubernetes.io/hostname=node-1
kubernetes.io/os=linux
kubernetes.io/arch=amd64
node.kubernetes.io/instance-type=m5.xlarge
topology.kubernetes.io/zone=us-west-2a
topology.kubernetes.io/region=us-west-2

# Custom labels
kubectl label nodes node-1 workload=gpu
kubectl label nodes node-2 storage=ssd
kubectl label nodes node-3 team=backend

Use Cases

GPU Scheduling:

# Require GPU nodes
requiredDuringSchedulingIgnoredDuringExecution:
  nodeSelectorTerms:
  - matchExpressions:
    - key: accelerator
      operator: In
      values:
      - nvidia-tesla-v100

SSD Storage:

# Prefer SSD nodes for databases
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
  preference:
    matchExpressions:
    - key: storage
      operator: In
      values:
      - ssd

Cost Optimization:

# Prefer spot instances (non-critical workloads)
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
  preference:
    matchExpressions:
    - key: node.kubernetes.io/instance-lifecycle
      operator: In
      values:
      - spot

Taints and Tolerations

Taint Effects

NoSchedule:

• New pods won't schedule
• Existing pods remain

PreferNoSchedule:

• Scheduler tries to avoid
• Will schedule if no other options

NoExecute:

• Evict existing pods immediately
• New pods won't schedule

Applying Taints

# Taint GPU nodes
kubectl taint nodes gpu-node-1 workload=gpu:NoSchedule

# Taint spot instances (may be terminated)
kubectl taint nodes spot-node-1 instance-type=spot:NoSchedule

# Taint nodes for maintenance
kubectl taint nodes node-1 maintenance=true:NoExecute

# Remove taint
kubectl taint nodes node-1 maintenance-

Pod Tolerations

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  tolerations:
  - key: "workload"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"
  - key: "instance-type"
    operator: "Equal"
    value: "spot"
    effect: "NoSchedule"
  containers:
  - name: ml-training
    image: pytorch:latest

Toleration Operators

Equal:

tolerations:
- key: "key1"
  operator: "Equal"
  value: "value1"
  effect: "NoSchedule"

Exists:

# Tolerate any value for key1
tolerations:
- key: "key1"
  operator: "Exists"
  effect: "NoSchedule"

Wildcard:

# Tolerate all taints
tolerations:
- operator: "Exists"

Toleration with Timeout

apiVersion: v1
kind: Pod
metadata:
  name: fault-tolerant-app
spec:
  tolerations:
  - key: "node.kubernetes.io/unreachable"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 300  # Wait 5min before eviction
  - key: "node.kubernetes.io/not-ready"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 300
  containers:
  - name: app
    image: myapp:latest

Common Taint Use Cases

Dedicated Nodes for Team:

# Taint nodes for team-alpha only
kubectl taint nodes node-1 node-2 node-3 team=alpha:NoSchedule

# Team-alpha pods tolerate
tolerations:
- key: "team"
  operator: "Equal"
  value: "alpha"
  effect: "NoSchedule"

Isolate System Workloads:

# Taint nodes for system components only
kubectl taint nodes master-1 node-role.kubernetes.io/master:NoSchedule

Handle Hardware Failures:

# Taint nodes with disk issues
kubectl taint nodes node-1 disk=failing:NoExecute

Topology Spread Constraints

Even Distribution Across Zones

apiVersion: apps/v1
kind: Deployment
metadata:
  name: critical-app
spec:
  replicas: 9
  selector:
    matchLabels:
      app: critical-app
  template:
    metadata:
      labels:
        app: critical-app
    spec:
      topologySpreadConstraints:
      - maxSkew: 1              # Max difference in pod count
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule  # Hard constraint
        labelSelector:
          matchLabels:
            app: critical-app
      containers:
      - name: app
        image: myapp:latest

Result: 9 replicas spread evenly across 3 zones (3 per zone)

Multi-Level Spreading

topologySpreadConstraints:
# Level 1: Spread across zones (hard)
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app: critical-app

# Level 2: Spread across nodes within zones (soft)
- maxSkew: 2
  topologyKey: kubernetes.io/hostname
  whenUnsatisfiable: ScheduleAnyway
  labelSelector:
    matchLabels:
      app: critical-app

maxSkew Behavior

maxSkew: 1 (Strict)

• Zone A: 3 pods
• Zone B: 3 pods
• Zone C: 3 pods
• Cannot add pod to Zone A until B and C catch up

maxSkew: 2 (Relaxed)

• Zone A: 4 pods
• Zone B: 3 pods
• Zone C: 2 pods
• Max difference is 2 (4-2=2)

whenUnsatisfiable

DoNotSchedule (Hard):

• Pod remains Pending if constraint violated
• Strict high-availability requirement

ScheduleAnyway (Soft):

• Scheduler tries to satisfy but schedules anyway
• Preference, not requirement

minDomains

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app: critical-app
  minDomains: 3  # Require at least 3 zones

Topology Spread vs. Pod Anti-Affinity

Topology Spread (Modern):

• More intuitive (maxSkew semantics)
• Better control over distribution
• Recommended for new applications

Pod Anti-Affinity (Legacy):

• More complex configuration
• Less flexible
• Still widely used

Migration Example:

# Old: Pod anti-affinity
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        labelSelector:
          matchLabels:
            app: myapp
        topologyKey: kubernetes.io/hostname

# New: Topology spread
topologySpreadConstraints:
- maxSkew: 1
  topologyKey: kubernetes.io/hostname
  whenUnsatisfiable: ScheduleAnyway
  labelSelector:
    matchLabels:
      app: myapp

Pod Priority and Preemption

Priority Classes

# High priority for critical services
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "Critical production services"
---
# Medium priority (default)
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: medium-priority
value: 500000
globalDefault: true
description: "Standard production workloads"
---
# Low priority for batch jobs
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: low-priority
value: 100000
globalDefault: false
description: "Batch jobs and background tasks"

Using Priority Classes

apiVersion: v1
kind: Pod
metadata:
  name: critical-api
spec:
  priorityClassName: high-priority
  containers:
  - name: api
    image: api:latest

Preemption Behavior

When node resources exhausted: 1. Scheduler tries to find node for high-priority pod 2. If no node available, scheduler looks for preemption candidates 3. Lower-priority pods evicted to make room 4. High-priority pod scheduled

Preemption Example:

• Node capacity: 4 CPU cores
• Running: 2 medium-priority pods (2 cores each)
• Pending: 1 high-priority pod (2 cores)
• Result: 1 medium-priority pod evicted, high-priority scheduled

PodDisruptionBudget (Protect from Preemption)

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 2  # Keep at least 2 replicas running
  selector:
    matchLabels:
      app: api

Alternative (percentage):

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  maxUnavailable: 1  # Max 1 pod disrupted at a time
  selector:
    matchLabels:
      app: api

Priority Use Cases

Critical Services:

# Payment API - highest priority
priorityClassName: high-priority
value: 1000000

Standard Applications:

# Web frontend - medium priority
priorityClassName: medium-priority
value: 500000

Batch Jobs:

# ETL jobs - low priority (preemptible)
priorityClassName: low-priority
value: 100000

Development:

# Dev environments - lowest priority
priorityClassName: dev-priority
value: 0

Pod Affinity and Anti-Affinity

Pod Affinity (Co-locate Pods)

Schedule pods on same node or zone as other pods:

apiVersion: v1
kind: Pod
metadata:
  name: web-frontend
spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: cache
        topologyKey: kubernetes.io/hostname  # Same node
  containers:
  - name: frontend
    image: frontend:latest

Use Case: Co-locate frontend with cache for low latency

Pod Anti-Affinity (Spread Pods)

Schedule pods away from other pods:

apiVersion: v1
kind: Pod
metadata:
  name: api-pod
  labels:
    app: api
spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: api
        topologyKey: kubernetes.io/hostname  # Different nodes
  containers:
  - name: api
    image: api:latest

Use Case: Spread API replicas across nodes for high availability

Preferred Anti-Affinity

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 5
  template:
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: web-app
              topologyKey: topology.kubernetes.io/zone  # Prefer different zones
          - weight: 50
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: web-app
              topologyKey: kubernetes.io/hostname  # Then different nodes

Scheduling Decision Framework

Decision Tree

START: How should I schedule this workload?

Q1: Does it need special hardware?
  ├─ YES → Use Node Affinity (GPU, SSD, etc.)
  └─ NO  → Q2

Q2: Should it be isolated from other workloads?
  ├─ YES → Use Taints + Tolerations
  └─ NO  → Q3

Q3: Does it need high availability?
  ├─ YES → Use Topology Spread Constraints
  └─ NO  → Q4

Q4: Is it more critical than other workloads?
  ├─ YES → Use Priority Class
  └─ NO  → Default scheduling

Pattern Combinations

High-Availability API:

spec:
  # 1. Topology spread (even distribution)
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: api

  # 2. High priority
  priorityClassName: high-priority

  # 3. PodDisruptionBudget
  # (separate resource)

ML Training Job:

spec:
  # 1. Node affinity (GPU nodes)
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: accelerator
            operator: In
            values:
            - nvidia-tesla-v100

  # 2. Toleration (GPU taints)
  tolerations:
  - key: workload
    operator: Equal
    value: gpu
    effect: NoSchedule

  # 3. Low priority (preemptible)
  priorityClassName: low-priority

Cost-Optimized Batch Job:

spec:
  # 1. Prefer spot instances
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: node.kubernetes.io/instance-lifecycle
            operator: In
            values:
            - spot

  # 2. Tolerate spot interruptions
  tolerations:
  - key: instance-type
    operator: Equal
    value: spot
    effect: NoSchedule

  # 3. Lowest priority
  priorityClassName: batch-priority

Summary

Scheduling Pattern Selection:

Best Practices: 1. Use topology spread over pod anti-affinity 2. Combine node affinity with taints for isolation 3. Set PodDisruptionBudgets with priority classes 4. Test scheduling constraints in dev/staging 5. Monitor pod Pending events 6. Use preferred constraints when possible (more flexible)

Security

Table of Contents

1. RBAC (Role-Based Access Control) 2. Pod Security Standards 3. Policy Enforcement 4. Secrets Management 5. Image Security 6. Network Security

RBAC (Role-Based Access Control)

Roles and ClusterRoles

Role (Namespace-scoped):

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: production
rules:
- apiGroups: [""]  # Core API group
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list"]

ClusterRole (Cluster-wide):

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-admin
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: [""]
  resources: ["persistentvolumes"]
  verbs: ["get", "list", "watch", "create", "delete"]

RoleBindings and ClusterRoleBindings

RoleBinding (Namespace-scoped):

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: production
subjects:
- kind: User
  name: jane@example.com
  apiGroup: rbac.authorization.k8s.io
- kind: Group
  name: developers
  apiGroup: rbac.authorization.k8s.io
- kind: ServiceAccount
  name: app-sa
  namespace: production
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

ClusterRoleBinding (Cluster-wide):

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: node-admins
subjects:
- kind: Group
  name: sre-team
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: node-admin
  apiGroup: rbac.authorization.k8s.io

ServiceAccounts

Create ServiceAccount:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app-sa
  namespace: production
automountServiceAccountToken: false  # Don't auto-mount unless needed

Use in Pod:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  namespace: production
spec:
  serviceAccountName: my-app-sa
  containers:
  - name: app
    image: myapp:latest

Grant Permissions:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: configmap-reader
  namespace: production
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-configmap-access
  namespace: production
subjects:
- kind: ServiceAccount
  name: my-app-sa
  namespace: production
roleRef:
  kind: Role
  name: configmap-reader
  apiGroup: rbac.authorization.k8s.io

Common RBAC Patterns

Read-Only Access:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: readonly
  namespace: production
rules:
- apiGroups: ["", "apps", "batch"]
  resources: ["*"]
  verbs: ["get", "list", "watch"]

Deployment Manager:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deployment-manager
  namespace: production
rules:
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]

Debug Access (Exec into Pods):

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: debug-access
  namespace: production
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]

RBAC Best Practices

Least Privilege:

# Bad: Too permissive
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]

# Good: Specific permissions
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list"]
  resourceNames: ["app-config"]  # Further restrict to specific resource

Avoid Default ServiceAccount:

# Bad: Using default ServiceAccount
spec:
  serviceAccountName: default

# Good: Dedicated ServiceAccount
spec:
  serviceAccountName: my-app-sa

Audit RBAC:

# Check user permissions
kubectl auth can-i create deployments --namespace production --as jane@example.com

# List all RoleBindings
kubectl get rolebindings -A

# Audit who can delete pods
kubectl get rolebindings,clusterrolebindings -A -o json | \
  jq -r '.items[] | select(.roleRef.name=="cluster-admin")'

Pod Security Standards

Namespace Enforcement

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Enforcement Modes:

• enforce: Block pod creation if violates standard
• audit: Log violations but allow
• warn: Show warning but allow

Security Levels:

• privileged: Unrestricted (system workloads only)
• baseline: Minimally restrictive (prevents known escalations)
• restricted: Most secure (removes all privilege escalations)

Restricted Pod Configuration

apiVersion: v1
kind: Pod
metadata:
  name: secure-app
  namespace: production
spec:
  # Pod-level security context
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
    seccompProfile:
      type: RuntimeDefault

  containers:
  - name: app
    image: myapp:latest

    # Container-level security context
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 1000
      capabilities:
        drop:
        - ALL
      seccompProfile:
        type: RuntimeDefault

    # Writable directories
    volumeMounts:
    - name: tmp
      mountPath: /tmp
    - name: cache
      mountPath: /app/cache

  volumes:
  - name: tmp
    emptyDir: {}
  - name: cache
    emptyDir: {}

Security Context Options

runAsNonRoot:

securityContext:
  runAsNonRoot: true  # Prevent root user

readOnlyRootFilesystem:

securityContext:
  readOnlyRootFilesystem: true  # Immutable filesystem

Capabilities:

securityContext:
  capabilities:
    drop:
    - ALL  # Drop all capabilities
    add:
    - NET_BIND_SERVICE  # Add only specific capability

seccomp Profile:

securityContext:
  seccompProfile:
    type: RuntimeDefault  # Apply default seccomp profile

AppArmor:

metadata:
  annotations:
    container.apparmor.security.beta.kubernetes.io/app: runtime/default

Pod Security Standard Migration

# Audit current violations
kubectl label namespace production \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn=restricted

# Review warnings
kubectl get pods -n production

# Fix violations, then enforce
kubectl label namespace production \
  pod-security.kubernetes.io/enforce=restricted

Policy Enforcement

Kyverno Policies

Installation:

helm repo add kyverno https://kyverno.github.io/kyverno/
helm install kyverno kyverno/kyverno --namespace kyverno --create-namespace

Require Resource Limits:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resources
spec:
  validationFailureAction: enforce
  background: true
  rules:
  - name: check-resources
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "CPU and memory resource requests and limits are required"
      pattern:
        spec:
          containers:
          - resources:
              requests:
                memory: "?*"
                cpu: "?*"
              limits:
                memory: "?*"
                cpu: "?*"

Block Latest Tag:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  validationFailureAction: enforce
  rules:
  - name: require-image-tag
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Using 'latest' image tag is not allowed"
      pattern:
        spec:
          containers:
          - image: "!*:latest"

Require Labels:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-labels
spec:
  validationFailureAction: enforce
  rules:
  - name: check-labels
    match:
      any:
      - resources:
          kinds:
          - Deployment
    validate:
      message: "Deployments must have 'app' and 'team' labels"
      pattern:
        metadata:
          labels:
            app: "?*"
            team: "?*"

Mutate (Add Default Resources):

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-default-resources
spec:
  background: true
  rules:
  - name: add-resources
    match:
      any:
      - resources:
          kinds:
          - Pod
    mutate:
      patchStrategicMerge:
        spec:
          containers:
          - (name): "*"
            resources:
              requests:
                +(memory): "256Mi"
                +(cpu): "250m"
              limits:
                +(memory): "512Mi"
                +(cpu): "500m"

OPA Gatekeeper

Installation:

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml

Constraint Template:

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        openAPIV3Schema:
          type: object
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
  - target: admission.k8s.gatekeeper.sh
    rego: |
      package k8srequiredlabels
      violation[{"msg": msg}] {
        provided := {label | input.review.object.metadata.labels[label]}
        required := {label | label := input.parameters.labels[_]}
        missing := required - provided
        count(missing) > 0
        msg := sprintf("Missing required labels: %v", [missing])
      }

Constraint:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: require-app-team-labels
spec:
  match:
    kinds:
    - apiGroups: ["apps"]
      kinds: ["Deployment"]
  parameters:
    labels: ["app", "team"]

Secrets Management

Kubernetes Secrets

Create Secret:

kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --from-literal=password=securepassword \
  --namespace production

# YAML definition
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
type: Opaque
stringData:
  username: admin
  password: securepassword

Use in Pod:

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: myapp:latest
    env:
    - name: DB_USERNAME
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: username
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: password

Mount as Volume:

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: myapp:latest
    volumeMounts:
    - name: secrets
      mountPath: /etc/secrets
      readOnly: true
  volumes:
  - name: secrets
    secret:
      secretName: db-credentials

Encryption at Rest

Enable encryption (kube-apiserver flag):

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <base64-encoded-32-byte-key>
  - identity: {}

External Secrets Operator

Installation:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets-system --create-namespace

SecretStore (AWS Secrets Manager):

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets

ExternalSecret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: db-secret
    creationPolicy: Owner
  data:
  - secretKey: password
    remoteRef:
      key: production/postgres/password

Image Security

Image Scanning

Trivy (Open Source):

# Scan image
trivy image myapp:latest

# Scan for HIGH/CRITICAL only
trivy image --severity HIGH,CRITICAL myapp:latest

# Output JSON
trivy image --format json -o results.json myapp:latest

Admission Controller (Scan on Deploy):

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: scan-images
spec:
  validationFailureAction: enforce
  webhookTimeoutSeconds: 30
  rules:
  - name: scan-image
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Image has HIGH/CRITICAL vulnerabilities"
      foreach:
      - list: "request.object.spec.containers"
        deny:
          conditions:
            any:
            - key: "{{ scan('{{ element.image }}').critical }}"
              operator: GreaterThan
              value: 0

Image Pull Secrets

# Create registry secret
kubectl create secret docker-registry regcred \
  --docker-server=myregistry.azurecr.io \
  --docker-username=myuser \
  --docker-password=mypassword \
  --docker-email=myemail@example.com \
  --namespace production

# Use in Pod
spec:
  imagePullSecrets:
  - name: regcred
  containers:
  - name: app
    image: myregistry.azurecr.io/myapp:v1.0

Image Signing (Sigstore/Cosign)

# Sign image
cosign sign --key cosign.key myregistry.io/myapp:v1.0

# Verify signature
cosign verify --key cosign.pub myregistry.io/myapp:v1.0

Kyverno Verification:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signature
spec:
  validationFailureAction: enforce
  rules:
  - name: verify-signature
    match:
      any:
      - resources:
          kinds:
          - Pod
    verifyImages:
    - imageReferences:
      - "myregistry.io/*"
      attestors:
      - entries:
        - keys:
            publicKeys: |-
              -----BEGIN PUBLIC KEY-----
              ...
              -----END PUBLIC KEY-----

Network Security

TLS for In-Cluster Communication

cert-manager Installation:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.0/cert-manager.yaml

ClusterIssuer:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx

Certificate:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: myapp-tls
  namespace: production
spec:
  secretName: myapp-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - myapp.example.com

Summary

Security Checklist:

• [ ] Enable RBAC and follow least-privilege principle
• [ ] Create dedicated ServiceAccounts (don't use default)
• [ ] Enforce Pod Security Standards (Restricted for apps)
• [ ] Implement policy enforcement (Kyverno or OPA)
• [ ] Use External Secrets Operator for sensitive data
• [ ] Enable encryption at rest for Secrets
• [ ] Scan images for vulnerabilities
• [ ] Use image pull secrets for private registries
• [ ] Implement NetworkPolicies (default-deny)
• [ ] Enable TLS for ingress traffic (cert-manager)
• [ ] Regularly audit RBAC permissions
• [ ] Monitor security events and policy violations

Recommended Tools:

• RBAC: Built-in Kubernetes
• Pod Security: Pod Security Standards (built-in)
• Policy: Kyverno (easier) or OPA Gatekeeper (more powerful)
• Secrets: External Secrets Operator + cloud provider
• Image Scanning: Trivy
• TLS: cert-manager
• Audit: Falco, kube-bench