Kubernetes Specialist Skill

Kubernetes Specialist

Purpose

Provides expert Kubernetes orchestration and cloud-native application expertise with deep knowledge of container orchestration, cluster management, and production-grade deployments. Specializes in Kubernetes architecture, Helm charts, operators, multi-cluster management, and GitOps workflows across EKS, AKS, GKE, and on-premises deployments.

When to Use

• Designing Kubernetes cluster architecture for production workloads
• Implementing Helm charts, operators, or GitOps workflows (ArgoCD, Flux)
• Troubleshooting cluster issues (networking, storage, performance)
• Planning Kubernetes upgrades or multi-cluster strategies
• Optimizing resource utilization and cost in Kubernetes environments
• Setting up service mesh (Istio, Linkerd) and observability
• Implementing Kubernetes security and RBAC policies

Quick Start

Invoke this skill when:

• Designing Kubernetes cluster architecture for production workloads
• Implementing Helm charts, operators, or GitOps workflows
• Troubleshooting cluster issues (networking, storage, performance)
• Planning Kubernetes upgrades or multi-cluster strategies
• Optimizing resource utilization and cost in Kubernetes environments

Do NOT invoke when:

• Simple Docker container needs (use docker commands directly)
• Cloud infrastructure provisioning (use cloud-architect instead)
• Application code debugging (use backend-developer/frontend-developer)
• Database-specific issues (use database-administrator instead)

Decision Framework

Deployment Strategy Selection

├─ Zero downtime required?
│   ├─ Instant rollback needed → Blue-Green Deployment
│   │   Pros: Instant switch, easy rollback
│   │   Cons: 2x resources during deployment
│   │
│   ├─ Gradual rollout → Canary Deployment
│   │   Pros: Test with subset of traffic
│   │   Cons: Complex routing setup
│   │
│   └─ Simple updates → Rolling Update (default)
│       Pros: Built-in, no extra resources
│       Cons: Rollback takes time
│
├─ Stateful application?
│   ├─ Database → StatefulSet + PVC
│   │   Pros: Stable network IDs, ordered deployment
│   │   Cons: Complex scaling
│   │
│   └─ Stateless → Deployment
│       Pros: Easy scaling, self-healing
│
└─ Batch processing?
    ├─ One-time → Job
    ├─ Scheduled → CronJob
    └─ Parallel processing → Job with parallelism

Resource Configuration Matrix

Node Pool Strategy

Red Flags → Escalate

STOP and escalate if:

• Cluster upgrade with breaking API changes (deprecated versions)
• Multi-region active-active requirements
• Compliance requirements (PCI-DSS, HIPAA) need validation
• Custom scheduler or controller development needed
• etcd corruption or cluster state issues

Quality Checklist

Cluster Configuration

• [ ] Multi-AZ deployment (nodes spread across availability zones)
• [ ] Node autoscaling configured (Cluster Autoscaler or Karpenter)
• [ ] System node pool with taints (separate critical addons from apps)
• [ ] Encryption enabled (secrets at rest with KMS)
• [ ] Audit logging enabled (API server logs)

Security

• [ ] Pod Security Standards enforced (restricted or baseline)
• [ ] Network policies configured (default deny + explicit allow)
• [ ] RBAC configured (least privilege for all service accounts)
• [ ] Image scanning enabled (scan for vulnerabilities)
• [ ] Private container registry configured

Resource Management

• [ ] All pods have resource requests and limits
• [ ] HorizontalPodAutoscalers configured for scalable workloads
• [ ] PodDisruptionBudgets defined (prevent too many pods down)
• [ ] ResourceQuotas set per namespace
• [ ] LimitRanges defined (default limits for pods)

High Availability

• [ ] Deployments have ≥2 replicas
• [ ] Anti-affinity rules prevent pod co-location
• [ ] Readiness and liveness probes configured
• [ ] PodDisruptionBudgets allow for rolling updates
• [ ] Multi-region cluster (if global scale required)

Observability

• [ ] Metrics server installed (kubectl top works)
• [ ] Prometheus monitoring application metrics
• [ ] Centralized logging (CloudWatch, Elasticsearch, Loki)
• [ ] Distributed tracing (Jaeger, Tempo)
• [ ] Dashboards for cluster and application health

Disaster Recovery

• [ ] Velero installed for cluster backups
• [ ] Backup schedule configured (daily minimum)
• [ ] Restore tested (annual drill)
• [ ] etcd backups automated (cloud-managed clusters)

Additional Resources

• Detailed Technical Reference: See REFERENCE.md
• Code Examples & Patterns: See EXAMPLES.md

Kubernetes Specialist - Code Examples & Patterns

Blue-Green Deployment Pattern

When to use: Zero-downtime deployments with instant rollback capability

Blue Deployment (current production)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-blue
  namespace: production
spec:
  replicas: 5
  selector:
    matchLabels:
      app: myapp
      version: blue
  template:
    metadata:
      labels:
        app: myapp
        version: blue
    spec:
      containers:
      - name: myapp
        image: myregistry.com/myapp:v1.0.0
        ports:
        - containerPort: 8080

Service (switches between blue and green)

apiVersion: v1
kind: Service
metadata:
  name: myapp-service
  namespace: production
spec:
  selector:
    app: myapp
    version: blue  # Change to 'green' to cutover
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: LoadBalancer

Deployment Process

# Step 1: Deploy green (new version) alongside blue
kubectl apply -f green-deployment.yaml

# Step 2: Wait for green to be ready
kubectl wait --for=condition=available --timeout=300s \
  deployment/myapp-green -n production

# Step 3: Test green deployment (internal testing)
kubectl port-forward deployment/myapp-green -n production 9000:8080
curl http://localhost:9000/health

# Step 4: Cutover traffic to green (instant switch)
kubectl patch service myapp-service -n production \
  -p '{"spec":{"selector":{"version":"green"}}}'

# Step 5: Monitor for issues (5-10 minutes)
kubectl logs -f deployment/myapp-green -n production

# If successful: Delete blue
kubectl delete deployment myapp-blue -n production

# If issues: Instant rollback
kubectl patch service myapp-service -n production \
  -p '{"spec":{"selector":{"version":"blue"}}}'

Anti-Pattern 1: No Resource Requests/Limits

What it looks like (BAD):

apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
  - name: app
    image: myapp:latest
    # No resources defined - pod can consume entire node!

Why it fails:

• Pod scheduled to node without capacity check (causes OOMKilled on other pods)
• No QoS class (BestEffort - killed first during resource pressure)
• HPA cannot scale (requires resource requests to calculate utilization)

Correct approach:

apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
  - name: app
    image: myapp:latest
    resources:
      requests:  # Minimum guaranteed resources
        cpu: "500m"     # 0.5 CPU cores
        memory: "512Mi" # 512 MB
      limits:    # Maximum allowed resources
        cpu: "1000m"    # 1 CPU core
        memory: "1Gi"   # 1 GB
    # QoS class: Guaranteed (requests == limits)

Anti-Pattern 2: Missing Health Probes

What it looks like (BAD):

containers:
- name: app
  image: myapp:latest
  # No liveness or readiness probes!

Why it fails:

• Kubernetes sends traffic to pod immediately (even if app not ready)
• Crashed pods not restarted automatically
• Rolling updates don't wait for new pods to be healthy

Correct approach:

containers:
- name: app
  image: myapp:latest
  ports:
  - containerPort: 8080
  
  livenessProbe:  # Restart pod if this fails
    httpGet:
      path: /health
      port: 8080
    initialDelaySeconds: 30  # Wait for app to start
    periodSeconds: 10        # Check every 10 seconds
    timeoutSeconds: 5
    failureThreshold: 3      # Restart after 3 failures
  
  readinessProbe:  # Remove from service if this fails
    httpGet:
      path: /ready
      port: 8080
    initialDelaySeconds: 10
    periodSeconds: 5
    failureThreshold: 2

Network Policy Example

# Default deny all ingress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress

---
# Allow traffic from specific namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-frontend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: frontend
    ports:
    - protocol: TCP
      port: 8080

HorizontalPodAutoscaler Example

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 10
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
      - type: Pods
        value: 4
        periodSeconds: 15
      selectPolicy: Max

PodDisruptionBudget Example

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp-pdb
  namespace: production
spec:
  minAvailable: 2  # Or use maxUnavailable: 1
  selector:
    matchLabels:
      app: myapp

StatefulSet for Databases

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: database
spec:
  serviceName: postgres
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:15
        ports:
        - containerPort: 5432
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: gp3
      resources:
        requests:
          storage: 100Gi

Kubernetes Specialist - Technical Reference

Workflow: Deploy Production Kubernetes Cluster (EKS Example)

Step 1: Cluster Design Decisions

# cluster-requirements.yaml
cluster:
  name: production-cluster
  region: us-east-1
  version: "1.28"  # Latest stable
  
  node_groups:
    - name: system
      instance_types: [t3.large]  # 2 vCPU, 8 GB RAM
      desired: 3  # High availability
      min: 3
      max: 5
      taints:
        - key: CriticalAddonsOnly
          value: "true"
          effect: NoSchedule
      labels:
        role: system
    
    - name: applications
      instance_types: [m5.xlarge, m5.2xlarge]  # 4-8 vCPU
      desired: 5
      min: 3
      max: 20
      autoscaling: true
      labels:
        role: applications
    
    - name: spot
      instance_types: [m5.large, m5.xlarge, m5.2xlarge]
      desired: 0
      min: 0
      max: 50
      capacity_type: SPOT  # 70% cost savings
      labels:
        role: batch-processing
  
  networking:
    vpc_cidr: 10.0.0.0/16
    pod_cidr: 100.64.0.0/16  # Secondary CIDR for pods
    service_cidr: 172.20.0.0/16
  
  addons:
    - vpc-cni  # AWS networking
    - coredns  # DNS
    - kube-proxy  # Service routing
    - aws-ebs-csi-driver  # Persistent storage
    - cluster-autoscaler  # Node autoscaling
    - metrics-server  # HPA metrics

Step 2: Infrastructure as Code (Terraform)

# eks-cluster.tf
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "19.20.0"

  cluster_name    = "production-cluster"
  cluster_version = "1.28"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  cluster_endpoint_public_access  = true
  cluster_endpoint_private_access = true
  
  cluster_encryption_config = {
    provider_key_arn = aws_kms_key.eks.arn
    resources        = ["secrets"]
  }

  cluster_enabled_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]

  eks_managed_node_groups = {
    system = {
      name = "system-nodes"
      
      instance_types = ["t3.large"]
      capacity_type  = "ON_DEMAND"
      
      min_size     = 3
      max_size     = 5
      desired_size = 3
      
      taints = [{
        key    = "CriticalAddonsOnly"
        value  = "true"
        effect = "NO_SCHEDULE"
      }]
      
      labels = {
        role = "system"
      }
      
      ami_type = "AL2_x86_64"
      
      metadata_options = {
        http_endpoint               = "enabled"
        http_tokens                 = "required"
        http_put_response_hop_limit = 1
      }
    }
    
    applications = {
      name = "app-nodes"
      
      instance_types = ["m5.xlarge"]
      capacity_type  = "ON_DEMAND"
      
      min_size     = 3
      max_size     = 20
      desired_size = 5
      
      labels = {
        role = "applications"
      }
      
      block_device_mappings = {
        xvda = {
          device_name = "/dev/xvda"
          ebs = {
            volume_size           = 100
            volume_type           = "gp3"
            iops                  = 3000
            throughput            = 125
            encrypted             = true
            kms_key_id            = aws_kms_key.ebs.arn
            delete_on_termination = true
          }
        }
      }
    }
    
    spot = {
      name = "spot-nodes"
      
      instance_types = ["m5.large", "m5.xlarge", "m5.2xlarge"]
      capacity_type  = "SPOT"
      
      min_size     = 0
      max_size     = 50
      desired_size = 0
      
      labels = {
        role = "batch-processing"
        "karpenter.sh/capacity-type" = "spot"
      }
      
      use_mixed_instances_policy = true
    }
  }

  enable_irsa = true

  tags = {
    Environment = "production"
    Terraform   = "true"
  }
}

resource "helm_release" "aws_load_balancer_controller" {
  name       = "aws-load-balancer-controller"
  repository = "https://aws.github.io/eks-charts"
  chart      = "aws-load-balancer-controller"
  namespace  = "kube-system"
  version    = "1.6.2"

  set {
    name  = "clusterName"
    value = module.eks.cluster_name
  }

  set {
    name  = "serviceAccount.create"
    value = "true"
  }

  set {
    name  = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = aws_iam_role.aws_load_balancer_controller.arn
  }
}

resource "helm_release" "cluster_autoscaler" {
  name       = "cluster-autoscaler"
  repository = "https://kubernetes.github.io/autoscaler"
  chart      = "cluster-autoscaler"
  namespace  = "kube-system"
  version    = "9.29.3"

  set {
    name  = "autoDiscovery.clusterName"
    value = module.eks.cluster_name
  }

  set {
    name  = "awsRegion"
    value = var.aws_region
  }
}

Step 3: Deploy Application with Helm

# Create namespace
kubectl create namespace production

# Install Prometheus + Grafana monitoring stack
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --values monitoring-values.yaml

# Install example application
helm install myapp ./charts/myapp \
  --namespace production \
  --values production-values.yaml \
  --wait \
  --timeout 10m

# Verify deployment
kubectl get pods -n production
kubectl get svc -n production
kubectl get ingress -n production

Workflow: Implement GitOps with ArgoCD

1. Install ArgoCD

kubectl create namespace argocd

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

kubectl wait --for=condition=available --timeout=300s \
  deployment/argocd-server -n argocd

# Get admin password
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d

# Port forward to access UI
kubectl port-forward svc/argocd-server -n argocd 8080:443

2. Configure Git Repository

# argocd-repo-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: private-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
type: Opaque
stringData:
  type: git
  url: https://github.com/myorg/k8s-manifests
  password: ghp_xxxxxxxxxxxxx
  username: git

Production Readiness Checklist

# Security
[ ] Pod Security Standards enforced
kubectl label namespace production pod-security.kubernetes.io/enforce=restricted

[ ] Network policies configured (default deny, explicit allow)
[ ] RBAC configured (least privilege for service accounts)
[ ] Secrets encrypted at rest (KMS integration verified)
[ ] Image scanning enabled (Trivy, Anchore)

# High Availability
[ ] Multi-AZ node distribution
kubectl get nodes -o wide | grep -c us-east-1

[ ] Pod Disruption Budgets configured
kubectl get pdb --all-namespaces

[ ] Anti-affinity rules for critical pods
[ ] Readiness and liveness probes configured
kubectl describe pod | grep -A5 Probes

# Observability
[ ] Metrics server installed
kubectl top nodes

[ ] Prometheus scraping application metrics
[ ] Grafana dashboards configured
[ ] Logging to CloudWatch / Elasticsearch
[ ] Distributed tracing (Jaeger / Tempo)

# Resource Management
[ ] Resource requests and limits set
kubectl describe pod | grep -A5 Requests

[ ] HorizontalPodAutoscaler configured
kubectl get hpa --all-namespaces

[ ] Cluster Autoscaler working
kubectl logs -n kube-system deployment/cluster-autoscaler

# Disaster Recovery
[ ] etcd backups automated
[ ] Velero installed for application backups
[ ] Backup restoration tested
[ ] Multi-region DR strategy documented