Cluster Deployment Engineer

Cluster Deployment Engineer

When to Use

• Bootstrap, upgrade, or resize a Kubernetes cluster (managed or self-hosted)
• Install or upgrade cluster add-ons: ingress controller, CNI, metrics, DNS, cert-manager
• Deploy workloads via Helm, Kustomize, or raw manifests with health checks
• Configure namespaces, resource quotas, LimitRanges, PriorityClasses
• Design cluster RBAC, ServiceAccounts, and namespace isolation
• Operate GitOps controllers (Argo CD, Flux) targeting cluster state
• Troubleshoot Pending pods, ImagePullBackOff, networking, and control-plane issues
• Plan multi-cluster or environment-per-cluster topology

When NOT to Use

• Design VPC, subnets, IAM, or Terraform modules for cloud foundation → infrastructure-engineer
• Author or fix generic CI/CD pipelines and build systems → devops
• Choose canary, blue-green, or feature-flag rollout strategy → deployment-strategist
• Build developer portals, scaffolders, or platform product roadmap → platform-engineer
• Pipeline SAST, SBOM, signing gates → devsecops
• Cross-service architecture ADRs → senior-system-architecture
• SEV program and paging policy → incident-management-engineer
• Host/GPU inventory, stranded kW, hardware refresh → data-center-compute-supply-efficiency

Related skills

Core Workflows

1. Cluster lifecycle

1. Requirements — version skew policy, HA, regions, node SKUs, autoscaling bounds 2. Bootstrap — control plane, node groups, OIDC for workloads 3. Add-ons — ordered install (CNI → DNS → ingress → certs → metrics) 4. Upgrade — control plane first, node cordon/drain, validate API and workloads 5. Decommission — drain, backup etcd/state if self-managed, destroy order

See `references/cluster_lifecycle.md`.

2. Workload delivery

1. Package — Helm chart or Kustomize overlay per env 2. Values — resources, replicas, probes, affinity, topology spread 3. Apply — dry-run, diff, apply; watch rollout status 4. Verify — readiness, Service endpoints, ingress, smoke test 5. Rollback — helm rollback or GitOps revert commit

See `references/workload_delivery.md`.

3. Networking and ingress

• Services, Endpoints, EndpointSlices
• Ingress / Gateway API, TLS termination
• NetworkPolicy default-deny + allow lists
• Service mesh only when required (complexity cost)

See `references/networking_ingress.md`.

4. Security and tenancy

• Namespace per team/env; quotas and LimitRanges
• RBAC: least privilege; no cluster-admin for apps
• Secrets: external secrets operator; no plain Secrets in git
• Pod Security Admission (restricted baseline)

Pair with devsecops for admission controllers and image policy.

See `references/cluster_security_rbac.md`.

5. GitOps at cluster scope

• One repo or path per cluster/environment
• App-of-apps or directory per add-on
• Sync waves for ordering; health checks and sync hooks
• Drift: prefer Git as source of truth

See `references/gitops_cluster.md`.

6. Troubleshooting

See `references/troubleshooting.md`.

Output standards

• Manifest or Helm change with env-specific values table
• Pre/post upgrade checklist with rollback step
• Runbook snippet: symptoms, commands, escalation to infrastructure-engineer if cloud API/IaC

When to load references

• Bootstrap, upgrade, nodes → references/cluster_lifecycle.md
• Helm, Kustomize, rollouts → references/workload_delivery.md
• Ingress, CNI, policies → references/networking_ingress.md
• RBAC, quotas, PSA → references/cluster_security_rbac.md
• Argo CD / Flux → references/gitops_cluster.md
• Incidents and debug → references/troubleshooting.md

Cluster lifecycle

Table of contents

1. Bootstrap order 2. Upgrades 3. Node operations

Bootstrap order

Typical managed-cluster sequence:

1. Cluster API available (version pinned) 2. CNI — pods must schedule 3. CoreDNS — in-cluster DNS 4. metrics-server — HPA and kubectl top 5. Ingress controller — HTTP(S) entry 6. cert-manager — TLS certificates 7. GitOps controller — optional if not pipeline-only 8. Observability agents — DaemonSets last if they tolerate NotReady nodes

Document versions in a cluster inventory table (component, chart/app version, owner).

Upgrades

Maintain n-1 skew policy between control plane and nodes per vendor guidance.

Node operations

• Scale out — increase pool; verify labels/taints for workload placement
• Scale in — drain with kubectl drain --ignore-daemonsets --delete-emptydir
• Taints — dedicated pools for GPU, system, or batch only when needed

Self-managed: backup etcd before minor upgrades; test restore annually.

Cluster security and RBAC

Table of contents

1. Namespace tenancy 2. RBAC 3. Secrets 4. Pod Security Admission

Namespace tenancy

Per team or env:

# ResourceQuota example fields
requests.cpu, requests.memory, limits.cpu, limits.memory, pods, services

Use LimitRange for default container limits in dev namespaces.

RBAC

• Bind Role/RoleBinding in namespace; avoid ClusterRoleBinding to cluster-admin for apps
• CI deploy SA: create/update in target namespace only
• Humans: read-only default; break-glass cluster-admin via PAM group

Audit: kubectl auth can-i --list -n <ns> --as=system:serviceaccount:<ns>:<sa>

Secrets

• Do not commit Secrets; use External Secrets / CSI driver / cloud secret manager
• Rotate on compromise; prefer short-lived tokens (IRSA, workload identity)

Pod Security Admission

Enforce at namespace labels; validate with kubectl label dry-run before cutover.

Admission webhooks and image policies → devsecops.

GitOps at cluster scope

Table of contents

1. Repository layout 2. Sync ordering 3. Drift and rollback

Repository layout

clusters/
  prod/
    infrastructure/    # add-ons, CRDs
    apps/                # team workloads
  staging/
    ...

Or app-of-apps root Application pointing at child paths.

Separate platform repo (cluster infra) from application repos when blast radius differs.

Sync ordering

Argo CD sync waves (annotation argocd.argoproj.io/sync-wave):

Use Sync hooks for DB migrations only when idempotent and owned by app team.

Drift and rollback

• Manual kubectl edit creates drift—revert Git or disable auto-sync temporarily with incident record
• Rollback = revert Git commit or Helm revision in GitOps values
• Pipeline that only pushes images must still update Git tag for GitOps flow

CI pipeline mechanics → devops.

Networking and ingress

Table of contents

1. Services 2. Ingress and Gateway API 3. NetworkPolicy

Services

Verify: kubectl get endpointslices -n <ns> matches pod IPs.

Ingress and Gateway API

• One ingress class per environment (e.g. nginx, alb)
• TLS: cert-manager Certificate + issuer; or cloud-managed certs
• Annotate timeouts and body size for long uploads

Debug path: DNS → LB → ingress controller → Service → Pod.

NetworkPolicy

Start default-deny in namespace, then allow:

• Ingress from ingress controller namespace
• Egress to DNS (kube-system), APIs, databases on known ports

Label selectors must match pod templates exactly—typos cause silent drops.

Service mesh adds mTLS and traffic policy; adopt only with platform team (platform-engineer).

Troubleshooting

Table of contents

1. Quick commands 2. Symptom matrix 3. Escalation

Quick commands

kubectl get events -A --sort-by='.lastTimestamp' | tail -20
kubectl describe pod <pod> -n <ns>
kubectl logs <pod> -n <ns> --previous
kubectl top nodes
kubectl get pods -A -o wide | grep -v Running

Symptom matrix

Escalation

Capture: timestamp, cluster, namespace, workload, events snippet, change that preceded failure.

Workload delivery

Table of contents

1. Helm 2. Kustomize 3. Rollout verification

Helm

helm upgrade --install <release> <chart> -n <ns> -f values.<env>.yaml --atomic --timeout 10m
helm history <release> -n <ns>
helm rollback <release> <revision> -n <ns>

Values discipline:

• Base values.yaml + env overlays (values.prod.yaml)
• Pin chart version in CI or GitOps; do not float latest
• Set resources.requests/limits, liveness/readiness probes, podDisruptionBudget where HA

Kustomize

base/
  deployment.yaml
overlays/
  staging/kustomization.yaml
  prod/kustomization.yaml

• Use images: for tag promotion; patches for replicas and env
• kubectl apply -k overlays/prod --dry-run=server before merge

Rollout verification

kubectl rollout status deployment/<name> -n <ns>
kubectl get pods -n <ns> -o wide
kubectl describe pod <pod> -n <ns>   # Events

Exit criteria: all replicas ready, Service has endpoints, ingress returns 200 on health path.

Application rollout strategy (canary %, traffic split) → deployment-strategist.