Eks Best Practices

EKS Best Practices

Comprehensive guidance for designing, deploying, and operating Amazon EKS clusters. Consolidates guidance from the AWS EKS Best Practices Guide, AWS EKS HA/Resiliency Guide, and terraform-aws-modules/terraform-aws-eks examples.

When to Use This Skill

Activate this skill when:

• Designing a new EKS cluster architecture
• Choosing between EKS compute options (Fargate, MNG, Karpenter, Auto Mode)
• Configuring EKS networking (VPC CNI, ingress, service mesh)
• Implementing EKS security (IAM, pod security, secrets)
• Planning cluster upgrades or migrations
• Reviewing EKS architecture decisions
• Working with terraform-aws-modules/terraform-aws-eks examples
• Optimizing EKS cost or scaling to large clusters

Don't use this skill for:

• Generic Kubernetes concepts (Claude knows these)
• Provider-specific API reference (link to AWS docs)
• Non-EKS container orchestration (ECS, Lambda)
• Step-by-step EKS upgrade execution — this skill covers upgrade strategy and architectural decisions, not the per-version procedures themselves.

EKS Architecture Decision Framework

When to Use EKS

EKS Deployment Models

Shared Responsibility

Compute Selection Matrix

Decision Table

Quick Decision Guide

• Default choice: Karpenter — best balance of flexibility, cost, and automation
• Zero ops priority: EKS Auto Mode — AWS manages nodes, add-ons, and scaling via managed Karpenter. Best for teams that want Kubernetes benefits without operational overhead around upgrades, autoscaling, load balancing, and storage
• Serverless/batch: Fargate — no nodes to manage, per-pod billing
• Predictable, stable: MNG — familiar ASG model, managed updates
• Custom requirements: Self-managed — full control, highest overhead

✅ DO:

• Use Karpenter as the default node autoscaler for new clusters
• Run system components (CoreDNS, Karpenter) on MNG or Fargate
• Use multiple instance types for availability and cost optimization

❌ DON'T:

• Use self-managed nodes without a specific technical requirement
• Run Fargate for GPU or DaemonSet-dependent workloads
• Mix Karpenter and Cluster Autoscaler on the same node groups

Networking Quick Reference

VPC CNI Mode Decision

Ingress Pattern Selection

IPv4 vs IPv6

For detailed networking guidance, see: Networking — VPC CNI & IP | Networking — Ingress & DNS

Security Essentials

IAM Strategy

Key rules:

• ✅ Use Pod Identity for new workloads — simpler setup, session tags, role chaining
• ✅ Use EKS access entries (API mode) over aws-auth ConfigMap
• ✅ Move VPC CNI permissions from node role to Pod Identity/IRSA
• ❌ Don't use wildcard conditions in IRSA trust policies
• ❌ Don't attach application permissions to node IAM roles

Pod Security Baseline

Apply Pod Security Admission (PSA) labels to all namespaces:

# Minimum: enforce baseline, warn on restricted
metadata:
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/warn: restricted

Secrets Management

Always enable KMS envelope encryption for Kubernetes secrets.

For detailed security guidance, see: Security Reference | Runtime & Network | Supply Chain & Compliance

Reliability Essentials

Pod Disruption Budgets

Create PDBs for every production workload with >1 replica:

Health Probe Strategy

Critical rule: Liveness probes must NOT check external dependencies. If the database goes down and liveness checks the DB, ALL pods restart — causing cascading failure.

Graceful Shutdown Pattern

spec:
  terminationGracePeriodSeconds: 60
  containers:
  - lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 15"]

Why `sleep 15`: Gives kube-proxy and load balancer time to remove the pod from traffic routing before SIGTERM.

Multi-AZ Distribution

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule

For detailed reliability guidance, see: Reliability & Resiliency — Core (see also reliability-advanced.md for DR, deployment strategies, and large-cluster guidance)

Cluster Upgrade Strategy

Upgrade Sequence (Strict Order)

1. Control Plane → 2. EKS Add-ons → 3. Data Plane → 4. Custom Add-ons

Pre-Upgrade Checklist

1. Check EKS Cluster Insights for upgrade readiness 2. Scan for deprecated APIs (Pluto, kube-no-trouble) 3. Verify add-on compatibility with target version 4. Test in non-prod environment first 5. Ensure PDBs are configured for graceful node drain 6. Back up cluster state (Velero or GitOps repo)

Upgrade Strategy Decision

Data Plane with Karpenter

Karpenter automatically replaces nodes via drift detection after control plane upgrade. Control the speed with disruption.budgets:

disruption:
  budgets:
  - nodes: "10%"  # Max 10% of nodes replaced at a time

For detailed upgrade guidance, see: Cluster Upgrades Reference

Autoscaling Quick Reference

Node Autoscaler Selection

Pod Autoscaler Selection

For detailed autoscaling guidance, see: Autoscaling Reference | Karpenter Reference

Terraform Examples Quick Start

Based on terraform-aws-modules/terraform-aws-eks.

Example Selection

Common Deployment Topologies

Private cluster with Karpenter:

VPC (3 AZs, terraform-aws-modules/vpc/aws)
├── Private subnets → EKS nodes (MNG for system, Karpenter for workloads)
├── Public subnets  → ALB (internet-facing)
├── Intra subnets   → EKS control plane ENIs
└── NAT Gateway     → 1 per AZ for production

Multi-tenant platform:

EKS Cluster (terraform-aws-modules/eks/aws)
├── kube-system         (platform: CoreDNS, kube-proxy, VPC CNI)
├── karpenter           (Karpenter controller on MNG)
├── monitoring          (shared: Prometheus, Grafana)
├── ingress             (shared: AWS LBC)
├── team-a namespace    (RBAC, NetworkPolicy, ResourceQuota)
├── team-b namespace    (RBAC, NetworkPolicy, ResourceQuota)
└── team-c namespace    (RBAC, NetworkPolicy, ResourceQuota)

For detailed examples and terraform patterns, see: Terraform Examples Reference

Cost Optimization Quick Wins

For detailed cost guidance, see: Cost Optimization Reference | For scalability guidance, see: Scalability Reference

Observability Quick Reference

Essential: Enable EKS audit logging and GuardDuty EKS Runtime Monitoring for security visibility.

For detailed observability guidance, see: Observability Reference

EKS Capabilities

EKS Capabilities are AWS-managed features installed and updated as part of the EKS platform. They run in AWS-owned infrastructure separate from your clusters, with AWS handling scaling, patching, and upgrading.

Combined pattern: ArgoCD deploys ACK resources + KRO compositions via GitOps, providing a single workflow for both infrastructure and applications.

For detailed ArgoCD patterns, see: ArgoCD Patterns Reference

Sources:

• EKS Capabilities Documentation
• AWS Blog — Deep dive: Simplifying resource orchestration with Amazon EKS Capabilities

Detailed References

This skill uses progressive disclosure — essential guidance is in this main file, detailed reference material is loaded on demand:

• [Security](references/security.md) — IAM, Cluster Access Manager, Pod Identity, IRSA, pod security standards, multi-tenancy, secrets management, data encryption
• [Security — Runtime & Network](references/security-runtime-network.md) — Runtime threat detection (GuardDuty, seccomp, AppArmor, Falco), network policies, SG for pods, encryption in transit, detective controls
• [Security — Supply Chain & Compliance](references/security-supply-chain.md) — Image security (SBOMs, attestations, ECR hardening), infrastructure hardening (Bottlerocket, CIS benchmarks), regulatory compliance, incident response
• [Networking](references/networking.md) — VPC CNI modes (secondary IP, prefix delegation, custom networking), subnet/CIDR planning, IPv4 vs IPv6, Security Groups for Pods, IP address management
• [Networking — Ingress & DNS](references/networking-ingress-dns.md) — Ingress patterns (ALB, NLB, Gateway API), AWS Load Balancer Controller, service mesh, DNS/CoreDNS tuning, private cluster connectivity
• [Reliability & Resiliency — Core](references/reliability-core.md) — HA patterns, PDBs, health probes, load balancer health checks, lifecycle hooks, topology spread, resource management
• [Reliability & Resiliency — Advanced](references/reliability-advanced.md) — disaster recovery, zonal shift, deployment strategies, large cluster guidance, chaos engineering, admission-controller topology enforcement
• [Autoscaling](references/autoscaling.md) — Autoscaler selection, Cluster Autoscaler (IAM, Spot, overprovisioning, parameter tuning), HPA, VPA, KEDA, CoreDNS autoscaling
• [Karpenter](references/karpenter.md) — Operational best practices, NodePools, EC2NodeClass, Spot/interruption handling, consolidation, multiple NodePool strategy, cost controls, resource management, private clusters, CoreDNS with Karpenter
• [Cluster Upgrades](references/cluster-upgrades.md) — In-place and blue-green upgrades, pre-upgrade validation, add-on management, API deprecation detection, version skew policy, Bottlerocket updates, rollback procedures
• [Cost Optimization](references/cost-optimization.md) — CFM framework, compute/networking/storage cost strategies, observability cost management, Spot, Graviton, tagging, Kubecost
• [Scalability](references/scalability.md) — Scaling theory (churn rate, QPS), control plane (APF, monitoring), data plane (node sizing, diversity), cluster services (CoreDNS, Metrics Server), workload patterns, IPVS, large-cluster guidance
• [Observability](references/observability.md) — Observability strategy, CloudWatch Container Insights & Application Signals, Prometheus/Grafana, control plane monitoring, network performance monitoring, logging architecture, distributed tracing, GPU/AI-ML observability, detective controls, alerting patterns
• [Terraform Examples](references/terraform-examples.md) — terraform-aws-modules/terraform-aws-eks examples, submodules, add-on management, Provisioned Control Plane, EFA, VPC patterns, deployment topologies
• [ArgoCD Patterns](references/argocd-patterns.md) — ArgoCD architecture, App of Apps, ApplicationSets, GitOps Bridge, multi-cluster patterns (hub-and-spoke, decentralized, hybrid), EKS ArgoCD Capability (managed vs self-managed, migration), ACK/KRO integration, multi-tenant RBAC
• [Container Registry](references/container-registry.md) — ECR architecture, operating models, image promotion, vulnerability scanning, base image curation, lifecycle policies, pull-through cache, repository creation templates, managed signing (AWS Signer), archival storage class, registry configuration
• [EKS Auto Mode](references/eks-auto-mode.md) — Auto Mode architecture, managed NodePools/NodeClasses, migration from standard EKS, comparison with self-managed Karpenter, limitations and FAQ

How to use: When you need detailed information on a topic, reference the appropriate guide. Claude will load it on demand.

Sources

• AWS EKS Best Practices Guide
• terraform-aws-modules/terraform-aws-eks

ArgoCD Patterns for EKS

Part of: eks-best-practices

Purpose: ArgoCD architecture patterns, deployment strategies, and multi-tenant RBAC for Amazon EKS

---

Table of Contents

1. ArgoCD Architecture on EKS 2. App of Apps Pattern 3. ApplicationSets 4. GitOps Bridge Pattern 5. Multi-Tenant RBAC 6. EKS ArgoCD Capability (Managed) 7. ACK and KRO Integration

---

ArgoCD Architecture on EKS

ArgoCD is a declarative GitOps continuous delivery tool for Kubernetes. It continuously monitors source repositories and reconciles the desired state with the live state in the cluster. ArgoCD supports multiple source types: Git repositories, Helm registries (HTTP and OCI), and OCI images — giving flexibility for different security and compliance requirements.

Core Components

Self-Managed vs EKS Managed ArgoCD

---

App of Apps Pattern

The App of Apps pattern uses a single "parent" ArgoCD Application that contains manifests defining other ArgoCD Applications. This enables declarative bootstrapping of an entire platform from a single entry point.

Structure

When to Use

Benefits and Limitations

---

ApplicationSets

ApplicationSets are a templating mechanism that generates multiple ArgoCD Applications from a single definition. They use generators to produce parameter sets, which are then applied to a template to create Applications.

Generators

Common Patterns

ApplicationSets vs App of Apps

---

GitOps Bridge Pattern

The GitOps Bridge pattern separates infrastructure provisioning (Terraform) from application/add-on management (ArgoCD). Terraform creates the cluster and bootstraps ArgoCD; ArgoCD manages everything else.

How It Works

Integration with terraform-aws-modules/eks

The terraform-aws-modules/eks module provisions the cluster. Terraform then installs ArgoCD and creates a bootstrap Application pointing to the GitOps repository. From that point, ArgoCD takes over management of add-ons and applications.

---

Multi-Tenant RBAC

ArgoCD Projects provide tenant isolation by restricting what each team can deploy, where they can deploy, and what cluster resources they can access.

AppProject Scoping

SSO Integration

RBAC Model

---

Multi-Cluster Architecture Patterns

When deploying ArgoCD (managed or self-managed) across multiple clusters, three patterns emerge:

Hub-and-Spoke (Centralized)

All capabilities run on a central management cluster that orchestrates workloads and infrastructure across spoke clusters.

Best for: Centralized platform teams, audit/compliance requirements, fleet management.

Decentralized

Each cluster runs its own capabilities independently.

Best for: Autonomous teams, independent clusters, minimal cross-cluster dependencies.

Hybrid (Hub + Local ACK)

Combines centralized GitOps delivery with local resource management based on scope.

Best for: Platform teams managing critical infrastructure centrally while enabling self-service for application teams.

Choosing a Pattern

You can start with one pattern and evolve — capabilities are independent and can be deployed differently across clusters.

---

EKS ArgoCD Capability (Managed)

The EKS ArgoCD Capability is an AWS-managed ArgoCD service that runs in AWS-owned infrastructure, separate from your cluster. AWS handles scaling, patching, and upgrading. It does not consume worker node resources.

Managed Capability Limitations

Features not available in the managed capability (use self-managed if you need these):

Key Operational Differences

• Single namespace: All ArgoCD CRs (Application, ApplicationSet, AppProject) must be created in one namespace (default: argocd). Workloads deploy to any namespace in any target cluster.
• Only EKS targets: Deployment targets must be EKS clusters identified by ARN (not arbitrary Kubernetes API server URLs).
• Local cluster not auto-registered: You must explicitly register the local cluster using its ARN to deploy to it.
• CLI differences: Use argocd app sync namespace/appname (namespace prefix required). argocd admin and argocd login are not supported — use account or project tokens.
• Namespace isolation: Keep only ArgoCD-relevant secrets in the ArgoCD namespace — the capability has access to all secrets in its namespace.

When to Use EKS Managed vs Self-Managed

Migrating from Self-Managed to EKS Managed

1. Review current config for unsupported features (CMPs, custom Lua, Notifications, UI extensions) 2. Scale self-managed ArgoCD controllers to zero replicas 3. Create ArgoCD capability resource on your cluster 4. Export existing Applications, ApplicationSets, and AppProjects 5. Migrate repository credentials and cluster secrets 6. Update destination.server fields to use cluster names or EKS cluster ARNs 7. Apply manifests to the managed instance 8. Verify applications sync correctly 9. Decommission self-managed installation

Existing Application/ApplicationSet manifests work with minimal modification — the managed capability uses the same Kubernetes APIs and CRDs.

---

ACK and KRO Integration

AWS Controllers for Kubernetes (ACK)

ACK lets you manage AWS resources using Kubernetes custom resources. Instead of using Terraform or CloudFormation for AWS resources, you define them as Kubernetes manifests that ArgoCD can manage via GitOps. ACK translates Kubernetes CR specs into AWS API calls and continuously reconciles to maintain desired state (detecting and correcting drift).

Kube Resource Orchestrator (KRO)

KRO provides platform abstractions via ResourceGraphDefinitions (RGDs). Platform teams define golden path templates that combine multiple resources (Kubernetes + AWS via ACK) into a single custom resource that tenant teams consume. KRO automatically determines interdependencies and resource ordering, and uses Common Expression Language (CEL) for injecting values between resources and conditional logic.

Combined Pattern: ArgoCD + ACK + KRO

---

Sources:

• ArgoCD Documentation
• EKS Workshop — ArgoCD
• AWS EKS Capabilities
• AWS Blog — GitOps with EKS Blueprints and ArgoCD
• AWS Blog — Deep dive: Streamlining GitOps with EKS ArgoCD Capability

EKS Autoscaling Best Practices

Part of: eks-best-practices

Purpose: Autoscaler selection, Cluster Autoscaler, HPA, VPA, KEDA, and CoreDNS autoscaling for Amazon EKS

---

Table of Contents

1. Autoscaler Selection 2. Cluster Autoscaler 3. EKS Auto Mode 4. Horizontal Pod Autoscaler 5. Vertical Pod Autoscaler 6. KEDA Event-Driven Autoscaling 7. CoreDNS Autoscaling

---

Autoscaler Selection

Node Autoscaler Decision Matrix

For detailed Karpenter guidance, see: Karpenter Reference

Pod Autoscaler Decision Matrix

---

Cluster Autoscaler

When to Use CAS Over Karpenter

• EKS on Outposts (Karpenter not supported)
• Self-managed node groups with specific AMI requirements
• Clusters already running CAS with complex ASG configurations
• Organizations requiring node group-level operational boundaries

Operating the Cluster Autoscaler

CAS runs as a single-replica Deployment using leader election for HA. It is not horizontally scalable -- scale it vertically for large clusters.

Key requirements:

• Version must match cluster version -- cross-version compatibility is not tested or supported
• Enable Auto Discovery -- unless you have specific advanced use cases requiring manual ASG configuration
• Use EKS Managed Node Groups -- they provide automatic ASG discovery and graceful node termination

IAM Least Privilege

Scope CAS's IAM permissions to the cluster's own ASGs. This prevents a CAS instance in one cluster from modifying node groups in another cluster:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:SetDesiredCapacity",
        "autoscaling:TerminateInstanceInAutoScalingGroup"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/k8s.io/cluster-autoscaler/enabled": "true",
          "aws:ResourceTag/k8s.io/cluster-autoscaler/my-cluster": "owned"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DescribeAutoScalingInstances",
        "autoscaling:DescribeLaunchConfigurations",
        "autoscaling:DescribeScalingActivities",
        "autoscaling:DescribeTags",
        "ec2:DescribeImages",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeLaunchTemplateVersions",
        "ec2:GetInstanceTypesFromInstanceRequirements",
        "eks:DescribeNodegroup"
      ],
      "Resource": "*"
    }
  ]
}

Node Group Configuration

Nodes within a node group must have identical scheduling properties (labels, taints, resources). For MixedInstancePolicies:

• Instance types must have the same shape for CPU, memory, and GPU
• The first instance type in the policy is used for scheduling simulation
• Larger subsequent types waste resources after scale-out; smaller ones cause scheduling failures

Design principles:

• Prefer fewer node groups with many nodes over many node groups with few nodes -- this has the biggest impact on CAS scalability
• Use Namespaces for pod isolation instead of dedicated node groups (except in low-trust multi-tenant clusters)
• Use node taints or selectors as the exception, not the rule
• Define regional resources as a single ASG spanning multiple AZs

Key CAS Configuration

# Helm values for Cluster Autoscaler
autoDiscovery:
  clusterName: my-cluster
extraArgs:
  balance-similar-node-groups: true
  skip-nodes-with-local-storage: false
  expander: least-waste           # or: priority, random, most-pods
  scale-down-utilization-threshold: 0.5
  scale-down-delay-after-add: 10m
  scale-down-unneeded-time: 10m
  max-graceful-termination-sec: 600

CAS Expander Selection

Priority Expander Example

Use the priority expander to prefer specific node groups with fallback:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-autoscaler-priority-expander
  namespace: kube-system
data:
  priorities: |-
    10:
      - .*p2-node-group.*
    50:
      - .*p3-node-group.*

CAS tries p3-node-group first. If provisioning doesn't succeed within --max-node-provision-time (default 15 minutes), it falls back to p2-node-group.

Spot Instances with CAS

Separate On-Demand and Spot capacity into different ASGs because their scheduling properties differ fundamentally (Spot nodes are typically tainted for preemption tolerance).

For MixedInstancePolicies with Spot:

• All instance types must have similar CPU and memory (e.g., m4, m5, m5a, m5n families)
• Use the ec2-instance-selector tool to identify similar instance types
• Maximize diversity across instance families and AZs to reduce interruption impact
• Use --expander=least-waste to further optimize cost across diverse node groups

Overprovisioning

CAS adds nodes only when needed, which means pods wait for node launch (~60-90s). For latency-sensitive workloads, use overprovisioning:

Deploy low-priority "pause" pods that occupy spare capacity. When real pods arrive with higher priority, the pause pods are preempted, and the real pods schedule immediately on the existing node. The now-unschedulable pause pods trigger CAS to scale out a new node in the background.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: overprovisioning
value: -1  # Lower than default (0)
globalDefault: false
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: overprovisioning
spec:
  replicas: 3  # One per AZ for optimal zone scheduling
  template:
    spec:
      priorityClassName: overprovisioning
      containers:
      - name: pause
        image: registry.k8s.io/pause
        resources:
          requests:
            cpu: "1"
            memory: 2Gi

Size the pause pods to match your typical workload. Set replicas equal to the number of AZs to ensure spare capacity in each zone.

Scaling from Zero

CAS can scale node groups to and from zero, providing significant cost savings for intermittent workloads. CAS detects node resources from the InstanceType in the LaunchConfiguration or LaunchTemplate.

For pods requiring additional resources, node selectors, or taints not discoverable from the launch config, use ASG tags:

k8s.io/cluster-autoscaler/node-template/resources/$RESOURCE_NAME: 5
k8s.io/cluster-autoscaler/node-template/label/$LABEL_KEY: $LABEL_VALUE
k8s.io/cluster-autoscaler/node-template/taint/$TAINT_KEY: NoSchedule

Note: when scaling to zero, capacity is returned to EC2 and may not be available when you scale back up.

Prevent Scale-Down Eviction

For expensive-to-restart workloads (batch jobs, ML training, long test runs), prevent CAS from scaling down the node:

metadata:
  annotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

Scalability Tuning

For clusters approaching 1000+ nodes:

• Vertically scale CAS -- increase CPU and memory requests. CAS stores all pods and nodes in memory, which can exceed 1GB for large clusters. Use the Addon Resizer or VPA to automate this.
• Reduce node groups -- fewer, larger node groups improve CAS performance. Many small node groups is a CAS anti-pattern.
• Tune scan interval -- the default 10s scan interval works for most clusters. For large clusters, increasing to 30-60s reduces API call volume (6x fewer calls) with moderate scale-up latency increase. Since node launch takes ~60-90s anyway, a 30s scan interval adds only ~50% to total scale-up time.
• Shard as last resort -- deploy multiple CAS instances, each managing different ASGs. Use separate namespaces to avoid leader election conflicts. Caveat: shards don't communicate, so multiple shards may scale out for the same unschedulable pod.

Advanced Use Cases

EBS Volumes and Stateful Workloads:

• Enable balance-similar-node-groups=true for cross-AZ balancing
• Configure identical node groups per AZ, each with its own EBS volumes

Accelerators/GPU:

• GPU device plugins can take minutes to advertise resources after node launch, causing repeated unnecessary scale-outs
• Label GPU nodes with --node-labels k8s.amazonaws.com/accelerator=$ACCELERATOR_TYPE on the kubelet
• CAS uses this label to trigger accelerator-optimized behavior (including scale-down of nodes with unused accelerators)

CAS Parameter Reference

---

EKS Auto Mode

What EKS Auto Mode Manages

EKS Auto Mode provides AWS-managed:

• Node provisioning and scaling (no node groups to manage)
• Node OS patching and upgrades
• Compute optimization (instance selection, Spot)
• Cluster add-ons (VPC CNI, CoreDNS, kube-proxy)

When to Use Auto Mode

Use Auto Mode when:

• Minimizing operational overhead is the top priority
• Standard compute patterns (web apps, APIs, microservices)
• Teams without deep Kubernetes node management expertise
• Greenfield clusters where you can adopt Auto Mode from the start

Don't use Auto Mode when:

• You need custom AMIs or specific kernel configurations
• GPU/ML workloads requiring specific instance types or drivers
• Windows containers are required
• You need fine-grained control over node configuration
• Running on EKS Outposts or EKS Anywhere

Auto Mode Configuration

# Terraform
resource "aws_eks_cluster" "this" {
  name = "my-cluster"

  compute_config {
    enabled       = true
    node_pools    = ["general-purpose", "system"]
    node_role_arn = aws_iam_role.node.arn
  }

  kubernetes_network_config {
    elastic_load_balancing {
      enabled = true
    }
  }

  storage_config {
    block_storage {
      enabled = true
    }
  }
}

For detailed Auto Mode guidance, see: EKS Auto Mode Reference

---

Horizontal Pod Autoscaler

HPA Configuration Patterns

# CPU-based HPA with scaling behavior
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 3
  maxReplicas: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Percent
        value: 100
        periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300    # 5 min cooldown
      policies:
      - type: Percent
        value: 10
        periodSeconds: 60               # Scale down 10% per minute

HPA Best Practices

DO:

• Set CPU target to 60-80% (leave headroom for bursts)
• Configure scaleDown.stabilizationWindowSeconds (prevent flapping)
• Set minReplicas >= 2 for HA (>= 3 for production)
• Use behavior policies to control scaling speed

DON'T:

• Set target utilization to 90%+ (no burst headroom)
• Use HPA without setting resource requests (HPA needs requests for CPU %)
• Combine HPA and VPA on the same CPU/memory metric
• Set minReplicas: 1 for production services

Custom Metrics with CloudWatch

# HPA with custom CloudWatch metric via KEDA or metrics-adapter
metrics:
- type: External
  external:
    metric:
      name: sqs-queue-depth
      selector:
        matchLabels:
          queue: order-processing
    target:
      type: AverageValue
      averageValue: "5"

---

Vertical Pod Autoscaler

VPA Modes

VPA Configuration

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: app-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Off"  # Start with recommendations only
  resourcePolicy:
    containerPolicies:
    - containerName: app
      minAllowed:
        cpu: 100m
        memory: 128Mi
      maxAllowed:
        cpu: 4
        memory: 8Gi
      controlledResources: ["cpu", "memory"]

VPA + HPA Coexistence

Rule: Never scale on the same metric.

Safe combinations:

• VPA manages memory requests + HPA scales on CPU
• VPA manages CPU/memory + HPA scales on custom metrics (QPS, queue depth)

Conflict:

• VPA manages CPU + HPA scales on CPU (fight each other)

---

KEDA Event-Driven Autoscaling

KEDA with SQS

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sqs-consumer
spec:
  scaleTargetRef:
    name: queue-processor
  minReplicaCount: 0    # Scale to zero when no messages
  maxReplicaCount: 100
  cooldownPeriod: 300
  triggers:
  - type: aws-sqs-queue
    metadata:
      queueURL: https://sqs.us-east-1.amazonaws.com/123456789012/orders
      queueLength: "5"
      awsRegion: us-east-1
    authenticationRef:
      name: keda-aws-credentials

Common KEDA Triggers for EKS

---

CoreDNS Autoscaling

Proportional Autoscaler

# Scale CoreDNS proportionally to cluster size
apiVersion: v1
kind: ConfigMap
metadata:
  name: dns-autoscaler
  namespace: kube-system
data:
  linear: |-
    {
      "coresPerReplica": 256,
      "nodesPerReplica": 16,
      "min": 2,
      "max": 20,
      "preventSinglePointFailure": true
    }

Scaling formula: replicas = max(ceil(cores / coresPerReplica), ceil(nodes / nodesPerReplica))

For a 100-node cluster with 400 cores: max(ceil(400/256), ceil(100/16)) = max(2, 7) = 7 replicas

CoreDNS Tuning

Set lameduck duration to 30 seconds -- delays CoreDNS shutdown during pod termination, allowing iptables rule propagation and in-flight request completion. Especially critical with Karpenter's rapid node churn.

Optimize ndots setting -- default ndots: 5 causes excessive DNS queries (tries multiple search domain suffixes before resolving external names). Set to 2 for pods making external DNS calls:

spec:
  dnsConfig:
    options:
    - name: ndots
      value: "2"

Alternatively, use trailing dots for FQDNs in application config: api.example.com.

Use NodeLocal DNSCache with Karpenter to prevent DNS failures during node scaling events.

---

Sources:

• AWS EKS Best Practices Guide -- Cluster Autoscaler
• AWS EKS Best Practices Guide -- Auto Mode
• AWS EKS Best Practices Guide -- Karpenter

EKS Cluster Upgrade Best Practices

Part of: eks-best-practices

Purpose: Upgrade planning, in-place and blue-green strategies, add-on management, API deprecation detection, and version support for Amazon EKS

---

Table of Contents

1. Upgrade Planning 2. In-Place Upgrade Procedure 3. Blue-Green Cluster Upgrade 4. Add-On Version Management 5. API Deprecation Detection 6. Data Plane Upgrades 7. Version Support Policy 8. Bottlerocket-Specific Guidance 9. Emergency Rollback Procedures

---

Upgrade Planning

Pre-Upgrade Checklist

Verify Infrastructure Requirements

AWS requires these resources to complete the control plane upgrade:

# 1. Verify at least 5 free IPs in cluster subnets
CLUSTER=<cluster-name>
aws ec2 describe-subnets --subnet-ids \
  $(aws eks describe-cluster --name ${CLUSTER} \
  --query 'cluster.resourcesVpcConfig.subnetIds' --output text) \
  --query 'Subnets[*].[SubnetId,AvailabilityZone,AvailableIpAddressCount]' \
  --output table

# 2. Verify EKS IAM role exists with correct trust policy
ROLE_ARN=$(aws eks describe-cluster --name ${CLUSTER} \
  --query 'cluster.roleArn' --output text)
aws iam get-role --role-name ${ROLE_ARN##*/} \
  --query 'Role.AssumeRolePolicyDocument'
# Should show: Principal: eks.amazonaws.com, Action: sts:AssumeRole

# 3. If secret encryption is enabled, verify KMS key access
aws eks describe-cluster --name ${CLUSTER} \
  --query 'cluster.encryptionConfig'

If cluster subnets are running low on IPs, add new subnets in the same AZs via UpdateClusterConfiguration before upgrading. Consider associating additional CIDR blocks to expand the IP pool.

Enable Control Plane Logging

Enable logging before the upgrade to capture any errors during the process:

aws eks update-cluster-config --name my-cluster \
  --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

EKS Cluster Insights

# List upgrade insights with issues
aws eks list-insights \
  --cluster-name my-cluster \
  --filter 'statuses=ERROR,WARNING'

# Get detailed remediation advice for a specific insight
aws eks describe-insight \
  --cluster-name my-cluster \
  --id <insight-id>
# Returns: affected resources, deprecated APIs, recommended actions

Cluster Insights automatically detects:

• Deprecated API usage in your cluster (last 30 days)
• Add-on compatibility issues
• Known upgrade blockers

If an insight shows "status": ERROR, you must resolve it before upgrading.

Upgrade Strategy Decision

---

In-Place Upgrade Procedure

Upgrade Sequence (Strict Order)

1. Control Plane    (AWS-managed, ~15-30 min)
     ↓
2. EKS Add-ons     (VPC CNI, CoreDNS, kube-proxy, EBS CSI)
     ↓
3. Data Plane       (Node groups, Karpenter nodes, or Fargate restart)
     ↓
4. Custom Add-ons   (Ingress controller, cert-manager, monitoring, etc.)
     ↓
5. Update kubectl   (Match client to cluster version)

Step 1: Upgrade Control Plane

# Check current version
aws eks describe-cluster --name my-cluster \
  --query 'cluster.version'

# Upgrade control plane (one minor version at a time)
aws eks update-cluster-version \
  --name my-cluster \
  --kubernetes-version 1.31

# Monitor upgrade status
aws eks describe-update \
  --name my-cluster \
  --update-id <update-id>

Key constraints:

• Can only upgrade one minor version at a time (1.29 → 1.30, not 1.29 → 1.31)
• Control plane upgrade takes 15-30 minutes
• API server remains available during upgrade (brief API errors possible)
• Cannot rollback control plane version

Step 2: Upgrade Add-ons

# Check current add-on versions
aws eks describe-addon --cluster-name my-cluster --addon-name vpc-cni
aws eks describe-addon --cluster-name my-cluster --addon-name coredns
aws eks describe-addon --cluster-name my-cluster --addon-name kube-proxy

# Upgrade each add-on
aws eks update-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --addon-version v1.18.0-eksbuild.1 \
  --resolve-conflicts OVERWRITE

VPC CNI constraint: When installed as an EKS managed add-on, VPC CNI can only be upgraded one minor version at a time (same as the cluster).

EKS add-ons are not automatically upgraded during a control plane upgrade — you must initiate each add-on update separately.

Step 3: Upgrade Data Plane

For Managed Node Groups:

aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name default \
  --kubernetes-version 1.31

# Monitor rolling update
aws eks describe-nodegroup \
  --cluster-name my-cluster \
  --nodegroup-name default \
  --query 'nodegroup.updateConfig'

For EKS Auto Mode: No action needed. After the control plane upgrade, Auto Mode incrementally updates managed nodes while respecting PDBs. Monitor to verify compliance with your operational requirements.

For Karpenter: See Data Plane Upgrades section.

For Fargate: Redeploy workloads to pick up the new version. Identify Fargate pods:

kubectl get pods -A -o wide | grep fargate-
# Restart each deployment running on Fargate
kubectl rollout restart deployment <name> -n <namespace>

Step 4: Update kubectl

After the cluster upgrade, update your kubectl client to match:

# Verify kubectl version matches cluster
kubectl version --short

Ensure Availability During Upgrade

Configure PDBs and topology spread to prevent downtime during data plane rolling updates:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp
spec:
  minAvailable: "80%"
  selector:
    matchLabels:
      app: myapp
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 10
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      topologySpreadConstraints:
      - maxSkew: 2
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: myapp
      - maxSkew: 2
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: myapp

Spreading across zones and hosts ensures pods migrate to new nodes automatically during rolling replacements.

---

Blue-Green Cluster Upgrade

When to Use Blue-Green

• Major version jumps (skipping multiple minor versions via new cluster)
• Zero-downtime requirement for the upgrade itself
• Significant architectural changes alongside version upgrade
• Compliance requirement for rollback capability

Blue-Green Procedure

1. Create "green" cluster at target version
2. Deploy all workloads to green (via GitOps)
3. Run smoke tests on green cluster
4. Shift traffic gradually (DNS weighted / ALB weighted)
5. Monitor for issues
6. Decommission "blue" cluster after validation

Traffic Shifting Patterns

Blue-Green Downsides to Consider

• API endpoint and OIDC change — all consumers (kubectl, CI/CD, IRSA trust policies) must be updated to the new cluster's endpoint
• Load balancers and external DNS cannot easily span both clusters simultaneously
• 2× cluster cost during the migration period, which may also limit region EC2 capacity
• Dependent workloads need coordination to migrate together (e.g., services that call each other)
• Stateful workloads require backup/restore or shared storage (EFS, managed databases)

Stateful Workload Migration

For workloads with PersistentVolumes: 1. Back up data with Velero or application-level backup 2. Restore in new cluster 3. For EBS: Snapshot → Create volume in new cluster's AZs 4. For EFS: Mount same file system from both clusters 5. For databases: Use managed service (RDS, DynamoDB) — no migration needed

Note: Velero backs up Kubernetes resources and PV data, but not AWS resources (IAM roles, security groups, VPC config). These must be recreated separately (Terraform/CloudFormation).

---

Add-On Version Management

Core EKS Add-Ons

Add-On Compatibility Matrix Check

# List compatible versions for an add-on
aws eks describe-addon-versions \
  --addon-name vpc-cni \
  --kubernetes-version 1.31 \
  --query 'addons[0].addonVersions[*].{Version:addonVersion,Default:compatibilities[0].defaultVersion}' \
  --output table

Inventory All Components Using K8s API

Before upgrading, identify every component that uses the Kubernetes API directly:

# Find critical cluster components (often in *-system namespaces)
kubectl get ns | grep '-system'

Common components to verify compatibility: AWS LBC, Karpenter, Cluster Autoscaler, cert-manager, metrics-server, monitoring agents, ingress controllers, CSI drivers.

Karpenter is tightly coupled to the Kubernetes version — always check Karpenter release notes for target version support.

Cluster Autoscaler must match the cluster minor version — upgrade it when you upgrade the cluster.

Self-Managed Add-On Upgrades

For add-ons not managed by EKS (ingress controllers, cert-manager, etc.): 1. Check the add-on's compatibility matrix for the target K8s version 2. Upgrade the add-on before or after the control plane upgrade (per add-on docs) 3. Test in non-prod first

---

API Deprecation Detection

Detection Methods

Monitor Deprecated API Usage (Prometheus)

The apiserver_requested_deprecated_apis metric (since K8s 1.19) tracks real-time usage of deprecated APIs:

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis
# Example output:
# apiserver_requested_deprecated_apis{group="policy",removed_release="1.25",
#   resource="podsecuritypolicies",version="v1beta1"} 1

Query Audit Logs for Deprecated API Calls

CLUSTER="<cluster_name>"
QUERY_ID=$(aws logs start-query \
  --log-group-name /aws/eks/${CLUSTER}/cluster \
  --start-time $(date -u --date="-30 minutes" "+%s") \
  --end-time $(date "+%s") \
  --query-string 'fields @message | filter `annotations.k8s.io/deprecated`="true"' \
  --query queryId --output text)

sleep 5
aws logs get-query-results --query-id $QUERY_ID

Using Pluto

# Install
brew install FairwindsOps/tap/pluto

# Scan Helm releases in cluster
pluto detect-helm --target-versions k8s=v1.31

# Scan manifest files (more accurate — recommended for CI)
pluto detect-files -d manifests/ --target-versions k8s=v1.31

# Scan live cluster
pluto detect-api-resources --target-versions k8s=v1.31

Using kube-no-trouble

sh -c "$(curl -sSL https://git.io/install-kubent)"
kubent --target-version 1.31

Scanning static manifests is generally more accurate than live cluster scanning (fewer false positives). Run kubent/pluto in CI pipelines to catch issues before deployment.

Key API Removals by Version

Feature-Specific Migration Guidance

Dockershim removal (1.25): EKS Optimized AMI for 1.25+ uses containerd, not Docker. If you mount the Docker socket (/var/run/docker.sock), detect dependencies with the Detector for Docker Socket (DDS) kubectl plugin before upgrading nodes.

PodSecurityPolicy removal (1.25): Migrate to built-in Pod Security Standards (PSS) or a policy-as-code solution (Kyverno, OPA/Gatekeeper) before upgrading to 1.25.

In-tree storage driver deprecation (1.23): Install the Amazon EBS CSI driver before upgrading to 1.23+ to avoid service interruption for EBS-backed workloads. The in-tree to CSI migration is enabled by default in EKS 1.23+.

Convert Manifests

Use kubectl convert to automatically update API versions in manifest files:

kubectl-convert -f old-deployment.yaml --output-version apps/v1

---

Data Plane Upgrades

Version Skew Policy

This applies to MNG, self-managed nodes, and Fargate. However, keep AMI versions current for security — older kubelet versions may have unpatched CVEs.

Karpenter Node Upgrades

Automatic via drift detection:

When you update the control plane version, Karpenter detects AMI drift and automatically replaces nodes:

1. Karpenter detects the node's AMI doesn't match the latest EKS-optimized AMI 2. Karpenter provisions a new node with the updated AMI 3. Karpenter cordons the old node 4. Karpenter drains the old node (respecting PDBs) 5. Pods reschedule on the new node 6. Old node is terminated

Control drain speed with NodePool disruption settings:

spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    budgets:
    - nodes: "10%"    # Max 10% of nodes disrupted at a time
    - nodes: "0"
      schedule: "0 9 * * 1-5"  # No disruptions during business hours
      duration: 8h

Karpenter node expiry as an alternative to drift — set expireAfter on the NodePool to automatically replace nodes after a time period, ensuring regular AMI refresh:

spec:
  template:
    spec:
      expireAfter: 720h  # 30 days — nodes replaced with latest AMI

Karpenter does not add jitter to expiry — configure PDBs to prevent simultaneous expiration from disrupting workloads.

Force immediate node replacement:

kubectl annotate nodes --all karpenter.sh/voluntary-disruption=drifted --overwrite

Managed Node Group Upgrades

# Rolling update (default strategy)
aws eks update-nodegroup-version \
  --cluster-name my-cluster \
  --nodegroup-name default

# Configure update behavior
aws eks update-nodegroup-config \
  --cluster-name my-cluster \
  --nodegroup-name default \
  --update-config '{"maxUnavailable": 1}'
  # Or: {"maxUnavailablePercentage": 33}

Self-Managed Node Group Upgrades

For nodes deployed outside the EKS managed service, use your provisioning tool:

---

Version Support Policy

EKS Version Lifecycle

Always get exact dates from the EKS API — do not compute them from release dates. Standard and extended support windows have historically shifted; the API is the only trustworthy source.

aws eks describe-cluster-versions --region <region> \
  --query 'clusterVersions[*].[clusterVersion,status,endOfStandardSupportDate,endOfExtendedSupportDate]' \
  --output table

You can disable extended support so auto-upgrade happens at end of standard support instead.

Planning Timeline

• New K8s minor on EKS: ~3 releases per year
• End of standard support: query the API per cluster version
• End of extended support: query the API per cluster version
• After: EKS auto-upgrades your cluster (may disrupt workloads)

Recommendation: Upgrade every 3-4 months to stay within standard support. Budget one upgrade cycle per quarter. Look beyond the next version — review upcoming K8s releases to identify major changes early (e.g., Dockershim removal was announced well before 1.25).

Additional Upgrade Tools

✅ DO:

• Subscribe to EKS version release notifications
• Maintain a documented upgrade runbook
• Test upgrades in non-prod environments first
• Use Cluster Insights to validate readiness
• Enable control plane logging before upgrading
• Use Managed Node Groups, Karpenter, or Auto Mode to simplify data plane upgrades
• Review the EKS release calendar

❌ DON'T:

• Skip more than 2 minor versions behind current
• Rely on extended support as a permanent solution (extra cost, then auto-upgrade)
• Upgrade production without testing in staging first
• Ignore API deprecation warnings from Cluster Insights
• Forget to restart Fargate deployments after control plane upgrade

---

Test Cluster Validation Phase

Before upgrading production, validate the target Kubernetes version in a dedicated test cluster.

9-Step Test Procedure

---

Bottlerocket-Specific Guidance

Bottlerocket Update Operator (BUO)

The Bottlerocket Update Operator automates OS-level updates for Bottlerocket nodes without requiring full node replacement. BUO runs as a DaemonSet (agent on each node) plus a controller that coordinates updates in waves to avoid disrupting too many nodes simultaneously.

SSM Connectivity Verification

Bottlerocket uses AWS Systems Manager (SSM) for administrative access — there is no SSH. Bottlerocket provides two special containers:

To verify SSM connectivity: check that the SSM agent is running on the node, the node's IAM role has AmazonSSMManagedInstanceCore policy, and VPC endpoints for SSM are configured (if private subnets without NAT).

OS Update vs K8s Version Upgrade

---

Emergency Rollback Procedures

Rollback Matrix

Full Cluster Rebuild from Backup

Use when: catastrophic cluster failure, corrupted etcd state, or failed upgrade with no rollback path.

Prerequisites:

• Velero backups (K8s resources + PV snapshots) in a separate account/region
• GitOps repository with all application and add-on manifests
• Terraform code for cluster infrastructure
• Note: Velero does not back up AWS resources (IAM roles, SGs, VPC config) — these must be recreated via IaC

High-level steps:

✅ DO:

• Test full cluster rebuild quarterly in an isolated environment
• Keep Terraform state and Velero backups in a separate account
• Document the rebuild runbook with exact commands and validation steps

❌ DON'T:

• Attempt to repair a corrupted cluster for hours — rebuild is often faster
• Skip the validation step before switching traffic
• Forget to update DNS TTLs in advance (low TTL enables faster failover)

---

Sources:

• AWS EKS Best Practices Guide — Cluster Upgrades
• EKS Version Lifecycle
• Karpenter Upgrade Guide

Container Registry Best Practices

Part of: eks-best-practices

Purpose: ECR architecture, operating models, image promotion, vulnerability scanning, base image curation, lifecycle policies, pull-through cache, managed signing, archival storage, and registry configuration for Amazon EKS

---

Table of Contents

1. ECR Architecture 2. Operating Models 3. Image Promotion Pipeline 4. Vulnerability Scanning 5. Base Image Curation 6. ECR Lifecycle Policies 7. Pull-Through Cache 8. Repository Creation Templates 9. Managed Signing 10. Archival Storage Class 11. Registry Configuration

---

ECR Architecture

Private vs Public Repositories

Repository Naming Conventions

Use a consistent naming pattern that encodes ownership and purpose:

Cross-Account Access

VPC Endpoints for ECR

For private clusters or security-sensitive environments, configure VPC endpoints to avoid routing image pulls through the internet:

---

Operating Models

When to Use Each

---

Image Promotion Pipeline

Promotion Flow

Tag Strategy

Promotion Methods

DO:

• Use immutable tags (Git SHA or semver) -- never latest in production
• Enable immutable tag setting on ECR repositories to prevent overwrites
• Include image digest (@sha256:...) in production deployments for guaranteed immutability

DON'T:

• Use latest tag in production -- it's mutable and non-deterministic
• Rebuild images for promotion -- re-tag or replicate the exact same digest
• Skip scanning between promotion stages

---

Vulnerability Scanning

ECR Scanning Options

Severity Gating

Integration with Security Hub

Enhanced scanning findings are automatically sent to Security Hub, providing centralized visibility across all accounts. Configure Security Hub automations to:

• Notify teams of critical findings via SNS
• Create Jira/ServiceNow tickets for high findings
• Track remediation SLAs

DO:

• Enable enhanced scanning (Inspector) for production repositories
• Set up continuous scanning -- new CVEs are disclosed daily
• Gate CI/CD pipelines on scan results -- block critical/high before push
• Integrate with Security Hub for centralized finding management

DON'T:

• Rely on basic scanning for production -- it misses language-level vulnerabilities
• Scan only at push time -- images become vulnerable as new CVEs are disclosed
• Ignore medium-severity findings indefinitely -- track and remediate on a schedule

---

Base Image Curation

Why Curate Base Images

Using uncurated public images introduces risk: unknown vulnerabilities, unnecessary packages (shells, curl, build tools), and inconsistent patching. A curated base image pipeline provides a controlled, scanned, and patched foundation for all application images.

Minimal Base Image Options

Base Image Pipeline

Multi-Architecture Images

For Graviton (arm64) support, build multi-arch images using Docker buildx or CI pipeline matrix builds:

DO:

• Maintain a curated set of approved base images in a dedicated ECR repository
• Rebuild base images weekly to pick up security patches
• Use multi-stage builds to exclude build tools from final images
• Build multi-arch images if using Graviton

DON'T:

• Pull base images directly from Docker Hub in production -- use pull-through cache or internal copies
• Include shells, curl, or package managers in production images unless required
• Skip scanning base images -- they're the foundation of your security posture

---

ECR Lifecycle Policies

Lifecycle policies automatically clean up old or untagged images, reducing storage costs and keeping repositories manageable.

Recommended Rules

Count Types

Lifecycle rules support different ways to measure image age:

Tag Filtering

Use tagPatternList with wildcards to target specific images:

{
  "tagStatus": "tagged",
  "tagPatternList": ["release-*", "v*"],
  "countType": "sinceImagePushed",
  "countNumber": 90,
  "action": { "type": "expire" }
}

This is more flexible than tagPrefixList -- patterns like *-rc or dev-* let you target release candidates, dev builds, or any naming convention.

DO:

• Apply lifecycle policies to every repository -- don't let images accumulate indefinitely
• Keep at least 30 recent tagged images for rollback capability
• Remove untagged images aggressively (1 day retention)
• Use sinceImagePulled for shared base images to preserve actively-used versions

DON'T:

• Delete all old images without considering rollback needs
• Apply lifecycle policies that conflict with compliance retention requirements
• Forget to set lifecycle policies on pull-through cache repositories -- they accumulate images quickly

---

Pull-Through Cache

ECR pull-through cache rules automatically cache images from upstream public registries in your private ECR. When a pod pulls an image through the cache, ECR fetches it from the upstream registry, stores it locally, and serves subsequent pulls from the cache.

Supported Upstream Registries

How It Works

1. Pod requests image via ECR pull-through cache URI (e.g., <acct>.dkr.ecr.<region>.amazonaws.com/docker-hub/library/nginx:1.25) 2. ECR checks if image exists in cache 3. If missing or stale (>24 hours since last check), ECR pulls from upstream -- this requires internet access via NAT gateway or VPC endpoint 4. ECR stores the image (including multi-arch manifests) and serves it locally 5. Subsequent pulls come from cache with no upstream dependency

When to Use

DO:

• Enable pull-through cache for Docker Hub at minimum -- rate limiting is the most common issue
• Store upstream credentials in Secrets Manager for registries that require authentication
• Apply vulnerability scanning and lifecycle policies to cache repositories
• Use repository creation templates to auto-configure cache repositories

DON'T:

• Assume cached images are scanned automatically -- configure scanning rules for cache repositories
• Use pull-through cache as a substitute for curated base images -- it caches everything, including vulnerable images
• Forget that the first pull requires internet access -- air-gapped clusters need initial seeding

---

Repository Creation Templates

Repository creation templates automatically configure new repositories as they're created -- whether through pull-through cache, create-on-push, or replication. Without templates, new repositories get default settings and miss critical configurations like scanning, encryption, and lifecycle policies.

How Templates Work

Templates match repository names by prefix. When a new repository is created (by any mechanism), ECR checks for a matching template and applies its configuration:

Template Matching

Templates use prefix matching with a priority order: 1. Longest matching prefix wins 2. If no prefix matches, the ROOT template applies (if configured)

Example: For repository docker-hub/library/nginx, a template with prefix docker-hub/library/ takes priority over one with prefix docker-hub/.

Create-on-Push

Create-on-push allows repositories to be created automatically when an image is pushed to a repository name that doesn't exist yet. Combined with templates, this means new services can push images without any pre-provisioning -- the repository is created with the correct configuration automatically.

Enable create-on-push either as a registry default or per-template.

DO:

• Create a ROOT template as a catch-all to ensure every repository gets baseline configuration
• Use specific prefix templates for pull-through cache registries (e.g., docker-hub/, ghcr/)
• Include lifecycle policies in templates so cache repositories don't accumulate images endlessly
• Enable create-on-push for development environments to reduce friction

DON'T:

• Skip templates for pull-through cache -- without them, cached repos have no scanning or lifecycle policies
• Enable create-on-push in production without templates -- you'll get misconfigured repositories

---

Managed Signing

ECR managed signing automatically signs container images on push using AWS Signer, providing cryptographic proof that an image was built and pushed through your pipeline. This supports verification at deploy time via admission controllers like Kyverno or OPA Gatekeeper.

How It Works

1. Configure signing rules at the registry level (up to 10 rules per registry) 2. Each rule specifies a repository filter (prefix match) and an AWS Signer signing profile 3. When an image is pushed to a matching repository, ECR automatically creates a Notation-format signature 4. The signature is stored alongside the image in the same repository 5. Admission controllers verify the signature before allowing the image to run

Configuration

Integration with Admission Control

Managed signing pairs with Kubernetes admission controllers for deploy-time verification:

DO:

• Enable managed signing for production repositories to establish image provenance
• Use repository prefix filters to sign only images that need verification (avoids signing dev/test images)
• Combine with admission controllers to enforce signature verification at deploy time

DON'T:

• Treat signing as a substitute for vulnerability scanning -- signing proves provenance, not safety
• Use the same signing profile for all environments -- separate dev and prod signing identities

See also: Security -- Supply Chain for admission control patterns and image verification policies

---

Archival Storage Class

ECR archival storage provides a low-cost tier for images you need to retain but rarely access -- compliance snapshots, audit artifacts, or old release images. Archival images cost significantly less than standard storage but must be restored before they can be pulled.

How It Works

Lifecycle Policy Integration

Use lifecycle policies to automatically archive images after a retention period:

{
  "rules": [
    {
      "rulePriority": 1,
      "selection": {
        "tagStatus": "tagged",
        "tagPatternList": ["release-*"],
        "countType": "sinceImagePushed",
        "countNumber": 180
      },
      "action": { "type": "archive" }
    },
    {
      "rulePriority": 2,
      "selection": {
        "tagStatus": "tagged",
        "tagPatternList": ["release-*"],
        "countType": "sinceImageTransitioned",
        "countNumber": 730
      },
      "action": { "type": "expire" }
    }
  ]
}

This archives release images after 180 days and permanently deletes them 2 years after archival -- a typical compliance lifecycle.

DO:

• Use archival storage for images required by compliance but rarely pulled
• Chain lifecycle rules: archive after N days, expire after M days from archival
• Test restore times before relying on archived images for disaster recovery

DON'T:

• Archive images you may need for rapid rollback -- 20-minute restore is too slow for incidents
• Forget that archived images can't be scanned -- restore and scan if you need to assess vulnerabilities

---

Registry Configuration

ECR has registry-level settings that affect all repositories in the account/region. Two settings are particularly useful for large registries.

Blob Mounting

Blob mounting allows image layers that already exist in one repository to be referenced (mounted) when pushing to another repository in the same registry, instead of re-uploading them. This is significant when many images share common base layers.

Keep blob mounting enabled unless you have a specific security requirement to isolate layer access between repositories.

Pull-Time Update Exclusions

When pull-through cache is enabled, ECR checks the upstream registry for updates every 24 hours. Pull-time update exclusions let you pin specific repositories so ECR never re-checks upstream -- the cached version is treated as authoritative.

Use this for:

• Known-good images you've validated and don't want upstream changes to override
• Air-gapped environments where you've seeded images and upstream is unreachable
• Compliance scenarios where you need a frozen, auditable copy

---

Helm Chart Management

ECR OCI Support vs S3 Helm Repository

Pushing and Consuming Helm Charts via ECR

The workflow for Helm charts stored in ECR OCI follows three steps:

1. Authenticate: Obtain an ECR authorization token and pass it to helm registry login. The same ECR IAM credentials used for container images work for Helm charts. 2. Package and push: Package the chart directory into a .tgz archive, then push it to an OCI URI in ECR (e.g., oci://<account-id>.dkr.ecr.<region>.amazonaws.com/charts/). 3. Install from ECR: Reference the OCI URI directly in helm install or in ArgoCD Application source configuration with a specific version tag.

Design considerations:

• Use a dedicated charts/ prefix in ECR to separate Helm charts from container images
• Apply the same ECR lifecycle policies to chart repositories to clean up old versions
• ECR cross-account replication works for Helm charts — spoke accounts get chart replicas automatically
• ArgoCD natively supports OCI Helm sources — no extra configuration needed beyond ECR auth

---

Sources:

• Amazon ECR Documentation
• Designing a secure container image registry
• Amazon Inspector container scanning
• ECR Pull-Through Cache
• ECR Managed Signing
• ECR Archival Storage
• ECR Repository Creation Templates

EKS Cost Optimization

Part of: eks-best-practices

Purpose: Cost optimization framework, compute/networking/storage cost strategies, observability cost management, tagging, and cost visibility tools for Amazon EKS

---

Table of Contents

1. Cost Optimization Framework 2. Compute Cost Optimization 3. Networking Cost Optimization 4. Storage Cost Optimization 5. Observability Cost Optimization 6. Tagging & Cost Visibility

---

Cost Optimization Framework

AWS Cloud Financial Management (CFM) organizes cost optimization into four pillars:

The "See" pillar comes first because you can't optimize what you can't measure. Start with tagging and cost visibility before pursuing compute or networking savings.

EKS Cost Components

Quick Wins

---

Compute Cost Optimization

Compute is typically the largest cost driver for EKS. Optimize in this order:

1. Right-size workloads -- match requests to actual usage 2. Reduce unused capacity -- autoscale and consolidate 3. Optimize capacity types -- Spot, Graviton, Savings Plans

Right-Sizing Workloads

Requests should align with actual utilization. Overprovisioned requests waste capacity -- the largest factor in total cluster costs. Each container (including sidecars) should have its own requests and limits.

Right-sizing tools:

# Deploy VPA in recommendation-only mode
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: app-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Off"  # Recommendations only

# View recommendations
kubectl get vpa app-vpa -o jsonpath='{.status.recommendation.containerRecommendations[*]}' | jq

Right-sizing decision framework:

Reducing Unused Capacity

Use HPA to scale pods based on demand, then let node autoscalers remove empty or underutilized nodes. Restrictive PodDisruptionBudgets can block node scale-down -- set minAvailable well below your replica count (e.g. minAvailable: 4 for a 6-pod deployment).

For event-driven scaling (SQS queues, Kafka, CloudWatch metrics), use KEDA instead of HPA's built-in metrics.

Karpenter Consolidation

Karpenter continuously monitors and bin-packs workloads onto fewer, optimally-sized instances:

# Enable consolidation in NodePool
spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30s  # Fast for non-prod; use longer (e.g. 5m) in prod

Karpenter selects the most cost-effective instance from your allowed types. It replaces underutilized nodes with smaller ones and removes empty nodes automatically.

For workloads that shouldn't be interrupted (long batch jobs without checkpointing), use the karpenter.sh/do-not-disrupt: "true" annotation.

See also: Karpenter Reference for detailed NodePool configuration, consolidation tuning, and Spot handling.

Cluster Autoscaler Priority Expander

If using Cluster Autoscaler instead of Karpenter, the priority expander lets you prefer cheaper capacity:

# Priority expander ConfigMap -- scale reserved/Spot groups before on-demand
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-autoscaler-priority-expander
  namespace: kube-system
data:
  priorities: |-
    10:
      - .*ondemand.*
    50:
      - .*reserved.*

Also consider the Kubernetes Descheduler alongside CAS -- it rebalances pod placement after scheduling to improve cluster-wide utilization, which CAS alone does not do.

Graviton (arm64) Migration

Graviton instances deliver 20-40% better price/performance than equivalent x86:

# Karpenter NodePool supporting both architectures
spec:
  template:
    spec:
      requirements:
      - key: kubernetes.io/arch
        operator: In
        values: ["amd64", "arm64"]  # Karpenter prefers cheaper Graviton
      - key: karpenter.k8s.aws/instance-category
        operator: In
        values: ["c", "m", "r"]

DO:

• Build multi-arch container images (docker buildx)
• Test on arm64 in staging before production
• Use Karpenter -- it automatically selects the most cost-effective architecture

DON'T:

• Assume all container images support arm64 (check base images)
• Mix architectures within a single deployment without affinity rules

Spot Instance Strategies

# Karpenter: Diversified Spot strategy
spec:
  template:
    spec:
      requirements:
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["spot", "on-demand"]
      - key: karpenter.k8s.aws/instance-category
        operator: In
        values: ["c", "m", "r"]     # Multiple families
      - key: karpenter.k8s.aws/instance-generation
        operator: Gt
        values: ["5"]                # Current gen only
      - key: karpenter.k8s.aws/instance-size
        operator: In
        values: ["large", "xlarge", "2xlarge"]  # Multiple sizes

Spot suitability:

Karpenter handles Spot interruptions automatically (receives 2-min notice, cordons, launches replacement, drains respecting PDBs). For MNG/self-managed nodes, deploy AWS Node Termination Handler.

Savings Plans & Reserved Instances

For stable, predictable baseline capacity, Compute Savings Plans provide up to 66% savings over On-Demand. Layer them with Spot for variable workloads:

• Baseline (always running): Savings Plans or Reserved Instances
• Variable (scales up/down): Spot with On-Demand fallback

Downscaling Patterns

# KEDA cron-based scaling -- scale to zero at night
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: dev-app
spec:
  scaleTargetRef:
    name: dev-app
  minReplicaCount: 0
  maxReplicaCount: 5
  triggers:
  - type: cron
    metadata:
      timezone: America/New_York
      start: "0 8 * * 1-5"    # Scale up at 8 AM weekdays
      end: "0 20 * * 1-5"     # Scale down at 8 PM weekdays
      desiredReplicas: "3"

When all pods are evicted from a node, Karpenter removes it. Combined with HPA/KEDA scaling pods to zero, nodes automatically scale to zero.

---

Networking Cost Optimization

Cross-AZ data transfer is a significant cost in multi-AZ EKS clusters. AWS charges for data crossing AZ boundaries, so keeping traffic local reduces costs.

Pod-to-Pod Traffic: Topology-Aware Routing

By default, kube-proxy distributes traffic across all pods regardless of AZ placement, causing cross-AZ charges.

Topology-aware routing (beta) allocates endpoints proportionally across zones:

apiVersion: v1
kind: Service
metadata:
  name: orders-service
  annotations:
    service.kubernetes.io/topology-mode: Auto
spec:
  selector:
    app: orders
  type: ClusterIP

Works best with evenly distributed workloads. Use with pod topology spread constraints to keep replicas balanced across zones. Hints may not be assigned when capacity fluctuates across zones (e.g., with Spot instances).

Traffic Distribution (GA in K8s 1.33) is a simpler, more predictable alternative:

apiVersion: v1
kind: Service
metadata:
  name: orders-service
spec:
  trafficDistribution: PreferClose
  selector:
    app: orders
  type: ClusterIP

PreferClose routes to same-zone endpoints first, falling back to any endpoint when none are local. Can overload endpoints in high-traffic zones -- mitigate with per-zone deployments with independent HPAs, or topology spread constraints.

Service Internal Traffic Policy restricts traffic to the originating node:

spec:
  internalTrafficPolicy: Local

Use for tightly coupled services with frequent inter-communication. Requires co-located replicas via pod affinity rules -- traffic is dropped when no local endpoint exists. Cannot be combined with topology-aware routing.

Load Balancer to Pod Communication

The AWS Load Balancer Controller supports two traffic modes:

Use IP mode to eliminate data transfer charges from LB-to-Pod traffic. Ensure the LB is deployed across all subnets in your VPC.

Network Cost Quick Reference

For detailed networking configuration, see: Networking Reference | Networking -- Ingress & DNS

---

Storage Cost Optimization

Ephemeral Storage

Instance stores are physically attached to the host -- free, but data is lost on termination. Use HostPath or the Local Persistent Volume Static Provisioner to expose them in Kubernetes.

Persistent Volumes: EBS

Start with gp3 -- 20% cheaper per GB than gp2 and allows independent IOPS/throughput scaling:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  fsType: ext4
volumeBindingMode: WaitForFirstConsumer

Migration paths from gp2 to gp3:

For mission-critical workloads needing >16K IOPS or >1 GiB/s throughput, use io2 Block Express (up to 256K IOPS, 4 GiB/s, 64 TiB).

Dynamically resize volumes as data grows rather than overprovisioning upfront. Use AWS Trusted Advisor or Popeye to find dangling/unused volumes.

Persistent Volumes: EFS

EFS charges only for stored data with no upfront provisioning. Use Intelligent-Tiering to automatically move infrequently accessed files to cheaper storage (up to 92% savings).

EFS lifecycle policies and Intelligent-Tiering must be configured outside the CSI driver (console or EFS API).

Persistent Volumes: FSx

For FSx for Lustre, link to S3 for long-term storage -- lazy-load data into Lustre for processing, write results back to S3, then delete the filesystem.

Storage Quick Reference

---

Observability Cost Optimization

Observability costs scale with data volume. Optimize by collecting only what matters and retaining intelligently.

Logging

Control plane logs: Evaluate which log types are needed per environment. Non-production clusters may only need API server logs enabled selectively. Production clusters benefit from all types for incident investigation. EKS control plane logs are classified as Vended Logs with volume discount pricing.

Metrics

Identify high-cardinality offenders:

# Top 5 scrape targets by metric count
topk_max(5, max_over_time(scrape_samples_scraped[1h]))

# Top 5 by churn rate (new series created per scrape)
topk_max(5, max_over_time(scrape_series_added[1h]))

Use Grafana Mimirtool to find metrics collected but never used in dashboards or alerts.

Traces

For high-volume services, implement sampling strategies:

• Head-based sampling: Decide at trace start (simple, but may miss important traces)
• Tail-based sampling: Decide after trace completes (captures errors/slow requests, more complex)

Use the ADOT Collector's tail sampling processor to retain only traces that exceed latency thresholds or contain errors.

---

Tagging & Cost Visibility

Tagging Strategy

Tags are the foundation of cost allocation. Without them, you can see total spend but not who's spending it.

# Karpenter EC2NodeClass tags
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  tags:
    Team: platform
    CostCenter: engineering
    Environment: production
    ManagedBy: karpenter
    kubernetes.io/cluster/my-cluster: owned

AWS resource tags don't directly correlate with Kubernetes labels. Use Kubernetes labels on pods/namespaces for in-cluster cost attribution (via Kubecost), and AWS tags for billing/Cost Explorer views.

Cost Visibility Tools

Kubecost

Kubecost provides real-time cost monitoring, namespace/label allocation, and right-sizing recommendations:

helm install kubecost kubecost/cost-analyzer \
  --namespace kubecost --create-namespace \
  --set kubecostProductConfigs.clusterName=my-cluster \
  --set kubecostProductConfigs.cloudIntegrationSecret=cloud-integration

DO:

• Use namespace-level cost allocation for multi-tenant clusters
• Enable CUR integration for accurate AWS pricing (not list price estimates)
• Set up idle cost alerts -- unused resources are the biggest waste
• Use right-sizing recommendations to adjust resource requests

DON'T:

• Rely solely on CloudWatch for K8s cost attribution -- it lacks namespace-level granularity
• Skip resource requests on pods -- Kubecost needs requests to calculate allocation
• Ignore shared costs (control plane, monitoring, ingress) -- allocate proportionally

---

Sources:

• AWS EKS Best Practices Guide -- Cost Optimization
• Kubecost Documentation