Eks Build

EKS Build

Generate complete, production-ready EKS infrastructure projects. All generated code is terraform apply-ready with zero manual fixups.

All generated Terraform code MUST follow the conventions from these companion skills:

• terraform-skill -- resource block ordering, variable conventions, file organization, version constraints

When to Use

• Generating EKS Terraform code from requirements (new project)
• Scaffolding EKS infrastructure with air-gapped, proxy, or compliance constraints
• Creating ArgoCD GitOps integration for EKS addon management
• Implementing two-phase webhook ordering for eks-blueprints-addons
• Configuring IRSA or Pod Identity for EKS workloads
• Generating ArgoCD ApplicationSets for addon lifecycle
• Creating Ralph loop validation configurations for EKS projects
• Adding custom addons (Kyverno, Prisma, New Relic) to an EKS project
• Choosing between or comparing deployment patterns (full Terraform vs ArgoCD+Terraform vs ArgoCD+ACK/KRO) for an actual build

Don't Use

• Designing EKS architecture from requirements (use eks-design)
• General Terraform module development or testing (use terraform-skill)
• EKS cluster reconnaissance or discovery (use eks-recon)
• EKS operational best practices reference (use eks-best-practices)

Internet Search Requirements

This skill generates code from scratch every time. Always search the internet for:

• Latest versions of all addons, Helm charts, and Terraform modules before generating code -- never use hardcoded versions from cached knowledge
• Latest EKS best practices when requirements don't exactly match existing patterns
• Addon-specific configuration for addons not fully covered in existing patterns
• Breaking changes in new versions of addons or Terraform modules
• Pattern 2b: KRO, ACK, and Crossplane -- these evolve rapidly. Always search for latest RGD schema syntax, supported CRDs, ProviderConfig format, and known limitations

The references in this skill cover the most common scenarios. For anything beyond that, research first, then adapt.

Pattern Decision Matrix

Key distinction: All patterns use Terraform for the EKS cluster and VPC infrastructure. The patterns differ in how K8s addons and AWS addon resources are managed after the cluster exists.

For detailed architecture, trade-offs, and selection criteria see references/pattern-guide.md.

Pattern 2a vs 2b -- The Defining Difference

The ONLY difference between 2a and 2b is WHO creates the AWS resources that addons need (S3 buckets, IAM roles, policies, Route53 zones):

Self-check: If generated code still uses Terraform for IRSA roles or S3 buckets for addons, that is Pattern 2a, NOT 2b.

Pattern 2b does NOT prescribe which controller. The choice is: ACK, Crossplane, or any K8s-native AWS controller. KRO provides orchestration on top (composing multiple resources into RGDs). Always search the internet for latest KRO docs before generating Pattern 2b code.

Workflow

Step 1: Gather Requirements

Accept requirements via interactive questionnaire or a requirements YAML file. If YAML is supplied, validate coverage (cluster, compute, networking, addons, auth, compliance, multi-tenancy) and prompt for gaps.

Step 2: Select Pattern

Based on requirements, select Pattern 1, 2a, or 2b from the decision matrix.

Step 3: Create Project Structure

projects/<project-name>/
├── design/        # Architecture docs (eks-design skill)
└── code/          # Terraform code (this skill)

Step 4: Generate Pattern Scaffold

Generate root Terraform files under projects/<project-name>/code/ following terraform-skill conventions.

• Pattern 1 -- main.tf implements two-phase module architecture (Critical Rule 1). Phase 1: LBC + Gatekeeper with wait = true. Phase 2: all remaining addons with depends_on = [module.eks_addons_webhooks].
• Pattern 2a/2b -- main.tf provisions EKS cluster and GitOps Bridge. For 2a include IRSA roles. For 2b include only controller bootstrap IAM.

Before generating, search the internet for latest stable versions of terraform-aws-modules/eks, terraform-aws-modules/vpc, aws-ia/eks-blueprints-addons, and all Helm charts. Pin exact versions. See references/version-matrix.md.

Step 5: Generate Required Modules

Under projects/<project-name>/code/modules/:

• eks-cluster/ -- always. Wraps terraform-aws-modules/eks with opinionated defaults from references/baseline-defaults.md.
• custom-addons/ -- if custom addons enabled. See references/addon-catalog.md.
• kyverno-policies/ -- if Kyverno CIS benchmark enabled.
• eks-tenants/ -- if multi-tenancy enabled.
• eks-gitops-bridge/ -- Pattern 2 only.

Step 6-7: Configure and Customize

Edit configs/*.yaml based on requirements and apply customization patches per references/customization-guide.md:

• Air-gapped, Enterprise proxy, Private registry, Compliance-strict

Step 8: Generate GitOps Artifacts (Pattern 2 only)

Generate the gitops/ tree with ArgoCD ApplicationSets. Use cluster ARN as destination server (Critical Rule 10). For OCI Helm charts, omit oci:// prefix in repoURL (Critical Rule 10b).

Step 9-10: README and Validate

Generate README and run scripts/validate_project.sh to verify structure, formatting, and critical configuration.

Critical Build Rules

Non-negotiable lessons from production deployments. Violating any causes deployment failures. See references/lessons-learned.md for full context.

1. Two-Phase Module Architecture (Pattern 1)

Split eks-blueprints-addons into TWO module calls:

• Phase 1: LBC + Gatekeeper only, wait: true
• Phase 2: All other addons, depends_on = [module.eks_addons_webhooks]

LBC and Gatekeeper register webhooks (failurePolicy: Fail) before pods are ready. Without two-phase, these webhooks block all other addon deployments.

2. before_compute: true

MANDATORY for vpc-cni and eks-pod-identity-agent. Without this, nodes fail with NodeCreationFailure: NetworkPluginNotReady.

3. Cluster-Autoscaler Version Match

Image tag MUST match EKS K8s minor version (e.g., v1.x.y for EKS 1.x).

4. LBC Requires Explicit vpcId

IMDS fallback fails with hop-limit restrictions. Always set vpcId in LBC Helm values.

5. Multus DISABLED

Thick-plugin pod-lookup race blocks ALL new pod creation. Never enable.

6. Version Pinning -- Always Search Internet

Never rely on module defaults or hardcoded versions. Search for every addon and module version before generating. See references/version-matrix.md.

7. Kyverno syncOptions (Pattern 2)

Use Replace=true, NOT ServerSideApply=true. SSA conflicts with ArgoCD selfHeal: true + --force.

8. Velero Configuration

Three-part config: S3 bucket (encrypted) + Auth (Pod Identity or IRSA) + Helm values (upgradeCRDs: false, pin kubectl.image.tag).

9. ACK/KRO Are Capabilities, Crossplane Is Not

ACK and KRO run as EKS Capabilities (no pods). Validate via CRDs, not kubectl pods. Crossplane runs as pods via Helm chart.

10. ArgoCD Destination Server (Pattern 2)

Use cluster ARN as destination server, NOT https://kubernetes.default.svc. AppProjects require sourceNamespaces: [argocd].

10b. ArgoCD OCI Helm Chart repoURL

Do NOT include oci:// prefix in repoURL. Use public.ecr.aws/karpenter with chart: karpenter.

10c. S3 Bucket Persistence in Destroy/Apply Cycles (Pattern 2b)

S3 buckets persist after terraform destroy because the controller is deleted first. Add aws s3 rb s3://<bucket> --force || true as pre-destroy step.

11. Use HashiCorp Terraform, NOT OpenTofu

OpenTofu v1.10.6 has severe depends_on performance regression (30+ min plan vs < 2 min with Terraform v1.14.5). Require HashiCorp Terraform v1.14.5+.

12. ArgoCD EKS Capability Requires IAM Identity Center

The AWS-managed ArgoCD EKS Capability requires an existing IAM Identity Center instance. Self-managed ArgoCD can use any auth method.

Customization Rules

Output Structure

Pattern 1

projects/<project-name>/code/
├── main.tf                    # Two-phase module architecture
├── locals.tf, data.tf, providers.tf, variables.tf, outputs.tf, versions.tf
├── modules/
│   ├── eks-cluster/
│   ├── custom-addons/
│   └── kyverno-policies/
├── configs/
│   ├── cluster.yaml, compute.yaml, addons.yaml, backend.hcl
└── validation-checklist.md

Pattern 2

projects/<project-name>/code/
├── main.tf                    # Cluster + GitOps Bridge
├── locals.tf, data.tf, providers.tf, variables.tf, outputs.tf, versions.tf
├── modules/
│   ├── eks-cluster/
│   ├── eks-gitops-bridge/
│   └── custom-addons/
├── configs/
│   ├── cluster.yaml, compute.yaml, addons.yaml, backend.hcl
├── gitops/
│   ├── addons/applicationset.yaml
│   ├── custom-addons/applicationset.yaml
│   ├── bootstrap/argocd-projects.yaml
│   └── tenants/applicationset.yaml
└── validation-checklist.md

Deployment

Pattern 1:

cd projects/<project-name>/code
terraform init -backend-config=configs/backend.hcl
terraform plan && terraform apply

Pattern 2:

cd projects/<project-name>/code
terraform init -backend-config=configs/backend.hcl
terraform plan && terraform apply
aws eks update-kubeconfig --name <CLUSTER_NAME> --region <REGION>
kubectl apply -f gitops/bootstrap/argocd-projects.yaml
kubectl apply -f gitops/addons/applicationset.yaml
kubectl apply -f gitops/custom-addons/applicationset.yaml

Expected Timing (HashiCorp Terraform v1.14.5+)

Completion Checklist

Every item must be done before handoff:

• [ ] Internet search for latest versions (not from cached knowledge)
• [ ] Pattern selected and confirmed
• [ ] Project structure created with pattern scaffold
• [ ] Required modules generated
• [ ] Configuration customized with project-specific values
• [ ] Customization patches applied (if applicable)
• [ ] GitOps artifacts generated (Pattern 2 only)
• [ ] terraform fmt -check passes
• [ ] before_compute: true set for vpc-cni and eks-pod-identity-agent
• [ ] Two-phase module architecture present (Pattern 1)
• [ ] All versions explicitly pinned
• [ ] Multus NOT enabled
• [ ] validation-checklist.md generated

References

Read these as needed based on the task at hand:

• Baseline Defaults -- Read when generating cluster, compute, networking, or security configuration. Covers all default values applied to every project.
• Pattern Guide -- Read when selecting between patterns or implementing pattern-specific architecture.
• Customization Guide -- Read when applying air-gapped, proxy, private registry, or compliance constraints.
• Addon Catalog -- Read when configuring specific addons or adding custom addons. Covers all 29+ supported addons.
• Lessons Learned -- Read when troubleshooting deployment failures or understanding why a Critical Build Rule exists.
• Version Matrix -- Read when looking up authoritative version sources for addons and Terraform modules.
• Checkov Config -- Read when setting up security scanning for generated Terraform code.

Addon Catalog

All addons supported by this framework, organized by category. Each addon is toggled via YAML configuration -- no Terraform code changes required.

Addon Summary Table

\* In Pattern 2, custom addons deploy via gitops/custom-addons/ directory-based ApplicationSet.

EKS Managed Addons

Deployed through the EKS API (not Helm), managed identically in both patterns. Configure under eks_addons in addons.yaml:

eks_addons:
  vpc-cni:
    most_recent: true
    before_compute: true          # MANDATORY -- prevents NodeCreationFailure
  coredns:
    most_recent: true
  kube-proxy:
    most_recent: true
  aws-ebs-csi-driver:
    most_recent: true
  aws-efs-csi-driver:
    most_recent: true
  eks-pod-identity-agent:
    most_recent: true
    before_compute: true          # MANDATORY -- for Pod Identity workloads

before_compute Requirements

Pod Identity for EKS Managed Addons

EKS managed addons (EBS CSI, EFS CSI, FSx CSI, Mountpoint S3 CSI, CloudWatch) use Pod Identity instead of IRSA. IAM roles are created by terraform-aws-modules/eks-pod-identity/aws modules. Associations are separate `aws_eks_pod_identity_association` resources -- not inline in addon config.

Why separate resources (not inline `pod_identity_association`): 1. Terraform v1.14's strict for_each rejects unknown module outputs in the upstream EKS module's filter expression 2. Separate resources create correct destroy ordering -- associations destroyed before IAM roles

# IAM role (no association -- created separately)
module "ebs_csi_pod_identity" {
  source  = "terraform-aws-modules/eks-pod-identity/aws"
  version = "<LOOK_UP>"  # Look up latest from Terraform Registry
  name            = "${local.cluster_config.name}-ebs-csi"
  use_name_prefix = false
  attach_aws_ebs_csi_policy  = true
  aws_ebs_csi_policy_name    = "${local.cluster_config.name}-EBS_CSI"  # Avoid cross-project collision
  tags = local.tags
}

# Separate association (depends on both cluster and IAM role)
resource "aws_eks_pod_identity_association" "ebs_csi" {
  cluster_name    = module.eks_cluster.cluster_name
  namespace       = "kube-system"
  service_account = "ebs-csi-controller-sa"
  role_arn        = module.ebs_csi_pod_identity[0].iam_role_arn
}

Blueprints Addons (Pattern 1)

The aws-ia/eks-blueprints-addons module deploys these as Helm releases. Each addon has an enabled flag and optional config block:

aws_load_balancer_controller:
  enabled: true
  config:
    wait: true                     # Required for Phase 1 webhook ordering
    chart_version: "<LOOK_UP>"     # Always override module default
    set:
      - name: vpcId
        value: "vpc-0abc123"       # Explicit -- IMDS fallback can fail
      - name: replicaCount
        value: "2"

Critical: Two-Phase Module Architecture

LBC and Gatekeeper must deploy in Phase 1 (with wait: true) before all other addons in Phase 2. See lessons-learned.md for details.

ArgoCD-Managed Addons (Pattern 2)

Addon enable flags pass to ArgoCD via the GitOps Bridge metadata secret. ArgoCD reads enable_<addon_name> annotations and conditionally creates Helm Applications.

Karpenter in Pattern 2 (validated): Terraform creates the IAM roles and Pod Identity associations via the dedicated submodule. The Helm chart deploys via ArgoCD Application (OCI chart: oci://public.ecr.aws/karpenter/karpenter). NodePool/EC2NodeClass CRDs can be deployed via a custom-addons chart. Discovery tags must be applied by Terraform before ArgoCD syncs.

addons:
  aws_load_balancer_controller:
    enabled: true
  metrics_server:
    enabled: true
  cluster_autoscaler:
    enabled: true

Tenant Workloads (Pattern 2)

Tenants use a Kustomize-based ApplicationSet with a Git Directory Generator that auto-discovers tenant overlay directories (gitops/tenants/*/overlays/*). Each tenant gets its own AppProject for RBAC isolation.

Custom Addons

Prisma Cloud Defender

Runtime container security agent from Palo Alto Networks.

custom_addons:
  prisma_defender:
    enabled: true
    chart: twistlock-defender
    chart_version: "<LOOK_UP>"
    repository: "https://<REGISTRY_URL>"
    namespace: twistlock

New Relic

Infrastructure monitoring bundle (K8s integration, Prometheus agent, logging).

custom_addons:
  new_relic:
    enabled: true
    chart: nri-bundle
    chart_version: "<LOOK_UP>"     # Look up from https://artifacthub.io/packages/helm/newrelic/nri-bundle
    repository: https://helm-charts.newrelic.com
    namespace: newrelic
    set:
      - name: global.licenseKey
        value: "<YOUR_LICENSE_KEY>"

Flux CD

GitOps toolkit. Installs Flux controllers; configure GitRepository and Kustomization CRDs separately.

custom_addons:
  flux:
    enabled: true
    chart: flux2
    chart_version: "<LOOK_UP>"     # Look up from https://artifacthub.io/packages/helm/fluxcd-community/flux2
    repository: https://fluxcd-community.github.io/helm-charts
    namespace: flux-system

Multus CNI -- BROKEN

Do NOT enable. Thick-plugin has a pod-lookup race condition that blocks ALL new pod creation.

custom_addons:
  multus:
    enabled: false  # BROKEN: thick-plugin pod-lookup race

AWS PCA Issuer

Bridges cert-manager with AWS Private Certificate Authority for org-trusted TLS.

custom_addons:
  pca_issuer:
    enabled: true
    chart: aws-privateca-issuer
    chart_version: "<LOOK_UP>"     # Look up from https://artifacthub.io/packages/helm/cert-manager/aws-privateca-issuer
    repository: https://cert-manager.github.io/aws-privateca-issuer
    namespace: cert-manager
    service_account_role_arn: "arn:aws:iam::<ACCOUNT_ID>:role/pca-issuer-role"
    cluster_issuer_arn: "arn:aws:acm-pca:<REGION>:<ACCOUNT_ID>:certificate-authority/<CA_ID>"
    cluster_issuer_name: pca-cluster-issuer

Adding a New Custom Addon

Pattern 1 (Terraform)

1. Add a module block in modules/custom-addons/main.tf:

module "my_addon" {
  count   = try(var.custom_addons_config.my_addon.enabled, false) ? 1 : 0
  source  = "aws-ia/eks-blueprints-addon/aws"
  version = "<LOOK_UP>"  # Look up from Terraform Registry

  chart            = try(var.custom_addons_config.my_addon.chart, "my-addon")
  chart_version    = try(var.custom_addons_config.my_addon.chart_version, "<LOOK_UP>")
  repository       = try(var.custom_addons_config.my_addon.repository, "https://charts.example.com")
  namespace        = try(var.custom_addons_config.my_addon.namespace, "my-addon")
  create_namespace = true

  values = try(var.custom_addons_config.my_addon.values, [])
  set    = try(var.custom_addons_config.my_addon.set, [])
  tags   = var.tags
}

2. Add config in addons.yaml:

custom_addons:
  my_addon:
    enabled: true
    chart: my-addon
    chart_version: "<LOOK_UP>"
    repository: "https://charts.example.com"
    namespace: my-addon

Pattern 2 (ArgoCD)

1. Create gitops/custom-addons/charts/my-addon/ with Chart.yaml + values.yaml. 2. The directory-based ApplicationSet auto-discovers and deploys it.

Baseline Defaults

Table of Contents

• Cluster
• Compute
• Networking
• Namespace Security
• Addon Resilience
• IAM
• Observability
• Upgrade Path
• Cost Optimization

Apply these to ALL generated projects regardless of environment (production, staging, development) unless explicitly overridden. Sourced from validated deployments and EKS best practices.

---

Cluster

• Authentication mode: API_AND_CONFIG_MAP (not legacy CONFIG_MAP)
• Endpoint access: Private-only for production, private+public for dev/staging
• Encryption: KMS envelope encryption for etcd secrets (mandatory for production)
• Logging: Enable all 5 log types: api, audit, authenticator, controllerManager, scheduler
• Log retention: Audit 365 days, control plane 90 days, application 30 days
• Access entries: Use EKS access entries with scoped AWS-managed policies (AmazonEKSClusterAdminPolicy, AmazonEKSAdminPolicy, AmazonEKSEditPolicy, AmazonEKSViewPolicy)

Compute

Node Groups (MNG)

• EBS volume: gp3, encrypted, 100Gi, iops 3000, throughput 125 (never gp2)
• Graviton: Suggest arm64 (m7g, c7g, r7g) when workloads support it -- 20-40% savings. Default to x86 (m6i, m7i) when uncertain
• Spot: Recommend for dev/staging. Never for production stateful workloads
• Multiple instance types: Always 3+ per node group for AZ availability
• IMDSv2: Enforce hop limit 1 in launch template -- prevents pods from accessing node IMDS credentials

# Launch template metadata options
metadata_options = {
  http_endpoint               = "enabled"
  http_tokens                 = "required"
  http_put_response_hop_limit = 1
}

# EBS block device
block_device_mappings = {
  xvda = {
    device_name = "/dev/xvda"
    ebs = {
      volume_size = 100
      volume_type = "gp3"
      iops        = 3000
      throughput  = 125
      encrypted   = true
    }
  }
}

Node IAM Role

Only attach these policies -- never application permissions:

• AmazonEKSWorkerNodePolicy
• AmazonEKS_CNI_Policy (move to IRSA/Pod Identity when possible)
• AmazonEC2ContainerRegistryReadOnly
• AmazonSSMManagedInstanceCore (for SSM access, replaces SSH)

Karpenter (When Selected)

Architecture: Use the dedicated terraform-aws-modules/eks/aws//modules/karpenter submodule (NOT eks-blueprints-addons). This submodule creates the controller IAM role with Pod Identity and the node IAM role. In Pattern 1, deploy the Helm chart via helm_release. In Pattern 2, deploy via ArgoCD Application (validated).

Prerequisites: The account must have the AWSServiceRoleForEC2Spot service-linked role for Spot instances. Create with aws iam create-service-linked-role --aws-service-name spot.amazonaws.com or via Terraform aws_iam_service_linked_role.

Key components: 1. module "karpenter" -- controller IAM role (Pod Identity) + node IAM role 2. helm_release "karpenter" -- Karpenter controller (chart: oci://public.ecr.aws/karpenter/karpenter) 3. kubectl_manifest -- NodePool + EC2NodeClass CRDs 4. aws_ec2_tag -- discovery tags on private subnets and cluster primary security group

Discovery tags: Karpenter finds subnets and security groups via karpenter.sh/discovery: <cluster-name> tags. Tag ALL private subnets and the cluster primary security group.

NodePool defaults:

apiVersion: karpenter.sh/v1
kind: NodePool
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
      - key: kubernetes.io/arch
        operator: In
        values: ["amd64", "arm64"]       # Multi-arch for cost savings
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["on-demand", "spot"]     # Karpenter handles fallback
      - key: karpenter.k8s.aws/instance-category
        operator: In
        values: ["c", "m", "r"]           # Diverse instance families
      - key: karpenter.k8s.aws/instance-generation
        operator: Gt
        values: ["5"]                     # Gen 6+ only
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
    budgets:
    - nodes: "10%"                        # Max 10% replaced at once
  limits:
    cpu: "1000"
    memory: 2000Gi
  weight: 50

EC2NodeClass defaults:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
spec:
  role: <CLUSTER_NAME>-karpenter-node    # Must match node_iam_role_name in module
  amiSelectorTerms:
  - alias: al2023@latest
  subnetSelectorTerms:
  - tags:
      karpenter.sh/discovery: <CLUSTER_NAME>
  securityGroupSelectorTerms:
  - tags:
      karpenter.sh/discovery: <CLUSTER_NAME>
  blockDeviceMappings:
  - deviceName: /dev/xvda
    ebs:
      volumeSize: 100Gi
      volumeType: gp3
      encrypted: true
      iops: 3000
      throughput: 125

Networking

VPC CNI

# vpc-cni addon configuration_values
configuration_values = jsonencode({
  env = {
    ENABLE_PREFIX_DELEGATION = "true"    # 4-16x more pods/node
    WARM_PREFIX_TARGET       = "1"
  }
  enableNetworkPolicy = "true"           # Native network policy (v1.14+)
})

Subnet Design

• Private subnets: Worker nodes, tagged kubernetes.io/role/internal-elb = 1
• Public subnets: ALB only (internet-facing), tagged kubernetes.io/role/elb = 1
• Intra subnets: EKS control plane ENIs (when available)
• All subnets: Tagged kubernetes.io/cluster/<cluster-name> = shared
• NAT Gateway: One per AZ for production (eliminates inter-AZ NAT traversal)

kube-proxy

Set via kube-proxy ConfigMap: mode: "ipvs", ipvs.scheduler: "lc"

CoreDNS Tuning

• ndots: Set to 2 in pod dnsConfig (default 5 causes 4 extra lookups per external DNS query)
• Proportional autoscaler: coresPerReplica: 256, nodesPerReplica: 16, min: 2, max: 20
• Lameduck duration: 30s (critical for Karpenter -- delays shutdown for iptables propagation)
• NodeLocal DNSCache: Deploy for clusters >100 nodes (reduces CoreDNS load 80%+)

Ingress

• ALB target type: target-type: ip (eliminates cross-AZ LB-to-pod charges)
• SSL policy: ELBSecurityPolicy-TLS13-1-2-2021-06
• Health check alignment: ALB health check path = readiness probe path, interval >= readiness probe period

Namespace Security

Pod Security Admission Labels

Apply to ALL generated namespaces (addon namespaces, tenant namespaces):

metadata:
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted

For hardened workloads, use enforce: restricted.

Default-Deny NetworkPolicy

Generate for each namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

Then add explicit allow rules for required traffic (DNS on port 53/UDP, HTTPS on 443/TCP).

Resource Quotas (Tenants)

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-quota
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    pods: "50"
    persistentvolumeclaims: "10"

Addon Resilience

Replica Count and Topology

For production environments:

Topology Spread Template

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
- maxSkew: 1
  topologyKey: kubernetes.io/hostname
  whenUnsatisfiable: ScheduleAnyway

Graceful Shutdown

For addons behind ALB:

• terminationGracePeriodSeconds: 60
• preStop: sleep 15 (allows kube-proxy and LB to deregister before SIGTERM)

HPA Defaults (When Applicable)

• minReplicas: 3 (production)
• averageUtilization: 70 (30% headroom for bursts)
• scaleUp.stabilizationWindowSeconds: 60
• scaleDown.stabilizationWindowSeconds: 300 (5 min cooldown, prevent flapping)
• Never run VPA and HPA on the same metric (VPA cpu + HPA cpu = conflict)

Health Probes

Liveness probes must NOT check external dependencies -- if the DB goes down and liveness checks it, all pods restart, causing cascading failure.

Resource Requests

• Requests: Always set for CPU and memory (scheduling guarantee)
• Limits: Always set memory (OOM protection). Usually omit CPU limits (avoid throttling)
• Memory limit = 1.5-2x memory requests

IAM

• Pod Identity is default. IRSA as fallback only
• Pod Identity associations for EKS managed addons: use separate `aws_eks_pod_identity_association` resources (not inline in addon config) for correct destroy ordering
• Node role: Only AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy, AmazonEC2ContainerRegistryReadOnly
• Access entries: Use scoped AWS-managed policies, not aws-auth ConfigMap

Observability

Container Insights

Enable via EKS addon: amazon-cloudwatch-observability

Fluent Bit

• Log group: /eks/<cluster-name>/application
• Stream prefix: pod-
• Auto-create groups: true

Alert Thresholds

GuardDuty

EKS Runtime Monitoring is auto-enabled at account level when GuardDuty is active. Not managed by Terraform. Key finding types: CryptocurrencyMining, PrivilegeEscalation, ReverseShell.

Upgrade Path

Strict sequence:

1. Control Plane -> 2. EKS Managed Add-ons -> 3. Data Plane (nodes) -> 4. Custom Add-ons (Helm)

Pre-Upgrade Checklist

1. Check EKS Cluster Insights: aws eks list-insights --cluster-name <name> 2. Scan deprecated APIs: Pluto or kube-no-trouble 3. Verify addon compatibility: aws eks describe-addon-versions --kubernetes-version <target> 4. Ensure PDBs configured (won't block node drains) 5. Back up via Velero or GitOps repo 6. Test in non-prod first

Key API Removals

Version Support

• 14 months standard support
• +12 months extended support (additional fees)
• After: Auto-upgrade to oldest supported version

Cost Optimization

Tagging Strategy

All generated resources should include:

tags = {
  Project     = var.project_name
  Environment = var.environment
  ManagedBy   = "terraform"
  Cluster     = var.cluster_name
}

Karpenter propagates tags to EC2 instances, EBS volumes automatically.

Checkov Configuration

Checkov is a static analysis tool for Terraform that detects security misconfigurations and compliance violations before deployment. This configuration file (.checkov.yaml) should be placed at the project root.

What It Validates

• Security best practices for AWS resources (encryption, public access, IAM)
• Terraform configuration correctness
• Compliance with CIS, SOC2, HIPAA, and PCI-DSS benchmarks
• Network security (security groups, NACLs, public exposure)

How to Use

Place the .checkov.yaml file at the root of your generated project (projects/<project-name>/code/.checkov.yaml). Run checkov from that directory:

cd projects/<project-name>/code
checkov -d .

The configuration uses soft-fail mode so CI pipelines report findings without blocking. Review the JUnit XML output (checkov-report.xml) for integration with CI test report widgets.

Configuration

###############################################################################
# Checkov Configuration
# https://www.checkov.io/2.Basics/CLI%20Command%20Reference.html
###############################################################################

# Scan only Terraform files
framework:
  - terraform

# Skip provider cache and module cache directories
skip-path:
  - "*/.terraform"
  - ".terraform"

# Soft-fail: report findings but exit 0 (non-blocking in CI)
soft-fail: true

# Skip CKV_TF_1 (require commit hash for module sources) -- we use Terraform
# registry modules with version pins, which is standard practice.
# CKV_TF_2 (require version tag) already covers our use case.
skip-check:
  - CKV_TF_1

# Compact output for CI logs
compact: true
quiet: true

# JUnit XML output for CI test report widget
output:
  - junitxml

output-file-path: checkov-report.xml

Skipped Checks

Adding Custom Skips

When generated code intentionally deviates from a checkov rule (e.g., a public-facing ALB that must exist), add the check ID to skip-check with a comment explaining why:

skip-check:
  - CKV_TF_1    # Registry modules use version pins
  - CKV_AWS_91  # ALB must be public-facing per requirements

Alternatively, use inline suppression in Terraform:

resource "aws_lb" "public" {
  #checkov:skip=CKV_AWS_91:ALB is intentionally public-facing
  internal = false
  # ...
}

EKS Infrastructure Customization Guide

Covers environment-specific constraints that require modifications to the baseline patterns.

---

1. Air-Gapped / VPC-Endpoint-Only Networks

Clusters with no internet egress must reach AWS services through VPC endpoints. Every image pull, Helm fetch, and API call must stay within the VPC.

Required VPC endpoints (all Interface type except S3 which is Gateway): eks, eks-auth, ecr.api, ecr.dkr, s3 (Gateway), sts, ec2, elasticloadbalancing, autoscaling, logs, ebs, ssm. All interface endpoints must have private DNS enabled and share the worker node subnets and security groups.

ECR pull-through cache -- mirror every public registry into private ECR:

aws ecr create-pull-through-cache-rule --ecr-repository-prefix ecr-public  --upstream-registry-url public.ecr.aws
aws ecr create-pull-through-cache-rule --ecr-repository-prefix docker-hub  --upstream-registry-url registry-1.docker.io
aws ecr create-pull-through-cache-rule --ecr-repository-prefix quay        --upstream-registry-url quay.io
aws ecr create-pull-through-cache-rule --ecr-repository-prefix ghcr        --upstream-registry-url ghcr.io

containerd mirror config via AL2023 nodeadm in compute.yaml:

cloudinit_pre_nodeadm:
  - content_type: application/node.eks.aws
    content: |
      [settings.container-registry.mirrors]
        [settings.container-registry.mirrors."docker.io"]
          endpoints = ["https://<ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com/docker-hub"]
        [settings.container-registry.mirrors."quay.io"]
          endpoints = ["https://<ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com/quay"]
        [settings.container-registry.mirrors."ghcr.io"]
          endpoints = ["https://<ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com/ghcr"]

Helm chart mirroring -- push OCI artifacts to ECR or host tarballs in S3 (reachable via the S3 gateway endpoint):

helm push cert-manager-<VERSION>.tgz oci://<ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com/charts

Override each addon's repository in addons.yaml. No public registry references are permitted in any Helm values, pod specs, or init containers.

---

2. Enterprise Proxy

Three layers need proxy injection: node OS, container runtime, and pods.

Node bootstrap (AL2023 launch template user data in compute.yaml):

pre_bootstrap_user_data: |
  cat <<'PROXY' > /etc/profile.d/proxy.sh
  export HTTP_PROXY="http://<PROXY_HOST>:<PROXY_PORT>"
  export HTTPS_PROXY="http://<PROXY_HOST>:<PROXY_PORT>"
  export NO_PROXY="169.254.169.254,169.254.170.2,10.0.0.0/8,.internal,.eks.amazonaws.com,.s3.amazonaws.com,.ecr.amazonaws.com"
  PROXY
  source /etc/profile.d/proxy.sh

containerd systemd override (append to pre_bootstrap_user_data):

mkdir -p /etc/systemd/system/containerd.service.d
cat <<'EOF' > /etc/systemd/system/containerd.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://<PROXY_HOST>:<PROXY_PORT>"
Environment="HTTPS_PROXY=http://<PROXY_HOST>:<PROXY_PORT>"
Environment="NO_PROXY=169.254.169.254,169.254.170.2,10.0.0.0/8,.internal,.eks.amazonaws.com,.s3.amazonaws.com,.ecr.amazonaws.com"
EOF
systemctl daemon-reload && systemctl restart containerd

NO_PROXY must include: 169.254.169.254 (IMDS), 169.254.170.2 (credential provider), 10.0.0.0/8 (VPC CIDR -- adjust as needed), .internal, .eks.amazonaws.com, .s3.amazonaws.com, .ecr.amazonaws.com, .dkr.ecr.amazonaws.com.

Proxy-aware addon config -- inject env vars via Helm set in addons.yaml:

aws_load_balancer_controller:
  config:
    set:
      - { name: env.HTTP_PROXY,  value: "http://<PROXY_HOST>:<PROXY_PORT>" }
      - { name: env.HTTPS_PROXY, value: "http://<PROXY_HOST>:<PROXY_PORT>" }
      - { name: env.NO_PROXY,    value: "169.254.169.254,169.254.170.2,10.0.0.0/8,.internal,.eks.amazonaws.com" }

Apply the same pattern to: cluster_autoscaler, external_dns, external_secrets, cert_manager, velero, and ingress_nginx.

---

3. Private Container Registry

All images must come from an approved private registry. Override every addon's image references and restrict what the cluster can pull.

Per-addon image repository overrides in addons.yaml:

aws_load_balancer_controller:
  config:
    set:
      - { name: image.repository, value: "<REGISTRY_HOST>/eks/aws-load-balancer-controller" }
cluster_autoscaler:
  config:
    set:
      - { name: image.repository, value: "<REGISTRY_HOST>/k8s/cluster-autoscaler" }
cert_manager:
  config:
    set:
      - { name: image.repository,            value: "<REGISTRY_HOST>/jetstack/cert-manager-controller" }
      - { name: webhook.image.repository,    value: "<REGISTRY_HOST>/jetstack/cert-manager-webhook" }
      - { name: cainjector.image.repository, value: "<REGISTRY_HOST>/jetstack/cert-manager-cainjector" }
external_secrets:
  config:
    set:
      - { name: image.repository, value: "<REGISTRY_HOST>/ghcr/external-secrets" }
velero:
  config:
    set:
      - { name: image.repository,        value: "<REGISTRY_HOST>/docker/velero/velero" }
      - { name: kubectl.image.repository, value: "<REGISTRY_HOST>/docker/bitnami/kubectl" }
metrics_server:
  config:
    set:
      - { name: image.repository, value: "<REGISTRY_HOST>/k8s/metrics-server" }
external_dns:
  config:
    set:
      - { name: image.repository, value: "<REGISTRY_HOST>/k8s/external-dns" }

ImagePullSecret -- create a kubernetes.io/dockerconfigjson Secret in each namespace, then inject it via Kyverno mutation policy or by patching the default ServiceAccount.

Kyverno registry restriction policy -- deploy a ClusterPolicy with validationFailureAction: Enforce that matches all Pods and validates both containers[*].image and initContainers[*].image against an allow-list: <REGISTRY_HOST>/* and <ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com/*. Reject anything else at admission.

---

4. Compliance-Strict

For FedRAMP, PCI-DSS, HIPAA -- hardening at every layer.

Required security addons (all must be enabled in addons.yaml):

custom_addons:
  kyverno:
    enabled: true
    pod_security_standard: restricted        # Not 'baseline'
    validation_failure_action: Enforce       # Not 'Audit'
gatekeeper:   { enabled: true }
cert_manager: { enabled: true }
external_secrets: { enabled: true }
cis_benchmark:
  enabled: true
  mode: Enforce                              # Reject non-compliant workloads at admission
  exclude_namespaces: [kube-system, kube-public, kube-node-lease, kyverno, gatekeeper-system]

EKS control plane hardening in cluster.yaml:

endpoint_public_access: false                # Private endpoint only
endpoint_private_access: true
kms_key_arn: "arn:aws:kms:<REGION>:<ACCOUNT_ID>:key/<KEY_ID>"
cluster_enabled_log_types: [api, audit, authenticator, controllerManager, scheduler]

CloudWatch log retention at 365 days:

resource "aws_cloudwatch_log_group" "eks_audit" {
  name              = "/aws/eks/${local.cluster_config.name}/cluster"
  retention_in_days = 365
  kms_key_id        = local.cluster_config.kms_key_arn
}

Node hardening in compute.yaml:

managed_node_groups:
  default:
    ami_type: AL2023_x86_64_STANDARD
    metadata_options:
      http_endpoint: enabled
      http_tokens: required                  # IMDSv2 enforced
      http_put_response_hop_limit: 1         # Blocks containers from IMDS
    block_device_mappings:
      xvda:
        device_name: /dev/xvda
        ebs: { volume_size: 100, volume_type: gp3, encrypted: true, kms_key_id: "<KMS_ARN>" }
    iam_role_additional_policies:
      AmazonSSMManagedInstanceCore: "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
    remote_access: {}                        # No SSH -- SSM Session Manager only

PSS Baseline enforcement -- set pss_level: baseline and default_deny: true on all platform namespaces. The default_deny flag auto-creates a NetworkPolicy that blocks all ingress and egress (podSelector: {}, policyTypes: [Ingress, Egress]). Addon-specific allow rules must be layered on top per namespace.

KMS envelope encryption: the kms_key_arn in cluster.yaml enables encryption for all Kubernetes Secrets at rest. The KMS key must grant eks.amazonaws.com in its key policy, reside in the same region, and have automatic rotation enabled.

Lessons Learned: EKS Deployment Patterns

Table of Contents

• 1. before_compute Is Mandatory
• 2. Two-Phase Module Architecture
• 3. Addon Version Pinning
• 4. Multus CNI Is BROKEN
• 5. Kyverno Gotchas
• 6. ArgoCD EKS Capability
• 7. GitOps Bridge Pattern
• 8. IRSA vs Pod Identity
• 9. Velero Configuration
• 10. ACK and KRO
• 11. Deployment Order
• 12. Use HashiCorp Terraform, NOT OpenTofu
• 13. Terraform Operational Notes
• 14. Automated Validation Findings
• 15. Pod Identity Destroy Ordering
• 16. EFS CSI Driver Operational Notes
• 17. Karpenter Dedicated Submodule
• 18. Pod Identity IAM Policy Name Collision
• 19. ArgoCD Namespace Race Condition
• 20. EKS Capability Destroy Hangs Terraform
• 21. CIS Policies Cannot Be Deployed as Helm Chart
• 22. Pattern 2a Custom-Addons Require Real Git Repo

Consolidated findings from two EKS deployment patterns:

• Pattern 1 (terraform-eks): Full Terraform with eks-blueprints-addons, two-phase module architecture
• Pattern 2 (argocd-eks): ArgoCD EKS Capability + ACK + GitOps Bridge

---

1. before_compute Is Mandatory

Patterns: Both

vpc-cni and eks-pod-identity-agent MUST have before_compute: true. Without it, EKS managed addons and node groups are created in parallel. Nodes finish before the CNI installs, fail health checks with NodeCreationFailure: NetworkPluginNotReady, and stay NotReady.

eks_addons:
  vpc-cni:
    most_recent: true
    before_compute: true
  eks-pod-identity-agent:
    most_recent: true
    before_compute: true

State migration note: Adding before_compute: true to an existing addon moves the Terraform resource key. Use terraform import for in-place migration, or do a clean destroy+apply cycle.

---

2. Two-Phase Module Architecture

Patterns: Pattern 1 only

Problem

LBC and Gatekeeper register webhooks with failurePolicy: Fail before their pods have ready endpoints. The single eks-blueprints-addons module deploys all Helm releases in parallel. Other addons hit these webhooks and get rejected.

Why wait: true alone fails

wait: true only blocks when Terraform marks that resource as done. It does NOT prevent sibling helm_release resources in the same module from starting simultaneously.

Fix

Split into two module calls: 1. Phase 1 (module.eks_addons_webhooks): LBC + Gatekeeper only, with wait: true 2. Phase 2 (module.eks_addons): Everything else, with depends_on = [module.eks_addons_webhooks]

Phase 1 also needs depends_on = [module.eks_cluster] so all EKS managed addons are ACTIVE before any Helm release starts.

Trade-off

The module graph is instantiated twice, but with HashiCorp Terraform v1.14.5, plan still completes in under 2 minutes. Do NOT use OpenTofu -- see Section 12.

---

3. Addon Version Pinning

Patterns: Both

Module defaults are stale

The eks-blueprints-addons module ships chart defaults that lag significantly behind current releases. Always look up the latest versions from authoritative sources (see version-matrix.md).

Rules

• ALWAYS override chart versions from config YAML; never rely on module defaults
• Versions pinned at plan time go stale by deployment day. Re-verify before every apply.
• cluster-autoscaler image.tag MUST match EKS K8s minor version
• LBC requires explicit vpcId -- IMDS fallback fails with hop-limit restrictions

---

4. Multus CNI Is BROKEN

Patterns: Both (currently DISABLED)

• Thick-plugin pod-lookup race: Multus queries the K8s API for pod metadata before the pod is registered. ALL new pod creation fails.
• Impact: Existing pods survive, but no new pods can start anywhere in the cluster.

Recovery (if accidentally enabled)

1. SSM to all nodes: rm -f /etc/cni/net.d/00-multus.conf 2. Delete DaemonSet: kubectl delete ds kube-multus-ds -n kube-system 3. Restart stuck pods, disable in config, re-apply

Do not enable until upstream fixes the thick-plugin race.

---

5. Kyverno Gotchas

Patterns: Both

Webhook blocks other addons

During install/upgrade, Kyverno's validating webhook has no endpoints while pods start. This blocks pod creation for other addons. If a deployment gets stuck, wait for Kyverno to stabilize, then kubectl rollout restart the stuck deployment.

TLS cert incompatibility on major version upgrades

Fix: delete ALL secrets AND all pods in the kyverno namespace simultaneously:

kubectl delete secrets --all -n kyverno && kubectl delete pods --all -n kyverno

ArgoCD sync strategy (Pattern 2)

Use Replace=true syncOption, NOT ServerSideApply=true. SSA conflicts with selfHeal: true + --force.

---

6. ArgoCD EKS Capability

Patterns: Pattern 2 only

• Fully managed external service -- no pods in the cluster.
• ArgoCD CRDs are installed automatically; ApplicationSets are applied via kubectl apply.
• Use cluster ARN as the destination server (not https://kubernetes.default.svc).
• AppProjects need sourceNamespaces: [argocd].

Go templates

• missingkey=zero is essential. Without it, any missing annotation fails the entire ApplicationSet.
• Use index .metadata.annotations "key-name" for keys with dots or hyphens.

---

7. GitOps Bridge Pattern

Patterns: Pattern 2 only

Terraform writes cluster metadata to a K8s Secret annotated with argocd.argoproj.io/secret-type: cluster. ArgoCD's clusters generator reads these annotations and exposes them as template variables in ApplicationSets.

• eks-blueprints-addons provides gitops_metadata output with all IRSA ARNs, SA names, and namespaces.
• Annotation keys use module-specific names (e.g., cluster_autoscaler_iam_role_arn). Always check the module's gitops_metadata output for exact key names.

---

8. IRSA vs Pod Identity

Patterns: Both

• Pod Identity is preferred for new workloads (simpler, better auditing).
• EKS managed addons: use Pod Identity with separate aws_eks_pod_identity_association resources -- see Section 15.
• Helm-based addons (LBC, External DNS, Velero): IRSA via eks-blueprints-addons.
• Karpenter: Pod Identity via dedicated submodule -- see Section 17.
• When both are configured for the same SA, Pod Identity takes precedence.

---

9. Velero Configuration

Patterns: Both

Three-part config: S3 bucket + auth (IRSA or Pod Identity) + Helm chart.

CRD name collision

ACK DynamoDB controller installs a Backup CRD that conflicts with Velero's. Always use the full API group: backup.velero.io.

---

10. ACK and KRO

Patterns: Both

ACK

• EKS Capability: no pods in cluster. Creates and reconciles AWS resources from K8s CRs.
• Field naming follows AWS SDK Go convention: blockPublicACLs (not blockPublicAcls). Always verify against the installed CRD schema.
• Use services.k8s.aws/deletion-policy=retain when migrating from ACK-managed to Terraform-managed resources.

KRO

• EKS Capability available but the controller may not reconcile new ResourceGraphDefinition resources in all cases. RGDs may stay Inactive.
• Workaround: Use ACK resources directly instead of KRO orchestration if RGDs are not activating.

---

11. Deployment Order

Patterns: Pattern 2

1. terraform apply          (cluster, IAM, capabilities, GitOps Bridge secret)
2. kubectl apply            argocd-projects.yaml
3. kubectl apply            addons applicationset.yaml
4. kubectl apply            custom-addons applicationset.yaml
5. kubectl apply            tenants applicationset.yaml

For Pattern 1, a single terraform apply handles everything (two-phase module architecture manages internal ordering).

---

12. Use HashiCorp Terraform, NOT OpenTofu

Patterns: Both

Problem

OpenTofu v1.10.6 has a severe performance regression with depends_on module graphs. The two-phase module architecture causes tofu plan to take 30+ minutes (vs < 2 min with Terraform v1.14.5).

Evidence

Fix

• Use HashiCorp Terraform v1.14.5+
• Verify: terraform --version must show Terraform v1.x.x, NOT OpenTofu
• Common gotcha: macOS Homebrew aliases terraform to tofu. Fix with brew install hashicorp/tap/terraform && brew link --overwrite terraform
• After switching: run terraform init -reconfigure -upgrade to re-register providers

---

13. Terraform Operational Notes

Patterns: Both

depends_on vs implicit references

module.eks_cluster.cluster_name    -> waits only for the cluster resource
depends_on = [module.eks_cluster]  -> waits for ALL resources in the module

wait: true scope

Within a single module, wait: true on one helm_release does NOT delay sibling helm_release resources. Ordering requires separate module calls with depends_on.

Deployment Timing (HashiCorp Terraform v1.14.5)

Pattern 1: Full Terraform

Pattern 2: ArgoCD + Terraform

---

14. Automated Validation Findings

Patterns: Both

GuardDuty addon auto-installed

AWS auto-installs aws-guardduty-agent when GuardDuty EKS Runtime Monitoring is enabled. Not managed by Terraform. Expect 5 or 6 managed addons depending on GuardDuty config.

ACK S3 bucket re-deploy handling (Pattern 2)

When running destroy/apply cycles, ACK S3 buckets may persist. On re-deploy, ACK enters ACK.Terminal with Resource already exists.

Fix: Delete the pre-existing bucket before re-deploy:

aws s3 rb s3://<CLUSTER_NAME>-velero-backups-<ACCOUNT_ID> --force

LBC webhook TLS CA mismatch (Pattern 2) -- ~20% recurrence rate

The LBC Helm chart uses genCA/genSignedCert. ArgoCD's partial sync can regenerate CA and cert independently, causing caBundle mismatch.

Fix: Delete stuck ArgoCD apps -- ApplicationSet recreates them:

kubectl delete application aws-load-balancer-controller gatekeeper -n argocd

external-secrets startup latency

The cert-controller and webhook pods show 0/1 Ready for ~90 seconds. This is normal startup behavior during initial certificate generation.

---

15. Pod Identity Destroy Ordering

Patterns: Pattern 1

Problem

When Pod Identity IAM roles are created by separate modules and role ARNs are passed to the EKS addon config, terraform destroy may delete IAM roles before the addons that reference them.

Fix

Create Pod Identity associations as separate `aws_eks_pod_identity_association` resources:

resource "aws_eks_pod_identity_association" "ebs_csi" {
  cluster_name    = module.eks_cluster.cluster_name
  namespace       = "kube-system"
  service_account = "ebs-csi-controller-sa"
  role_arn        = module.ebs_csi_pod_identity[0].iam_role_arn
}

This creates correct dependency graph: Create (role -> addon -> association), Destroy (association first -> role and addon in parallel).

---

16. EFS CSI Driver Operational Notes

Patterns: Both

DNS propagation delay

After creating EFS mount targets, DNS resolution takes ~5 minutes to propagate. Pods mounting EFS immediately will fail with Failed to resolve.

Security groups

EFS mount targets must allow NFS (TCP 2049) inbound from the node security group.

TLS on arm64

The efs-proxy binary does not work reliably on arm64 Karpenter-provisioned nodes. Workaround: use nodeSelector: kubernetes.io/arch: amd64 for pods that require EFS TLS.

EFS access points

Non-root containers cannot write to root-owned EFS directories. Create an EFS access point with PosixUser and CreationInfo set to the container's uid/gid.

---

17. Karpenter Dedicated Submodule

Patterns: Both (validated)

Why not eks-blueprints-addons

The eks-blueprints-addons Karpenter integration has issues: stale chart default, missing CRDs, deprecated settings keys.

Recommended approach

Use terraform-aws-modules/eks/aws//modules/karpenter:

module "karpenter" {
  source  = "terraform-aws-modules/eks/aws//modules/karpenter"
  version = "<LOOK_UP>"  # Look up latest from Terraform Registry
  cluster_name                    = module.eks_cluster.cluster_name
  create_pod_identity_association = true
  node_iam_role_use_name_prefix   = false
  node_iam_role_name              = "${local.cluster_config.name}-karpenter-node"
  node_iam_role_additional_policies = {
    AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
  }
  tags = local.tags
}

Then deploy the Helm chart via helm_release and NodePool/EC2NodeClass via kubectl_manifest. Tag private subnets and cluster primary SG with karpenter.sh/discovery: <cluster-name>.

EC2 Spot Service-Linked Role

Karpenter needs AWSServiceRoleForEC2Spot. Create before deploying:

aws iam create-service-linked-role --aws-service-name spot.amazonaws.com

---

18. Pod Identity IAM Policy Name Collision

Patterns: Both

Problem

The terraform-aws-modules/eks-pod-identity/aws module creates IAM policies with hardcoded default names. Multiple projects in the same account collide.

Fix

Always set a cluster-prefixed policy name:

module "ebs_csi_pod_identity" {
  source  = "terraform-aws-modules/eks-pod-identity/aws"
  version = "~> 2.7"
  name            = "${local.cluster_config.name}-ebs-csi"
  use_name_prefix = false
  attach_aws_ebs_csi_policy  = true
  aws_ebs_csi_policy_name    = "${local.cluster_config.name}-EBS_CSI"
  tags = local.tags
}

---

19. ArgoCD Namespace Race Condition

Patterns: Pattern 2a

Problem

The ArgoCD EKS Capability creates the argocd namespace asynchronously. If the GitOps Bridge secret creation races it, the apply fails.

Fix

Make the namespace resource conditional -- only create when ArgoCD capability is disabled:

resource "kubernetes_namespace_v1" "argocd" {
  count = local.argocd_idc_enabled ? 0 : 1
  metadata { name = local.argocd_namespace }
  depends_on = [module.eks_cluster]
}

---

20. EKS Capability Destroy Hangs Terraform

Patterns: All patterns using EKS Capabilities

Problem: The provider's delete operation for capabilities hangs indefinitely during terraform destroy.

Fix: Add a null_resource with destroy-time provisioner that deletes capabilities via the EKS API before Terraform attempts deletion. Include sleep 180 for ArgoCD capability deletion time.

Workaround: Manually delete capabilities via API, remove from state, then destroy:

terraform state list | grep capability | xargs -I{} terraform state rm {}
terraform destroy -auto-approve

---

21. CIS Policies Cannot Be Deployed as Helm Chart

Patterns: Pattern 2a, 2b

Problem: Kyverno CIS policies use JMESPath {{ }} delimiters that Helm interprets as Go templates.

Fix: Deploy via kubectl_manifest resources in the custom-addons Terraform module.

---

22. Pattern 2a Custom-Addons Require Real Git Repo

Patterns: Pattern 2a

Problem: Custom-addons ApplicationSets use ArgoCD's git directory generator, which requires a real, accessible git repository URL.

Fix: For resources needed at deploy time (Velero S3, CIS policies), create them in Terraform rather than relying on ArgoCD custom-addons.

EKS Infrastructure Pattern Comparison and Selection Guide

Table of Contents

• Pattern Overview
• Decision Matrix
• Pattern 1: Full Terraform
• Pattern 2a: ArgoCD + Terraform AWS
• Pattern 2b: ArgoCD + ACK/KRO
• Pattern 3: ArgoCD Cluster Creation (Placeholder)
• Trade-offs
• Migration Paths
• Shared Infrastructure

Pattern Overview

Decision Matrix

Use Pattern 1 when the team is Terraform-native, needs deterministic plan/apply cycles, or operates in strict change-management environments.

Use Pattern 2a when the team wants GitOps for addon lifecycle but prefers Terraform for AWS IAM resources (IRSA roles).

Use Pattern 2b when the team wants Kubernetes-native AWS resource management via ACK CRDs, reducing Terraform surface area.

Use Pattern 3 (future) when full cluster lifecycle management through ArgoCD and ACK is desired.

Pattern 1: Full Terraform

Architecture

Terraform manages the entire stack: VPC, EKS cluster, EKS managed addons, Helm-based addons (via eks-blueprints-addons), IRSA roles, and AWS resources like S3 buckets.

Two-Phase Module Architecture

Addons that register webhooks with failurePolicy: Fail must be fully running before other addons attempt to create Services or Namespaces:

• Phase 1 (`eks_addons_webhooks`): Deploys only AWS Load Balancer Controller and Gatekeeper with wait: true.
• Phase 2 (`eks_addons`): Deploys all remaining addons with depends_on on Phase 1. LBC and Gatekeeper set to enabled = false in Phase 2.

Both phases use aws-ia/eks-blueprints-addons/aws (look up latest version from Terraform Registry).

before_compute for Critical Addons

EKS managed addons vpc-cni and eks-pod-identity-agent set before_compute: true to ensure they are active before node groups launch.

Karpenter (When Selected)

Uses dedicated terraform-aws-modules/eks/aws//modules/karpenter submodule. See baseline-defaults.md for NodePool/EC2NodeClass defaults and lessons-learned.md Section 17.

AWS Resource Management

Terraform directly creates AWS resources (Velero S3 bucket, etc.). The eks-blueprints-addons module handles IRSA role creation. EKS managed addons use Pod Identity with separate aws_eks_pod_identity_association resources -- see lessons-learned.md Section 15.

Pattern 2a: ArgoCD + Terraform AWS

Architecture

Terraform creates the EKS cluster, EKS managed addons, IRSA roles, and the ArgoCD EKS Capability. ArgoCD deploys all Helm-based addons via ApplicationSets.

ArgoCD EKS Capability

ArgoCD is deployed as a fully managed AWS EKS Capability (type = "ARGOCD"), rather than a self-managed Helm release. Authentication uses AWS Identity Center (SSO) with group-based RBAC mapping.

GitOps Bridge Pattern

Terraform creates a Kubernetes secret in the argocd namespace with label argocd.argoproj.io/secret-type: cluster. This secret carries:

• Cluster identity: cluster_name, aws_region, aws_account_id, vpc_id
• IRSA role ARNs from module.eks_blueprints_addons.gitops_metadata
• Addon enable flags: enable_aws_load_balancer_controller, enable_velero, etc.
• GitOps repo coordinates: gitops_repo_url, gitops_repo_path, gitops_repo_revision

The secret server field is set to the cluster ARN (not https://kubernetes.default.svc).

IRSA via eks-blueprints-addons

The module is called with create_kubernetes_resources = false -- creates only IAM roles without deploying Helm charts. The gitops_metadata output provides all IRSA ARNs for the GitOps Bridge secret.

ApplicationSets with Go Templates

The cluster-addons ApplicationSet uses a matrix generator combining the cluster secret selector with a static addon list. Go templates with missingkey=zero inject per-addon Helm values from cluster secret annotations.

Pattern 2b: ArgoCD + ACK/KRO

Architecture

Same base as Pattern 2a, but AWS resources that would normally be Terraform-managed are instead created by ACK controllers running as EKS Capabilities. KRO provides custom resource composition.

ACK for AWS Resources

The ACK EKS Capability provisions AWS resources via Kubernetes CRDs. For example, the velero-infra custom addon chart contains an ACK Bucket resource that creates the Velero S3 backup bucket.

Pod Identity for Velero

Pattern 2b uses Pod Identity for Velero (not IRSA). Terraform creates the IAM role with pods.eks.amazonaws.com as trusted principal and a aws_eks_pod_identity_association.

KRO Capability

KRO provides Kubernetes Resource Orchestration for composing higher-level abstractions from ACK and native resources. KRO needs no IAM permissions; it operates through Kubernetes RBAC.

Pattern 3: ArgoCD Cluster Creation (Placeholder)

This pattern extends Pattern 2b so that ArgoCD manages the full cluster lifecycle through ACK. Not yet validated.

Conceptual Architecture

• A management cluster runs ArgoCD and ACK controllers
• Workload clusters are defined as ACK Cluster CRDs in the GitOps repository
• ArgoCD syncs cluster definitions, ACK provisions them
• Terraform scope is reduced to the management cluster and foundational networking only

Trade-offs

Migration Paths

Pattern 1 to Pattern 2a

1. Add ArgoCD EKS Capability module and GitOps Bridge secret to Terraform config 2. Re-run eks-blueprints-addons with create_kubernetes_resources = false 3. Create the cluster-addons ApplicationSet matching current chart versions 4. Apply Terraform to create ArgoCD capability and GitOps Bridge 5. Remove Phase 1/Phase 2 Helm deployment blocks from Terraform

Pattern 2a to Pattern 2b

1. Enable ACK and KRO EKS Capabilities 2. Create custom addon charts containing ACK CRD manifests 3. Switch from IRSA to Pod Identity for addons interacting with ACK-managed resources 4. Add custom-addons ApplicationSet to GitOps repository 5. Remove equivalent Terraform resources after ACK resources are healthy

Pattern 2b to Pattern 3 (Future)

1. Provision a management cluster with ArgoCD and ACK capabilities 2. Define EKS cluster configuration as ACK CRDs in GitOps repository 3. Create ArgoCD Application managing the cluster CRD 4. Validate cluster creation, upgrade, and deletion workflows 5. Gradually move workload clusters from Terraform-managed to ACK-managed

Shared Infrastructure

All patterns share:

• EKS cluster module: terraform-aws-modules/eks/aws with API-based access, KMS encryption, private endpoint
• EKS managed addons: vpc-cni, coredns, kube-proxy, aws-ebs-csi-driver, eks-pod-identity-agent with before_compute ordering. Pod Identity via separate association resources.
• Platform namespaces: Pod Security Standards labels + default-deny NetworkPolicy
• EKS Capabilities: ACK and KRO available in all patterns via capability submodule

Version Management

Do NOT use hardcoded versions. Always look up current versions from the authoritative sources below before generating code. Pin the verified version in the generated addons.yaml or ApplicationSet.

Version Lookup Process

Before generating a project, look up every addon and module version. Never reuse versions from a previous generation -- they go stale.

How to Look Up Versions

Use these methods in order of preference:

1. Web search (recommended) -- Search the internet for the addon name + "latest version" or "helm chart version". Examples:

• Search: aws-load-balancer-controller helm chart latest version
• Search: cert-manager latest stable release
• Search: terraform-aws-modules/eks latest version

2. ArtifactHub pages -- Fetch the ArtifactHub URL from the table below for current chart version, app version, and changelog.

3. GitHub releases -- For addons not on ArtifactHub (e.g., cluster-autoscaler), check the GitHub releases page.

4. Terraform Registry -- For Terraform modules, fetch the registry page.

5. Helm CLI (if available) -- Run helm search repo <chart> after adding the repo.

Lookup Rules

• EKS managed addons: Use most_recent: true in Terraform -- EKS auto-selects compatible versions. No manual lookup needed.
• Helm chart addons: Look up each enabled addon's latest stable chart version.
• cluster-autoscaler: Image tag MUST match the EKS K8s minor version (e.g., v1.x.y for EKS 1.x).
• Terraform modules: Check registry for latest compatible version. Use ~> constraint for minor version flexibility.

Authoritative Version Sources

Look up current versions here before every project generation:

Known Stale Defaults (eks-blueprints-addons Module)

The eks-blueprints-addons module ships chart defaults that lag behind. These addons MUST be overridden:

Addon-Specific Version Rules

cluster-autoscaler

• Image tag MUST match EKS K8s minor version (e.g., v1.x.0 for EKS 1.x)
• The eks-blueprints-addons module auto-selects correct tag for Pattern 1
• For Pattern 2, set image.tag explicitly in the ApplicationSet
• Verify via: kubectl get deploy -n kube-system cluster-autoscaler-aws-cluster-autoscaler -o jsonpath='{.spec.template.spec.containers[0].image}'

aws-load-balancer-controller

• Requires explicit vpcId in Helm values -- IMDS fallback fails with hop-limit
• Watch for major chart version bumps that change value schemas

velero

• Set upgradeCRDs: false -- bitnami/kubectl image for latest K8s may not exist
• Pin kubectl.image.tag to the latest available -- search Docker Hub for bitnami/kubectl
• Set credentials.useSecret: false when using Pod Identity

kyverno

• Override custom-addons module chart version to latest stable
• Major version upgrades may need TLS secret + pod deletion

Terraform Module Versioning

EKS Capabilities