Eks Upgrade Check

EKS Upgrade Readiness Skill

Overview

This skill assesses your live EKS cluster's readiness for a Kubernetes version upgrade. It connects to your cluster via AWS CLI and kubectl, runs automated checks across 8 assessment areas, calculates a readiness score (0-100%), and produces a detailed report with prioritized remediation steps and pre-filled AWS CLI commands.

This skill is laser-focused on upgrade safety — answering the question: "Is it safe to upgrade this cluster to the next version?"

What Gets Assessed

Readiness Score

The skill calculates a weighted readiness score:

Hard Blocker Override: If any hard blocker is detected (e.g., incompatible Karpenter, critical add-on DEGRADED, subnet IPs < 5, cluster not ACTIVE), the score is capped at ≤ 59% (NOT READY) regardless of other findings. See references/report-generation.md for the full list.

Score Interpretation:

• 90-100: READY — Safe to proceed
• 80-89: GOOD — Minor issues, can proceed with caution
• 70-79: FAIR — Several issues need attention first
• 60-69: RISKY — Significant issues, not recommended yet
• 0-59: NOT READY — Critical blockers, must resolve first

Prerequisites

1. AWS credentials configured — aws configure or ~/.aws/credentials with EKS access 2. kubectl access to the target cluster (for Kubernetes API queries) 3. Required AWS Permissions:

• eks:DescribeCluster, eks:ListClusters, eks:ListNodegroups, eks:DescribeNodegroup
• eks:ListAddons, eks:DescribeAddon, eks:ListInsights, eks:DescribeInsight
• ec2:DescribeSubnets
• iam:GetRole, iam:ListAttachedRolePolicies, iam:ListRolePolicies, iam:GetRolePolicy

MCP Server Setup

This skill works without any MCP server — it falls back to AWS CLI and kubectl commands. That fallback path is the default in apex.

For richer EKS operations (live cluster reads, upgrade insights, K8s resource introspection), enable the EKS MCP server via the apex eks-mcp-server skill — it walks you through both AWS-hosted and self-hosted setup options. Once configured, this skill will prefer MCP tools over CLI for EKS operations.

Note: Apex does NOT ship a project-root .mcp.json. MCP setup is opt-in and user-driven through the eks-mcp-server skill.

Configuration

The skill uses your existing AWS credentials. No additional configuration needed if aws eks list-clusters works from your terminal.

To use a specific profile or region, set environment variables:

export AWS_PROFILE=your-profile-name
export AWS_REGION=your-region

Getting Started

Invoke the skill: /eks-upgrade-check

Or simply ask: "Run an EKS upgrade readiness assessment"

The skill will discover your clusters, ask which one to assess and what target version, then run the full assessment.

---

Assessment Workflow

Step 0: Pre-flight

Action 1 — List clusters (test connectivity & discover clusters)

Run aws eks list-clusters to discover available clusters.

• ✅ Success → Show the cluster list. Ask which cluster to assess. If only one cluster, confirm it.
• ❌ Failure → STOP. Do NOT retry more than once. Show:

Cannot access EKS clusters. Try these steps:

1. Check that AWS credentials are configured: aws sts get-caller-identity

2. Check your region: aws eks list-clusters --region <region>

3. Check that MCP servers are enabled in Claude Code

Wait for the user to resolve the issue.

Action 2 — Describe the selected cluster

Run aws eks describe-cluster --name <cluster> and show: cluster name, Kubernetes version, platform version, region, status, account ID.

Action 2b — Validate cluster status

Check the status field from the cluster description. If status is NOT ACTIVE:

• CREATING/UPDATING/DELETING → STOP. Show: "Cluster is currently in <status> state. The EKS API will reject an upgrade request. Wait for the operation to complete, then re-run this assessment."
• FAILED → STOP. Show: "Cluster is in FAILED state. This is a hard blocker — the cluster must be recovered before an upgrade can be attempted. Contact AWS Support if the cluster is stuck in FAILED."

Do NOT proceed with the assessment if cluster status is not ACTIVE. This is a hard blocker (see report-generation.md).

Action 3 — Validate permissions

After describing the cluster, verify key permissions by attempting: 1. aws eks list-nodegroups --cluster-name <cluster> 2. aws eks list-addons --cluster-name <cluster> 3. aws eks list-insights --cluster-name <cluster>

If any fail with AccessDenied, show the user exactly which permission is missing and list the required IAM actions. Do NOT proceed until permissions are confirmed.

Action 4 — Determine target version

Ask: "Your cluster is on v[current]. The next version is v[current+1]. Shall I assess upgrade readiness to v[current+1]?"

If the user specifies a version more than 1 minor version ahead, explain that EKS requires one-version-at-a-time upgrades and show the required path (e.g., 1.29 → 1.30 → 1.31 → 1.32). Offer to assess the first hop.

Action 5 — Confirm and proceed

Steps 1-8: Run Assessment

Read each steering file in order from ${CLAUDE_SKILL_DIR}/references/. For each section: 1. Read the steering file 2. Execute the checks described in it using AWS CLI and kubectl commands 3. Collect findings with severity ratings

Steering file loading guide:

Step 9: Calculate Score & Generate Report

Read ${CLAUDE_SKILL_DIR}/references/report-generation.md and produce the report.

---

Tool Usage Rules

1. Do NOT call any tools when this skill is first activated. Wait for the user to ask. 2. Do NOT hardcode or guess cluster names. Always discover by listing first. 3. Do NOT retry a failed command more than once. 4. Always read the relevant steering file before executing checks for that section. 5. Use `aws` CLI and `kubectl` for cluster queries. If MCP servers are available, prefer them for EKS operations.

Data Files

• OSS Add-on Registry: ${CLAUDE_SKILL_DIR}/data/oss_addon_registry.json — identifiers and authoritative upstream URLs for common OSS add-ons. This file does NOT contain compatibility data. Compatibility is always verified live via the registry's compatibility_url and releases_url fields. If an add-on is not in the registry or the upstream source is unreachable, report UNKNOWN — never guess.
• HTML Converter: ${CLAUDE_SKILL_DIR}/tools/md_to_html.py — converts markdown reports to HTML

Report Output

• Markdown: EKS-Upgrade-Assessment-<cluster>-<version>-<YYYY-MM-DD>-<HHMM>.md
• HTML: Run python3 ${CLAUDE_SKILL_DIR}/tools/md_to_html.py <report>.md to convert

Do NOT generate HTML manually. Always use the conversion script.

Add-on Compatibility

Purpose

Assess all EKS managed add-ons, discovered OSS add-ons, and Karpenter for compatibility with the target Kubernetes version.

Checks to Execute

4.1 — Core EKS Managed Add-ons

The 4 core add-ons that MUST be checked:

• vpc-cni (Amazon VPC CNI)
• coredns
• kube-proxy
• aws-ebs-csi-driver (if installed)

How to check: 1. List all EKS managed add-ons → describe each for version, status, health 2. For each core add-on:

• Record installed version
• Check health status and any issues
• Note if it's self-managed (not in the managed add-on list but running in kube-system)

Key talking point: EKS does NOT auto-update add-ons when you upgrade the control plane. This is the #1 thing customers forget. A cluster upgraded to 1.33 can still be running vpc-cni from 1.29.

Rating per add-on:

• Compatible + healthy → PASS
• Behind but compatible → WARN (update recommended)
• Incompatible or unhealthy → FAIL
• Self-managed (not EKS managed) → WARN (recommend converting to managed)

4.2 — Additional Managed Add-ons

Check any other installed managed add-ons:

• amazon-cloudwatch-observability
• aws-efs-csi-driver
• adot (AWS Distro for OpenTelemetry)
• eks-pod-identity-agent
• aws-guardduty-agent
• eks-node-monitoring-agent
• snapshot-controller

How to check: 1. List all add-ons → describe each 2. Record version, status, health for each

4.3 — OSS Add-on Discovery & Compatibility Verification

Scan workloads to discover non-AWS add-ons running in the cluster, then verify their compatibility with the target Kubernetes version via web search.

Step 1: Discover OSS add-ons 1. List Deployments, DaemonSets, StatefulSets across all namespaces 2. For each workload, extract add-on identity from (in priority order):

• Labels: app.kubernetes.io/name, app.kubernetes.io/version
• Helm labels: helm.sh/chart, app.kubernetes.io/managed-by
• Container image repo + tag (e.g., quay.io/jetstack/cert-manager-controller:v1.15.0)

3. Exclude AWS-managed add-ons (vpc-cni, coredns, kube-proxy, ebs-csi) and Karpenter (checked separately) 4. Exclude workloads in these system namespaces (treat as user apps, not add-ons only if they clearly match a known add-on identifier): kube-system is included for add-on scan; default and application namespaces are EXCLUDED unless the workload matches a known add-on identifier in the registry. 5. For each workload examined, classify it into exactly one bucket:

6. For each IDENTIFIED add-on, record: name, version, namespace, identification method (which label/image matched). 7. For each UNIDENTIFIED ADD-ON, record: workload kind, name, namespace, image(s), any labels present, and why it couldn't be identified.

Common OSS add-ons to look for:

• cert-manager
• external-dns
• metrics-server
• cluster-autoscaler
• aws-load-balancer-controller
• ingress-nginx (retired March 2026)
• istio / envoy
• prometheus / grafana
• argocd / flux

Step 2: Verify compatibility via upstream sources (MANDATORY for each discovered OSS add-on)

Compatibility data is NEVER read from a local file. OSS projects ship faster than any shipped data file can keep up with, and stale data produces unsafe upgrade advice. Always fetch compatibility information live from the upstream project.

Lookup order (stop at the first that yields a definitive answer):

1. Check the local registry for the authoritative URL Read ${CLAUDE_SKILL_DIR}/data/oss_addon_registry.json. If the add-on is listed, use its compatibility_url (primary) and releases_url (fallback). The registry contains identifiers and URLs — it does NOT contain compatibility data itself.

2. Fetch the compatibility page Use webFetch on the registry's compatibility_url. Look for a supported-versions table or statement that covers both the installed add-on version and the target Kubernetes version.

3. Fetch release notes if no compatibility page exists Use webFetch on releases_url and inspect the relevant release for "Kubernetes compatibility" or "breaking changes" sections.

4. Fall back to web search only if the above fail Use remote_web_search with queries like:

• "<addon-name> <addon-version> supported Kubernetes versions"
• "<addon-name> compatibility matrix"

Prefer results from the project's own domain or GitHub org.

5. If the add-on is not in the registry Search with remote_web_search first to identify the authoritative source (project docs or GitHub releases), then apply steps 2–3 against that source.

If no authoritative source can be reached or the answer is ambiguous: Report the add-on as "compatibility UNKNOWN — manual verification required" with MEDIUM severity and include the URL(s) consulted. Do NOT assume compatibility. Do NOT fall back to LLM training data — it is likely outdated.

Verdict states (use exactly one per add-on):

Every discovered add-on MUST end with exactly one of these verdicts. "Probably fine" is not an allowed outcome.

Unidentified Workloads

Workloads classified as UNIDENTIFIED ADD-ON in Step 1 are a distinct concern from UNKNOWN_VERIFIABLE add-ons. The skill found something add-on-shaped but cannot name it, which means the user likely knows what it is and the skill does not.

For each unidentified workload, collect:

• Workload kind (Deployment/DaemonSet/StatefulSet) and name
• Namespace
• Container image(s) including registry, repo, and tag
• All present labels (to help the user recognize it)
• Replica count

Report these in a dedicated "Unidentified Workloads" table in the report. Do NOT silently drop them. The user needs to know which of their workloads the skill could not assess, so they can provide context or verify compatibility manually.

Do NOT guess the identity from image names alone if the match is ambiguous. For example, myregistry.internal/platform/controller:v2.1 is unidentified — not "probably a custom controller, assumed compatible". Ambiguity is reported, not resolved.

Registry notes field: Some add-ons in the registry have a notes field flagging special handling (e.g., ingress-nginx retirement, cluster-autoscaler K8s version pinning). Always read and apply these notes.

Output per OSS add-on:

| Add-on | Version | Verdict | Source URL | Notes |

The Verdict column uses the exact states defined above (COMPATIBLE, UPDATE_RECOMMENDED, INCOMPATIBLE, UNKNOWN_VERIFIABLE, UNKNOWN_UNIDENTIFIED). The Source URL column is mandatory — it shows the user exactly where the verdict came from (or which URL failed to load) and lets them verify it.

Output for unidentified workloads:

| Kind | Name | Namespace | Image | Labels | Why unidentified |

4.4 — Karpenter Compatibility

How to check: 1. List Deployments in the karpenter namespace, or check for NodePool CRDs (nodepools.karpenter.sh) 2. If installed, find the Karpenter deployment → extract version from:

• Labels: app.kubernetes.io/version or helm.sh/chart
• Container image tag (e.g., public.ecr.aws/karpenter/controller:0.37.0)

3. Check compatibility against the official matrix from https://karpenter.sh/docs/upgrading/compatibility/:

Official Karpenter Compatibility Matrix (source: karpenter.sh):

IMPORTANT: Do NOT rely on the approximate ranges previously listed here. Always use the matrix above. Note the jump from 0.37 (for 1.30) to 1.0.5 (for 1.31) — this is a major version boundary that requires API migration (v1beta1 → v1).

If the target Kubernetes version or installed Karpenter version is NOT in the matrix above: You MUST perform a web search to verify compatibility: 1. Search: "Karpenter compatibility matrix Kubernetes <target-version>" using remote_web_search 2. Fetch the official page: https://karpenter.sh/docs/upgrading/compatibility/ using webFetch 3. Do NOT guess or assume compatibility. Report as UNKNOWN if you can't verify.

Rating:

• Compatible version per matrix → PASS
• Installed but version unknown → WARN (manual review)
• Incompatible version per matrix → FAIL (must upgrade Karpenter BEFORE control plane)

Key talking point: Karpenter must be upgraded BEFORE the control plane, not after. The order matters. The 0.x → 1.x migration requires migrating from Provisioner to NodePool v1 APIs. See https://karpenter.sh/v1.0/upgrading/v1-migration/

Score Impact

Canonical scoring is defined in `references/report-generation.md` §Category 4 (Add-on Compatibility) and §Category 5 (Karpenter).

Breaking Changes Detection

Purpose

Identify version-specific breaking changes that affect ACTUAL resources in the cluster. Only flag a breaking change if the cluster has resources that will be impacted.

Principle

Every breaking change entry must be written in consultant-advisory style:

• What we found in YOUR cluster and why it matters
• Real-world impact if not addressed before upgrade
• Concrete remediation with commands where applicable

Do NOT list generic Kubernetes release notes. Only report changes that affect resources actually present in the cluster.

Version-Specific Breaking Changes

Target >= 1.25: PodSecurityPolicy Removed

Check: List PodSecurityPolicy resources via Kubernetes API

• If PSPs exist → HIGH severity. PSPs will cease to exist after upgrade.
• Remediation: Migrate to Pod Security Standards (PSS) by labeling namespaces: kubectl label namespace <ns> pod-security.kubernetes.io/enforce=restricted

Target >= 1.29: FlowSchema API v1beta2 Removed

Check: Scan cluster resources for apiVersion: flowcontrol.apiserver.k8s.io/v1beta2

• Look at FlowSchema and PriorityLevelConfiguration resources
• If found → MEDIUM severity. Update to flowcontrol.apiserver.k8s.io/v1

Target >= 1.32: FlowSchema API v1beta3 Removed

Check: Scan for apiVersion: flowcontrol.apiserver.k8s.io/v1beta3

• If found → HIGH severity. Update to flowcontrol.apiserver.k8s.io/v1

Target >= 1.32: Anonymous Auth Restricted

Always flag (MEDIUM severity) — affects all clusters upgrading to 1.32+.

• Anonymous requests only allowed to /healthz, /livez, /readyz
• Check: kubectl get clusterrolebindings -o json | jq '.items[] | select(.subjects[]?.name=="system:unauthenticated")'
• Impact: Monitoring tools or LB health checks hitting non-health endpoints will get 401

Target >= 1.33: Endpoints API Deprecated

Check: List Endpoints resources (exclude the default kubernetes endpoint)

• If custom Endpoints exist → MEDIUM severity
• Remediation: Migrate to EndpointSlices API (discovery.k8s.io/v1)

Target >= 1.33: AL2 AMI Not Available

Check: List nodes → inspect status.nodeInfo.kernelVersion for amzn2 or osImage for Amazon Linux 2

• If AL2 nodes found → HIGH severity. Cannot create new AL2 node groups for 1.33+
• Remediation: Migrate to AL2023 or Bottlerocket BEFORE upgrading control plane

Target >= 1.34: AppArmor Deprecated

Check: Scan deployments/daemonsets/statefulsets for AppArmor annotations in pod template

• If found → MEDIUM severity
• Remediation: Migrate to seccomp profiles

Target >= 1.35: Cgroup v1 Support Removed

Always flag (HIGH severity) for 1.35 targets.

• kubelet refuses to start on cgroup v1 nodes unless failCgroupV1=false
• AL2 uses cgroup v1 by default; AL2023 and Bottlerocket use cgroup v2
• Check node OS to determine impact

Target >= 1.35: Containerd 1.x End of Support

Check: List nodes → inspect status.nodeInfo.containerRuntimeVersion

• If any node shows containerd 1.x → MEDIUM severity
• Last release supporting containerd 1.x; next version requires 2.0+

Target >= 1.35: Ingress NGINX Retired

Check: List deployments/daemonsets with ingress-nginx or nginx-ingress in name

• If found → HIGH severity. No more security patches.
• Remediation: Migrate to Gateway API or AWS Load Balancer Controller

Target >= 1.35: IPVS Proxy Mode Deprecated

Check: Read kube-proxy ConfigMap → check mode field

• If mode: ipvs → MEDIUM severity. Removal planned for 1.36.
• Remediation: Switch to iptables or nftables mode

Target >= 1.35: --pod-infra-container-image Flag Removed

Always flag (LOW severity) for 1.35 targets.

• Affects custom AMIs with this kubelet flag in bootstrap scripts
• EKS-managed AMIs are not affected

Target > 1.35: Live Lookup Required

This file does not cover breaking changes for versions beyond 1.35. If the target version is > 1.35, you MUST perform a live lookup before reporting "no breaking changes found."

How to check: 1. Search AWS docs: search_documentation for "EKS Kubernetes <target> breaking changes" 2. Search AWS docs: search_documentation for "Kubernetes <target> removed APIs" 3. Fetch the Kubernetes changelog: read_documentation on the K8s CHANGELOG for the target minor version (e.g., CHANGELOG-1.36.md) 4. Check for EKS-specific changes: search_documentation for "EKS <target> release notes"

If no breaking changes are found after live lookup: Report "No breaking changes identified for <target> based on available documentation" with a note that the user should re-check closer to their upgrade date as documentation may be updated.

If live sources are unreachable: Report "Breaking changes for <target> could not be verified — AWS documentation unavailable" with MEDIUM severity. Do NOT assume no breaking changes exist.

Score Impact

Canonical scoring is defined in `references/report-generation.md` §Category 1 (Breaking Changes).

Deprecated API Detection

Purpose

Scan live cluster resources for usage of deprecated or removed Kubernetes APIs that will break during or after the upgrade.

How to Check

Step 1: Get EKS Upgrade Insights

Use the EKS Insights API with category UPGRADE_READINESS — this is the most reliable source for deprecated API detection as AWS scans the audit logs.

1. Get EKS Insights → filter for UPGRADE_READINESS 2. For any non-PASSING insights → get detailed description 3. Record: insight status, affected resources, recommended action

Step 2: Scan Live Resources

Run two scans in parallel for each resource type. Both are required because they catch different failure modes.

Resource types to scan:

• Deployments, DaemonSets, StatefulSets, ReplicaSets
• CronJobs, Jobs
• Ingresses
• NetworkPolicies
• PodDisruptionBudgets
• HorizontalPodAutoscalers
• CustomResourceDefinitions
• ValidatingWebhookConfigurations, MutatingWebhookConfigurations
• FlowSchemas, PriorityLevelConfigurations

Step 2a: Object apiVersion scan

For each resource type, list resources and check the live apiVersion field against the deprecation table in Step 3.

Step 2b: managedFields apiVersion scan

For each resource, inspect every entry in metadata.managedFields[] and check its apiVersion against the deprecation table in Step 3. The API server may auto-convert resources to the storage version, so Step 2a alone misses manifests originally applied under a deprecated apiVersion. managedFields preserves the apiVersion used by every writer (kubectl, controllers, Argo CD, Flux, Helm), so this scan covers all configuration sources.

kubectl get <kind> --all-namespaces -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\t"}{range .metadata.managedFields[*]}{.manager}{"="}{.apiVersion}{","}{end}{"\n"}{end}'

Output is namespace/name<TAB>manager1=apiVersion1,manager2=apiVersion2,.... The manager portion identifies which writer used each apiVersion (e.g., kubectl-client-side-apply, argocd-application-controller, controller names) — this points to where the source manifest needs to be updated.

Anti-pattern — do not pre-filter with naïve substring greps.

# WRONG — `v1` is a prefix of `v1beta3`, so `grep -v` strips both lines.
... | grep -v "flowcontrol.apiserver.k8s.io/v1"

A single resource often has multiple manager=apiVersion entries on the same line (e.g., a controller writing v1 plus the user writing v1beta3). Filter-then-decide pipelines drop the line entirely as soon as any benign apiVersion matches. Walk the full output line by line and check each manager=apiVersion pair against the deprecation table in Step 3 instead.

Anti-pattern — do not substitute `-o yaml` or `-o json`.

# WRONG — kubectl 1.21+ hides managedFields from -o yaml / -o json by default,
# so this scan returns false negatives.
kubectl get <kind> -A -o yaml | grep apiVersion

Use the -o jsonpath form above. It accesses managedFields directly and is not affected by the default-hide behavior.

Step 3: Check for Removed APIs by Target Version

Step 4: Classify Findings

For each deprecated API found, record the source (object from Step 2a / managedFields from Step 2b) and severity:

• Removed in target version → HIGH severity, action required
• Deprecated but still available in target → LOW severity, plan migration
• Removed in future version → INFO, awareness only

If a single resource is flagged by both Step 2a and Step 2b, report it once with source: object+managedFields. Counting at the API-path level (not the resource level) is canonical — see references/report-generation.md Category 2.

Output Format

For each finding, report:

• API version and kind
• Resource name and namespace
• Source (object / managedFields / object+managedFields)
• Whether it's removed in the target version or just deprecated
• Specific migration command (e.g., update apiVersion field, re-apply manifests

with the new apiVersion)

Score Impact

Canonical scoring is defined in `references/report-generation.md` §Category 2 (Deprecated APIs).

Node Readiness

Purpose

Assess node groups, AMI types, version alignment, and migration requirements for the target version.

Checks to Execute

5.1 — Node Group Inventory

How to check: 1. List all managed node groups → describe each for:

• Kubernetes version
• AMI type (AL2, AL2023, AL2_ARM_64, BOTTLEROCKET_x86_64, etc.)
• Instance types
• Scaling config (min/max/desired)
• Capacity type (ON_DEMAND, SPOT)
• Health status

2. List nodes via Kubernetes API → get:

• status.nodeInfo.kubeletVersion
• status.nodeInfo.osImage
• status.nodeInfo.kernelVersion
• status.nodeInfo.containerRuntimeVersion
• Labels: topology.kubernetes.io/zone, node.kubernetes.io/instance-type

3. Check for Karpenter NodePools (nodepools.karpenter.sh) 4. Check for EKS Auto Mode (computeConfig in cluster describe)

Output per node group:

• Name, version, AMI type, instance types, scaling config
• Version skew against target (calculated in version-validation)

5.2 — AL2 to AL2023 Migration Assessment

Why this matters:

• AL2 standard support ended June 2025
• EKS 1.33+ does NOT publish AL2 AMIs — cannot create new AL2 node groups
• AL2 uses cgroup v1; AL2023 uses cgroup v2 (required for EKS 1.35+)

How to check: 1. From node group descriptions, identify AMI type 2. From node Kubernetes API, check kernelVersion for amzn2 or osImage for Amazon Linux 2 3. Count AL2 nodes and node groups

Rating:

• No AL2 nodes → PASS
• AL2 nodes present, target < 1.33 → WARN (plan migration)
• AL2 nodes present, target >= 1.33 → FAIL (blocker — no AL2 AMI available)

Migration guidance: 1. Create new node group with AL2023 AMI type 2. Cordon old AL2 nodes: kubectl cordon <node-name> 3. Drain workloads: kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data 4. Delete old node group after all pods rescheduled 5. Key differences: cgroup v2 default, dnf instead of yum, different kernel

5.3 — Container Runtime Version

How to check: 1. List nodes → status.nodeInfo.containerRuntimeVersion 2. Check for containerd 1.x vs 2.x

Rating:

• All nodes on containerd 2.x → PASS
• Any node on containerd 1.x, target < 1.35 → WARN (plan upgrade)
• Any node on containerd 1.x, target >= 1.35 → WARN (last supported version, next will block)

5.4 — Self-Managed Nodes

How to check: 1. List all nodes 2. Compare against managed node group nodes (by labels or node group membership) 3. Nodes not in any managed node group or Karpenter → self-managed

Rating:

• No self-managed nodes → PASS
• Self-managed nodes present → WARN (no automated upgrade path, manual AMI update required)

5.5 — Subnet IP Capacity

Why this matters:

• EKS requires at least 5 available IPs in each cluster subnet to update the control plane

(EKS creates new ENIs for the upgraded API server). If any subnet has < 5 IPs, the update-cluster-version API call will fail immediately.

• During node group rolling updates, new nodes are launched before old nodes are terminated

(surge). Each new node consumes 1 IP for its primary ENI plus additional IPs for the VPC CNI warm pool (pod IPs). Insufficient capacity causes the node group update to hang.

How to check: 1. Get the cluster subnet IDs from the cluster description (already retrieved in pre-flight Action 2 — resourcesVpcConfig.subnetIds). 2. Run:

   aws ec2 describe-subnets --subnet-ids <subnet-id-1> <subnet-id-2> ... \
     --query 'Subnets[].{SubnetId:SubnetId,AZ:AvailabilityZone,AvailableIPs:AvailableIpAddressCount,CIDR:CidrBlock}' \
     --output table

3. For each subnet, evaluate AvailableIpAddressCount against thresholds.

Thresholds:

Important context for the 5–15 warning: The exact number of IPs needed during node group surge depends on:

• Instance type (determines max ENIs and IPs per ENI)
• VPC CNI configuration (WARM_IP_TARGET, MINIMUM_IP_TARGET, ENABLE_PREFIX_DELEGATION)
• Node group maxSurge setting (default: 1 additional node)

Do NOT report a precise "you need X IPs" number — instead flag the risk and advise the user to verify capacity is sufficient for their instance type and CNI config.

If subnet has < 5 IPs, report:

❌ Subnet IP exhaustion — control plane upgrade will fail

>

Subnet <subnet-id> in <az> has only <N> available IPs (CIDR: <cidr>).

EKS requires at least 5 free IPs per subnet to place control plane ENIs during an upgrade.

>

Remediation (choose one):

1. Remove unused ENIs: aws ec2 describe-network-interfaces --filters Name=subnet-id,Values=<subnet-id> Name=status,Values=available --query 'NetworkInterfaces[].NetworkInterfaceId'

2. Add a new subnet to the cluster: aws eks update-cluster-config --name <cluster> --resources-vpc-config subnetIds=<existing>,<new-subnet>

3. Expand the subnet CIDR (if VPC allows)

If subnet has 5–15 IPs, report:

⚠️ Low subnet IP capacity — node group upgrade may stall

>

Subnet <subnet-id> in <az> has <N> available IPs. While this is sufficient for the

control plane upgrade (minimum 5), the node group rolling update launches new nodes before

terminating old ones. If your instance type + VPC CNI warm pool requires more IPs than are

available, the surge node will fail to launch.

>

Before upgrading: Verify capacity is sufficient for your configuration, or consider

adding subnets / enabling VPC CNI prefix delegation to reduce per-pod IP consumption.

Score Impact

Canonical scoring is defined in `references/report-generation.md` §Category 3 (Node Readiness) and §Category 8 (AL2 Nodes).

Report Generation

Purpose

After all assessment checks are complete, calculate the readiness score and generate the upgrade assessment report.

Step 1: Calculate Readiness Score

You MUST follow this algorithm exactly. Do NOT interpret loosely. Every rule below is deterministic.

1.1 — Scoring Algorithm (Pseudocode)

score = 100

# --- Category 1: Breaking Changes (max deduction: 25) ---
# COUNTING UNIT: each distinct breaking change TYPE that affects at least one resource.
# Example: "FlowSchema v1beta2 removed" = 1 item (even if 17 FlowSchema resources use it).
# Example: "PSP removed" = 1 item (even if 5 PSPs exist).
breaking_changes_deduction = 0
for each breaking_change_type found in cluster:
    if severity == HIGH:   breaking_changes_deduction += 10
    if severity == MEDIUM: breaking_changes_deduction += 4
    if severity == LOW:    breaking_changes_deduction += 2
breaking_changes_deduction = min(breaking_changes_deduction, 25)

# --- Category 2: Deprecated APIs (max deduction: 20) ---
# COUNTING UNIT: each distinct API path (e.g., flowschemas and prioritylevelconfigurations
# are 2 separate API paths even though they share the same API group).
# Count API paths, NOT individual resources using that path.
#
# An API path is "found in cluster" if surfaced by EITHER Step 2a (live object
# apiVersion) OR Step 2b (any entry in metadata.managedFields[].apiVersion) in
# references/deprecated-apis.md. A path is counted ONCE regardless of step.
deprecated_apis_deduction = 0
for each deprecated_api_path found in cluster:
    if removed_in_target_version:    deprecated_apis_deduction += 5
    if deprecated_but_still_served:  deprecated_apis_deduction += 1
deprecated_apis_deduction = min(deprecated_apis_deduction, 20)

# --- Category 3: Node Readiness (max deduction: 20) ---
# Includes version skew AND subnet IP capacity.
# COUNTING UNIT: each node group (skew) + each subnet (IP check).
node_skew_deduction = 0
for each node_group:
    skew = target_minor_version - node_group_minor_version
    if skew > 2:  node_skew_deduction += 20   # blocker — immediately caps
    if skew == 2: node_skew_deduction += 5
for each subnet in cluster_subnets:
    if subnet.available_ips < 5:   node_skew_deduction += 5   # hard blocker
    elif subnet.available_ips <= 15: node_skew_deduction += 2  # warning
node_skew_deduction = min(node_skew_deduction, 20)

# --- Category 4: Add-on Compatibility (max deduction: 15) ---
# COUNTING UNIT: each add-on (by name) + each unidentified workload.
# CLASSIFICATION RULES:
#   - "critical add-on" = vpc-cni, coredns, kube-proxy, aws-ebs-csi-driver
#   - "optional add-on" = all other managed add-ons and identified OSS add-ons
#   - Status DEGRADED or FAILED with correct version = treat as critical/optional
#     incompatible (same deduction as version incompatibility)
#   - Status ACTIVE but version behind = "update recommended"
#   - UNKNOWN_VERIFIABLE = identified but upstream compat source unreachable/ambiguous
#   - UNKNOWN_UNIDENTIFIED = workload looks like an add-on but couldn't be identified
addon_deduction = 0
for each addon:
    if addon.verdict == "INCOMPATIBLE" or addon.status in [DEGRADED, FAILED]:
        if addon.name in [vpc-cni, coredns, kube-proxy, aws-ebs-csi-driver]:
            addon_deduction += 5   # critical add-on
        else:
            addon_deduction += 3   # optional add-on
    elif addon.verdict == "UNKNOWN_VERIFIABLE":
        addon_deduction += 2       # identified, compatibility unverified
    elif addon.verdict == "UPDATE_RECOMMENDED":
        addon_deduction += 1       # version behind but compatible
for each unidentified_workload:
    addon_deduction += 2           # UNKNOWN_UNIDENTIFIED
addon_deduction = min(addon_deduction, 15)

# --- Category 5: Karpenter (max deduction: 10) ---
# COUNTING UNIT: binary — installed and incompatible, or not.
karpenter_deduction = 0
if karpenter_installed and karpenter_version_incompatible_with_target:
    karpenter_deduction = 10

# --- Category 6: Workload Risks (max deduction: 10) ---
# COUNTING UNIT: each individual Deployment/StatefulSet/DaemonSet affected.
# Only count workloads in non-system namespaces (exclude: kube-system, kube-public,
# kube-node-lease, karpenter, amazon-cloudwatch, amazon-guardduty).
# A single workload can trigger MULTIPLE risk types — count each risk separately.
#
# HIGH-severity risks (3 pts each, sub-cap 8 pts):
#   - Deployment with replicas == 1
#   - Deployment with strategy.type == Recreate
#
# MEDIUM-severity risks (1 pt each unless noted, sub-cap 4 pts):
#   - Deployment missing readinessProbe on ANY container (1 pt)
#   - Deployment missing resources.requests (cpu or memory) on ANY container (1 pt)
#   - Multi-replica Deployment without a matching PodDisruptionBudget (1 pt)
#   - Drain-blocking PDB (disruptionsAllowed == 0) (2 pts each)
#
# IMPORTANT: If one workload has BOTH single-replica AND missing probes,
# that is 1 HIGH (3 pts) + 1 MEDIUM (1 pt) = 4 pts for that workload.
workload_high = 0
workload_medium = 0
for each workload in non_system_namespaces:
    if workload.replicas == 1:                workload_high += 3
    if workload.strategy == "Recreate":       workload_high += 3
    if workload.missing_readiness_probe:      workload_medium += 1
    if workload.missing_resource_requests:    workload_medium += 1
    if workload.replicas > 1 and no_matching_pdb: workload_medium += 1
for each pdb where disruptionsAllowed == 0:
    workload_medium += 2                      # drain-blocking PDB
workload_high = min(workload_high, 8)
workload_medium = min(workload_medium, 4)
workload_deduction = min(workload_high + workload_medium, 10)

# --- Category 7: AWS Upgrade Insights (max deduction: 10) ---
# COUNTING UNIT: each insight ID from the EKS Insights API.
# Map insight status to severity:
#   FAILING → 5 pts
#   WARNING → 2 pts
#   ERROR   → 3 pts
#   PASSING → 0 pts
#   UNKNOWN → 0 pts
insights_deduction = 0
for each insight:
    if insight.status == "FAILING":  insights_deduction += 5
    if insight.status == "WARNING":  insights_deduction += 2
    if insight.status == "ERROR":    insights_deduction += 3
insights_deduction = min(insights_deduction, 10)

# --- Category 8: AL2 Nodes (max deduction: 5) ---
# COUNTING UNIT: count of individual AL2 nodes.
al2_deduction = 0
al2_node_count = count of nodes where osImage contains "Amazon Linux 2" (not "2023")
                 or kernelVersion contains "amzn2"
if al2_node_count > 0:
    al2_deduction = 2 + (al2_node_count // 3)   # integer division
al2_deduction = min(al2_deduction, 5)

# --- Category 9: Behavioral Changes (max deduction: 5) ---
# COUNTING UNIT: each distinct behavioral change TYPE that applies to the target version.
behavioral_deduction = 0
for each behavioral_change applicable to target:
    if severity == MEDIUM: behavioral_deduction += 2
    if severity == LOW:    behavioral_deduction += 1
behavioral_deduction = min(behavioral_deduction, 5)

# --- Category 10: Unsupported Version (max deduction: 15) ---
# TRIGGER: cluster's current version has passed its Extended Support Until date.
# This is a binary check — either the version is unsupported or it isn't.
# NOTE: If the target version does not exist on EKS, the assessment is ABORTED
# in Step 1.0 (version-validation.md) — no score is produced at all.
unsupported_deduction = 0
if cluster_version_extended_support_end_date < assessment_date:
    unsupported_deduction = 15

# --- Final Score ---
total_deductions = (breaking_changes_deduction + deprecated_apis_deduction
                    + node_skew_deduction + addon_deduction + karpenter_deduction
                    + workload_deduction + insights_deduction + al2_deduction
                    + behavioral_deduction + unsupported_deduction)
score = max(0, 100 - total_deductions)

# --- Hard Blocker Override (apply AFTER arithmetic) ---
# If ANY hard blocker is present, the upgrade CANNOT proceed safely.
# Cap score at 59 (NOT READY) regardless of the arithmetic result.
#
# Hard blockers (exhaustive list):
#   1. Node version skew > 2 (K8s API server rejects the upgrade)
#   2. Karpenter version incompatible with target (node provisioning breaks)
#   3. Critical add-on INCOMPATIBLE with target version (networking/storage breaks)
#   4. Critical add-on DEGRADED or FAILED (node drain stalls — volumes, DNS, or
#      networking broken during reschedule)
#   5. API removed in target version AND actively used in cluster (workloads fail)
#   6. Cluster status != ACTIVE (EKS API rejects update-cluster-version)
#   7. AL2-only node groups AND target >= 1.33 (no AL2 AMI available for target)
#   8. Any cluster subnet has < 5 available IPs (EKS API rejects update-cluster-version)
#
# NOTE: "Critical add-on" = vpc-cni, coredns, kube-proxy, aws-ebs-csi-driver
has_hard_blocker = False
if node_skew_any_group > 2:                           has_hard_blocker = True
if karpenter_installed and karpenter_incompatible:    has_hard_blocker = True
if any critical_addon.verdict == "INCOMPATIBLE":      has_hard_blocker = True
if any critical_addon.status in [DEGRADED, FAILED]:   has_hard_blocker = True
if any api_removed_in_target_and_in_use:              has_hard_blocker = True
if cluster_status != "ACTIVE":                        has_hard_blocker = True
if al2_only_node_groups and target >= 1.33:           has_hard_blocker = True
if any subnet.available_ips < 5:                      has_hard_blocker = True

if has_hard_blocker:
    score = min(score, 59)

1.2 — Score Interpretation

1.3 — Worked Example

Cluster: example-cluster, upgrading 1.30 → 1.31

Findings:

• EBS CSI driver DEGRADED (IAM issue) → critical add-on, status DEGRADED → 5 pts
• 17 FlowSchema resources using flowcontrol.apiserver.k8s.io/v1beta3 (2 API paths: flowschemas + prioritylevelconfigurations, deprecated but available in 1.31) → 1 + 1 = 2 pts
• 1 AWS Insight WARNING (deprecated APIs for v1.32) → 2 pts
• legacy-app: 1 replica (HIGH=3) + Recreate strategy (HIGH=3) + missing probes (MED=1) + missing requests (MED=1) = 8 pts
• single-replica-app: 1 replica (HIGH=3) + missing probes (MED=1) + missing requests (MED=1) = 5 pts
• recreate-app: Recreate strategy (HIGH=3) + missing probes (MED=1) = 4 pts
• no-resources-app: missing probes (MED=1) + missing requests (MED=1) = 2 pts
• insufficient-replicas-app: missing probes (MED=1) = 1 pt
• karpenter-test-app: missing probes (MED=1) = 1 pt

Workload risk calculation:

• HIGH sub-total: 3+3+3+3 = 12 → capped at 8
• MEDIUM sub-total: 1+1+1+1+1+1+1+1+1+1 = 10 → capped at 4
• Workload total: 8+4 = 12 → capped at 10

Score (arithmetic):

100 - 0 (breaking) - 2 (deprecated) - 0 (skew) - 5 (addon) - 0 (karpenter)
    - 10 (workload) - 2 (insights) - 0 (AL2) - 0 (behavioral) - 0 (unsupported)
= 100 - 19 = 81%

Hard blocker override:

EBS CSI driver DEGRADED → critical add-on DEGRADED → has_hard_blocker = True
score = min(81, 59) = 59% → NOT READY

Final score: 59% — NOT READY (hard blocker: critical add-on DEGRADED)

Step 2: Build Master Finding List (MANDATORY — do this BEFORE calculating the score)

Before calculating the score, you MUST compile a complete finding table. This table is the single source of truth for scoring. Every row must map to exactly one line in the pseudocode above.

| # | Category | Finding | Counting Unit | Severity | Pts | Rule Applied |
|---|----------|---------|---------------|----------|-----|--------------|
| 1 | Deprecated APIs | flowschemas v1beta3 | API path | LOW | 1 | deprecated_but_still_served |
| 2 | Deprecated APIs | prioritylevelconfigurations v1beta3 | API path | LOW | 1 | deprecated_but_still_served |
| 3 | Add-on | aws-ebs-csi-driver DEGRADED | add-on | HIGH | 5 | critical addon DEGRADED |
| ... | ... | ... | ... | ... | ... | ... |

After building this table: 1. Sum each category column 2. Apply the per-category cap from the pseudocode 3. Sum all capped category totals 4. Subtract from 100

Include this table in the report under "Score Breakdown" so users can audit the math.

Step 3: Consistency Checks (MANDATORY)

3.1 Structural contract (check FIRST, before content checks)

Before returning the report, verify it contains exactly these top-level sections in this order:

1. # EKS Upgrade Readiness Assessment 2. ## Readiness Score: ... 3. ## Blockers & Critical Actions 4. ## Recommended Actions 5. ## Informational Findings 6. ## Evidence 7. ## Upgrade Plan 8. ## AWS Reference Links

If ANY of sections 3, 4, 5, 7, or 8 is missing, the report is invalid — add the missing section (with "No blockers identified." / "No recommended actions." / "None." placeholder text if empty) before returning it to the user.

Sections 3, 4, and 5 MUST appear before section 6 (Evidence). If they appear after Evidence, the report is invalid — reorder before returning.

3.2 Content checks

1. Every HIGH/CRITICAL finding must appear in "Blockers & Critical Actions" 2. Every MEDIUM finding must appear in "Recommended Actions" 3. Every LOW finding must appear in "Informational Findings" 4. The executive summary must match the findings — don't call something critical if it's medium 5. Score components must add up correctly 6. CROSS-CHECK RULE: Before writing any count (e.g., "5 deployments missing probes"), go back to the raw data and list the names. If the count of names doesn't match the number in your heading, fix it. Never write a count from memory. 7. NO HALLUCINATED NUMBERS: For any dollar amount, percentage, or numeric claim, show the arithmetic inline or in a comment. If you can't show the math, don't state the number. 8. WORKLOAD TABLE REQUIRED: The master workload table from workload-risks.md Step A MUST be produced before any workload risk findings are written. All workload counts in the report must be traceable to rows in that table.

Step 4: Write the Report

Filename Format

EKS-Upgrade-Assessment-<cluster>-<current>-to-<target>-<YYYY-MM-DD>-<HHMM>.md

Example: EKS-Upgrade-Assessment-my-cluster-1.30-to-1.31-2026-03-26-1430.md

Report Template

The report structure is a contract, not a suggestion. Every report MUST contain the sections below, in exactly this order, with exactly these headings. Do not reorder, rename, or omit required sections. Sections marked OPTIONAL are included only when their condition is met; if the condition isn't met, omit the section entirely (do not leave it as "N/A" or "None found").

Required section order (every report, every time):

1. # EKS Upgrade Readiness Assessment — title + metadata table 2. ## Readiness Score: XX% — [LEVEL] — summary sentence + Score Breakdown table 3. ## Blockers & Critical Actions — MUST appear even if empty (write "No blockers identified.") 4. ## Recommended Actions — MUST appear even if empty (write "No recommended actions.") 5. ## Informational Findings — MUST appear even if empty (write "None.") 6. ## Evidence — container for the detailed tables below

• ### Add-on Inventory
• ### Unknown & Unidentified Add-ons — OPTIONAL (only if any UNKNOWN_* verdicts exist)
• ### Node Group Summary
• ### Workload Risk Summary

7. ## Upgrade Plan — always required 8. ## AWS Reference Links — always required

The three action sections (Blockers, Recommended, Informational) come BEFORE the Evidence tables. This is intentional — readers open the report to answer "what do I need to do?", not "what did the tool find?". Evidence supports the action items; it doesn't precede them.

# EKS Upgrade Readiness Assessment

| Field | Value |
|-------|-------|
| Cluster | [name] |
| Region | [region] |
| Account | [account-id] |
| Current Version | [current] |
| Target Version | [target] |
| Assessment Date | [YYYY-MM-DD HH:MM] |

---

## Readiness Score: [XX]% — [READY/GOOD/FAIR/RISKY/NOT READY]

[2-3 sentence summary. What's the bottom line? Can they upgrade safely?]

### Score Breakdown

| Category | Status | Deduction | Details |
|----------|--------|-----------|---------|
| Breaking Changes | ✅/⚠️/❌ | -X pts | [summary] |
| Deprecated APIs | ✅/⚠️/❌ | -X pts | [summary] |
| Node Readiness | ✅/⚠️/❌ | -X pts | [summary] |
| Add-on Compatibility | ✅/⚠️/❌ | -X pts | [summary] |
| Karpenter | ✅/⚠️/❌/N/A | -X pts | [summary] |
| Workload Risks | ✅/⚠️/❌ | -X pts | [summary] |
| AWS Upgrade Insights | ✅/⚠️/❌ | -X pts | [summary] |
| AL2 / AMI | ✅/⚠️/❌ | -X pts | [summary] |
| Behavioral Changes | ✅/⚠️/❌ | -X pts | [summary] |
| Unsupported Version | ✅/❌/N/A | -X pts | [summary — omit row if version is supported] |
| **Total** | | **-X pts** | **Score: XX%** |

---

## Blockers & Critical Actions

[Items that MUST be resolved before upgrading. If none, write: "No blockers identified."]

### [Finding Title]
- **Severity:** HIGH/CRITICAL
- **What we found:** [specific to this cluster]
- **Impact if not addressed:** [real-world consequence]
- **Remediation:**

[pre-filled command with actual cluster name and region]

- **Reference:** [AWS doc link]

---

## Recommended Actions

[Items that SHOULD be addressed but won't block the upgrade. If none, write: "No recommended actions."]

### [Finding Title]
- **Severity:** MEDIUM
- **What we found:** [details]
- **Remediation:** [steps]

---

## Informational Findings

[LOW severity items and behavioral changes — awareness only. If none, write: "None."]

---

## Evidence

### Add-on Inventory

| Add-on | Type | Version | Status | Verdict | Source |
|--------|------|---------|--------|---------|--------|
| [name] | Managed/Self-managed/OSS | [ver] | [health] | COMPATIBLE/UPDATE_RECOMMENDED/INCOMPATIBLE/UNKNOWN_VERIFIABLE | [URL or "managed"] |

### Unknown & Unidentified Add-ons

Include this subsection only if ANY add-on has verdict `UNKNOWN_VERIFIABLE` or
`UNKNOWN_UNIDENTIFIED`. Omit it entirely if everything was resolved.

#### Compatibility Unverified (UNKNOWN_VERIFIABLE)

Add-ons the skill identified but could not verify against the target Kubernetes
version. The user must check these manually before upgrading.

| Add-on | Version | URL(s) Consulted | Why Unverified |
|--------|---------|------------------|----------------|
| [name] | [ver] | [url] | [e.g., page 404, no compat matrix found, ambiguous wording] |

#### Unidentified Workloads (UNKNOWN_UNIDENTIFIED)

Workloads that appear to be add-ons (based on namespace or shape) but could not be
identified. The user likely knows what these are — please review and confirm
compatibility with the target version manually.

| Kind | Name | Namespace | Image | Labels |
|------|------|-----------|-------|--------|
| [Deployment/DaemonSet/StatefulSet] | [name] | [ns] | [full image:tag] | [key labels present] |

### Node Group Summary

| Node Group | Version | AMI Type | Instances | Skew | Status |
|------------|---------|----------|-----------|------|--------|
| [name] | [ver] | [ami] | [min/max] | [N] | ✅/⚠️/❌ |

### Workload Risk Summary

| Risk | Severity | Count | Details |
|------|----------|-------|---------|
| Single replica deployments | HIGH | [N] | [names] |
| Missing PDBs | MEDIUM | [N] | [names] |
| Missing readiness probes | MEDIUM | [N] | [names] |
| Missing resource requests | HIGH | [N]% | [percentage] |

---

## Upgrade Plan

[Step-by-step upgrade sequence with pre-filled commands.]

### Pre-Upgrade Checklist
- [ ] All blockers resolved
- [ ] Add-ons updated to compatible versions
- [ ] Node groups ready (AL2023/Bottlerocket)
- [ ] PDBs in place for critical workloads
- [ ] Backup/snapshot taken

### Step 1: Update Add-ons (if needed)

aws eks update-addon --cluster-name [CLUSTER] --addon-name [ADDON] --addon-version [VERSION] --region [REGION]

### Step 2: Upgrade Control Plane

aws eks update-cluster-version --name [CLUSTER] --kubernetes-version [TARGET] --region [REGION]

### Step 3: Monitor Upgrade Progress

aws eks describe-update --name [CLUSTER] --update-id [UPDATE_ID] --region [REGION]

### Step 4: Upgrade Node Groups

aws eks update-nodegroup-version --cluster-name [CLUSTER] --nodegroup-name [NODEGROUP] --region [REGION]

### Step 5: Verify

kubectl get nodes kubectl get pods -A | grep -v Running | grep -v Completed

---

## AWS Reference Links

[All links verified via web search or AWS documentation. Do NOT fabricate URLs.]

Step 5: Look Up AWS References

Use web search or AWS documentation to find verified URLs. Prefer:

• https://docs.aws.amazon.com/eks/latest/best-practices/
• https://docs.aws.amazon.com/eks/latest/userguide/
• https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html

Do NOT fabricate deep-link URLs. When in doubt, link to the broad section page.

Step 6: Write the Report File

Write to the workspace root.

Step 7: Offer HTML Conversion

After writing the markdown report, ask: "Would you like me to convert the report to HTML? Run: `python3 tools/md_to_html.py <report-filename>.md`"

Do NOT generate HTML manually. Always use the conversion script.

AWS Upgrade Insights

Purpose

Retrieve and interpret official AWS EKS Upgrade Insights — pre-upgrade checks that AWS runs against your cluster.

How to Check

Step 1: Get All Insights

1. Call get_eks_insights with the cluster name 2. Filter for category UPGRADE_READINESS 3. Record each insight: ID, status, name, description

Step 2: Get Details for Non-Passing Insights

For any insight with status other than PASSING: 1. Call get_eks_insights with the specific insight_id 2. Record: detailed description, recommendation, affected resources

Step 3: Classify Findings

Step 4: Cross-Reference with Other Sections

AWS Upgrade Insights often overlap with findings from other sections (deprecated APIs, add-on compatibility). When reporting:

• Note if an insight confirms a finding from another section
• Do NOT double-count in the score — the insight score is separate from other categories
• Highlight any insights that reveal issues NOT caught by other checks

Important Context for Users

AWS Upgrade Insights checks multiple versions ahead, not just the immediate target. For example, if upgrading from 1.30 → 1.31, AWS may flag deprecated APIs that are removed in 1.33. This is valuable forward-looking information but should not be confused with immediate blockers.

Explain this distinction clearly in the report:

• "Blocked for target version" = must fix before upgrading
• "Flagged by AWS for future version" = plan to fix, but not a blocker for this upgrade

Score Impact

Canonical scoring is defined in `references/report-generation.md` §Category 7.

Quick reference: FAILING = 5 pts, ERROR = 3 pts, WARNING = 2 pts. Max category = 10 pts.

Version Validation & Upgrade Path

Purpose

Validate the upgrade path, determine support status, and enforce EKS upgrade rules.

EKS Version Support Calendar (fallback reference, as of March 2026)

Freshness gate — apply BEFORE using this table:

1. If the cluster version or target version is NOT in the table below → fetch live

data from AWS docs (search_documentation for "EKS Kubernetes versions") before proceeding.

2. If today's assessment date is past the "Extended Support Until" date for the cluster's

current version → that version's status may have changed to UNSUPPORTED. Verify live before

reporting support status.

3. If live lookup fails or the MCP server is unavailable → use the table as fallback, but add

a note in the report: "Support status unverified — table data may be stale."

CRITICAL: The upgradePolicy.supportType field from the API is a CONFIGURATION PREFERENCE, not the current billing status. Always determine actual support status from the calendar above or from live AWS documentation.

Cost impact: Extended support costs $0.60/hr vs $0.10/hr for standard support.

IMPORTANT — Cost Calculation Formula (do NOT estimate, always compute):

extra_cost_per_month = (0.60 - 0.10) × 730 = $365/month per cluster
total_extended_cost  = 0.60 × 730 = $438/month per cluster
total_standard_cost  = 0.10 × 730 = $73/month per cluster

Always use this formula. Do NOT round, estimate, or hallucinate cost figures. 730 = average hours per month (365 days × 24 hours ÷ 12 months).

Checks to Execute

1.0 — Target Version Existence (MUST run before other checks)

Why: EKS releases versions incrementally. A target version that doesn't exist on EKS yet cannot be assessed. The arithmetic check (target - current == 1) is necessary but NOT sufficient.

How to check: 1. Confirm the target version exists in the calendar table above. 2. If NOT in the table → search AWS docs (search_documentation for "EKS Kubernetes versions") to confirm whether the version has been released on EKS. 3. If live lookup also finds no evidence the version exists on EKS → ABORT the assessment.

If target version does not exist on EKS — STOP and report:

## Assessment Cannot Proceed

**Target version <target> is not yet available on Amazon EKS.**

The latest supported EKS version is <latest_known>. Kubernetes <target> has not been
released on EKS as of this assessment date (<date>).

**What you can do:**
- Assess upgrade readiness to <latest_known> instead (if your cluster is on <latest - 1>)
- Monitor the EKS release calendar for <target> availability:
  https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html

Do NOT proceed with Steps 1-8. Do NOT produce a readiness score. End the assessment here.

1.1 — Current Version & Support Status

How to check: 1. Describe the cluster → get version and platformVersion 2. Match version against the calendar table above (applying the freshness gate) 3. Determine the support status. Possible states:

• STANDARD — within standard support window
• EXTENDED — past standard support, within extended support window
• UNSUPPORTED — past extended support end date (see handling below)

4. Report: version, support status, when current support period ends (or already ended)

UNSUPPORTED version handling:

If the cluster version's Extended Support Until date has passed:

• Status = UNSUPPORTED
• Severity = CRITICAL
• Flag as a blocker in the report (see report-generation.md for template)
• The cluster no longer receives security patches or bug fixes from AWS
• AWS may force-upgrade the cluster with limited notice
• Extended support billing ($0.60/hr) still applies even past the end date until the cluster is upgraded
• Score impact: 15 pts deduction (see report-generation.md §Category 10)

Output: Current version, support tier, cost implications. If UNSUPPORTED, include urgency callout.

1.2 — Upgrade Path Validation

Rules:

• EKS requires upgrading one minor version at a time (e.g., 1.30 → 1.31, not 1.30 → 1.32)
• Downgrades are not supported
• Same-version "upgrades" are invalid

How to check: 1. Parse current version (from cluster describe) and target version (from user input) 2. Calculate version difference: target_minor - current_minor 3. If difference == 1: valid direct upgrade 4. If difference > 1: show required upgrade path (e.g., 1.29 → 1.30 → 1.31 → 1.32) 5. If difference <= 0: invalid (same version or downgrade)

Output: Valid/invalid path, required intermediate steps if multi-hop.

1.3 — Version Skew Policy Check

Rules (Kubernetes version skew policy):

• kubelet can be at most N-2 relative to the control plane
• If control plane is upgraded to target version, nodes must be within 2 minor versions

How to check: 1. List all node groups → describe each for Kubernetes version 2. List nodes via Kubernetes API → get kubelet versions from status.nodeInfo.kubeletVersion 3. For each node/node group version, calculate skew against the TARGET version (not current) 4. Skew > 2: BLOCKER — nodes must be upgraded first 5. Skew == 2: WARNING — at maximum skew, upgrade nodes promptly after control plane

Output: Per-node-group version, skew against target, blocker/warning status.

Score Impact

Canonical scoring is defined in `references/report-generation.md` §Category 3 (Node Readiness) and §Category 10 (Unsupported Version).

Workload Risks

Purpose

Assess workload resilience during the upgrade process. These are not upgrade blockers but affect the safety and smoothness of the upgrade.

CRITICAL: Systematic Enumeration Rule

You MUST follow this process to avoid miscounting. Do NOT count from memory.

Step A: Build the Master Workload Table

Before checking ANY risk, build a single table of ALL workloads in non-system namespaces.

Non-system namespaces to EXCLUDE: kube-system, kube-public, kube-node-lease, karpenter, amazon-cloudwatch, amazon-guardduty, aws-observability.

Workload types to INCLUDE: Deployments, StatefulSets, DaemonSets.

How to build the table: 1. List ALL Deployments across all namespaces 2. List ALL StatefulSets across all namespaces 3. List ALL DaemonSets across all namespaces 4. Filter out workloads in system namespaces listed above 5. For EACH remaining workload, extract from its spec:

• name, namespace, kind (Deployment/StatefulSet/DaemonSet)
• replicas (for Deployments/StatefulSets; DaemonSets run on all nodes)
• strategy.type (Deployments only: RollingUpdate or Recreate)
• For EACH container: readinessProbe (present/absent), livenessProbe (present/absent),

resources.requests.cpu (value or absent), resources.requests.memory (value or absent)

Output format — you MUST produce this table before proceeding:

| # | Name | Kind | NS | Replicas | Strategy | Probes | Requests | Notes |
|---|------|------|----|----------|----------|--------|----------|-------|
| 1 | app-a | Deployment | default | 3 | RollingUpdate | ✅ readiness+liveness | ✅ cpu+mem | |
| 2 | app-b | Deployment | default | 1 | Recreate | ❌ none | ❌ none | single-replica, recreate |
| 3 | mon-agent | DaemonSet | default | N/A | N/A | ❌ none | ✅ cpu+mem | |

Step B: Check Each Risk Against the Table

Walk through each check below. For every finding, reference the row number from the table. This prevents miscounting and ensures no workload is missed.

Checks to Execute

6.1 — Single Replica Deployments

Why this matters: Node drains during upgrade will cause downtime for single-replica workloads.

How to check: From the master table, filter for kind == Deployment AND replicas == 1.

Rating: Each match = HIGH severity (3 pts in score).

6.2 — Missing Pod Disruption Budgets

Why this matters: Without PDBs, node drain can evict all pods simultaneously.

How to check: 1. List PodDisruptionBudgets across all namespaces 2. From the master table, filter for kind == Deployment AND replicas > 1 in non-system namespaces 3. Cross-reference: which multi-replica deployments have NO matching PDB? 4. Check for drain-blocking PDBs (see 6.2b below)

IMPORTANT: Only flag missing PDBs for workloads with replicas > 1. A PDB on a single-replica deployment is meaningless — do NOT flag single-replica workloads for missing PDBs.

Rating: Each missing PDB on multi-replica deployment = MEDIUM severity (1 pt).

6.2b — Drain-Blocking PDBs (upgrade stall risk)

Why this matters: A PDB that allows zero disruptions will cause kubectl drain to hang indefinitely during node group upgrades. The node group upgrade will eventually time out (typically after 1+ hours), failing the rolling update. This is the #1 cause of "upgrade stuck" support tickets.

How to check: 1. For each PDB found in step 6.2, inspect:

• status.disruptionsAllowed == 0, OR
• spec.maxUnavailable == 0, OR
• spec.minAvailable == total replicas of the target workload

2. If any of the above conditions is true AND the target workload has pods running on nodes that will be drained during the upgrade → flag it.

Report message (use this exact framing):

⚠️ PDB may stall node group upgrade

>

<pdb-name> in namespace <ns> currently allows 0 disruptions for <workload-name>.

During a node group rolling update, EKS drains each node before replacing it. If this PDB

cannot be satisfied (e.g., not enough capacity on remaining nodes to reschedule pods), the

drain will hang until the node group upgrade times out (~1 hour).

>

Before upgrading:

1. Verify sufficient cluster capacity exists for pods to reschedule to other nodes

2. Consider temporarily relaxing the PDB: kubectl patch pdb <name> -n <ns> -p '{"spec":{"maxUnavailable":1}}'

3. Or ensure the workload has enough replicas spread across multiple nodes

>

If you skip this: The node group upgrade will likely time out and require manual

intervention. The control plane upgrade itself will succeed, but node rotation will stall.

Rating: Each drain-blocking PDB = MEDIUM severity (2 pts).

This is NOT a hard blocker because:

• The control plane upgrade itself will succeed
• The issue only manifests during node group rolling update
• It can be resolved mid-upgrade by patching the PDB
• But it WILL cause significant delay and potential manual intervention if not addressed

6.3 — Missing Health Probes

Why this matters: Without readiness probes, traffic is sent to pods before they're ready.

How to check: From the master table, filter for workloads where ANY container is missing a readinessProbe. Count ALL workload types (Deployments, StatefulSets, AND DaemonSets).

Rating: Each workload missing probes = MEDIUM severity (1 pt).

6.4 — Missing Resource Requests

Why this matters: Without resource requests, pods can't be properly rescheduled during node drains.

How to check: From the master table, filter for workloads where ANY container is missing resources.requests.cpu OR resources.requests.memory.

IMPORTANT: Check the ACTUAL spec data. Do NOT assume a workload has or lacks requests without verifying. If the deployment spec shows requests: {cpu: "100m", memory: "128Mi"}, that workload HAS requests — do not flag it.

Rating: Each workload missing requests = MEDIUM severity (1 pt).

6.5 — Recreate Update Strategy

Why this matters: Recreate strategy causes full downtime during any rollout.

How to check: From the master table, filter for kind == Deployment AND strategy == Recreate.

Rating: Each match = HIGH severity (3 pts in score).

6.6 — Graceful Shutdown Configuration

Why this matters: Without preStop hooks, there's a race condition during node drain.

How to check: 1. From the master table, identify workloads exposed via Services (especially LoadBalancer type) 2. Check if those workloads have lifecycle.preStop hooks 3. Check terminationGracePeriodSeconds

Rating: Missing preStop on externally-facing workloads = MEDIUM severity (1 pt).

Step C: Compile Findings with Row References

After all checks, produce a findings list that references the master table row numbers:

| Finding | Severity | Workloads (by row #) | Count |
|---------|----------|---------------------|-------|
| Single replica | HIGH | #2, #7 | 2 |
| Recreate strategy | HIGH | #2, #5 | 2 |
| Missing probes | MEDIUM | #2, #3, #5, #6, #8 | 5 |
| Missing requests | MEDIUM | #2, #6 | 2 |

This makes the count verifiable. If the count doesn't match the listed row numbers, something is wrong.

Score Impact

Canonical scoring is defined in `references/report-generation.md` §Category 6 (Workload Risks).

Upstream Provenance

This skill is vendored from an upstream repo. Do not edit files here directly — your changes will be overwritten by the next sync.

Local modifications applied at sync time

The sync script applies three deterministic edits to upstream content:

1. `### MCP Server Setup` section is replaced. Apex does not ship a project-root .mcp.json; MCP setup is delegated to the eks-mcp-server skill in this repo. The upstream's fallback note ("falls back to AWS CLI and kubectl") is preserved. 2. `steering/` -> `references/` rename. Upstream's progressive-disclosure docs live under steering/, but apex already uses a top-level steering/ directory at the repo root for workflow orchestration (different concept). The sync script renames the directory on copy and rewrites all internal cross-refs from steering/ to references/ inside SKILL.md and the 8 progressive-disclosure files. This aligns the layout with the Anthropic skill spec's canonical name for "additional documentation agents read on demand." 3. `description:` frontmatter is replaced with a "pushy" wording. Upstream's description is a keyword list. Apex review feedback (#36) calls for natural-question phrasings ("can I upgrade my cluster?", "is my cluster ready for 1.32?", etc.) that mirror sibling skills like eks-best-practices and eks-recon. The sync script replaces the whole description: line on every run.

Everything else is byte-for-byte from upstream.

To propose changes

Open a PR against the upstream repo: https://github.com/aws-samples/sample-eks-upgrade-skill.git

Then re-run the sync script here.