Eks Resilience Checker

Language / 语言

• If the user speaks English, follow SKILL_EN.md
• 如果用户使用中文，请遵循 SKILL_ZH.md

Detect the language from the user's message and load the corresponding instruction file.

<!-- AGENT: Do not load this file during assessment execution. This is a design document for human reference only. -->

📄 Product Requirement Document (PRD)

EKS Resilience Checker — Agent Skill

版本: 0.2 日期: 2026-04-03 状态: Draft 仓库: sample-aws-resilience-skill/eks-resilience-checker 变更: v0.2 — 日志分析移至 chaos-engineering-on-aws；聚焦 26 项评估；assessment.json → chaos skill 集成接口 变更: v0.3 — 新增分发安装方式（npx skills add）；更新 Lifecycle 图加入第四个 Skill 变更: v0.4 — Section 10 改为"从未合并 PR 借鉴的内容"，去掉 MCP Server 方案，聚焦 Skill 本身

---

1. Executive Summary

1.1 产品愿景

开发一个面向 Claude Code / Kiro CLI / Codex 的 Agent Skill，对 Amazon EKS 集群执行全面的韧性架构评估。覆盖三层：应用工作负载（A1-A14）、控制平面（C1-C5）、数据平面（D1-D7），共 26 项检查。输出结构化评估结果，可直接作为 chaos-engineering-on-aws 的输入驱动混沌实验。

1.2 在 Resilience Lifecycle 中的定位

┌─────────────────────────────────────────────────────────────────────────────────────────┐
│                         AWS Resilience Lifecycle Framework                                │
│                                                                                          │
│  Stage 1: Set Objectives  Stage 2: Design & Implement  Stage 3: Evaluate & Test          │
│  ┌───────────────────┐    ┌────────────────────────┐   ┌──────────────────────────────┐  │
│  │ aws-rma-           │    │ aws-resilience-         │   │ chaos-engineering-on-aws      │  │
│  │ assessment          │───►│ modeling                │──►│ 混沌实验 + 指标 + 日志分析    │  │
│  │ "我们在哪?"         │    │ "什么可能出错?"          │   │ "真的会坏吗?"                 │  │
│  └───────────────────┘    └────────────────────────┘   └───────────┬──────────────────┘  │
│                                      ▲                              │                     │
│                                      │                              │                     │
│                                      │    ┌─────────────────────────┴──────────────────┐  │
│                                      │    │ eks-resilience-checker (本 Skill, 第 4 个)   │  │
│                                      │    │ ① 26 项 K8s 韧性评估 → assessment.json      │  │
│                                      │    │ ② FAIL → 实验推荐 → chaos skill 消费        │  │
│                                      │    └────────────────────────────────────────────┘  │
│                                      └──────────── Feedback Loop ────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────────────┘

四个 Skill 完整链路：

一句话定义：自动化评估 EKS 集群 26 项韧性最佳实践，输出结构化结果供混沌实验消费。

1.3 职责边界

1.4 核心价值

1.5 目标用户

1.6 使用工具

---

2. 功能范围

2.1 EKS 韧性基线评估（26 项检查）

自动化评估 EKS 集群的韧性配置，覆盖三个层面：

Application Checks（A1-A14）

Control Plane Checks（C1-C5）

Data Plane Checks（D1-D7）

评估输出

output/
├── assessment.json              # 结构化评估结果（26 项）— chaos skill 可消费
├── assessment-report.md         # 人类可读报告（Markdown）
├── assessment-report.html       # HTML 报告（内联 CSS，可独立打开）
└── remediation-commands.sh      # 一键修复脚本（可执行的 kubectl/aws 命令）

2.2 assessment.json 结构化输出

{
  "schema_version": "1.0",
  "cluster_name": "PetSite",
  "region": "ap-northeast-1",
  "kubernetes_version": "1.32",
  "platform_version": "eks.8",
  "timestamp": "2026-04-03T08:00:00Z",
  "target_namespaces": ["petadoptions", "default"],

  "summary": {
    "total_checks": 28,
    "passed": 20,
    "failed": 6,
    "info": 2,
    "critical_failures": 3,
    "compliance_score": 71.4
  },

  "checks": [
    {
      "id": "A2",
      "name": "Run Multiple Replicas",
      "category": "application",
      "severity": "critical",
      "status": "FAIL",
      "findings": [
        "Deployment petadoptions/payforadoption has replicas=1",
        "Deployment petadoptions/pethistory has replicas=1"
      ],
      "resources_affected": [
        "petadoptions/payforadoption",
        "petadoptions/pethistory"
      ],
      "remediation": "kubectl scale deployment payforadoption --replicas=2 -n petadoptions",
      "chaos_experiment_recommendation": {
        "description": "Kill single-replica pod to measure actual downtime",
        "fault_types": ["pod_kill", "fis_eks_pod_delete"],
        "priority": "P0",
        "rationale": "Single replica = guaranteed downtime on pod failure"
      }
    }
  ],

  "experiment_recommendations": [
    {
      "priority": "P0",
      "check_id": "A2",
      "target_resources": ["petadoptions/payforadoption"],
      "suggested_fault_type": "pod_kill",
      "suggested_backend": "chaosmesh",
      "hypothesis": "Killing the single-replica payforadoption pod will cause service unavailability until K8s recreates the pod (~30-60s)",
      "expected_rto_seconds": 60
    }
  ]
}

2.3 FAIL → 实验推荐映射表

---

3. 与 chaos-engineering-on-aws 的集成

3.1 作为第三输入源

chaos-engineering-on-aws 的 Step 1 "Define Experiment Targets" 增加输入方式：

Step 1 输入（三选一或组合）:
  Method 1: aws-resilience-modeling 报告     → AWS 资源级风险
  Method 2: 独立 chaos-input 文件           → 手动指定
  Method 3: eks-resilience-checker 的 assessment.json → K8s 配置风险 (新增)

消费方式： 1. 读取 assessment.json 的 experiment_recommendations 数组 2. 按 priority 排序（P0 > P1 > P2） 3. 每个推荐包含 suggested_fault_type（对应 fault-catalog.yaml）和 target_resources 4. 结合 Method 1 的 AWS 风险，合并去重后输出给用户确认

3.2 chaos-engineering-on-aws 日志分析扩展

日志分析功能作为 chaos-engineering-on-aws 的增强（不在本 Skill 中实现）：

chaos-engineering-on-aws Step 5 (改造后):
  Phase 2: Observation
    ├── monitor.sh → CloudWatch metrics       (已有)
    └── log-collector.sh → kubectl logs -f    (新增: 实时日志采集)
  Phase 3: Recovery
    └── 日志分析 + 错误分类                    (新增: timeout/connection/5xx/oom/other)

chaos-engineering-on-aws Step 6 (改造后):
  ├── 指标分析 (已有)
  ├── 应用日志分析章节 (新增)
  └── 事后日志分析独立入口 (新增: 用户给报告路径 → 直接做事后分析)

参考实现：

• 错误分类逻辑 → graph-driven-chaos/code/runner/log_collector.py
• 实时/事后双模式 + 交互设计 → panlm/skills/eks-app-log-analysis

---

4. 技术架构

4.1 工具依赖

MCP Server（可选增强）

当 MCP 不可用时，回退到 kubectl + aws CLI 直接调用。

4.2 EKS 认证方式

支持三种 kubectl → EKS 集群的认证路径：

已有 kubeconfig（最常用）

# 直接使用管理员提供的 kubeconfig
export KUBECONFIG=/path/to/admin-kubeconfig
kubectl get nodes

IAM kubeconfig

aws eks update-kubeconfig --name {CLUSTER_NAME} --region {REGION}

每次 kubectl 请求时自动通过 aws eks get-token 获取短期令牌。要求 IAM 身份在集群中有映射（EKS Access Entries 或 aws-auth ConfigMap）。

ServiceAccount Token（受限环境）

# 创建只读 ServiceAccount
kubectl create serviceaccount eks-resilience-checker -n kube-system
kubectl create clusterrolebinding eks-resilience-checker-readonly \
  --clusterrole=view \
  --serviceaccount=kube-system:eks-resilience-checker

# 生成令牌（有效期 1 年）
TOKEN=$(kubectl create token eks-resilience-checker -n kube-system --duration=8760h)

# 生成自包含 kubeconfig（无需 AWS CLI 即可使用）
ENDPOINT=$(aws eks describe-cluster --name {CLUSTER_NAME} --query 'cluster.endpoint' --output text)
CA_DATA=$(aws eks describe-cluster --name {CLUSTER_NAME} --query 'cluster.certificateAuthority.data' --output text)
kubectl config set-cluster eks-check --server=$ENDPOINT --certificate-authority-data=$CA_DATA --embed-certs=true --kubeconfig=./eks-resilience-kubeconfig
kubectl config set-credentials eks-checker --token=$TOKEN --kubeconfig=./eks-resilience-kubeconfig
kubectl config set-context eks-check --cluster=eks-check --user=eks-checker --kubeconfig=./eks-resilience-kubeconfig
kubectl config use-context eks-check --kubeconfig=./eks-resilience-kubeconfig

运行时零 AWS 依赖，只需 kubectl + 这个 kubeconfig 文件。适合 CI/CD 定期扫描。 使用 view ClusterRole（只读），符合本 Skill 纯只读评估的安全原则。

Skill 执行时：Step 1 应先询问用户 "你有现成的 kubeconfig 吗？还是需要生成一个？"

参考来源：RadiumGu/Chaosmesh-MCP EKS Authentication Methods

4.3 执行流程

Step 1: 集群发现
  ├── 用户提供 cluster name（或自动检测 current-context）
  ├── aws eks describe-cluster → 版本、VPC、endpoint、logging、addons
  └── 确认目标 namespace 列表（排除 kube-system 等系统 namespace）

Step 2: 自动化检查（26 项）
  ├── Application checks (A1-A14): kubectl 查询工作负载配置
  ├── Control Plane checks (C1-C5): aws eks API + kubectl 查询
  └── Data Plane checks (D1-D7): kubectl 查询节点 + 资源配置

Step 3: 生成报告
  ├── assessment.json — 结构化结果（chaos skill 可消费）
  ├── assessment-report.md — 人类可读报告
  ├── assessment-report.html — 带颜色的 HTML 报告
  └── remediation-commands.sh — 修复脚本

Step 4: 实验推荐（可选）
  ├── 基于 FAIL 项 + 映射表 → 生成 experiment_recommendations
  ├── 展示推荐列表，用户确认
  └── 输出供 chaos-engineering-on-aws Step 1 消费的格式

用户交互：

• Step 1: 确认集群和 namespace
• Step 3: 查看报告，选择是否生成修复脚本
• Step 4: 确认是否继续混沌实验（如果是 → 引导用户用 chaos-engineering-on-aws）

4.4 Skill 文件结构

eks-resilience-checker/
├── SKILL.md                            # 入口（语言检测 → 分流）
├── SKILL_EN.md                         # 英文版完整指令
├── SKILL_ZH.md                         # 中文版完整指令
├── README.md                           # 项目说明
├── README_zh.md                        # 中文说明
├── doc/
│   └── prd.md                          # 本文档
├── references/
│   ├── EKS-Resiliency-Checkpoints.md   # 26 项检查详细说明（已有）
│   ├── check-commands.md               # 每项检查对应的 kubectl/aws 命令
│   ├── remediation-templates.md        # 修复命令模板
│   └── fail-to-experiment-mapping.md   # FAIL → 实验推荐映射表
├── scripts/
│   └── assess.sh                       # 评估主脚本（可独立运行）
└── examples/
    └── petsite-assessment.md           # PetSite 集群评估示例

---

5. 安全和约束

5.1 权限要求

5.2 安全原则

1. 纯只读：评估阶段不做任何写操作 2. 修复脚本需确认：remediation-commands.sh 生成后需用户手动执行 3. Namespace 隔离：默认排除 kube-system、kube-public、kube-node-lease 4. 敏感信息：不在报告中暴露 Secret / ConfigMap 的值，只检查存在性

5.3 已知限制

5.4 设计决策

---

6. 输出语言规则

• 用户说英文 → 英文输出
• 用户说中文 → 中文输出
• SKILL.md 入口检测语言后分流到 SKILL_EN.md / SKILL_ZH.md

---

7. 里程碑

---

8. chaos-engineering-on-aws 日志分析扩展（单独跟踪）

以下内容不在本 Skill 范围内，但作为 chaos-engineering-on-aws 的增强需求记录：

8.1 需求

在 chaos-engineering-on-aws 的 Step 5/6 中增加应用日志分析能力：

8.2 实现要点

• monitor.sh 和 log-collector.sh 并行运行（metrics + logs 双通道）
• 错误分类：timeout / connection / 5xx / oom / other
• 日志采集使用 kubectl logs -f -l app={label} --prefix --timestamps
• 事后模式：kubectl logs --since-time={start} deployment/{name}
• 报告中加入"应用日志分析"章节：错误时间线 + 错误模式 + 恢复检测 + 跨服务关联

---

9. 分发和安装

9.1 npx skills add 一键安装

本 Skill 通过 vercel-labs/skills CLI 分发，用户可以一键安装到任何支持的 Agent（Claude Code / Codex / Kiro / OpenCode / Cursor 等 40+ 种）。

安装命令

# 列出仓库中所有可用 Skill
npx skills add panlm/sample-aws-resilience-skill --list

# 安装单个 Skill
npx skills add panlm/sample-aws-resilience-skill --skill eks-resilience-checker

# 安装所有 Skill（全套 4 个）
npx skills add panlm/sample-aws-resilience-skill --skill '*'

# 安装到指定 Agent
npx skills add panlm/sample-aws-resilience-skill --skill eks-resilience-checker -a claude-code -a codex

# 全局安装（跨项目可用）
npx skills add panlm/sample-aws-resilience-skill --skill eks-resilience-checker -g

# CI/CD 友好的非交互安装
npx skills add panlm/sample-aws-resilience-skill --skill eks-resilience-checker -a claude-code -g -y

安装原理

npx skills add 的工作机制：

1. 从 GitHub 仓库拉取 Skill 目录（不是 npm 包，是 Git 仓库） 2. 识别 SKILL.md 的 frontmatter（name + description）确定 Skill 名称 3. 检测本地已安装的 Agent（Claude Code → .claude/skills/、Codex → .agents/skills/ 等） 4. Symlink（默认）或 Copy 到对应 Agent 的 skills 目录 5. Agent 下次运行时自动加载新 Skill

仓库结构:
sample-aws-resilience-skill/
├── aws-rma-assessment/SKILL.md          → npx skills 识别为 "aws-rma-assessment"
├── aws-resilience-modeling/SKILL.md     → npx skills 识别为 "aws-resilience-modeling"
├── chaos-engineering-on-aws/SKILL.md    → npx skills 识别为 "chaos-engineering-on-aws"
└── eks-resilience-checker/SKILL.md      → npx skills 识别为 "eks-resilience-checker" (新增)

安装后:
~/.claude/skills/eks-resilience-checker → symlink → 仓库中的 eks-resilience-checker/

SKILL.md frontmatter 要求

---
name: eks-resilience-checker
description: >
  Assess Amazon EKS cluster resilience against 26 best practice checks
  covering application workloads, control plane, and data plane.
  Outputs structured assessment.json for chaos-engineering-on-aws integration.
  Use when the user wants to evaluate EKS cluster resilience, run resilience
  assessment, check EKS best practices, or prepare for chaos experiments.
  Triggers on: EKS resilience, 韧性评估, cluster assessment, resilience check,
  EKS best practices, 集群评估.
---

更新和管理

# 检查已安装 Skill 是否有更新
npx skills check

# 更新所有 Skill 到最新版
npx skills update

# 列出已安装的 Skill
npx skills list

# 卸载
npx skills remove eks-resilience-checker

9.2 手动安装（备选）

对于不使用 npx skills 的用户：

# Clone 仓库
git clone https://github.com/panlm/sample-aws-resilience-skill.git

# 手动 symlink 到 Agent skills 目录
ln -s $(pwd)/sample-aws-resilience-skill/eks-resilience-checker ~/.claude/skills/eks-resilience-checker

# 或者直接拷贝
cp -r sample-aws-resilience-skill/eks-resilience-checker ~/.claude/skills/

9.3 与其他 Skill 仓库的关系

两个仓库的 Skill 可以配合使用：

• panlm/skills 的 aws-fis-experiment-prepare + aws-fis-experiment-execute 是独立的 FIS 实验 prepare/execute 工具
• panlm/sample-aws-resilience-skill 的 chaos-engineering-on-aws 是端到端的混沌工程 Skill（6 步流程）
• 用户可以根据需要选择轻量的 prepare/execute 还是全流程的 chaos skill

---

10. 参考实现和执行路径

10.1 参考来源

本 Skill 实现参考了以下来源：

10.2 执行路径

用户触发 "EKS 韧性评估"
       │
       ▼
┌─ Path A: assess.sh 脚本（推荐）──────────────────────────────┐
│  bash scripts/assess.sh --cluster <name> --region <region>    │
│  纯 bash + kubectl + aws CLI + jq                             │
│  输出 4 个文件：                                              │
│    assessment.json          — 结构化结果 + 实验推荐           │
│    assessment-report.md     — Markdown 报告                   │
│    assessment-report.html   — HTML 报告（颜色编码）           │
│    remediation-commands.sh  — 修复脚本                        │
│  优点：一键完成，结构化输出，可重复，CI/CD 友好               │
└───────────────────────────────────────────────────────────────┘
       │ 如果 assess.sh 不适用
       ▼
┌─ Path B: Agent 逐条执行（默认）──────────────────────────────┐
│  Agent 按 SKILL_EN/ZH.md Step 2 执行 kubectl 命令            │
│  参考 references/check-commands.md                            │
│  优点：Agent 可按上下文跳过检查，灵活性最高                   │
└───────────────────────────────────────────────────────────────┘

10.3 验证记录

PetSite 集群实测（2026-04-04，EKS 1.34，ap-northeast-1）：

已修复的运行时问题：

• jq --argjson 在大集群上 Argument list too long → 改用 stdin 管道
• A6/A8 的 jq nested reference 需要 $root 模式

EKS Resilience Assessment Report — PetSite

Cluster: PetSite Region: ap-northeast-1 Kubernetes Version: 1.32 Platform Version: eks.8 Assessment Date: 2026-04-03T08:00:00Z Target Namespaces: petadoptions, default

---

Executive Summary

Score Breakdown:
  Application (A1-A14):   9/14 PASS   (64.3%)
  Control Plane (C1-C5):  4/5  PASS   (80.0%)
  Data Plane (D1-D7):     7/7  PASS*  (100.0%)
  * D4, D5 are FAIL but categorized under Application score adjustment below

Overall: 20/28 PASS = 71.4%

Risk Distribution

Critical:  ██████░░░░  3 FAIL / 7 total critical checks
Warning:   ███░░░░░░░  3 FAIL / 11 total warning checks
Info:      ██████████  0 FAIL / 4 total info checks (2 INFO status)

---

Detailed Findings

Application Checks (A1-A14)

A1: Avoid Singleton Pods — PASS

No standalone pods found. All pods are managed by controllers (Deployment, StatefulSet, DaemonSet).

A2: Run Multiple Replicas — FAIL (Critical)

Finding: 3 Deployments running with single replicas — guaranteed downtime on pod failure.

Remediation:

kubectl scale deployment payforadoption --replicas=2 -n petadoptions
kubectl scale deployment pethistory --replicas=2 -n petadoptions
kubectl scale deployment statusupdater --replicas=2 -n petadoptions

A3: Use Pod Anti-Affinity — FAIL (Warning)

Finding: All multi-replica Deployments lack podAntiAffinity — all replicas could be scheduled on the same node.

Remediation:

# Add to each Deployment spec.template.spec
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - <app-name>
        topologyKey: kubernetes.io/hostname

A4: Use Liveness Probes — PASS

All containers across Deployments, StatefulSets, and DaemonSets have livenessProbe configured.

A5: Use Readiness Probes — PASS

All containers have readinessProbe configured.

A6: Use Pod Disruption Budgets — FAIL (Warning)

Finding: No PodDisruptionBudgets found for multi-replica workloads. Node drain during maintenance could terminate all replicas simultaneously.

Remediation:

cat <<EOF | kubectl apply -f -
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: petlistadoptions-pdb
  namespace: petadoptions
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: petlistadoptions
EOF
# Repeat for petsearch, petsite

A7: Run Kubernetes Metrics Server — PASS

metrics-server running in kube-system. kubectl top nodes responds successfully.

A8: Use Horizontal Pod Autoscaler — FAIL (Warning)

Finding: Multi-replica workloads have no HPA — cannot auto-scale under load.

Remediation:

kubectl autoscale deployment petlistadoptions --cpu-percent=70 --min=2 --max=5 -n petadoptions
kubectl autoscale deployment petsearch --cpu-percent=70 --min=2 --max=5 -n petadoptions
kubectl autoscale deployment petsite --cpu-percent=70 --min=2 --max=5 -n petadoptions

A9: Use Custom Metrics Scaling — INFO

No custom metrics infrastructure (KEDA, Prometheus Adapter) detected. Basic CPU/memory scaling only.

This is informational — custom metrics are recommended but not required for all clusters.

A10: Use Vertical Pod Autoscaler — INFO

VPA CRD and controller not installed. Workloads rely on manually configured resource requests/limits.

This is informational — VPA is recommended for right-sizing but not critical.

A11: Use PreStop Hooks — PASS

All Deployment and StatefulSet containers have lifecycle.preStop hooks configured for graceful termination.

A12: Use a Service Mesh — PASS

AWS App Mesh detected (appmesh-controller running in appmesh-system namespace). Sidecar proxies injected in petadoptions namespace pods.

A13: Monitor Your Applications — PASS

CloudWatch Container Insights enabled. ADOT Collector DaemonSet running for metrics and traces collection.

A14: Use Centralized Logging — PASS

Fluent Bit DaemonSet running in amazon-cloudwatch namespace. Logs shipping to CloudWatch Logs.

---

Control Plane Checks (C1-C5)

C1: Monitor Control Plane Logs — PASS

Control plane logging enabled:

• api: Enabled
• audit: Enabled
• authenticator: Enabled
• controllerManager: Disabled
• scheduler: Disabled

Recommendation: Consider enabling controllerManager and scheduler logs for full visibility.

C2: Cluster Authentication — PASS

EKS Access Entries configured (modern API-based method). 3 access entries found with appropriate policies.

C3: Running Large Clusters — PASS

Total services: 42 (well below 1,000 threshold). No large-cluster optimizations needed.

C4: API Server Endpoint Access Control — FAIL (Critical)

Finding: API server publicly accessible from any IP address. This is a critical security risk.

Remediation:

aws eks update-cluster-config \
  --name PetSite \
  --region ap-northeast-1 \
  --resources-vpc-config \
    endpointPublicAccess=true,\
    publicAccessCidrs="203.0.113.0/24",\
    endpointPrivateAccess=true

C5: Avoid Catch-All Admission Webhooks — PASS

No overly broad MutatingWebhook or ValidatingWebhook configurations detected. All webhooks have proper namespace and object selectors.

---

Data Plane Checks (D1-D7)

D1: Use Cluster Autoscaler or Karpenter — PASS

Karpenter v0.37 detected (karpenter namespace). NodePool and EC2NodeClass resources configured.

D2: Worker Nodes Spread Across Multiple AZs — PASS

Distribution variance: 0% (within 20% threshold). Excellent AZ balance.

D3: Configure Resource Requests/Limits — PASS

All Deployments in petadoptions namespace have complete CPU and memory requests and limits.

D4: Namespace ResourceQuotas — PASS

D5: Namespace LimitRanges — PASS

D6: Monitor CoreDNS Metrics — PASS

CoreDNS running with metrics port 9153. ServiceMonitor configured for Prometheus scraping.

D7: CoreDNS Managed Configuration — PASS

CoreDNS managed via EKS Managed Add-on (coredns v1.11.4-eksbuild.2). Automatic updates enabled.

---

Experiment Recommendations

Based on FAIL findings, the following chaos experiments are recommended:

Recommended Experiment Sequence

Phase 1 (P0 — Immediate):
  1. Pod kill on payforadoption → measure actual downtime
  2. Pod kill on pethistory → measure actual downtime
  3. Restrict API server access (fix, not experiment)

Phase 2 (P1 — After fixing replicas):
  4. Terminate node hosting petlistadoptions pods → verify anti-affinity gap
  5. kubectl drain on a worker node → verify PDB gap

Phase 3 (P2 — After adding PDB + anti-affinity):
  6. CPU stress on petsearch → verify no auto-scaling response

Integration with chaos-engineering-on-aws

To run these experiments using the chaos skill:

1. Ensure chaos-engineering-on-aws skill is installed
2. Tell your AI agent:
   "Run chaos experiments based on the EKS assessment at output/assessment.json"
3. The chaos skill will read experiment_recommendations and guide you through
   the 6-step experiment lifecycle

---

Remediation Summary

Critical (Fix Immediately)

Warning (Fix Soon)

Info (Consider)

---

Generated by eks-resilience-checker v1.0 | Full structured data: output/assessment.json

中文 | English

EKS 韧性检查器

AI 驱动的 Agent Skill，对 Amazon EKS 集群执行自动化韧性评估，覆盖 应用工作负载（A1-A14）、控制平面（C1-C5）、数据平面（D1-D7）共 26 项最佳实践检查。输出结构化结果，可直接作为 chaos-engineering-on-aws Skill 的输入驱动混沌实验。

在韧性生命周期中的定位

┌──────────────────────────────────────────────────────────────────────────────────────┐
│                         AWS 韧性生命周期框架                                          │
│                                                                                      │
│  阶段 1: 目标设定       阶段 2: 设计与实施          阶段 3: 评估与测试                  │
│  ┌───────────────────┐  ┌────────────────────────┐  ┌──────────────────────────────┐  │
│  │ aws-rma-           │  │ aws-resilience-         │  │ chaos-engineering-on-aws      │  │
│  │ assessment          │─>│ modeling                │─>│ 混沌实验 + 指标 + 日志分析    │  │
│  │ "我们在哪?"         │  │ "什么可能出错?"          │  │ "真的会坏吗?"                 │  │
│  └───────────────────┘  └────────────────────────┘  └───────────┬──────────────────┘  │
│                                    ^                              │                     │
│                                    │    ┌─────────────────────────┴──────────────────┐  │
│                                    │    │ eks-resilience-checker（本 Skill）           │  │
│                                    │    │ ① 26 项 K8s 韧性评估 → assessment.json      │  │
│                                    │    │ ② FAIL → 实验推荐 → chaos skill 消费        │  │
│                                    │    └────────────────────────────────────────────┘  │
│                                    └──────────── 反馈循环 ───────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────────────────┘

安装

一键安装（推荐）

# 安装本 Skill 到你的 AI Agent
npx skills add aws-samples/sample-aws-resilience-skill --skill eks-resilience-checker

# 安装全部 4 个韧性 Skill
npx skills add aws-samples/sample-aws-resilience-skill --skill '*'

# 安装到指定 Agent
npx skills add aws-samples/sample-aws-resilience-skill --skill eks-resilience-checker -a claude-code

# 全局安装（跨项目可用）
npx skills add aws-samples/sample-aws-resilience-skill --skill eks-resilience-checker -g

手动安装

git clone https://github.com/aws-samples/sample-aws-resilience-skill.git
ln -s $(pwd)/sample-aws-resilience-skill/eks-resilience-checker ~/.claude/skills/eks-resilience-checker

快速开始

1. 配置 kubectl 访问 你的 EKS 集群 2. 告诉你的 AI Agent："对我的集群运行 EKS 韧性评估" 3. 查看 output/ 目录中生成的报告

前置条件

MCP Server（可选增强）

当 MCP 不可用时，回退到 kubectl + aws CLI 直接调用。

{
  "mcpServers": {
    "awslabs.eks-mcp-server": {
      "command": "uvx",
      "args": ["awslabs.eks-mcp-server@latest"],
      "env": { "AWS_REGION": "ap-northeast-1", "FASTMCP_LOG_LEVEL": "ERROR" }
    }
  }
}

检查分类

应用检查（A1-A14）

控制平面检查（C1-C5）

数据平面检查（D1-D7）

输出文件

output/
├── assessment.json              # 结构化评估结果（26 项）— chaos skill 可消费
├── assessment-report.md         # 人类可读报告（Markdown）
├── assessment-report.html       # HTML 报告（内联 CSS，可独立打开）
└── remediation-commands.sh      # 一键修复脚本（可执行的 kubectl/aws 命令）

assessment.json 结构

{
  "schema_version": "1.0",
  "cluster_name": "my-cluster",
  "region": "ap-northeast-1",
  "kubernetes_version": "1.32",
  "timestamp": "2026-04-03T08:00:00Z",
  "summary": {
    "total_checks": 28,
    "passed": 20,
    "failed": 6,
    "info": 2,
    "compliance_score": 71.4
  },
  "checks": [
    {
      "id": "A2",
      "name": "Run Multiple Replicas",
      "category": "application",
      "severity": "critical",
      "status": "FAIL",
      "findings": ["..."],
      "resources_affected": ["..."],
      "remediation": "Set spec.replicas > 1 for all production workloads.",
      "cost_impact": "+1 Pod per workload — doubles CPU/memory; may trigger additional node"
    }
  ],
  "experiment_recommendations": [ ... ]
}

与 chaos-engineering-on-aws 的集成

assessment.json 作为 chaos skill 步骤 1 的 方式 3 输入：

chaos-engineering-on-aws 步骤 1 — 输入源：
  方式 1: aws-resilience-modeling 报告     → AWS 资源级风险
  方式 2: 独立 chaos-input 文件           → 手动指定
  方式 3: eks-resilience-checker 评估      → K8s 配置风险（新增）

chaos skill 读取 assessment.json 中的 experiment_recommendations，按优先级排序（P0 > P1 > P2），将每项映射到 fault-catalog.yaml 中的故障类型。

四步工作流

示例报告

参见 examples/petsite-assessment.md 了解 PetSite EKS 集群的评估报告样例。

目录结构

eks-resilience-checker/
├── SKILL.md                            # 入口（语言检测 → 分流）
├── SKILL_EN.md / SKILL_ZH.md          # Skill 指令（双语，~226/221 行）
├── README.md / README_zh.md            # 说明文档（双语）
├── references/                         # 参考文档（Agent 按需加载）
│   ├── EKS-Resiliency-Checkpoints.md / _zh.md  # 26 项检查描述和原理
│   ├── check-commands.md / _zh.md      # 每项检查的 kubectl/aws 命令 + PASS/FAIL 标准
│   ├── eks-resiliency-checks-mcp.md / _zh.md   # MCP 方式执行检查（替代方案）
│   ├── remediation-templates.md / _zh.md        # 修复命令模板（含 YAML 示例）
│   ├── fail-to-experiment-mapping.md / _zh.md   # FAIL → 混沌实验映射表
│   └── eks-auth-setup.md / _zh.md      # EKS 认证配置指南（两种方式）
├── scripts/
│   ├── assess.sh                       # 自动化 26 项检查评估脚本（可独立运行）
│   └── README.md                       # 脚本使用说明
├── examples/
│   └── petsite-assessment.md           # PetSite 集群评估示例
└── doc/                                # 内部开发文档（Agent 不加载）
    └── prd.md                          # 产品需求文档

**中文** | English

EKS Resilience Checker

An AI-powered Agent Skill that performs automated resilience assessment of Amazon EKS clusters against 26 best practice checks covering Application workloads (A1-A14), Control Plane (C1-C5), and Data Plane (D1-D7). Outputs structured results that can directly feed into the chaos-engineering-on-aws skill to drive chaos experiments.

How It Fits — Resilience Lifecycle

┌──────────────────────────────────────────────────────────────────────────────────────┐
│                         AWS Resilience Lifecycle Framework                            │
│                                                                                      │
│  Stage 1: Set Objectives  Stage 2: Design & Implement  Stage 3: Evaluate & Test      │
│  ┌───────────────────┐    ┌────────────────────────┐   ┌──────────────────────────┐  │
│  │ aws-rma-           │    │ aws-resilience-         │   │ chaos-engineering-on-aws  │  │
│  │ assessment          │───>│ modeling                │──>│ Chaos experiments +       │  │
│  │ "Where are we?"    │    │ "What could go wrong?"  │   │ metrics + log analysis    │  │
│  └───────────────────┘    └────────────────────────┘   └───────────┬──────────────┘  │
│                                      ^                              │                 │
│                                      │    ┌─────────────────────────┴──────────────┐  │
│                                      │    │ eks-resilience-checker (this Skill)     │  │
│                                      │    │ 1. 26 K8s resilience checks            │  │
│                                      │    │ 2. FAIL -> experiment recommendations  │  │
│                                      │    └────────────────────────────────────────┘  │
│                                      └──────────── Feedback Loop ────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────────────────┘

Installation

One-Line Install (Recommended)

# Install this skill to your AI agent
npx skills add aws-samples/sample-aws-resilience-skill --skill eks-resilience-checker

# Install all 4 resilience skills
npx skills add aws-samples/sample-aws-resilience-skill --skill '*'

# Install to a specific agent
npx skills add aws-samples/sample-aws-resilience-skill --skill eks-resilience-checker -a claude-code

# Global install (available across projects)
npx skills add aws-samples/sample-aws-resilience-skill --skill eks-resilience-checker -g

Manual Install

git clone https://github.com/aws-samples/sample-aws-resilience-skill.git
ln -s $(pwd)/sample-aws-resilience-skill/eks-resilience-checker ~/.claude/skills/eks-resilience-checker

Quick Start

1. Configure kubectl access to your EKS cluster 2. Tell your AI agent: "Run EKS resilience assessment on my cluster" 3. Review the generated report in output/

Prerequisites

MCP Server (Optional Enhancement)

When MCP is unavailable, the skill falls back to kubectl + aws CLI.

{
  "mcpServers": {
    "awslabs.eks-mcp-server": {
      "command": "uvx",
      "args": ["awslabs.eks-mcp-server@latest"],
      "env": { "AWS_REGION": "ap-northeast-1", "FASTMCP_LOG_LEVEL": "ERROR" }
    }
  }
}

Check Categories

Application Checks (A1-A14)

Control Plane Checks (C1-C5)

Data Plane Checks (D1-D7)

Output Files

output/
├── assessment.json              # Structured results (26 checks) — consumable by chaos skill
├── assessment-report.md         # Human-readable report (Markdown)
├── assessment-report.html       # HTML report (inline CSS, standalone)
└── remediation-commands.sh      # One-click fix script (executable kubectl/aws commands)

assessment.json Structure

{
  "schema_version": "1.0",
  "cluster_name": "my-cluster",
  "region": "ap-northeast-1",
  "kubernetes_version": "1.32",
  "timestamp": "2026-04-03T08:00:00Z",
  "summary": {
    "total_checks": 28,
    "passed": 20,
    "failed": 6,
    "info": 2,
    "compliance_score": 71.4
  },
  "checks": [
    {
      "id": "A2",
      "name": "Run Multiple Replicas",
      "category": "application",
      "severity": "critical",
      "status": "FAIL",
      "findings": ["..."],
      "resources_affected": ["..."],
      "remediation": "Set spec.replicas > 1 for all production workloads.",
      "cost_impact": "+1 Pod per workload — doubles CPU/memory; may trigger additional node"
    }
  ],
  "experiment_recommendations": [ ... ]
}

Integration with chaos-engineering-on-aws

The assessment.json output serves as Method 3 input for the chaos skill's Step 1:

chaos-engineering-on-aws Step 1 — Input Sources:
  Method 1: aws-resilience-modeling report     → AWS resource-level risks
  Method 2: Standalone chaos-input file        → Manual specification
  Method 3: eks-resilience-checker assessment  → K8s configuration risks (NEW)

The chaos skill reads experiment_recommendations from assessment.json, sorts by priority (P0 > P1 > P2), and maps each to fault types in fault-catalog.yaml.

Four-Step Workflow

Example Report

See examples/petsite-assessment.md for a sample assessment report on a PetSite EKS cluster.

Directory Structure

eks-resilience-checker/
├── SKILL.md                            # Entry point (language detection → routing)
├── SKILL_EN.md / SKILL_ZH.md          # Instructions (bilingual, ~226/221 lines)
├── README.md / README_zh.md            # Documentation (bilingual)
├── references/                         # Reference docs (loaded on demand by Agent)
│   ├── EKS-Resiliency-Checkpoints.md / _zh.md  # 26-check descriptions & rationale
│   ├── check-commands.md / _zh.md      # kubectl/aws commands per check + PASS/FAIL criteria
│   ├── eks-resiliency-checks-mcp.md / _zh.md   # MCP-based check execution (alternative)
│   ├── remediation-templates.md / _zh.md        # Fix command templates with YAML examples
│   ├── fail-to-experiment-mapping.md / _zh.md   # FAIL → chaos experiment mapping table
│   └── eks-auth-setup.md / _zh.md      # EKS authentication setup guide (2 methods)
├── scripts/
│   ├── assess.sh                       # Automated 26-check assessment script (standalone)
│   └── README.md                       # Script usage guide
├── examples/
│   └── petsite-assessment.md           # PetSite cluster assessment example
└── doc/                                # Internal development docs (NOT loaded by Agent)
    └── prd.md                          # Product requirements

EKS 韧性检查命令

范围：本文件说明如何执行每项检查（精确命令 + PASS/FAIL 标准）。

关于每项检查做什么以及为什么重要，请参见 EKS-Resiliency-Checkpoints_zh.md。

每项检查的精确可运行命令、PASS 标准和严重级别。

排除系统命名空间：kube-system、kube-public、kube-node-lease。

---

应用检查 (A1–A14)

A1: Avoid Running Singleton Pods

严重级别: 🔴 关键 (Critical)

检查命令

kubectl get pods --all-namespaces -o json | jq '[.items[] | select((.metadata.ownerReferences // []) | length == 0) | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name}]'

PASS 标准

Empty array [] — no singleton pods found in non-system namespaces.

---

A2: Run Multiple Replicas

严重级别: 🔴 关键 (Critical)

检查命令

# Check Deployments
kubectl get deployments --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | select(.spec.replicas == 1) | {namespace: .metadata.namespace, name: .metadata.name, replicas: .spec.replicas}]'

# Check StatefulSets
kubectl get statefulsets --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | select(.spec.replicas == 1) | {namespace: .metadata.namespace, name: .metadata.name, replicas: .spec.replicas}]'

PASS 标准

Both commands return [] — all Deployments and StatefulSets have replicas > 1.

---

A3: Use Pod Anti-Affinity

严重级别: 🟡 警告 (Warning)

检查命令

kubectl get deployments --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | select(.spec.replicas > 1) | select(.spec.template.spec.affinity.podAntiAffinity == null) | {namespace: .metadata.namespace, name: .metadata.name, replicas: .spec.replicas}]'

PASS 标准

Empty array [] — all multi-replica Deployments have podAntiAffinity configured.

---

A4: Use Liveness Probes

严重级别: 🔴 关键 (Critical)

检查命令

# Check Deployments
kubectl get deployments --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name, containers_missing_probe: [.spec.template.spec.containers[] | select(.livenessProbe == null) | .name]} | select(.containers_missing_probe | length > 0)]'

# Check StatefulSets
kubectl get statefulsets --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name, containers_missing_probe: [.spec.template.spec.containers[] | select(.livenessProbe == null) | .name]} | select(.containers_missing_probe | length > 0)]'

# Check DaemonSets
kubectl get daemonsets --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name, containers_missing_probe: [.spec.template.spec.containers[] | select(.livenessProbe == null) | .name]} | select(.containers_missing_probe | length > 0)]'

PASS 标准

All three commands return [] — every container in every workload has a livenessProbe.

---

A5: Use Readiness Probes

严重级别: 🔴 关键 (Critical)

检查命令

# Check Deployments
kubectl get deployments --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name, containers_missing_probe: [.spec.template.spec.containers[] | select(.readinessProbe == null) | .name]} | select(.containers_missing_probe | length > 0)]'

# Check StatefulSets
kubectl get statefulsets --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name, containers_missing_probe: [.spec.template.spec.containers[] | select(.readinessProbe == null) | .name]} | select(.containers_missing_probe | length > 0)]'

# Check DaemonSets
kubectl get daemonsets --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name, containers_missing_probe: [.spec.template.spec.containers[] | select(.readinessProbe == null) | .name]} | select(.containers_missing_probe | length > 0)]'

PASS 标准

All three commands return [] — every container has a readinessProbe.

---

A6: Use Pod Disruption Budgets

严重级别: 🟡 警告 (Warning)

检查命令

# Step 1: List all PDBs and their selectors
kubectl get pdb --all-namespaces -o json | jq '[.items[] | {namespace: .metadata.namespace, name: .metadata.name, selector: .spec.selector.matchLabels}]'

# Step 2: Find multi-replica Deployments without a matching PDB
kubectl get deployments --all-namespaces -o json | jq --argjson pdbs "$(kubectl get pdb --all-namespaces -o json | jq '[.items[] | {namespace: .metadata.namespace, selector: .spec.selector.matchLabels}]')" '[.items[] | select(.metadata.namespace | test("^kube-") | not) | select(.spec.replicas > 1) | . as $deploy | {namespace: .metadata.namespace, name: .metadata.name} | select(. as $d | $pdbs | map(select(.namespace == $d.namespace)) | length == 0)]'

# Step 3: Find all StatefulSets without a matching PDB
kubectl get statefulsets --all-namespaces -o json | jq --argjson pdbs "$(kubectl get pdb --all-namespaces -o json | jq '[.items[] | {namespace: .metadata.namespace, selector: .spec.selector.matchLabels}]')" '[.items[] | select(.metadata.namespace | test("^kube-") | not) | . as $sts | {namespace: .metadata.namespace, name: .metadata.name} | select(. as $d | $pdbs | map(select(.namespace == $d.namespace)) | length == 0)]'

PASS 标准

Steps 2 and 3 return [] — all critical workloads (multi-replica Deployments and all StatefulSets) have corresponding PDBs.

---

A7: Run Kubernetes Metrics Server

严重级别: 🟡 警告 (Warning)

检查命令

# Check metrics-server deployment
kubectl get deployment metrics-server -n kube-system -o json 2>/dev/null | jq '{name: .metadata.name, available_replicas: .status.availableReplicas}'

# Verify metrics API is accessible
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" 2>/dev/null | jq '.items | length'

PASS 标准

• Deployment exists with availableReplicas >= 1
• Metrics API returns node count > 0

---

A8: Use Horizontal Pod Autoscaler

严重级别: 🟡 警告 (Warning)

检查命令

# List all HPAs
kubectl get hpa --all-namespaces -o json | jq '[.items[] | {namespace: .metadata.namespace, name: .metadata.name, target: .spec.scaleTargetRef.name}]'

# Find multi-replica workloads without HPA
kubectl get deployments --all-namespaces -o json | jq --argjson hpas "$(kubectl get hpa --all-namespaces -o json | jq '[.items[] | {namespace: .metadata.namespace, target: .spec.scaleTargetRef.name}]')" '[.items[] | select(.metadata.namespace | test("^kube-") | not) | select(.spec.replicas > 1) | {namespace: .metadata.namespace, name: .metadata.name} | select(. as $d | $hpas | map(select(.namespace == $d.namespace and .target == $d.name)) | length == 0)]'

PASS 标准

Second command returns [] — all multi-replica workloads have an HPA.

---

A9: Use Custom Metrics Scaling

Severity: 🟢 Info

检查命令

# Check custom metrics API
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" 2>/dev/null | jq '.resources | length'

# Check external metrics API
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" 2>/dev/null | jq '.resources | length'

# Check for Prometheus Adapter
kubectl get deployment -n kube-system -l app=prometheus-adapter -o json 2>/dev/null | jq '.items | length'

# Check for KEDA
kubectl get deployment -n keda -l app=keda-operator -o json 2>/dev/null | jq '.items | length'
kubectl get crd scaledobjects.keda.sh 2>/dev/null

# Find HPAs using custom or external metrics
kubectl get hpa --all-namespaces -o json | jq '[.items[] | select(.spec.metrics[]? | .type == "Pods" or .type == "Object" or .type == "External") | {namespace: .metadata.namespace, name: .metadata.name}]'

PASS 标准

At least one of the following is true:

• Custom metrics API is available
• Prometheus Adapter is deployed
• KEDA is installed
• HPAs with custom/external metrics exist

---

A10: Use Vertical Pod Autoscaler

Severity: 🟢 Info

检查命令

# Check VPA CRD
kubectl get crd verticalpodautoscalers.autoscaling.k8s.io 2>/dev/null

# Check VPA controller components
kubectl get deployment -n kube-system -l app=vpa-recommender -o json 2>/dev/null | jq '.items | length'
kubectl get deployment -n kube-system -l app=vpa-updater -o json 2>/dev/null | jq '.items | length'
kubectl get deployment -n kube-system -l app=vpa-admission-controller -o json 2>/dev/null | jq '.items | length'

# List existing VPA resources
kubectl get vpa --all-namespaces -o json 2>/dev/null | jq '[.items[] | {namespace: .metadata.namespace, name: .metadata.name, target: .spec.targetRef.name, mode: .spec.updatePolicy.updateMode}]'

# Check for Goldilocks
kubectl get deployment -n goldilocks -l app.kubernetes.io/name=goldilocks -o json 2>/dev/null | jq '.items | length'

PASS 标准

VPA CRD exists AND at least one VPA resource is configured, or Goldilocks is deployed.

---

A11: Use PreStop Hooks

严重级别: 🟡 警告 (Warning)

检查命令

# Check Deployments (DaemonSets intentionally excluded)
kubectl get deployments --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name, containers_missing_hook: [.spec.template.spec.containers[] | select(.lifecycle.preStop == null) | .name]} | select(.containers_missing_hook | length > 0)]'

# Check StatefulSets
kubectl get statefulsets --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name, containers_missing_hook: [.spec.template.spec.containers[] | select(.lifecycle.preStop == null) | .name]} | select(.containers_missing_hook | length > 0)]'

PASS 标准

Both commands return [] — all containers in Deployments and StatefulSets have preStop hooks.

---

A12: Use a Service Mesh

Severity: 🟢 Info

检查命令

# Check for Istio
kubectl get namespace istio-system 2>/dev/null && echo "Istio namespace found"
kubectl get crd virtualservices.networking.istio.io 2>/dev/null && echo "Istio CRDs found"

# Check for Linkerd
kubectl get namespace linkerd 2>/dev/null && echo "Linkerd namespace found"
kubectl get crd serviceprofiles.linkerd.io 2>/dev/null && echo "Linkerd CRDs found"

# Check for Consul
kubectl get namespace consul 2>/dev/null && echo "Consul namespace found"

# Check for sidecar proxies in application pods
kubectl get pods --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | select(.spec.containers | length > 1) | select(.spec.containers[].name | test("istio-proxy|linkerd-proxy|envoy-sidecar|consul-sidecar")) | {namespace: .metadata.namespace, name: .metadata.name}] | length'

PASS 标准

Any one of the service mesh namespaces/CRDs exists, or sidecar proxies are detected.

---

A13: Monitor Your Applications

严重级别: 🟡 警告 (Warning)

检查命令

# Check for Prometheus stack
kubectl get deployment -n monitoring -l app.kubernetes.io/name=prometheus -o json 2>/dev/null | jq '.items | length'
kubectl get crd prometheuses.monitoring.coreos.com 2>/dev/null && echo "Prometheus Operator CRD found"

# Check for CloudWatch Container Insights
kubectl get daemonset cloudwatch-agent -n amazon-cloudwatch 2>/dev/null && echo "CloudWatch agent found"
kubectl get daemonset fluent-bit -n amazon-cloudwatch 2>/dev/null && echo "Container Insights Fluent Bit found"

# Check for third-party monitoring
kubectl get pods --all-namespaces -o json | jq '[.items[] | select(.metadata.name | test("datadog|newrelic|dynatrace")) | {namespace: .metadata.namespace, name: .metadata.name}]'

PASS 标准

At least one monitoring solution detected (Prometheus, CloudWatch Container Insights, or third-party agent).

---

A14: Use Centralized Logging

严重级别: 🟡 警告 (Warning)

检查命令

# Check for Fluent Bit / Fluentd
kubectl get daemonset --all-namespaces -o json | jq '[.items[] | select(.metadata.name | test("fluent-bit|fluentd|fluent")) | {namespace: .metadata.namespace, name: .metadata.name}]'

# Check for CloudWatch Logs agent
kubectl get daemonset --all-namespaces -o json | jq '[.items[] | select(.metadata.name | test("cloudwatch")) | {namespace: .metadata.namespace, name: .metadata.name}]'

# Check for Elasticsearch / OpenSearch
kubectl get deployment --all-namespaces -o json | jq '[.items[] | select(.metadata.name | test("elasticsearch|opensearch|kibana")) | {namespace: .metadata.namespace, name: .metadata.name}]'

# Check for Loki
kubectl get deployment --all-namespaces -o json | jq '[.items[] | select(.metadata.name | test("loki")) | {namespace: .metadata.namespace, name: .metadata.name}]'

PASS 标准

At least one logging solution detected (Fluent Bit/Fluentd, CloudWatch Logs, Elasticsearch/OpenSearch, or Loki).

---

控制平面检查 (C1–C5)

C1: Monitor Control Plane Logs

严重级别: 🟡 警告 (Warning)

检查命令

CLUSTER_NAME="<cluster-name>"
REGION="ap-northeast-1"

aws eks describe-cluster --name "$CLUSTER_NAME" --region "$REGION" --query 'cluster.logging.clusterLogging[?enabled==`true`].types[]' --output json

PASS 标准

Output includes "api" in the enabled log types list (e.g., ["api", "audit", "authenticator"]).

---

C2: Cluster Authentication

严重级别: 🟡 警告 (Warning)

检查命令

CLUSTER_NAME="<cluster-name>"
REGION="ap-northeast-1"

# Method 1: Check EKS Access Entries (modern)
aws eks list-access-entries --cluster-name "$CLUSTER_NAME" --region "$REGION" --output json 2>/dev/null

# Method 2: Check aws-auth ConfigMap (traditional)
kubectl get configmap aws-auth -n kube-system -o json 2>/dev/null | jq '{mapRoles: .data.mapRoles, mapUsers: .data.mapUsers}'

PASS 标准

Either access entries exist (non-empty accessEntries list) OR aws-auth ConfigMap has mapRoles/mapUsers configured.

---

C3: Running Large Clusters

Severity: 🟢 Info

检查命令

# Count total services
SERVICE_COUNT=$(kubectl get services --all-namespaces --no-headers 2>/dev/null | wc -l)
echo "Total services: $SERVICE_COUNT"

# If > 1000, check kube-proxy mode
kubectl get configmap kube-proxy-config -n kube-system -o json 2>/dev/null | jq '.data."config"' | grep -o '"mode":"[^"]*"'

# Check VPC CNI WARM_IP_TARGET
kubectl get daemonset aws-node -n kube-system -o json 2>/dev/null | jq '[.spec.template.spec.containers[0].env[] | select(.name | test("WARM_IP_TARGET|WARM_ENI_TARGET|MINIMUM_IP_TARGET"))]'

PASS 标准

• If service count < 1000: automatic PASS (no optimization needed)
• If service count >= 1000: kube-proxy mode is "ipvs" AND WARM_IP_TARGET is set

---

C4: EKS Control Plane Endpoint Access Control

严重级别: 🔴 关键 (Critical)

检查命令

CLUSTER_NAME="<cluster-name>"
REGION="ap-northeast-1"

aws eks describe-cluster --name "$CLUSTER_NAME" --region "$REGION" --query 'cluster.resourcesVpcConfig.{endpointPublicAccess: endpointPublicAccess, endpointPrivateAccess: endpointPrivateAccess, publicAccessCidrs: publicAccessCidrs}' --output json

PASS 标准

One of:

• endpointPublicAccess: false (fully private) — PASS
• endpointPublicAccess: true AND publicAccessCidrs does NOT contain "0.0.0.0/0" — PASS
• endpointPublicAccess: true AND publicAccessCidrs contains "0.0.0.0/0" — FAIL

---

C5: Avoid Catch-All Admission Webhooks

严重级别: 🟡 警告 (Warning)

检查命令

# Check MutatingWebhookConfigurations
kubectl get mutatingwebhookconfigurations -o json | jq '[.items[] | {name: .metadata.name, webhooks: [.webhooks[] | select((.namespaceSelector == null) and (.objectSelector == null) and (.rules[]? | (.apiGroups[]? == "*") or (.apiVersions[]? == "*") or (.resources[]? == "*"))) | {name: .name, rules: .rules}]} | select(.webhooks | length > 0)]'

# Check ValidatingWebhookConfigurations
kubectl get validatingwebhookconfigurations -o json | jq '[.items[] | {name: .metadata.name, webhooks: [.webhooks[] | select((.namespaceSelector == null) and (.objectSelector == null) and (.rules[]? | (.apiGroups[]? == "*") or (.apiVersions[]? == "*") or (.resources[]? == "*"))) | {name: .name, rules: .rules}]} | select(.webhooks | length > 0)]'

PASS 标准

Both commands return [] — no webhooks with overly broad wildcard rules and missing selectors.

---

数据平面检查 (D1–D7)

D1: Use Kubernetes Cluster Autoscaler or Karpenter

严重级别: 🔴 关键 (Critical)

检查命令

# Check for Cluster Autoscaler
kubectl get deployment -n kube-system -l app=cluster-autoscaler -o json 2>/dev/null | jq '.items | length'

# Check for Karpenter
kubectl get namespace karpenter 2>/dev/null && echo "Karpenter namespace found"
kubectl get deployment -n karpenter -o json 2>/dev/null | jq '[.items[] | {name: .metadata.name, available: .status.availableReplicas}]'
kubectl get crd nodepools.karpenter.sh 2>/dev/null && echo "Karpenter CRDs found"
kubectl get crd ec2nodeclasses.karpenter.k8s.aws 2>/dev/null && echo "Karpenter AWS CRDs found"

PASS 标准

Either Cluster Autoscaler deployment exists with availableReplicas >= 1, OR Karpenter namespace/CRDs/deployments exist.

---

D2: Worker Nodes Spread Across Multiple AZs

严重级别: 🔴 关键 (Critical)

检查命令

# List nodes with AZ labels
kubectl get nodes -o json | jq '[.items[] | {name: .metadata.name, az: .metadata.labels["topology.kubernetes.io/zone"]}] | group_by(.az) | map({az: .[0].az, count: length})'

# Check AZ balance (variance within 20%)
kubectl get nodes -o json | jq '[.items[] | .metadata.labels["topology.kubernetes.io/zone"]] | group_by(.) | map({az: .[0], count: length}) | (map(.count) | (max - min) / max * 100) as $variance | {az_distribution: ., variance_percent: $variance, balanced: ($variance <= 20)}'

PASS 标准

• Nodes are spread across 2+ AZs
• Distribution variance is ≤ 20% (balanced: true)

---

D3: Configure Resource Requests/Limits

严重级别: 🔴 关键 (Critical)

检查命令

kubectl get deployments --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name, containers_missing_resources: [.spec.template.spec.containers[] | {name: .name, has_cpu_request: (.resources.requests.cpu != null), has_cpu_limit: (.resources.limits.cpu != null), has_mem_request: (.resources.requests.memory != null), has_mem_limit: (.resources.limits.memory != null)} | select(.has_cpu_request == false or .has_cpu_limit == false or .has_mem_request == false or .has_mem_limit == false)]} | select(.containers_missing_resources | length > 0)]'

PASS 标准

Empty array [] — all containers in all Deployments have CPU and memory requests AND limits.

---

D4: Namespace ResourceQuotas

严重级别: 🟡 警告 (Warning)

检查命令

# List user namespaces without ResourceQuota
kubectl get namespaces -o json | jq '[.items[] | select(.metadata.name | test("^kube-|^amazon-|^istio-|^linkerd") | not) | .metadata.name]' | while read -r ns; do
  ns=$(echo "$ns" | tr -d '"[] ')
  [ -z "$ns" ] && continue
  count=$(kubectl get resourcequota -n "$ns" --no-headers 2>/dev/null | wc -l)
  if [ "$count" -eq 0 ]; then echo "MISSING ResourceQuota: $ns"; fi
done

# Alternative single-command approach
for ns in $(kubectl get namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | grep -v -E '^kube-|^amazon-'); do
  if [ "$(kubectl get resourcequota -n "$ns" --no-headers 2>/dev/null | wc -l)" -eq 0 ]; then
    echo "MISSING: $ns"
  fi
done

PASS 标准

No output — all user namespaces (including default) have at least one ResourceQuota.

---

D5: Namespace LimitRanges

严重级别: 🟡 警告 (Warning)

检查命令

for ns in $(kubectl get namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | grep -v -E '^kube-|^amazon-'); do
  if [ "$(kubectl get limitrange -n "$ns" --no-headers 2>/dev/null | wc -l)" -eq 0 ]; then
    echo "MISSING: $ns"
  fi
done

PASS 标准

No output — all user namespaces have at least one LimitRange.

---

D6: Monitor CoreDNS Metrics

严重级别: 🟡 警告 (Warning)

检查命令

# Verify CoreDNS has metrics port 9153
kubectl get deployment coredns -n kube-system -o json 2>/dev/null | jq '[.spec.template.spec.containers[0].ports[] | select(.containerPort == 9153)]'

# Check for ServiceMonitor targeting CoreDNS
kubectl get servicemonitor -n kube-system -o json 2>/dev/null | jq '[.items[] | select(.spec.selector.matchLabels["k8s-app"] == "kube-dns" or .metadata.name | test("coredns|dns")) | {name: .metadata.name}]'

# Check for Prometheus scrape annotations on CoreDNS service
kubectl get service kube-dns -n kube-system -o json 2>/dev/null | jq '.metadata.annotations | with_entries(select(.key | test("prometheus")))'

PASS 标准

CoreDNS metrics port 9153 exists AND (ServiceMonitor targeting CoreDNS exists OR Prometheus scrape annotations present).

---

D7: CoreDNS Configuration

Severity: 🟢 Info

检查命令

CLUSTER_NAME="<cluster-name>"
REGION="ap-northeast-1"

# Check if EKS Auto Mode
aws eks describe-cluster --name "$CLUSTER_NAME" --region "$REGION" --query 'cluster.computeConfig.enabled' --output text 2>/dev/null

# Check if CoreDNS is an EKS managed add-on
aws eks describe-addon --cluster-name "$CLUSTER_NAME" --addon-name coredns --region "$REGION" --query 'addon.{status: status, version: addonVersion, configurationSchema: configurationSchema}' --output json 2>/dev/null

# Verify CoreDNS deployment exists
kubectl get deployment coredns -n kube-system -o json 2>/dev/null | jq '{name: .metadata.name, replicas: .spec.replicas, available: .status.availableReplicas}'

PASS 标准

• EKS Auto Mode enabled → automatic PASS
• OR CoreDNS is an EKS managed add-on (describe-addon returns status)
• FAIL if CoreDNS is self-managed in non-auto-mode clusters

EKS Resilience Check Commands

Scope: This file covers how to execute each check (exact commands + PASS/FAIL criteria).

For what each check does and why it matters, see EKS-Resiliency-Checkpoints.md.

Exact runnable commands, PASS criteria, and severity for each of the 26 checks.

System namespaces excluded: kube-system, kube-public, kube-node-lease.

---

Application Checks (A1–A14)

A1: Avoid Running Singleton Pods

Severity: 🔴 Critical

Check Command

kubectl get pods --all-namespaces -o json | jq '[.items[] | select((.metadata.ownerReferences // []) | length == 0) | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name}]'

PASS Criteria

Empty array [] — no singleton pods found in non-system namespaces.

---

A2: Run Multiple Replicas

Severity: 🔴 Critical

Check Command

# Check Deployments
kubectl get deployments --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | select(.spec.replicas == 1) | {namespace: .metadata.namespace, name: .metadata.name, replicas: .spec.replicas}]'

# Check StatefulSets
kubectl get statefulsets --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | select(.spec.replicas == 1) | {namespace: .metadata.namespace, name: .metadata.name, replicas: .spec.replicas}]'

PASS Criteria

Both commands return [] — all Deployments and StatefulSets have replicas > 1.

---

A3: Use Pod Anti-Affinity

Severity: 🟡 Warning

Check Command

kubectl get deployments --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | select(.spec.replicas > 1) | select(.spec.template.spec.affinity.podAntiAffinity == null) | {namespace: .metadata.namespace, name: .metadata.name, replicas: .spec.replicas}]'

PASS Criteria

Empty array [] — all multi-replica Deployments have podAntiAffinity configured.

---

A4: Use Liveness Probes

Severity: 🔴 Critical

Check Command

# Check Deployments
kubectl get deployments --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name, containers_missing_probe: [.spec.template.spec.containers[] | select(.livenessProbe == null) | .name]} | select(.containers_missing_probe | length > 0)]'

# Check StatefulSets
kubectl get statefulsets --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name, containers_missing_probe: [.spec.template.spec.containers[] | select(.livenessProbe == null) | .name]} | select(.containers_missing_probe | length > 0)]'

# Check DaemonSets
kubectl get daemonsets --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name, containers_missing_probe: [.spec.template.spec.containers[] | select(.livenessProbe == null) | .name]} | select(.containers_missing_probe | length > 0)]'

PASS Criteria

All three commands return [] — every container in every workload has a livenessProbe.

---

A5: Use Readiness Probes

Severity: 🔴 Critical

Check Command

# Check Deployments
kubectl get deployments --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name, containers_missing_probe: [.spec.template.spec.containers[] | select(.readinessProbe == null) | .name]} | select(.containers_missing_probe | length > 0)]'

# Check StatefulSets
kubectl get statefulsets --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name, containers_missing_probe: [.spec.template.spec.containers[] | select(.readinessProbe == null) | .name]} | select(.containers_missing_probe | length > 0)]'

# Check DaemonSets
kubectl get daemonsets --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name, containers_missing_probe: [.spec.template.spec.containers[] | select(.readinessProbe == null) | .name]} | select(.containers_missing_probe | length > 0)]'

PASS Criteria

All three commands return [] — every container has a readinessProbe.

---

A6: Use Pod Disruption Budgets

Severity: 🟡 Warning

Check Command

# Step 1: List all PDBs and their selectors
kubectl get pdb --all-namespaces -o json | jq '[.items[] | {namespace: .metadata.namespace, name: .metadata.name, selector: .spec.selector.matchLabels}]'

# Step 2: Find multi-replica Deployments without a matching PDB
kubectl get deployments --all-namespaces -o json | jq --argjson pdbs "$(kubectl get pdb --all-namespaces -o json | jq '[.items[] | {namespace: .metadata.namespace, selector: .spec.selector.matchLabels}]')" '[.items[] | select(.metadata.namespace | test("^kube-") | not) | select(.spec.replicas > 1) | . as $deploy | {namespace: .metadata.namespace, name: .metadata.name} | select(. as $d | $pdbs | map(select(.namespace == $d.namespace)) | length == 0)]'

# Step 3: Find all StatefulSets without a matching PDB
kubectl get statefulsets --all-namespaces -o json | jq --argjson pdbs "$(kubectl get pdb --all-namespaces -o json | jq '[.items[] | {namespace: .metadata.namespace, selector: .spec.selector.matchLabels}]')" '[.items[] | select(.metadata.namespace | test("^kube-") | not) | . as $sts | {namespace: .metadata.namespace, name: .metadata.name} | select(. as $d | $pdbs | map(select(.namespace == $d.namespace)) | length == 0)]'

PASS Criteria

Steps 2 and 3 return [] — all critical workloads (multi-replica Deployments and all StatefulSets) have corresponding PDBs.

---

A7: Run Kubernetes Metrics Server

Severity: 🟡 Warning

Check Command

# Check metrics-server deployment
kubectl get deployment metrics-server -n kube-system -o json 2>/dev/null | jq '{name: .metadata.name, available_replicas: .status.availableReplicas}'

# Verify metrics API is accessible
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" 2>/dev/null | jq '.items | length'

PASS Criteria

• Deployment exists with availableReplicas >= 1
• Metrics API returns node count > 0

---

A8: Use Horizontal Pod Autoscaler

Severity: 🟡 Warning

Check Command

# List all HPAs
kubectl get hpa --all-namespaces -o json | jq '[.items[] | {namespace: .metadata.namespace, name: .metadata.name, target: .spec.scaleTargetRef.name}]'

# Find multi-replica workloads without HPA
kubectl get deployments --all-namespaces -o json | jq --argjson hpas "$(kubectl get hpa --all-namespaces -o json | jq '[.items[] | {namespace: .metadata.namespace, target: .spec.scaleTargetRef.name}]')" '[.items[] | select(.metadata.namespace | test("^kube-") | not) | select(.spec.replicas > 1) | {namespace: .metadata.namespace, name: .metadata.name} | select(. as $d | $hpas | map(select(.namespace == $d.namespace and .target == $d.name)) | length == 0)]'

PASS Criteria

Second command returns [] — all multi-replica workloads have an HPA.

---

A9: Use Custom Metrics Scaling

Severity: 🟢 Info

Check Command

# Check custom metrics API
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" 2>/dev/null | jq '.resources | length'

# Check external metrics API
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" 2>/dev/null | jq '.resources | length'

# Check for Prometheus Adapter
kubectl get deployment -n kube-system -l app=prometheus-adapter -o json 2>/dev/null | jq '.items | length'

# Check for KEDA
kubectl get deployment -n keda -l app=keda-operator -o json 2>/dev/null | jq '.items | length'
kubectl get crd scaledobjects.keda.sh 2>/dev/null

# Find HPAs using custom or external metrics
kubectl get hpa --all-namespaces -o json | jq '[.items[] | select(.spec.metrics[]? | .type == "Pods" or .type == "Object" or .type == "External") | {namespace: .metadata.namespace, name: .metadata.name}]'

PASS Criteria

At least one of the following is true:

• Custom metrics API is available
• Prometheus Adapter is deployed
• KEDA is installed
• HPAs with custom/external metrics exist

---

A10: Use Vertical Pod Autoscaler

Severity: 🟢 Info

Check Command

# Check VPA CRD
kubectl get crd verticalpodautoscalers.autoscaling.k8s.io 2>/dev/null

# Check VPA controller components
kubectl get deployment -n kube-system -l app=vpa-recommender -o json 2>/dev/null | jq '.items | length'
kubectl get deployment -n kube-system -l app=vpa-updater -o json 2>/dev/null | jq '.items | length'
kubectl get deployment -n kube-system -l app=vpa-admission-controller -o json 2>/dev/null | jq '.items | length'

# List existing VPA resources
kubectl get vpa --all-namespaces -o json 2>/dev/null | jq '[.items[] | {namespace: .metadata.namespace, name: .metadata.name, target: .spec.targetRef.name, mode: .spec.updatePolicy.updateMode}]'

# Check for Goldilocks
kubectl get deployment -n goldilocks -l app.kubernetes.io/name=goldilocks -o json 2>/dev/null | jq '.items | length'

PASS Criteria

VPA CRD exists AND at least one VPA resource is configured, or Goldilocks is deployed.

---

A11: Use PreStop Hooks

Severity: 🟡 Warning

Check Command

# Check Deployments (DaemonSets intentionally excluded)
kubectl get deployments --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name, containers_missing_hook: [.spec.template.spec.containers[] | select(.lifecycle.preStop == null) | .name]} | select(.containers_missing_hook | length > 0)]'

# Check StatefulSets
kubectl get statefulsets --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name, containers_missing_hook: [.spec.template.spec.containers[] | select(.lifecycle.preStop == null) | .name]} | select(.containers_missing_hook | length > 0)]'

PASS Criteria

Both commands return [] — all containers in Deployments and StatefulSets have preStop hooks.

---

A12: Use a Service Mesh

Severity: 🟢 Info

Check Command

# Check for Istio
kubectl get namespace istio-system 2>/dev/null && echo "Istio namespace found"
kubectl get crd virtualservices.networking.istio.io 2>/dev/null && echo "Istio CRDs found"

# Check for Linkerd
kubectl get namespace linkerd 2>/dev/null && echo "Linkerd namespace found"
kubectl get crd serviceprofiles.linkerd.io 2>/dev/null && echo "Linkerd CRDs found"

# Check for Consul
kubectl get namespace consul 2>/dev/null && echo "Consul namespace found"

# Check for sidecar proxies in application pods
kubectl get pods --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | select(.spec.containers | length > 1) | select(.spec.containers[].name | test("istio-proxy|linkerd-proxy|envoy-sidecar|consul-sidecar")) | {namespace: .metadata.namespace, name: .metadata.name}] | length'

PASS Criteria

Any one of the service mesh namespaces/CRDs exists, or sidecar proxies are detected.

---

A13: Monitor Your Applications

Severity: 🟡 Warning

Check Command

# Check for Prometheus stack
kubectl get deployment -n monitoring -l app.kubernetes.io/name=prometheus -o json 2>/dev/null | jq '.items | length'
kubectl get crd prometheuses.monitoring.coreos.com 2>/dev/null && echo "Prometheus Operator CRD found"

# Check for CloudWatch Container Insights
kubectl get daemonset cloudwatch-agent -n amazon-cloudwatch 2>/dev/null && echo "CloudWatch agent found"
kubectl get daemonset fluent-bit -n amazon-cloudwatch 2>/dev/null && echo "Container Insights Fluent Bit found"

# Check for third-party monitoring
kubectl get pods --all-namespaces -o json | jq '[.items[] | select(.metadata.name | test("datadog|newrelic|dynatrace")) | {namespace: .metadata.namespace, name: .metadata.name}]'

PASS Criteria

At least one monitoring solution detected (Prometheus, CloudWatch Container Insights, or third-party agent).

---

A14: Use Centralized Logging

Severity: 🟡 Warning

Check Command

# Check for Fluent Bit / Fluentd
kubectl get daemonset --all-namespaces -o json | jq '[.items[] | select(.metadata.name | test("fluent-bit|fluentd|fluent")) | {namespace: .metadata.namespace, name: .metadata.name}]'

# Check for CloudWatch Logs agent
kubectl get daemonset --all-namespaces -o json | jq '[.items[] | select(.metadata.name | test("cloudwatch")) | {namespace: .metadata.namespace, name: .metadata.name}]'

# Check for Elasticsearch / OpenSearch
kubectl get deployment --all-namespaces -o json | jq '[.items[] | select(.metadata.name | test("elasticsearch|opensearch|kibana")) | {namespace: .metadata.namespace, name: .metadata.name}]'

# Check for Loki
kubectl get deployment --all-namespaces -o json | jq '[.items[] | select(.metadata.name | test("loki")) | {namespace: .metadata.namespace, name: .metadata.name}]'

PASS Criteria

At least one logging solution detected (Fluent Bit/Fluentd, CloudWatch Logs, Elasticsearch/OpenSearch, or Loki).

---

Control Plane Checks (C1–C5)

C1: Monitor Control Plane Logs

Severity: 🟡 Warning

Check Command

CLUSTER_NAME="<cluster-name>"
REGION="ap-northeast-1"

aws eks describe-cluster --name "$CLUSTER_NAME" --region "$REGION" --query 'cluster.logging.clusterLogging[?enabled==`true`].types[]' --output json

PASS Criteria

Output includes "api" in the enabled log types list (e.g., ["api", "audit", "authenticator"]).

---

C2: Cluster Authentication

Severity: 🟡 Warning

Check Command

CLUSTER_NAME="<cluster-name>"
REGION="ap-northeast-1"

# Method 1: Check EKS Access Entries (modern)
aws eks list-access-entries --cluster-name "$CLUSTER_NAME" --region "$REGION" --output json 2>/dev/null

# Method 2: Check aws-auth ConfigMap (traditional)
kubectl get configmap aws-auth -n kube-system -o json 2>/dev/null | jq '{mapRoles: .data.mapRoles, mapUsers: .data.mapUsers}'

PASS Criteria

Either access entries exist (non-empty accessEntries list) OR aws-auth ConfigMap has mapRoles/mapUsers configured.

---

C3: Running Large Clusters

Severity: 🟢 Info

Check Command

# Count total services
SERVICE_COUNT=$(kubectl get services --all-namespaces --no-headers 2>/dev/null | wc -l)
echo "Total services: $SERVICE_COUNT"

# If > 1000, check kube-proxy mode
kubectl get configmap kube-proxy-config -n kube-system -o json 2>/dev/null | jq '.data."config"' | grep -o '"mode":"[^"]*"'

# Check VPC CNI WARM_IP_TARGET
kubectl get daemonset aws-node -n kube-system -o json 2>/dev/null | jq '[.spec.template.spec.containers[0].env[] | select(.name | test("WARM_IP_TARGET|WARM_ENI_TARGET|MINIMUM_IP_TARGET"))]'

PASS Criteria

• If service count < 1000: automatic PASS (no optimization needed)
• If service count >= 1000: kube-proxy mode is "ipvs" AND WARM_IP_TARGET is set

---

C4: EKS Control Plane Endpoint Access Control

Severity: 🔴 Critical

Check Command

CLUSTER_NAME="<cluster-name>"
REGION="ap-northeast-1"

aws eks describe-cluster --name "$CLUSTER_NAME" --region "$REGION" --query 'cluster.resourcesVpcConfig.{endpointPublicAccess: endpointPublicAccess, endpointPrivateAccess: endpointPrivateAccess, publicAccessCidrs: publicAccessCidrs}' --output json

PASS Criteria

One of:

• endpointPublicAccess: false (fully private) — PASS
• endpointPublicAccess: true AND publicAccessCidrs does NOT contain "0.0.0.0/0" — PASS
• endpointPublicAccess: true AND publicAccessCidrs contains "0.0.0.0/0" — FAIL

---

C5: Avoid Catch-All Admission Webhooks

Severity: 🟡 Warning

Check Command

# Check MutatingWebhookConfigurations
kubectl get mutatingwebhookconfigurations -o json | jq '[.items[] | {name: .metadata.name, webhooks: [.webhooks[] | select((.namespaceSelector == null) and (.objectSelector == null) and (.rules[]? | (.apiGroups[]? == "*") or (.apiVersions[]? == "*") or (.resources[]? == "*"))) | {name: .name, rules: .rules}]} | select(.webhooks | length > 0)]'

# Check ValidatingWebhookConfigurations
kubectl get validatingwebhookconfigurations -o json | jq '[.items[] | {name: .metadata.name, webhooks: [.webhooks[] | select((.namespaceSelector == null) and (.objectSelector == null) and (.rules[]? | (.apiGroups[]? == "*") or (.apiVersions[]? == "*") or (.resources[]? == "*"))) | {name: .name, rules: .rules}]} | select(.webhooks | length > 0)]'

PASS Criteria

Both commands return [] — no webhooks with overly broad wildcard rules and missing selectors.

---

Data Plane Checks (D1–D7)

D1: Use Kubernetes Cluster Autoscaler or Karpenter

Severity: 🔴 Critical

Check Command

# Check for Cluster Autoscaler
kubectl get deployment -n kube-system -l app=cluster-autoscaler -o json 2>/dev/null | jq '.items | length'

# Check for Karpenter
kubectl get namespace karpenter 2>/dev/null && echo "Karpenter namespace found"
kubectl get deployment -n karpenter -o json 2>/dev/null | jq '[.items[] | {name: .metadata.name, available: .status.availableReplicas}]'
kubectl get crd nodepools.karpenter.sh 2>/dev/null && echo "Karpenter CRDs found"
kubectl get crd ec2nodeclasses.karpenter.k8s.aws 2>/dev/null && echo "Karpenter AWS CRDs found"

PASS Criteria

Either Cluster Autoscaler deployment exists with availableReplicas >= 1, OR Karpenter namespace/CRDs/deployments exist.

---

D2: Worker Nodes Spread Across Multiple AZs

Severity: 🔴 Critical

Check Command

# List nodes with AZ labels
kubectl get nodes -o json | jq '[.items[] | {name: .metadata.name, az: .metadata.labels["topology.kubernetes.io/zone"]}] | group_by(.az) | map({az: .[0].az, count: length})'

# Check AZ balance (variance within 20%)
kubectl get nodes -o json | jq '[.items[] | .metadata.labels["topology.kubernetes.io/zone"]] | group_by(.) | map({az: .[0], count: length}) | (map(.count) | (max - min) / max * 100) as $variance | {az_distribution: ., variance_percent: $variance, balanced: ($variance <= 20)}'

PASS Criteria

• Nodes are spread across 2+ AZs
• Distribution variance is ≤ 20% (balanced: true)

---

D3: Configure Resource Requests/Limits

Severity: 🔴 Critical

Check Command

kubectl get deployments --all-namespaces -o json | jq '[.items[] | select(.metadata.namespace | test("^kube-") | not) | {namespace: .metadata.namespace, name: .metadata.name, containers_missing_resources: [.spec.template.spec.containers[] | {name: .name, has_cpu_request: (.resources.requests.cpu != null), has_cpu_limit: (.resources.limits.cpu != null), has_mem_request: (.resources.requests.memory != null), has_mem_limit: (.resources.limits.memory != null)} | select(.has_cpu_request == false or .has_cpu_limit == false or .has_mem_request == false or .has_mem_limit == false)]} | select(.containers_missing_resources | length > 0)]'

PASS Criteria

Empty array [] — all containers in all Deployments have CPU and memory requests AND limits.

---

D4: Namespace ResourceQuotas

Severity: 🟡 Warning

Check Command

# List user namespaces without ResourceQuota
kubectl get namespaces -o json | jq '[.items[] | select(.metadata.name | test("^kube-|^amazon-|^istio-|^linkerd") | not) | .metadata.name]' | while read -r ns; do
  ns=$(echo "$ns" | tr -d '"[] ')
  [ -z "$ns" ] && continue
  count=$(kubectl get resourcequota -n "$ns" --no-headers 2>/dev/null | wc -l)
  if [ "$count" -eq 0 ]; then echo "MISSING ResourceQuota: $ns"; fi
done

# Alternative single-command approach
for ns in $(kubectl get namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | grep -v -E '^kube-|^amazon-'); do
  if [ "$(kubectl get resourcequota -n "$ns" --no-headers 2>/dev/null | wc -l)" -eq 0 ]; then
    echo "MISSING: $ns"
  fi
done

PASS Criteria

No output — all user namespaces (including default) have at least one ResourceQuota.

---

D5: Namespace LimitRanges

Severity: 🟡 Warning

Check Command

for ns in $(kubectl get namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | grep -v -E '^kube-|^amazon-'); do
  if [ "$(kubectl get limitrange -n "$ns" --no-headers 2>/dev/null | wc -l)" -eq 0 ]; then
    echo "MISSING: $ns"
  fi
done

PASS Criteria

No output — all user namespaces have at least one LimitRange.

---

D6: Monitor CoreDNS Metrics

Severity: 🟡 Warning

Check Command

# Verify CoreDNS has metrics port 9153
kubectl get deployment coredns -n kube-system -o json 2>/dev/null | jq '[.spec.template.spec.containers[0].ports[] | select(.containerPort == 9153)]'

# Check for ServiceMonitor targeting CoreDNS
kubectl get servicemonitor -n kube-system -o json 2>/dev/null | jq '[.items[] | select(.spec.selector.matchLabels["k8s-app"] == "kube-dns" or .metadata.name | test("coredns|dns")) | {name: .metadata.name}]'

# Check for Prometheus scrape annotations on CoreDNS service
kubectl get service kube-dns -n kube-system -o json 2>/dev/null | jq '.metadata.annotations | with_entries(select(.key | test("prometheus")))'

PASS Criteria

CoreDNS metrics port 9153 exists AND (ServiceMonitor targeting CoreDNS exists OR Prometheus scrape annotations present).

---

D7: CoreDNS Configuration

Severity: 🟢 Info

Check Command

CLUSTER_NAME="<cluster-name>"
REGION="ap-northeast-1"

# Check if EKS Auto Mode
aws eks describe-cluster --name "$CLUSTER_NAME" --region "$REGION" --query 'cluster.computeConfig.enabled' --output text 2>/dev/null

# Check if CoreDNS is an EKS managed add-on
aws eks describe-addon --cluster-name "$CLUSTER_NAME" --addon-name coredns --region "$REGION" --query 'addon.{status: status, version: addonVersion, configurationSchema: configurationSchema}' --output json 2>/dev/null

# Verify CoreDNS deployment exists
kubectl get deployment coredns -n kube-system -o json 2>/dev/null | jq '{name: .metadata.name, replicas: .spec.replicas, available: .status.availableReplicas}'

PASS Criteria

• EKS Auto Mode enabled → automatic PASS
• OR CoreDNS is an EKS managed add-on (describe-addon returns status)
• FAIL if CoreDNS is self-managed in non-auto-mode clusters

EKS 认证配置

kubectl 连接 EKS 集群有两种方式。

方式一：IAM kubeconfig（推荐）

使用 aws eks update-kubeconfig 生成 kubeconfig，每次请求时通过 aws eks get-token 获取令牌。

# 为目标集群生成 kubeconfig
aws eks update-kubeconfig --name {CLUSTER_NAME} --region {REGION}

# 验证访问
kubectl get nodes

如果用户已有 kubeconfig（如集群管理员提供的）：

# 指向已有 kubeconfig
export KUBECONFIG=/path/to/admin-kubeconfig
kubectl get nodes

# 或在命令中指定
kubectl --kubeconfig /path/to/admin-kubeconfig get nodes

在第 1 步中，询问用户："你是否已有 kubeconfig，还是需要我用 aws eks update-kubeconfig 生成一个？"

要求：

• AWS CLI 已安装并配置有效凭证（如需生成 kubeconfig）
• IAM 身份必须拥有 eks:DescribeCluster 权限
• IAM 身份必须在集群访问配置中映射（EKS Access Entries 或 aws-auth ConfigMap）

优点： 使用现有 AWS 凭证，标准 EKS 工作流，自动刷新令牌 缺点： 需要运行评估的机器上有 AWS CLI + IAM 凭证

方式二：静态 Service Account 令牌（受限环境）

创建具有只读权限的 Kubernetes ServiceAccount，生成自包含的 kubeconfig。运行时无需 AWS CLI 或 IAM 凭证。

# 1. 创建具有只读 RBAC 的 ServiceAccount
kubectl create serviceaccount eks-resilience-checker -n kube-system

# 2. 创建 ClusterRoleBinding（只读访问）
kubectl create clusterrolebinding eks-resilience-checker-readonly \
  --clusterrole=view \
  --serviceaccount=kube-system:eks-resilience-checker

# 3. 生成令牌（有效期 1 年）
TOKEN=$(kubectl create token eks-resilience-checker -n kube-system --duration=8760h)

# 4. 获取集群端点和 CA
ENDPOINT=$(aws eks describe-cluster --name {CLUSTER_NAME} --query 'cluster.endpoint' --output text)
CA_DATA=$(aws eks describe-cluster --name {CLUSTER_NAME} --query 'cluster.certificateAuthority.data' --output text)

# 5. 生成自包含 kubeconfig
kubectl config set-cluster eks-check --server=$ENDPOINT --certificate-authority-data=$CA_DATA --embed-certs=true --kubeconfig=./eks-resilience-kubeconfig
kubectl config set-credentials eks-checker --token=$TOKEN --kubeconfig=./eks-resilience-kubeconfig
kubectl config set-context eks-check --cluster=eks-check --user=eks-checker --kubeconfig=./eks-resilience-kubeconfig
kubectl config use-context eks-check --kubeconfig=./eks-resilience-kubeconfig

# 6. 使用生成的 kubeconfig
export KUBECONFIG=./eks-resilience-kubeconfig
kubectl get nodes

优点： 可移植，运行时无 AWS 依赖，最小权限（只读），适合 CI/CD 流水线 缺点： 令牌有固定到期时间（默认 1 年），需要续期，初始配置需要集群管理员权限

建议

• 交互式评估使用方式一
• CI/CD 流水线、自动化定期检查或无 AWS CLI 环境使用方式二

EKS Authentication Setup

Two methods to authenticate kubectl with an EKS cluster.

Method 1: IAM-based kubeconfig (Recommended)

Uses aws eks update-kubeconfig to generate a kubeconfig that obtains tokens via aws eks get-token on each request.

# Generate kubeconfig for the target cluster
aws eks update-kubeconfig --name {CLUSTER_NAME} --region {REGION}

# Verify access
kubectl get nodes

If the user already has a kubeconfig (e.g., provided by a cluster admin):

# Point to an existing kubeconfig
export KUBECONFIG=/path/to/admin-kubeconfig
kubectl get nodes

# Or specify per-command
kubectl --kubeconfig /path/to/admin-kubeconfig get nodes

In Step 1, ask the user: "Do you have an existing kubeconfig, or should I generate one with aws eks update-kubeconfig?"

Requirements:

• AWS CLI installed and configured with valid credentials (if generating kubeconfig)
• IAM identity must have eks:DescribeCluster permission
• IAM identity must be mapped in the cluster's access configuration (EKS Access Entries or aws-auth ConfigMap)

Pros: Uses existing AWS credentials, standard EKS workflow, automatic token refresh Cons: Requires AWS CLI + IAM credentials on the machine running the assessment

Method 2: Static Service Account Token (For restricted environments)

Creates a Kubernetes ServiceAccount with read-only permissions and generates a self-contained kubeconfig. No AWS CLI or IAM credentials required at runtime.

# 1. Create ServiceAccount with read-only RBAC
kubectl create serviceaccount eks-resilience-checker -n kube-system

# 2. Create ClusterRoleBinding (read-only access)
kubectl create clusterrolebinding eks-resilience-checker-readonly \
  --clusterrole=view \
  --serviceaccount=kube-system:eks-resilience-checker

# 3. Generate token (valid for 1 year)
TOKEN=$(kubectl create token eks-resilience-checker -n kube-system --duration=8760h)

# 4. Get cluster endpoint and CA
ENDPOINT=$(aws eks describe-cluster --name {CLUSTER_NAME} --query 'cluster.endpoint' --output text)
CA_DATA=$(aws eks describe-cluster --name {CLUSTER_NAME} --query 'cluster.certificateAuthority.data' --output text)

# 5. Generate self-contained kubeconfig
kubectl config set-cluster eks-check --server=$ENDPOINT --certificate-authority-data=$CA_DATA --embed-certs=true --kubeconfig=./eks-resilience-kubeconfig
kubectl config set-credentials eks-checker --token=$TOKEN --kubeconfig=./eks-resilience-kubeconfig
kubectl config set-context eks-check --cluster=eks-check --user=eks-checker --kubeconfig=./eks-resilience-kubeconfig
kubectl config use-context eks-check --kubeconfig=./eks-resilience-kubeconfig

# 6. Use the generated kubeconfig
export KUBECONFIG=./eks-resilience-kubeconfig
kubectl get nodes

Pros: Portable, no AWS dependency at runtime, least-privilege (read-only), works in CI/CD pipelines Cons: Token has fixed expiry (default 1 year), needs renewal, requires initial setup with cluster admin access

Recommendation

• Use Method 1 for interactive assessments
• Use Method 2 for CI/CD pipelines, automated periodic checks, or environments where AWS CLI is not available

assess.sh — EKS Resilience Assessment Script

Automated script that runs all 26 EKS resilience checks and generates reports.

Prerequisites

• kubectl configured with access to the target EKS cluster
• aws CLI configured with valid credentials
• jq installed
• Permissions: Kubernetes RBAC get/list on workload resources; IAM eks:DescribeCluster, eks:ListAddons

Usage

# Basic usage (auto-detect cluster from current context)
./scripts/assess.sh

# Specify cluster and region
./scripts/assess.sh --cluster-name my-cluster --region ap-northeast-1

# Specify target namespaces (comma-separated)
./scripts/assess.sh --cluster-name my-cluster --region ap-northeast-1 --namespaces "app1,app2,app3"

# Output to custom directory
./scripts/assess.sh --cluster-name my-cluster --output-dir ./my-output

Parameters

Output Files

Exit Codes

EKS Resilience Checker

Role Definition

You are a senior AWS EKS resilience assessment expert. You perform 26 automated checks across 3 categories — Application Workloads (A1-A14), Control Plane (C1-C5), and Data Plane (D1-D7) — against an Amazon EKS cluster. You output structured assessment results that can drive chaos experiments via chaos-engineering-on-aws.

Model Selection

Ask the user to select a model before starting:

• Sonnet 4.6 (default) — Faster, lower cost, suitable for routine assessments
• Opus 4.6 — Stronger reasoning, suitable for complex multi-cluster analysis

Default to Sonnet when not specified.

Prerequisites

Required Tools

EKS Authentication

Two methods available: IAM kubeconfig (recommended for interactive use) or Static Service Account Token (for CI/CD). See eks-auth-setup.md for detailed setup instructions.

Required Permissions

Optional MCP Server

When MCP is unavailable, fall back to kubectl + aws CLI direct calls.

State Persistence

All output goes to output/ directory:

output/
├── step1-cluster.json          # Cluster discovery results
├── assessment.json             # Structured results (26 checks) — chaos skill input
├── assessment-report.md        # Human-readable Markdown report
├── assessment-report.html      # HTML report (inline CSS, color-coded)
└── remediation-commands.sh     # Fix script (requires manual execution)

On startup, check for existing output/. If prior results exist, ask: continue from last run or start fresh.

---

Four-Step Workflow

Step 1: Cluster Discovery

1. Get cluster name — User provides it, or auto-detect:

   kubectl config current-context | sed 's|.*:cluster/||'

2. Describe cluster — Collect metadata:

   aws eks describe-cluster --name {CLUSTER_NAME} --region {REGION} --output json

Extract: kubernetesVersion, platformVersion, vpcId, endpoint config, logging, tags, addons.

3. List addons:

   aws eks list-addons --cluster-name {CLUSTER_NAME} --region {REGION} --output json

4. Determine target namespaces — List non-system namespaces, present to user for confirmation:

   kubectl get namespaces -o json | jq -r '[.items[].metadata.name | select(test("^kube-") | not) | select(. != "kube-system" and . != "kube-public" and . != "kube-node-lease")]'

5. Detect EKS Auto Mode:

   aws eks describe-cluster --name {CLUSTER_NAME} --query 'cluster.computeConfig.enabled' --output text

If true, flag for D7 auto-pass and adjust node-related checks.

6. Detect Fargate profiles:

   aws eks list-fargate-profiles --cluster-name {CLUSTER_NAME} --output json

If Fargate profiles exist, skip inapplicable checks (A3, D1) for Fargate workloads.

Output: Save to output/step1-cluster.json. Confirm cluster, region, namespaces with user.

---

Step 2: Automated Checks (26 Items)

Run all 26 checks against the confirmed cluster and namespaces.

Commands and PASS/FAIL criteria: See check-commands.md for exact kubectl/aws CLI commands for each check. MCP alternative: See eks-resiliency-checks-mcp.md for MCP-based execution. Check descriptions and rationale: See EKS-Resiliency-Checkpoints.md.

Namespace filter (used throughout):

TARGET_NS="namespace1,namespace2,..."  # from Step 1

For namespace-scoped checks, loop over TARGET_NS. For cluster-wide checks (A7, C1-C5, D1-D2, D6-D7), no filter needed.

26 Checks Overview:

Check Result Format (per check):

{
  "id": "A1",
  "name": "Avoid Running Singleton Pods",
  "category": "application",
  "severity": "critical",
  "status": "PASS",
  "findings": [],
  "resources_affected": [],
  "remediation": "",
  "chaos_experiment_recommendation": null
}

For FAIL results, populate findings with descriptions, resources_affected with namespace/resource-name, remediation with fix commands (see remediation-templates.md).

---

Step 3: Generate Reports

After all 26 checks, generate four output files to output/:

3.1 assessment.json

Structured results with schema:

• cluster_name, region, kubernetes_version, platform_version, timestamp, target_namespaces
• summary: total_checks, passed, failed, info, critical_failures, compliance_score
• checks[]: array of check results from Step 2
• experiment_recommendations[]: from Step 4 if executed

Compliance score: (passed / (total - info_only)) * 100. Info-severity FAILs (A9, A10, A12, D7) don't reduce score.

3.2 assessment-report.md

Markdown report: header → summary table → results by category → experiment recommendations.

3.3 assessment-report.html

Single-file HTML with inline CSS. Color coding: green=PASS, red=FAIL, blue=INFO. Severity badges, collapsible sections, compliance score gauge.

3.4 remediation-commands.sh

Executable script with fix commands for FAIL items only. Each section: check ID, explanation, command(s). Never auto-executed.

Present summary + compliance score. Ask if user wants Step 4.

Cost Impact Assessment

Each FAIL includes a cost estimate.

• With `awslabs.aws-pricing-mcp-server`: Query actual on-demand pricing for compute/monitoring fixes, show monthly dollar estimates.
• Without Pricing MCP (default): Qualitative descriptions (e.g., "Zero — config only" or "+1 Pod per workload").

---

Step 4: Experiment Recommendations (Optional)

Map FAIL items to chaos experiments. See fail-to-experiment-mapping.md for the complete mapping table.

Key mappings:

For each FAIL with a mapping, generate experiment recommendation entries in assessment.json. Sort by priority (P0 > P1 > P2).

Handoff: Guide user to invoke chaos-engineering-on-aws with output/assessment.json as Method 3 input.

---

Safety Principles

1. Read-only only: All checks use get, list, describe — no create/update/delete during assessment 2. Remediation requires manual execution: remediation-commands.sh is never auto-executed 3. System namespace exclusion: kube-system, kube-public, kube-node-lease excluded from workload checks 4. No secret exposure: Never read Secret/ConfigMap values — only check existence 5. Fargate awareness: Skip inapplicable checks for Fargate workloads 6. EKS Auto Mode awareness: Adjust node-related checks when Auto Mode detected

Error Handling

Never block entire assessment for a single check failure.

References

• EKS-Resiliency-Checkpoints.md — What each check does and why
• check-commands.md — How to execute each check (commands + PASS/FAIL)
• eks-resiliency-checks-mcp.md — MCP-based check execution
• remediation-templates.md — Fix command templates
• fail-to-experiment-mapping.md — FAIL → chaos experiment mapping
• eks-auth-setup.md — EKS authentication setup guide
• examples/petsite-assessment.md — Sample assessment output