Chaos Engineering On Aws

Language / 语言

• If the user speaks English, follow SKILL_EN.md
• 如果用户使用中文，请遵循 SKILL_ZH.md

Detect the language from the user's message and load the corresponding instruction file.

示例 1: EC2 实例终止 — Auto Scaling 恢复验证

架构模式：传统 EC2 + ALB + Auto Scaling Group FIS Action：aws:ec2:terminate-instances 验证点：ASG 自动启动替代实例、ALB 健康检查通过、请求不中断

---

稳态假设

当终止 ASG 中 1 个 EC2 实例后：

• ALB 请求成功率 >= 99.5%（5 分钟窗口）
• P99 延迟 <= 500ms
• ASG 在 300s 内补充新实例并通过健康检查
• 无数据丢失

验证要点

• Auto Scaling Group 替换速度和健康检查配置
• ALB 目标注销延迟和健康检查灵敏度
• 应用无状态性（实例丢失时无会话亲和问题）
• CloudWatch 告警 UnHealthyHostCount 在 1 分钟内触发
• 实例替换期间无客户可见错误

停止条件

{
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:{region}:{account}:alarm:chaos-stop-5xx-rate"
    }
  ]
}

对应 CloudWatch Alarm：

aws cloudwatch put-metric-alarm \
  --alarm-name "chaos-stop-5xx-rate" \
  --namespace "AWS/ApplicationELB" \
  --metric-name "HTTPCode_Target_5XX_Count" \
  --statistic Sum \
  --period 60 \
  --threshold 50 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2 \
  --treat-missing-data notBreaching \
  --alarm-actions "arn:aws:sns:{region}:{account}:chaos-alerts"

FIS 实验模板

{
  "description": "Terminate one EC2 instance in ASG to validate auto-recovery",
  "targets": {
    "ec2-instances": {
      "resourceType": "aws:ec2:instance",
      "resourceArns": [
        "arn:aws:ec2:{region}:{account}:instance/{instance-id}"
      ],
      "selectionMode": "COUNT(1)"
    }
  },
  "actions": {
    "terminate-instance": {
      "actionId": "aws:ec2:terminate-instances",
      "parameters": {},
      "targets": {
        "Instances": "ec2-instances"
      }
    }
  },
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:{region}:{account}:alarm:chaos-stop-5xx-rate"
    }
  ],
  "roleArn": "arn:aws:iam::{account}:role/FISExperimentRole",
  "tags": {
    "Purpose": "chaos-engineering",
    "RiskId": "R-001"
  }
}

执行命令

# 创建模板
aws fis create-experiment-template \
  --cli-input-json file://examples/ec2-terminate-template.json

# 启动实验
aws fis start-experiment --experiment-template-id <template-id>

# 监控实验状态
aws fis get-experiment --id <experiment-id> \
  --query 'experiment.state.status'

观测指标

预期结果

如果失败：说明 ASG 配置有问题（最小容量、健康检查间隔、启动模板），需检查 ASG 策略和 Launch Template。

Example 1: EC2 Instance Termination — Auto Scaling Recovery Validation

Architecture pattern: Traditional EC2 + ALB + Auto Scaling Group FIS Action: aws:ec2:terminate-instances Validation target: ASG automatically launches replacement instances, ALB health check passes, requests uninterrupted

---

Steady-State Hypothesis

After terminating 1 EC2 instance in the ASG:

• ALB request success rate >= 99.5% (5min window)
• P99 latency <= 500ms
• ASG replenishes a new instance and passes health check within 300s
• Zero data loss

What does this enable you to verify?

• Auto Scaling Group replacement speed and health check configuration
• ALB target deregistration delay and health check sensitivity
• Application statelessness (no session affinity issues when instance is lost)
• CloudWatch alarm fires for UnHealthyHostCount within 1 minute
• No customer-visible errors during instance replacement

Stop Conditions

{
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:{region}:{account}:alarm:chaos-stop-5xx-rate"
    }
  ]
}

Corresponding CloudWatch Alarm:

aws cloudwatch put-metric-alarm \
  --alarm-name "chaos-stop-5xx-rate" \
  --namespace "AWS/ApplicationELB" \
  --metric-name "HTTPCode_Target_5XX_Count" \
  --statistic Sum \
  --period 60 \
  --threshold 50 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2 \
  --treat-missing-data notBreaching \
  --alarm-actions "arn:aws:sns:{region}:{account}:chaos-alerts"

FIS Experiment Template

{
  "description": "Terminate one EC2 instance in ASG to validate auto-recovery",
  "targets": {
    "ec2-instances": {
      "resourceType": "aws:ec2:instance",
      "resourceArns": [
        "arn:aws:ec2:{region}:{account}:instance/{instance-id}"
      ],
      "selectionMode": "COUNT(1)"
    }
  },
  "actions": {
    "terminate-instance": {
      "actionId": "aws:ec2:terminate-instances",
      "parameters": {},
      "targets": {
        "Instances": "ec2-instances"
      }
    }
  },
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:{region}:{account}:alarm:chaos-stop-5xx-rate"
    }
  ],
  "roleArn": "arn:aws:iam::{account}:role/FISExperimentRole",
  "tags": {
    "Purpose": "chaos-engineering",
    "RiskId": "R-001"
  }
}

Execution Commands

# Create template
aws fis create-experiment-template \
  --cli-input-json file://examples/ec2-terminate-template.json

# Start experiment
aws fis start-experiment --experiment-template-id <template-id>

# Monitor experiment status
aws fis get-experiment --id <experiment-id> \
  --query 'experiment.state.status'

Observation Metrics

Expected Results

If failed: Indicates ASG configuration issues (min capacity, health check interval, launch template). Check ASG policies and Launch Template.

示例 2: RDS Aurora 故障转移 — 数据库 HA 验证

架构模式：Aurora Cluster（Writer + Reader） FIS Action：aws:rds:failover-db-cluster 验证点：Reader 提升为 Writer、应用连接自动恢复、零数据丢失

---

稳态假设

当触发 Aurora 集群故障转移后：

• 数据库写入恢复时间 <= 30s
• 应用请求成功率 >= 99%（故障转移期间允许短暂下降）
• 故障转移后数据完整性 100%
• 应用连接池自动重连，无需手动干预

验证要点

• 数据库连接池重连逻辑和 DNS TTL 设置
• 应用在短暂写不可用期间的错误处理
• Aurora 集群端点 DNS 传播速度
• 读副本提升和写入器/读取器角色交换正确性
• CloudWatch 告警 AuroraReplicaLag 和故障转移事件

停止条件

{
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:{region}:{account}:alarm:chaos-stop-db-connections"
    }
  ]
}

对应 Alarm（连接数归零超过 5 分钟则告警）：

aws cloudwatch put-metric-alarm \
  --alarm-name "chaos-stop-db-connections" \
  --namespace "AWS/RDS" \
  --metric-name "DatabaseConnections" \
  --dimensions Name=DBClusterIdentifier,Value={cluster-id} \
  --statistic Average \
  --period 60 \
  --threshold 0 \
  --comparison-operator LessThanOrEqualToThreshold \
  --evaluation-periods 5 \
  --treat-missing-data notBreaching \
  --alarm-actions "arn:aws:sns:{region}:{account}:chaos-alerts"

FIS 实验模板

{
  "description": "Failover Aurora cluster to validate HA and application reconnect",
  "targets": {
    "aurora-cluster": {
      "resourceType": "aws:rds:cluster",
      "resourceArns": [
        "arn:aws:rds:{region}:{account}:cluster:{cluster-id}"
      ],
      "selectionMode": "ALL"
    }
  },
  "actions": {
    "failover-cluster": {
      "actionId": "aws:rds:failover-db-cluster",
      "parameters": {},
      "targets": {
        "Clusters": "aurora-cluster"
      }
    }
  },
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:{region}:{account}:alarm:chaos-stop-db-connections"
    }
  ],
  "roleArn": "arn:aws:iam::{account}:role/FISExperimentRole",
  "tags": {
    "Purpose": "chaos-engineering",
    "RiskId": "R-002"
  }
}

执行命令

# 确认当前 Writer
aws rds describe-db-clusters --db-cluster-identifier {cluster-id} \
  --query 'DBClusters[0].DBClusterMembers[?IsClusterWriter==`true`].DBInstanceIdentifier'

# 创建并启动实验
aws fis create-experiment-template --cli-input-json file://examples/rds-failover-template.json
aws fis start-experiment --experiment-template-id <template-id>

# 验证 Writer 已切换
aws rds describe-db-clusters --db-cluster-identifier {cluster-id} \
  --query 'DBClusters[0].DBClusterMembers[?IsClusterWriter==`true`].DBInstanceIdentifier'

观测指标

预期结果

如果失败：常见原因 — 应用未使用 Aurora 集群端点（使用了实例端点）、连接池无重试逻辑、DNS TTL 过长。检查连接字符串和重试配置。

Example 2: RDS Aurora Failover — Database HA Validation

Architecture pattern: Aurora Cluster (Writer + Reader) FIS Action: aws:rds:failover-db-cluster Validation target: Reader promoted to Writer, application connections auto-recover, zero data loss

---

Steady-State Hypothesis

After triggering Aurora cluster failover:

• Database write recovery time <= 30s
• Application request success rate >= 99% (brief drop allowed during failover)
• Data integrity 100% after failover
• Application connection pool auto-reconnects without manual intervention

What does this enable you to verify?

• Database connection pool reconnection logic and DNS TTL settings
• Application error handling during brief write unavailability
• Aurora cluster endpoint DNS propagation speed
• Read replica promotion and writer/reader role swap correctness
• CloudWatch alarm for AuroraReplicaLag and failover events

Stop Conditions

{
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:{region}:{account}:alarm:chaos-stop-db-connections"
    }
  ]
}

Corresponding Alarm (alert if connections drop to zero for over 5 minutes):

aws cloudwatch put-metric-alarm \
  --alarm-name "chaos-stop-db-connections" \
  --namespace "AWS/RDS" \
  --metric-name "DatabaseConnections" \
  --dimensions Name=DBClusterIdentifier,Value={cluster-id} \
  --statistic Average \
  --period 60 \
  --threshold 0 \
  --comparison-operator LessThanOrEqualToThreshold \
  --evaluation-periods 5 \
  --treat-missing-data notBreaching \
  --alarm-actions "arn:aws:sns:{region}:{account}:chaos-alerts"

FIS Experiment Template

{
  "description": "Failover Aurora cluster to validate HA and application reconnect",
  "targets": {
    "aurora-cluster": {
      "resourceType": "aws:rds:cluster",
      "resourceArns": [
        "arn:aws:rds:{region}:{account}:cluster:{cluster-id}"
      ],
      "selectionMode": "ALL"
    }
  },
  "actions": {
    "failover-cluster": {
      "actionId": "aws:rds:failover-db-cluster",
      "parameters": {},
      "targets": {
        "Clusters": "aurora-cluster"
      }
    }
  },
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:{region}:{account}:alarm:chaos-stop-db-connections"
    }
  ],
  "roleArn": "arn:aws:iam::{account}:role/FISExperimentRole",
  "tags": {
    "Purpose": "chaos-engineering",
    "RiskId": "R-002"
  }
}

Execution Commands

# Confirm current Writer
aws rds describe-db-clusters --db-cluster-identifier {cluster-id} \
  --query 'DBClusters[0].DBClusterMembers[?IsClusterWriter==`true`].DBInstanceIdentifier'

# Create and start experiment
aws fis create-experiment-template --cli-input-json file://examples/rds-failover-template.json
aws fis start-experiment --experiment-template-id <template-id>

# Verify Writer has switched
aws rds describe-db-clusters --db-cluster-identifier {cluster-id} \
  --query 'DBClusters[0].DBClusterMembers[?IsClusterWriter==`true`].DBInstanceIdentifier'

Observation Metrics

Expected Results

If failed: Common causes — application not using Aurora cluster endpoint (using instance endpoint instead), connection pool lacks retry logic, DNS TTL too long. Check connection string and retry configuration.

示例 3: EKS Pod Kill — 微服务自愈验证

架构模式：EKS 微服务（Deployment + Service + Ingress） 工具：Chaos Mesh PodChaos（需集群已安装） 验证点：ReplicaSet 自动重建 Pod、流量通过 Service 无缝切换

---

前提

• 集群已安装 Chaos Mesh：kubectl get crd | grep chaos-mesh
• 目标 Deployment replicas >= 2

如果 Chaos Mesh 未安装，可用 FIS aws:eks:terminate-nodegroup-instances 做节点级故障替代（爆炸半径更大）。

⚠️ 不推荐用 FIS aws:eks:pod-delete 做 Pod 级故障 — 需额外配置 K8s ServiceAccount + RBAC + EKS access entry，且故障注入器 Pod 初始化慢（>2min）。Pod 级故障首选 Chaos Mesh。

稳态假设

当杀死目标服务的 1 个 Pod 后：

• Service 请求成功率 >= 99.9%
• P99 延迟 <= 300ms
• Pod 在 60s 内重建并进入 Ready 状态
• 无请求丢失（其他 Pod 接管流量）

验证要点

• Kubernetes liveness/readiness 探针配置有效性
• HPA 在 Pod 数量低于阈值时的扩容响应
• Service mesh / 负载均衡重新分配速度
• PodDisruptionBudget 在自愿中断时的执行情况
• 应用优雅关闭处理（SIGTERM → preStop hook）

Chaos Mesh Manifest

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: pod-kill-web-frontend
  namespace: chaos-testing
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces:
      - production
    labelSelectors:
      app: web-frontend
  duration: "30s"
  gracePeriod: 0

使用 MCP（如可用）

# chaosmesh-mcp 调用
pod_kill(
    service="web-frontend",
    duration="30s",
    mode="one",
    namespace="production"
)

执行命令

# 检查目标 Pod 数量
kubectl get pods -n production -l app=web-frontend

# 使用 experiment-runner.sh（推荐 — 自动轮询 + 状态管理）：
nohup bash scripts/experiment-runner.sh --mode chaosmesh \
    --manifest examples/pod-kill-web-frontend.yaml --namespace production \
    --one-shot --pod-label "app=web-frontend" --deployment "web-frontend" \
    --state-exp-id "EXP-001" --output-dir output/ &
RUNNER_PID=$!

# Monitor（Chaos Mesh 不设 EXPERIMENT_ID）：
export NAMESPACE="production" REGION="ap-northeast-1" DURATION=300
nohup bash scripts/monitor.sh &

# Log collector（所有实验必须启动）：
nohup bash scripts/log-collector.sh --namespace production \
    --services "web-frontend" --mode live --duration 300 --output-dir output/ &

wait $RUNNER_PID
# 退出码：0=完成（one-shot），1=失败，2=超时

# 手动方式（不使用脚本）：
kubectl apply -f examples/pod-kill-web-frontend.yaml
kubectl get pods -n production -l app=web-frontend -w
kubectl delete -f examples/pod-kill-web-frontend.yaml

观测指标

预期结果

如果失败：常见原因 — replicas=1 无冗余、readinessProbe 过长、PodDisruptionBudget 过严、镜像拉取慢（缺少 imagePullPolicy: IfNotPresent）。

Example 3: EKS Pod Kill — Microservice Self-Healing Validation

Architecture pattern: EKS microservices (Deployment + Service + Ingress) Tool: Chaos Mesh PodChaos (requires cluster installation) Validation target: ReplicaSet auto-recreates Pod, traffic switches seamlessly via Service

---

Prerequisites

• Cluster has Chaos Mesh installed: kubectl get crd | grep chaos-mesh
• Target Deployment replicas >= 2

If Chaos Mesh is not installed, use FIS aws:eks:terminate-nodegroup-instances for node-level fault as an alternative (larger blast radius).

⚠️ Not recommended to use FIS aws:eks:pod-delete for Pod-level faults — requires additional K8s ServiceAccount + RBAC + EKS access entry, and fault injector Pod initialization is slow (>2min). Prefer Chaos Mesh for Pod-level faults.

Steady-State Hypothesis

After killing 1 Pod of the target service:

• Service request success rate >= 99.9%
• P99 latency <= 300ms
• Pod rebuilt and enters Ready state within 60s
• Zero request loss (other Pods take over traffic)

What does this enable you to verify?

• Kubernetes liveness/readiness probe configuration effectiveness
• HPA scaling response when pod count drops below threshold
• Service mesh / load balancing redistribution speed
• PodDisruptionBudget enforcement during voluntary disruptions
• Application graceful shutdown handling (SIGTERM → preStop hook)

Chaos Mesh Manifest

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: pod-kill-web-frontend
  namespace: chaos-testing
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces:
      - production
    labelSelectors:
      app: web-frontend
  duration: "30s"
  gracePeriod: 0

Using MCP (if available)

# chaosmesh-mcp call
pod_kill(
    service="web-frontend",
    duration="30s",
    mode="one",
    namespace="production"
)

Execution Commands

# Check target Pod count
kubectl get pods -n production -l app=web-frontend

# Using experiment-runner.sh (recommended — handles polling + state):
nohup bash scripts/experiment-runner.sh --mode chaosmesh \
    --manifest examples/pod-kill-web-frontend.yaml --namespace production \
    --one-shot --pod-label "app=web-frontend" --deployment "web-frontend" \
    --state-exp-id "EXP-001" --output-dir output/ &
RUNNER_PID=$!

# Monitor (omit EXPERIMENT_ID for Chaos Mesh):
export NAMESPACE="production" REGION="ap-northeast-1" DURATION=300
nohup bash scripts/monitor.sh &

# Log collector (MANDATORY for all experiments):
nohup bash scripts/log-collector.sh --namespace production \
    --services "web-frontend" --mode live --duration 300 --output-dir output/ &

wait $RUNNER_PID
# Exit code: 0=completed (one-shot), 1=failed, 2=timeout

# Manual alternative (without scripts):
kubectl apply -f examples/pod-kill-web-frontend.yaml
kubectl get pods -n production -l app=web-frontend -w
kubectl delete -f examples/pod-kill-web-frontend.yaml

Observation Metrics

Expected Results

If failed: Common causes — replicas=1 (no redundancy), readinessProbe too long, PodDisruptionBudget too strict, slow image pull (missing imagePullPolicy: IfNotPresent).

示例 4: AZ 网络隔离 — 多 AZ 容错验证

架构模式：多 AZ 部署（ALB + EC2/EKS + RDS Multi-AZ） FIS Action：aws:network:disrupt-connectivity 验证点：单 AZ 网络中断后，流量自动切换到健康 AZ，服务保持可用

---

稳态假设

当隔离 1 个 AZ 的网络后：

• ALB 请求成功率 >= 99%（允许切换期间短暂下降）
• P99 延迟 <= 1000ms（单 AZ 承载全量可能略升）
• 恢复时间 <= 120s
• 数据库 Multi-AZ 故障转移成功（如 Primary 在被隔离 AZ）

验证要点

• 跨 AZ 流量路由和 ALB 健康检查的 AZ 感知能力
• AZ 网络中断时 EBS 卷可用性
• RDS/Aurora Multi-AZ 故障转移触发条件和时间
• 应用层重试和超时行为（AZ 隔离期间）
• 运维操作手册在 AZ 级事件中的有效性

停止条件

{
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:{region}:{account}:alarm:chaos-stop-availability"
    }
  ]
}

对应 Alarm：

aws cloudwatch put-metric-alarm \
  --alarm-name "chaos-stop-availability" \
  --namespace "AWS/ApplicationELB" \
  --metric-name "HTTPCode_ELB_5XX_Count" \
  --statistic Sum \
  --period 60 \
  --threshold 100 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 3 \
  --treat-missing-data notBreaching \
  --alarm-actions "arn:aws:sns:{region}:{account}:chaos-alerts"

FIS 实验模板

{
  "description": "Disrupt network connectivity for one AZ to validate multi-AZ failover",
  "targets": {
    "az-subnets": {
      "resourceType": "aws:ec2:subnet",
      "resourceArns": [
        "arn:aws:ec2:{region}:{account}:subnet/{subnet-az-a-1}",
        "arn:aws:ec2:{region}:{account}:subnet/{subnet-az-a-2}"
      ],
      "selectionMode": "ALL"
    }
  },
  "actions": {
    "disrupt-az-network": {
      "actionId": "aws:network:disrupt-connectivity",
      "parameters": {
        "scope": "all",
        "duration": "PT5M"
      },
      "targets": {
        "Subnets": "az-subnets"
      }
    }
  },
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:{region}:{account}:alarm:chaos-stop-availability"
    }
  ],
  "roleArn": "arn:aws:iam::{account}:role/FISExperimentRole",
  "tags": {
    "Purpose": "chaos-engineering",
    "RiskId": "R-004"
  }
}

执行命令

# 确认目标 AZ 的子网
aws ec2 describe-subnets \
  --filters "Name=availability-zone,Values={region}a" \
  --query 'Subnets[].{Id:SubnetId,AZ:AvailabilityZone,VPC:VpcId}'

# 确认各 AZ 实例分布
aws ec2 describe-instances \
  --filters "Name=tag:Environment,Values=production" \
  --query 'Reservations[].Instances[].{Id:InstanceId,AZ:Placement.AvailabilityZone,State:State.Name}'

# 创建并启动实验
aws fis create-experiment-template --cli-input-json file://examples/az-network-disrupt-template.json
aws fis start-experiment --experiment-template-id <template-id>

观测指标

预期结果

如果失败：常见原因：

• 实例全部部署在同一 AZ（无冗余）
• ALB 跨 AZ 健康检查间隔过长
• RDS 未开启 Multi-AZ
• 有状态服务依赖本地存储（EBS 不跨 AZ）
• Session sticky 导致切换后丢失会话

注意事项

⚠️ 这是爆炸半径最大的实验。建议： 1. 先在 Staging 验证 2. 确认每个 AZ 都有足够容量独立承载全量流量 3. 选择业务低峰时段 4. 确保 On-call 团队就位

Example 4: AZ Network Isolation — Multi-AZ Fault Tolerance Validation

Architecture pattern: Multi-AZ deployment (ALB + EC2/EKS + RDS Multi-AZ) FIS Action: aws:network:disrupt-connectivity Validation target: After single AZ network disruption, traffic automatically shifts to healthy AZ, service remains available

---

Steady-State Hypothesis

After isolating the network of 1 AZ:

• ALB request success rate >= 99% (brief drop allowed during switchover)
• P99 latency <= 1000ms (may increase slightly with single AZ bearing full load)
• Recovery time <= 120s
• Database Multi-AZ failover succeeds (if Primary is in the isolated AZ)

What does this enable you to verify?

• Cross-AZ traffic routing and ALB health check AZ-awareness
• EBS volume availability when AZ network is disrupted
• RDS/Aurora Multi-AZ failover trigger conditions and timing
• Application-level retry and timeout behavior during AZ isolation
• Operational runbook effectiveness for AZ-level incidents

Stop Conditions

{
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:{region}:{account}:alarm:chaos-stop-availability"
    }
  ]
}

Corresponding Alarm:

aws cloudwatch put-metric-alarm \
  --alarm-name "chaos-stop-availability" \
  --namespace "AWS/ApplicationELB" \
  --metric-name "HTTPCode_ELB_5XX_Count" \
  --statistic Sum \
  --period 60 \
  --threshold 100 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 3 \
  --treat-missing-data notBreaching \
  --alarm-actions "arn:aws:sns:{region}:{account}:chaos-alerts"

FIS Experiment Template

{
  "description": "Disrupt network connectivity for one AZ to validate multi-AZ failover",
  "targets": {
    "az-subnets": {
      "resourceType": "aws:ec2:subnet",
      "resourceArns": [
        "arn:aws:ec2:{region}:{account}:subnet/{subnet-az-a-1}",
        "arn:aws:ec2:{region}:{account}:subnet/{subnet-az-a-2}"
      ],
      "selectionMode": "ALL"
    }
  },
  "actions": {
    "disrupt-az-network": {
      "actionId": "aws:network:disrupt-connectivity",
      "parameters": {
        "scope": "all",
        "duration": "PT5M"
      },
      "targets": {
        "Subnets": "az-subnets"
      }
    }
  },
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:{region}:{account}:alarm:chaos-stop-availability"
    }
  ],
  "roleArn": "arn:aws:iam::{account}:role/FISExperimentRole",
  "tags": {
    "Purpose": "chaos-engineering",
    "RiskId": "R-004"
  }
}

Execution Commands

# Confirm target AZ subnets
aws ec2 describe-subnets \
  --filters "Name=availability-zone,Values={region}a" \
  --query 'Subnets[].{Id:SubnetId,AZ:AvailabilityZone,VPC:VpcId}'

# Confirm instance distribution across AZs
aws ec2 describe-instances \
  --filters "Name=tag:Environment,Values=production" \
  --query 'Reservations[].Instances[].{Id:InstanceId,AZ:Placement.AvailabilityZone,State:State.Name}'

# Create and start experiment
aws fis create-experiment-template --cli-input-json file://examples/az-network-disrupt-template.json
aws fis start-experiment --experiment-template-id <template-id>

Observation Metrics

Expected Results

If failed: Common causes:

• All instances deployed in the same AZ (no redundancy)
• ALB cross-AZ health check interval too long
• RDS Multi-AZ not enabled
• Stateful services depend on local storage (EBS does not span AZs)
• Session stickiness causes session loss after switchover

Caution

⚠️ This is the experiment with the largest blast radius. Recommendations: 1. Validate in Staging first 2. Confirm each AZ has sufficient capacity to independently handle full traffic 3. Choose a low-traffic time window 4. Ensure On-call team is in position

示例 05：组合 AZ 降级 — 多 Action FIS 实验

本示例展示如何使用 FIS 原生多 Action 模板 模拟复合 AZ 级故障，无需任何外部编排代码。

场景

模拟单个可用区的多维度降级： 1. 停止 EC2 实例（目标 AZ） 2. 暂停 EBS 卷 IO（目标 AZ） 3. 触发 RDS Aurora 故障转移

EC2 停止和 EBS 暂停同时启动（并行），RDS failover 延迟 30 秒后启动（通过 startAfter 串行依赖）。

架构

          AZ-a (目标)                 AZ-c (健康)
  ┌──────────────────────┐    ┌──────────────────────┐
  │  EC2: 已停止 ❌       │    │  EC2: 运行中 ✅       │
  │  EBS: IO 已暂停 ❌    │    │  EBS: 正常 ✅         │
  │  RDS Writer → 故障    │───►│  RDS Reader → Writer  │
  └──────────────────────┘    └──────────────────────┘

假设

假设陈述：当 AZ-a 的 EC2 实例被停止、EBS IO 被暂停、RDS 发生故障转移时，应用应当：

• 通过健康的 AZ-c 实例继续提供服务
• RDS failover 在 30 秒内完成
• 请求成功率保持 ≥ 95%
• 故障注入停止后 120 秒内完全恢复

验证要点

• 多 AZ 部署真正能承受 AZ 级降级（而不仅是单实例故障）
• 多服务协调故障行为（EC2 + EBS + RDS 同时故障）
• 跨 AZ 容量规划（剩余 AZ 承载全量负载）
• FIS 多 Action 模板中 startAfter 排序正确性
• 同一 AZ 多服务故障时的爆炸半径控制

前置条件

• [ ] 多 AZ 部署，至少 2 个 AZ 有实例
• [ ] 目标 EC2 实例和 EBS 卷已打标签 AzImpairmentPower: IceQualified
• [ ] RDS Aurora 集群在其他 AZ 有 Reader 实例
• [ ] FIS IAM Role 已创建（权限见下方）
• [ ] CloudWatch Alarm 已配置（用于 stop condition）
• [ ] 剩余 AZ 有足够容量承接流量

FIS 模板（多 Action + startAfter）

完整 JSON 模板：references/templates/az-power-interruption.json

关键 actions 部分（展示编排逻辑）：

{
  "actions": {
    "stop-ec2-az-a": {
      "actionId": "aws:ec2:stop-instances",
      "parameters": { "startInstancesAfterDuration": "PT5M" },
      "targets": { "Instances": "ec2-instances-az-a" }
    },
    "pause-ebs-az-a": {
      "actionId": "aws:ebs:pause-volume-io",
      "parameters": { "duration": "PT5M" },
      "targets": { "Volumes": "ebs-volumes-az-a" }
    },
    "failover-rds": {
      "actionId": "aws:rds:failover-db-cluster",
      "targets": { "Clusters": "rds-cluster" },
      "startAfter": ["stop-ec2-az-a"]
    },
    "wait-before-rds": {
      "actionId": "aws:fis:wait",
      "parameters": { "duration": "PT30S" },
      "startAfter": ["stop-ec2-az-a"]
    }
  }
}

关键设计要点

FIS startAfter 参考

执行

# 1. 创建多 action 模板
TEMPLATE_ID=$(aws fis create-experiment-template \
  --cli-input-json file://output/templates/composite-az-degradation.json \
  --region ap-northeast-1 \
  --query 'experimentTemplate.id' --output text)
echo "模板已创建: $TEMPLATE_ID"

# 2. 通过 experiment-runner.sh 运行（与单 action 实验完全相同）
nohup bash scripts/experiment-runner.sh \
  --mode fis \
  --template-id "$TEMPLATE_ID" \
  --region ap-northeast-1 \
  --timeout 720 \
  --poll-interval 15 \
  --output-dir output/ &
RUNNER_PID=$!

# 3. 启动监控
nohup bash scripts/monitor.sh &

# 4. 启动日志采集
nohup bash scripts/log-collector.sh \
  --namespace petadoptions \
  --services "petsite,petsearch,payforadoption" \
  --duration 720 \
  --output-dir output/ \
  --mode live &

# 5. 等待完成
wait $RUNNER_PID
echo "Runner 退出码: $?"

自定义实验时长

快速验证时缩短 duration：

# 将所有 action 时长改为 2 分钟
jq '
  .actions["stop-ec2-az-a"].parameters.startInstancesAfterDuration = "PT2M" |
  .actions["pause-ebs-az-a"].parameters.duration = "PT2M"
' output/templates/composite-az-degradation.json > output/templates/composite-az-degradation-short.json

成本估算

Example 05: Composite AZ Degradation — Multi-Action FIS Experiment

This example demonstrates how to use FIS native multi-action templates to simulate compound AZ-level failures without any external orchestration code.

Scenario

Simulate degradation in a single Availability Zone by simultaneously: 1. Stopping EC2 instances in the target AZ 2. Pausing EBS volume IO in the target AZ 3. Triggering RDS Aurora failover

EC2 stop and EBS pause start immediately (parallel). RDS failover starts 30 seconds later (serial dependency via startAfter).

Architecture

          AZ-a (target)              AZ-c (healthy)
  ┌──────────────────────┐    ┌──────────────────────┐
  │  EC2: stopped ❌      │    │  EC2: running ✅      │
  │  EBS: IO paused ❌    │    │  EBS: normal ✅       │
  │  RDS Writer → fails   │───►│  RDS Reader → Writer  │
  └──────────────────────┘    └──────────────────────┘

Hypothesis

Statement: When EC2 instances in AZ-a are stopped, EBS IO is paused, and RDS fails over, the application should:

• Continue serving requests via healthy AZ-c instances
• Complete RDS failover within 30 seconds
• Maintain request success rate ≥ 95% during the event
• Fully recover within 120 seconds after fault injection stops

What does this enable you to verify?

• Multi-AZ deployment truly survives AZ-level degradation (not just single-instance failure)
• Coordinated multi-service fault behavior (EC2 + EBS + RDS simultaneously)
• Cross-AZ capacity planning (remaining AZ handles full load)
• startAfter sequencing correctness in FIS multi-action templates
• Blast radius containment when multiple services fail in the same AZ

Prerequisites

• [ ] Multi-AZ deployment with instances in at least 2 AZs
• [ ] Target EC2 instances and EBS volumes tagged with AzImpairmentPower: IceQualified
• [ ] RDS Aurora cluster with reader instance in another AZ
• [ ] FIS IAM Role with required permissions (see below)
• [ ] CloudWatch Alarm for stop condition
• [ ] Sufficient capacity in remaining AZs

FIS Template (Multi-Action with startAfter)

Full JSON template: references/templates/az-power-interruption.json

The key actions section (showing orchestration logic):

{
  "actions": {
    "stop-ec2-az-a": {
      "actionId": "aws:ec2:stop-instances",
      "parameters": { "startInstancesAfterDuration": "PT5M" },
      "targets": { "Instances": "ec2-instances-az-a" }
    },
    "pause-ebs-az-a": {
      "actionId": "aws:ebs:pause-volume-io",
      "parameters": { "duration": "PT5M" },
      "targets": { "Volumes": "ebs-volumes-az-a" }
    },
    "failover-rds": {
      "actionId": "aws:rds:failover-db-cluster",
      "targets": { "Clusters": "rds-cluster" },
      "startAfter": ["stop-ec2-az-a"]
    },
    "wait-before-rds": {
      "actionId": "aws:fis:wait",
      "parameters": { "duration": "PT30S" },
      "startAfter": ["stop-ec2-az-a"]
    }
  }
}

Key Design Points

FIS startAfter Reference

Execution

# 1. Create the multi-action template
TEMPLATE_ID=$(aws fis create-experiment-template \
  --cli-input-json file://output/templates/composite-az-degradation.json \
  --region ap-northeast-1 \
  --query 'experimentTemplate.id' --output text)
echo "Template created: $TEMPLATE_ID"

# 2. Run via experiment-runner.sh (same as single-action — no changes needed)
nohup bash scripts/experiment-runner.sh \
  --mode fis \
  --template-id "$TEMPLATE_ID" \
  --region ap-northeast-1 \
  --timeout 720 \
  --poll-interval 15 \
  --output-dir output/ &
RUNNER_PID=$!

# 3. Start monitoring
nohup bash scripts/monitor.sh &

# 4. Start log collection
nohup bash scripts/log-collector.sh \
  --namespace petadoptions \
  --services "petsite,petsearch,payforadoption" \
  --duration 720 \
  --output-dir output/ \
  --mode live &

# 5. Wait
wait $RUNNER_PID
echo "Runner exit code: $?"

Stop Condition Alarm

aws cloudwatch put-metric-alarm \
  --alarm-name "chaos-stop-high-5xx" \
  --namespace "AWS/ApplicationELB" \
  --metric-name "HTTPCode_Target_5XX_Count" \
  --statistic Sum \
  --period 60 \
  --threshold 100 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2 \
  --treat-missing-data notBreaching \
  --dimensions Name=LoadBalancer,Value=app/my-alb/1234567890 \
  --region ap-northeast-1

FIS IAM Role Permissions

The FIS role needs permissions for all actions in the template:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:StopInstances",
        "ec2:StartInstances",
        "ec2:DescribeInstances",
        "ebs:PauseVolumeIO",
        "ebs:DescribeVolumes",
        "rds:FailoverDBCluster",
        "rds:DescribeDBClusters"
      ],
      "Resource": "*"
    }
  ]
}

Expected Results

PASSED ✅ Scenario

• EC2 instances in AZ-a stopped → traffic shifts to AZ-c
• EBS IO paused → no cascading failure (data layer isolated)
• RDS failover completes within 30s
• Success rate stays ≥ 95% throughout
• Full recovery within 120s after faults end

FAILED ❌ Scenarios

• Success rate drops below 95% → application not AZ-resilient
• RDS failover takes > 30s → database HA needs tuning
• Recovery time exceeds 120s → auto-scaling or health checks too slow
• Cascading failure to AZ-c → single points of failure exist

Duration Override

To run a shorter version for quick validation:

# Modify all action durations to 2 minutes
jq '
  .actions["stop-ec2-az-a"].parameters.startInstancesAfterDuration = "PT2M" |
  .actions["pause-ebs-az-a"].parameters.duration = "PT2M"
' output/templates/composite-az-degradation.json > output/templates/composite-az-degradation-short.json

Cost Estimate

示例 6: 数据库连接池耗尽 — 连接池韧性验证

架构模式：应用 → 连接池 → RDS/Aurora（PostgreSQL、MySQL 或 SQL Server） 注入方式：SSM 自动化（动态创建 EC2 负载生成器 → 耗尽连接 → 清理） 验证点：熔断器激活、优雅降级、连接池监控、自动恢复

模板来源：aws-samples/fis-template-library/database-connection-limit-exhaustion

内嵌模板：references/fis-templates/database-connection-exhaustion/

---

假设

当数据库连接数接近上限时，应用应当：

• 通过监控检测连接压力（CloudWatch DatabaseConnections 指标）
• 在连接完全耗尽前触发前导告警
• 连接仍可用时熔断器保持关闭

当连接完全耗尽时：

• 熔断器应打开，阻止新的连接尝试
• UI 应优雅降级（受影响功能不可用，其他功能正常）
• 告警应在 {y} 分钟内触发并通知运维团队
• 不共享该数据库的其他服务不受影响

连接释放后：

• 熔断器应在 {z} 分钟内关闭
• 稳态 {n} 事务/秒应恢复

验证要点

• 数据库连接池监控和告警是否正确配置
• 应用熔断器在连接耗尽时是否按预期工作
• 优雅降级是否防止了级联故障
• 连接可用后恢复是否自动
• 临时负载生成器是否正确清理（无资源泄漏）

前置条件

• [ ] VPC 子网可访问的 RDS 或 Aurora 实例
• [ ] 目标数据库已打标签 FIS-Ready=True
• [ ] 已创建 FIS IAM Role（使用 fis-role-iam-policy.json）
• [ ] 已创建 SSM Automation IAM Role（使用 ssm-role-iam-policy.json）
• [ ] 已部署 SSM 自动化文档（ssm-automation.yaml）
• [ ] 已配置 DatabaseConnections CloudWatch 告警
• [ ] 应用已配置数据库连接熔断器

部署

1. 部署 IAM 角色

# FIS 角色
aws iam create-role \
  --role-name FIS-DbConnExhaustion \
  --assume-role-policy-document file://references/fis-templates/database-connection-exhaustion/fis-iam-trust-relationship.json

aws iam put-role-policy \
  --role-name FIS-DbConnExhaustion \
  --policy-name fis-policy \
  --policy-document file://references/fis-templates/database-connection-exhaustion/fis-role-iam-policy.json

# SSM 角色
aws iam create-role \
  --role-name SSM-DbConnExhaustion \
  --assume-role-policy-document file://references/fis-templates/database-connection-exhaustion/ssm-iam-trust-relationship.json

aws iam put-role-policy \
  --role-name SSM-DbConnExhaustion \
  --policy-name ssm-policy \
  --policy-document file://references/fis-templates/database-connection-exhaustion/ssm-role-iam-policy.json

2. 部署 SSM 自动化文档

aws ssm create-document \
  --name db-connection-exhaustion \
  --document-type Automation \
  --content file://references/fis-templates/database-connection-exhaustion/ssm-automation.yaml \
  --document-format YAML

3. 创建 FIS 实验

编辑 experiment-template.json，替换：

• {ACCOUNT_ID} — AWS 账户 ID
• {REGION} — 目标区域
• {FIS_ROLE_ARN} — FIS 角色 ARN
• 数据库端点、凭证、引擎类型和连接数参数

aws fis create-experiment-template \
  --cli-input-json file://references/fis-templates/database-connection-exhaustion/experiment-template.json

执行

# 启动实验
aws fis start-experiment --experiment-template-id {TEMPLATE_ID}

# 监控数据库连接数
watch -n 10 'aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name DatabaseConnections \
  --dimensions Name=DBInstanceIdentifier,Value={INSTANCE_ID} \
  --start-time $(date -u -d "10 minutes ago" +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 60 \
  --statistics Maximum \
  --output table'

观测指标

清理

SSM 自动化文档会自动处理清理：

• 释放所有持有的数据库连接
• 终止临时 EC2 负载生成器实例
• 正常情况下无需手动清理

如果实验被手动停止或失败：

# 检查残留 EC2 实例
aws ec2 describe-instances \
  --filters "Name=tag:Purpose,Values=FIS-Connection-Exhaustion" "Name=instance-state-name,Values=running" \
  --query 'Reservations[].Instances[].InstanceId'

# 如有则终止
aws ec2 terminate-instances --instance-ids {INSTANCE_ID}

Example 6: Database Connection Limit Exhaustion — Connection Pool Resilience

Architecture pattern: Application → Connection Pool → RDS/Aurora (PostgreSQL, MySQL, or SQL Server) Injection method: SSM Automation (dynamic EC2 load generator → exhaust connections → cleanup) Validates: Circuit breaker activation, graceful degradation, connection pool monitoring, auto-recovery

Template source: aws-samples/fis-template-library/database-connection-limit-exhaustion

Embedded template: references/fis-templates/database-connection-exhaustion/

---

Hypothesis

When the database connection limit is approaching its limit, the application should:

• Detect connection pressure via monitoring (CloudWatch DatabaseConnections metric)
• Trigger a leading alarm before connections are fully exhausted
• Circuit breaker should remain closed while connections are available

When the connection limit is fully exhausted:

• Circuit breaker should open, preventing new connection attempts
• UI should degrade gracefully (affected features unavailable, other features continue)
• Alarm should fire and DevOps team notified within {y} minutes
• Other services not sharing this database should remain unaffected

After connections are released:

• Circuit breaker should close within {z} minutes
• Steady state of {n} transactions per second should resume

What does this enable you to verify?

• Database connection pool monitoring and alerting is properly configured
• Application circuit breaker works as expected under connection exhaustion
• Graceful degradation prevents cascading failures to unrelated services
• Recovery is automatic when connections become available again
• Ephemeral load generator is properly cleaned up (no resource leak)

Prerequisites

• [ ] RDS or Aurora instance accessible from a VPC subnet
• [ ] Target database tagged with FIS-Ready=True
• [ ] FIS IAM Role created with fis-role-iam-policy.json
• [ ] SSM Automation IAM Role created with ssm-role-iam-policy.json
• [ ] SSM Automation Document deployed from ssm-automation.yaml
• [ ] CloudWatch alarm for DatabaseConnections metric
• [ ] Application circuit breaker configured for database connections

Setup

1. Deploy IAM Roles

# FIS Role
aws iam create-role \
  --role-name FIS-DbConnExhaustion \
  --assume-role-policy-document file://references/fis-templates/database-connection-exhaustion/fis-iam-trust-relationship.json

aws iam put-role-policy \
  --role-name FIS-DbConnExhaustion \
  --policy-name fis-policy \
  --policy-document file://references/fis-templates/database-connection-exhaustion/fis-role-iam-policy.json

# SSM Role
aws iam create-role \
  --role-name SSM-DbConnExhaustion \
  --assume-role-policy-document file://references/fis-templates/database-connection-exhaustion/ssm-iam-trust-relationship.json

aws iam put-role-policy \
  --role-name SSM-DbConnExhaustion \
  --policy-name ssm-policy \
  --policy-document file://references/fis-templates/database-connection-exhaustion/ssm-role-iam-policy.json

2. Deploy SSM Automation Document

aws ssm create-document \
  --name db-connection-exhaustion \
  --document-type Automation \
  --content file://references/fis-templates/database-connection-exhaustion/ssm-automation.yaml \
  --document-format YAML

3. Create FIS Experiment

Edit experiment-template.json to replace:

• {ACCOUNT_ID} with your AWS account ID
• {REGION} with target region
• {FIS_ROLE_ARN} with the FIS role ARN
• Database endpoint, credentials, engine type, and connection count parameters

aws fis create-experiment-template \
  --cli-input-json file://references/fis-templates/database-connection-exhaustion/experiment-template.json

Execution

# Start the experiment
aws fis start-experiment --experiment-template-id {TEMPLATE_ID}

# Monitor database connections
watch -n 10 'aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name DatabaseConnections \
  --dimensions Name=DBInstanceIdentifier,Value={INSTANCE_ID} \
  --start-time $(date -u -d "10 minutes ago" +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 60 \
  --statistics Maximum \
  --output table'

Observation Metrics

Cleanup

The SSM Automation document handles cleanup automatically:

• Releases all held database connections
• Terminates the ephemeral EC2 load generator instance
• No manual cleanup required under normal operation

If the experiment is manually stopped or fails:

# Check for leftover EC2 instances
aws ec2 describe-instances \
  --filters "Name=tag:Purpose,Values=FIS-Connection-Exhaustion" "Name=instance-state-name,Values=running" \
  --query 'Reservations[].Instances[].InstanceId'

# Terminate if found
aws ec2 terminate-instances --instance-ids {INSTANCE_ID}

示例 7: ElastiCache Redis 连接中断 — 缓存层韧性验证

架构模式：应用 → ElastiCache Redis（复制组） 注入方式：SSM 自动化（安全组规则删除 → 恢复） 验证点：熔断器、重试风暴抑制、降级模式、缓存重建

模板来源：aws-samples/fis-template-library/elasticache-redis-connection-failure

内嵌模板：references/fis-templates/redis-connection-failure/

---

假设

当 Redis 连接被中断时：

• 应用熔断器应在 30 秒内激活
• 客户端重试风暴应被抑制（无指数放大）
• 应用应在降级模式下继续运行（回退到数据库查询或返回缓存默认值）
• 不向上下游服务产生级联故障
• Redis 连接 CloudWatch 告警应在 2 分钟内触发

当 Redis 连接恢复后：

• 正常运行应在 60 秒内恢复
• 缓存预热/重建应完成且不影响性能
• 熔断器应经历：Open → Half-Open → Closed

验证要点

• Redis 客户端熔断器和重试配置
• Cache-aside 模式的回退行为（缓存未命中时查数据库）
• 缓存层不可用时的应用降级策略
• 安全组故障注入作为可复用模式
• 恢复后的缓存重建行为和性能影响

前置条件

• [ ] ElastiCache Redis 复制组已打标签 FIS-Ready=True
• [ ] 应用实例在同一 VPC，安全组允许 Redis 访问
• [ ] 已创建 FIS IAM Role
• [ ] 已创建 SSM Automation IAM Role
• [ ] 已部署 SSM 自动化文档
• [ ] 应用已配置 Redis 连接熔断器
• [ ] 已配置 Redis 连接相关 CloudWatch 告警

部署

1. 部署 IAM 角色

aws iam create-role \
  --role-name FIS-RedisConnFailure \
  --assume-role-policy-document file://references/fis-templates/redis-connection-failure/fis-iam-trust-relationship.json

aws iam put-role-policy \
  --role-name FIS-RedisConnFailure \
  --policy-name fis-policy \
  --policy-document file://references/fis-templates/redis-connection-failure/redis-connection-failure-fis-role-iam-policy.json

aws iam create-role \
  --role-name SSM-RedisConnFailure \
  --assume-role-policy-document file://references/fis-templates/redis-connection-failure/ssm-iam-trust-relationship.json

aws iam put-role-policy \
  --role-name SSM-RedisConnFailure \
  --policy-name ssm-policy \
  --policy-document file://references/fis-templates/redis-connection-failure/redis-connection-failure-ssm-role-iam-policy.json

2. 部署 SSM 自动化文档

aws ssm create-document \
  --name redis-connection-failure \
  --document-type Automation \
  --content file://references/fis-templates/redis-connection-failure/redis-connection-failure-automation.yaml \
  --document-format YAML

3. 创建 FIS 实验

aws fis create-experiment-template \
  --cli-input-json file://references/fis-templates/redis-connection-failure/redis-connection-failure-experiment-template.json

执行

aws fis start-experiment --experiment-template-id {TEMPLATE_ID}

# 监控 Redis 连接
watch -n 5 'redis-cli -h {REDIS_ENDPOINT} ping 2>&1'

# 监控安全组变更
watch -n 10 'aws ec2 describe-security-groups \
  --group-ids {SG_ID} \
  --query "SecurityGroups[0].IpPermissions" \
  --output table'

观测指标

清理

SSM 自动化文档会自动恢复安全组规则。

如需手动清理：

# 检查安全组规则 — 验证 Redis 端口 (6379) 入站规则已恢复
aws ec2 describe-security-groups \
  --group-ids {SG_ID} \
  --query "SecurityGroups[0].IpPermissions[?FromPort==\`6379\`]"

# 如缺失则手动恢复
aws ec2 authorize-security-group-ingress \
  --group-id {SG_ID} \
  --protocol tcp \
  --port 6379 \
  --source-group {APP_SG_ID}

Example 7: ElastiCache Redis Connection Failure — Cache Layer Resilience

Architecture pattern: Application → ElastiCache Redis (replication group) Injection method: SSM Automation (Security Group rule removal → restore) Validates: Circuit breaker, retry storm prevention, degraded mode, cache rebuild

Template source: aws-samples/fis-template-library/elasticache-redis-connection-failure

Embedded template: references/fis-templates/redis-connection-failure/

---

Hypothesis

When Redis connections are disrupted:

• Application circuit breaker should activate within 30 seconds
• Client retry storms should be prevented (no exponential amplification)
• Application should continue in degraded mode (fall back to database or return cached defaults)
• No cascading failures to upstream or downstream services
• CloudWatch alarm for Redis connectivity should fire within 2 minutes

When Redis connectivity is restored:

• Normal operations should resume within 60 seconds
• Cache warming/rebuild should complete without impacting performance
• Circuit breaker should transition: Open → Half-Open → Closed

What does this enable you to verify?

• Redis client circuit breaker and retry configuration
• Cache-aside pattern fallback behavior (DB query on cache miss)
• Application degradation strategy when cache layer is unavailable
• Security Group-based fault injection as a reusable pattern
• Cache rebuild behavior and performance impact after restoration

Prerequisites

• [ ] ElastiCache Redis replication group tagged with FIS-Ready=True
• [ ] Application instances in same VPC with Security Group allowing Redis access
• [ ] FIS IAM Role created with redis-connection-failure-fis-role-iam-policy.json
• [ ] SSM Automation IAM Role created with redis-connection-failure-ssm-role-iam-policy.json
• [ ] SSM Automation Document deployed from redis-connection-failure-automation.yaml
• [ ] Application circuit breaker configured for Redis connections
• [ ] CloudWatch alarm for Redis ReplicationLag or custom connectivity metric

Setup

1. Deploy IAM Roles

# FIS Role
aws iam create-role \
  --role-name FIS-RedisConnFailure \
  --assume-role-policy-document file://references/fis-templates/redis-connection-failure/fis-iam-trust-relationship.json

aws iam put-role-policy \
  --role-name FIS-RedisConnFailure \
  --policy-name fis-policy \
  --policy-document file://references/fis-templates/redis-connection-failure/redis-connection-failure-fis-role-iam-policy.json

# SSM Role
aws iam create-role \
  --role-name SSM-RedisConnFailure \
  --assume-role-policy-document file://references/fis-templates/redis-connection-failure/ssm-iam-trust-relationship.json

aws iam put-role-policy \
  --role-name SSM-RedisConnFailure \
  --policy-name ssm-policy \
  --policy-document file://references/fis-templates/redis-connection-failure/redis-connection-failure-ssm-role-iam-policy.json

2. Deploy SSM Automation Document

aws ssm create-document \
  --name redis-connection-failure \
  --document-type Automation \
  --content file://references/fis-templates/redis-connection-failure/redis-connection-failure-automation.yaml \
  --document-format YAML

3. Create FIS Experiment

# Update template with your role ARNs and region
aws fis create-experiment-template \
  --cli-input-json file://references/fis-templates/redis-connection-failure/redis-connection-failure-experiment-template.json

Execution

# Start the experiment
aws fis start-experiment --experiment-template-id {TEMPLATE_ID}

# Monitor Redis connectivity
watch -n 5 'redis-cli -h {REDIS_ENDPOINT} ping 2>&1'

# Monitor Security Group changes
watch -n 10 'aws ec2 describe-security-groups \
  --group-ids {SG_ID} \
  --query "SecurityGroups[0].IpPermissions" \
  --output table'

Observation Metrics

Cleanup

The SSM Automation document automatically restores Security Group rules.

If manual cleanup is needed:

# Check current SG rules — verify Redis port (6379) inbound rule is restored
aws ec2 describe-security-groups \
  --group-ids {SG_ID} \
  --query "SecurityGroups[0].IpPermissions[?FromPort==\`6379\`]"

# Manually restore if missing
aws ec2 authorize-security-group-ingress \
  --group-id {SG_ID} \
  --protocol tcp \
  --port 6379 \
  --source-group {APP_SG_ID}

示例 8: SQS 队列不可用 — 消息队列韧性验证

架构模式：生产者 → SQS 队列 → 消费者（异步消息） 注入方式：SSM 自动化（SQS 队列策略拒绝 → 恢复，4 轮递增） 验证点：死信队列处理、消息背压、组件隔离、渐进式故障容忍

模板来源：aws-samples/fis-template-library/sqs-queue-impairment

内嵌模板：references/fis-templates/sqs-queue-impairment/

---

假设

当 SQS 队列访问被拒绝时（渐进式故障 — 4 轮）：

• 第一轮故障后 5 分钟内告警触发
• 组件 A（依赖此队列）应对终端用户不可用
• 其他组件（B、C）应继续正常运行
• 生产者应实现背压（停止接受无法入队的工作）
• 死信队列应捕获失败消息（如已配置）

故障轮次间（恢复窗口）：

• 消息处理应恢复正常
• 积压消息应被处理且不压垮消费者

所有故障轮次完成后：

• 5 分钟内完全恢复
• 无消息丢失（所有消息最终被处理或进入死信队列）

验证要点

• SQS 队列不可用时的监控和告警
• 生产者端错误处理和背压机制
• 消费者对间歇性队列访问故障的韧性
• 死信队列配置和消息捕获
• 组件隔离 — 一个队列故障不级联
• 渐进式故障容忍（系统能否应对恶化条件？）

前置条件

• [ ] SQS 队列已打标签 FIS-Ready=True
• [ ] 已创建 FIS IAM Role
• [ ] 已创建 SSM Automation IAM Role
• [ ] 已部署 SSM 自动化文档
• [ ] 已配置 NumberOfMessagesSent 或 ApproximateNumberOfMessagesVisible CloudWatch 告警
• [ ] 目标队列已配置死信队列（推荐）
• [ ] 应用健康检查端点可访问

渐进式故障时间表

实验总时长：约 37 分钟

部署

1. 部署 IAM 角色

aws iam create-role \
  --role-name FIS-SqsImpairment \
  --assume-role-policy-document file://references/fis-templates/sqs-queue-impairment/fis-iam-trust-relationship.json

aws iam put-role-policy \
  --role-name FIS-SqsImpairment \
  --policy-name fis-policy \
  --policy-document file://references/fis-templates/sqs-queue-impairment/sqs-queue-impairment-tag-based-fis-role-iam-policy.json

aws iam create-role \
  --role-name SSM-SqsImpairment \
  --assume-role-policy-document file://references/fis-templates/sqs-queue-impairment/ssm-iam-trust-relationship.json

aws iam put-role-policy \
  --role-name SSM-SqsImpairment \
  --policy-name ssm-policy \
  --policy-document file://references/fis-templates/sqs-queue-impairment/sqs-queue-impairment-tag-based-ssm-automation-role-iam-policy.json

2. 部署 SSM 自动化文档

aws ssm create-document \
  --name sqs-queue-impairment \
  --document-type Automation \
  --content file://references/fis-templates/sqs-queue-impairment/sqs-queue-impairment-tag-based-automation.yaml \
  --document-format YAML

3. 创建 FIS 实验

aws fis create-experiment-template \
  --cli-input-json file://references/fis-templates/sqs-queue-impairment/sqs-queue-impairment-tag-based-experiment-template.json

执行

aws fis start-experiment --experiment-template-id {TEMPLATE_ID}

# 监控队列可访问性（故障轮次期间应出现 AccessDenied）
watch -n 5 'aws sqs send-message \
  --queue-url "https://sqs.{REGION}.amazonaws.com/{ACCOUNT}/{QUEUE_NAME}" \
  --message-body "chaos-test-$(date +%s)" \
  --region {REGION} \
  --no-cli-pager 2>&1 | tail -1'

# 监控队列深度
watch -n 10 'aws sqs get-queue-attributes \
  --queue-url "https://sqs.{REGION}.amazonaws.com/{ACCOUNT}/{QUEUE_NAME}" \
  --attribute-names ApproximateNumberOfMessagesVisible ApproximateNumberOfMessagesNotVisible \
  --output table'

观测指标

清理

SSM 自动化文档会自动从 SQS 队列策略中移除拒绝声明。

如需手动清理：

# 检查当前队列策略是否有拒绝声明
aws sqs get-queue-attributes \
  --queue-url "https://sqs.{REGION}.amazonaws.com/{ACCOUNT}/{QUEUE_NAME}" \
  --attribute-names Policy

# 如仍有拒绝声明，清空策略
aws sqs set-queue-attributes \
  --queue-url "https://sqs.{REGION}.amazonaws.com/{ACCOUNT}/{QUEUE_NAME}" \
  --attributes '{"Policy": ""}'

Example 8: SQS Queue Impairment — Message Queue Resilience

Architecture pattern: Producer → SQS Queue → Consumer (async messaging) Injection method: SSM Automation (SQS queue policy deny → restore, 4 escalating rounds) Validates: DLQ handling, message backpressure, component isolation, progressive failure tolerance

Template source: aws-samples/fis-template-library/sqs-queue-impairment

Embedded template: references/fis-templates/sqs-queue-impairment/

---

Hypothesis

When SQS queue access is denied (progressive impairment — 4 rounds):

• Alarm should fire within 5 minutes of first impairment round
• Component A (dependent on this queue) should become unavailable to end users
• Other components (B, C) should continue operating normally
• Producer should implement backpressure (stop accepting work it cannot queue)
• Dead Letter Queue should capture failed messages (if configured)

Between impairment rounds (recovery windows):

• Message processing should resume normally
• Accumulated messages should be processed without overwhelming consumers

After all impairment rounds complete:

• Full recovery within 5 minutes
• No message loss (all messages eventually processed or in DLQ)

What does this enable you to verify?

• SQS monitoring and alerting for queue unavailability
• Producer-side error handling and backpressure mechanisms
• Consumer resilience to intermittent queue access failures
• Dead Letter Queue configuration and message capture
• Component isolation — failure in one queue doesn't cascade
• Progressive failure tolerance (can the system handle worsening conditions?)

Prerequisites

• [ ] SQS queue tagged with FIS-Ready=True
• [ ] FIS IAM Role created with sqs-queue-impairment-tag-based-fis-role-iam-policy.json
• [ ] SSM Automation IAM Role created with sqs-queue-impairment-tag-based-ssm-automation-role-iam-policy.json
• [ ] SSM Automation Document deployed from sqs-queue-impairment-tag-based-automation.yaml
• [ ] CloudWatch alarm for NumberOfMessagesSent or ApproximateNumberOfMessagesVisible
• [ ] Dead Letter Queue configured on target queue (recommended)
• [ ] Application health check endpoints accessible

Progressive Impairment Schedule

Total experiment duration: ~37 minutes

Setup

1. Deploy IAM Roles

aws iam create-role \
  --role-name FIS-SqsImpairment \
  --assume-role-policy-document file://references/fis-templates/sqs-queue-impairment/fis-iam-trust-relationship.json

aws iam put-role-policy \
  --role-name FIS-SqsImpairment \
  --policy-name fis-policy \
  --policy-document file://references/fis-templates/sqs-queue-impairment/sqs-queue-impairment-tag-based-fis-role-iam-policy.json

aws iam create-role \
  --role-name SSM-SqsImpairment \
  --assume-role-policy-document file://references/fis-templates/sqs-queue-impairment/ssm-iam-trust-relationship.json

aws iam put-role-policy \
  --role-name SSM-SqsImpairment \
  --policy-name ssm-policy \
  --policy-document file://references/fis-templates/sqs-queue-impairment/sqs-queue-impairment-tag-based-ssm-automation-role-iam-policy.json

2. Deploy SSM Automation Document

aws ssm create-document \
  --name sqs-queue-impairment \
  --document-type Automation \
  --content file://references/fis-templates/sqs-queue-impairment/sqs-queue-impairment-tag-based-automation.yaml \
  --document-format YAML

3. Create FIS Experiment

aws fis create-experiment-template \
  --cli-input-json file://references/fis-templates/sqs-queue-impairment/sqs-queue-impairment-tag-based-experiment-template.json

Execution

# Start the experiment
aws fis start-experiment --experiment-template-id {TEMPLATE_ID}

# Monitor queue accessibility (should see AccessDenied during impairment rounds)
watch -n 5 'aws sqs send-message \
  --queue-url "https://sqs.{REGION}.amazonaws.com/{ACCOUNT}/{QUEUE_NAME}" \
  --message-body "chaos-test-$(date +%s)" \
  --region {REGION} \
  --no-cli-pager 2>&1 | tail -1'

# Monitor queue depth
watch -n 10 'aws sqs get-queue-attributes \
  --queue-url "https://sqs.{REGION}.amazonaws.com/{ACCOUNT}/{QUEUE_NAME}" \
  --attribute-names ApproximateNumberOfMessagesVisible ApproximateNumberOfMessagesNotVisible \
  --output table'

Observation Metrics

Cleanup

The SSM Automation document automatically removes deny statements from the SQS queue policy.

If manual cleanup is needed:

# Check current queue policy for deny statements
aws sqs get-queue-attributes \
  --queue-url "https://sqs.{REGION}.amazonaws.com/{ACCOUNT}/{QUEUE_NAME}" \
  --attribute-names Policy

# If deny statement remains, remove it by setting the policy without the deny
aws sqs set-queue-attributes \
  --queue-url "https://sqs.{REGION}.amazonaws.com/{ACCOUNT}/{QUEUE_NAME}" \
  --attributes '{"Policy": ""}'

示例 9：ElastiCache AZ 电源中断 — 缓存层 AZ 韧性

架构模式：Application → ElastiCache Redis/Valkey（复制组、Multi-AZ） FIS Action：aws:elasticache:replicationgroup-interrupt-az-power 验证点：AZ 级别故障转移、副本提升、降低容量运行、连接恢复

---

稳态假设

中断目标 AZ 中 ElastiCache 节点的电源后：

• 主节点故障转移在 30 秒内完成（复制延迟最小的副本被提升）
• 应用连接池在 60 秒内重新连接到新主节点
• 缓存命中率在 5 分钟内恢复至 >= 90%
• 故障转移期间无数据丢失（实验前复制延迟接近零）
• 目标 AZ 中的只读副本替换在整个持续期间被阻止 — 集群以降低容量运行

验证要点

• ElastiCache Multi-AZ 自动故障转移是否正确工作
• 应用是否能够优雅地处理主端点变更
• AZ 级别故障下的连接池重试和重连逻辑
• 降低容量（受损 AZ 中副本被阻止）下的集群行为
• CloudWatch 告警是否检测到 ReplicationLag 尖峰和 IsPrimary 角色变更

停止条件

{
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:{region}:{account}:alarm:chaos-stop-cache-connections"
    }
  ]
}

对应告警：

aws cloudwatch put-metric-alarm \
  --alarm-name "chaos-stop-cache-connections" \
  --namespace "AWS/ElastiCache" \
  --metric-name "CurrConnections" \
  --dimensions Name=CacheClusterId,Value={cache-cluster-id} \
  --statistic Average \
  --period 60 \
  --threshold 0 \
  --comparison-operator LessThanOrEqualToThreshold \
  --evaluation-periods 3 \
  --treat-missing-data notBreaching

FIS 实验模板

{
  "description": "Interrupt ElastiCache AZ power to validate cache layer AZ resilience",
  "targets": {
    "elasticache-rg": {
      "resourceType": "aws:elasticache:replicationgroup",
      "resourceTags": {
        "AzImpairmentPower": "ElasticacheImpact"
      },
      "parameters": {
        "availabilityZoneIdentifier": "{target-az}"
      },
      "selectionMode": "ALL"
    }
  },
  "actions": {
    "interrupt-az-power": {
      "actionId": "aws:elasticache:replicationgroup-interrupt-az-power",
      "parameters": {
        "duration": "PT10M"
      },
      "targets": {
        "ReplicationGroups": "elasticache-rg"
      }
    }
  },
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:{region}:{account}:alarm:chaos-stop-cache-connections"
    }
  ],
  "roleArn": "arn:aws:iam::{account}:role/FISExperimentRole",
  "tags": {
    "Purpose": "chaos-engineering",
    "RiskId": "R-009"
  }
}

重要说明：

• 目标选择仅使用 resourceTags — ElastiCache 复制组不支持 resourceArns 和 filters
• 该操作已从 aws:elasticache:interrupt-cluster-az-power 重命名 — 请始终使用新名称
• 需要在复制组上启用 Multi-AZ；不支持 ElastiCache Serverless

IAM 权限

ElastiCache FIS 操作没有 AWS 托管策略。请附加内联策略：

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ElastiCacheActions",
      "Effect": "Allow",
      "Action": [
        "elasticache:InterruptClusterAzPower",
        "elasticache:DescribeReplicationGroups"
      ],
      "Resource": "*"
    },
    {
      "Sid": "TagResolution",
      "Effect": "Allow",
      "Action": [
        "tag:GetResources"
      ],
      "Resource": "*"
    }
  ]
}

执行命令

# Tag the replication group
aws elasticache add-tags-to-resource \
  --resource-name "arn:aws:elasticache:{region}:{account}:replicationgroup:{rg-id}" \
  --tags Key=AzImpairmentPower,Value=ElasticacheImpact

# Check pre-experiment state
aws elasticache describe-replication-groups \
  --replication-group-id {rg-id} \
  --query 'ReplicationGroups[0].NodeGroups[].NodeGroupMembers[].[CacheClusterId,CurrentRole,PreferredAvailabilityZone]' \
  --output table

# Create and start experiment
aws fis create-experiment-template --cli-input-json file://examples/09-elasticache-az-power-template.json
aws fis start-experiment --experiment-template-id {TEMPLATE_ID}

# Monitor failover
watch -n 5 'aws elasticache describe-replication-groups \
  --replication-group-id {rg-id} \
  --query "ReplicationGroups[0].NodeGroups[].NodeGroupMembers[].[CacheClusterId,CurrentRole,PreferredAvailabilityZone]" \
  --output table'

观测指标

预期结果

如果失败：常见原因 — 未启用 Multi-AZ、应用使用节点特定端点而非主端点、连接池缺少重试逻辑、复制组为 ElastiCache Serverless（不支持）。

Example 9: ElastiCache AZ Power Interruption — Cache Layer AZ Resilience

Architecture pattern: Application → ElastiCache Redis/Valkey (replication group, Multi-AZ) FIS Action: aws:elasticache:replicationgroup-interrupt-az-power Validation target: AZ-level failover, replica promotion, reduced-capacity operation, connection recovery

---

Steady-State Hypothesis

After interrupting power to ElastiCache nodes in the target AZ:

• Primary node failover completes within 30 seconds (replica with least replication lag is promoted)
• Application connection pool reconnects to the new primary within 60 seconds
• Cache hit rate recovers to >= 90% within 5 minutes
• No data loss during failover (replication lag near zero pre-experiment)
• Read replica replacements in the target AZ are blocked for the entire duration — cluster operates at reduced capacity

What does this enable you to verify?

• ElastiCache Multi-AZ automatic failover works correctly
• Application handles primary endpoint changes gracefully
• Connection pool retry and reconnect logic under AZ-level failure
• Cluster behavior under reduced capacity (replicas blocked in impaired AZ)
• CloudWatch alarm detection for ReplicationLag spike and IsPrimary role change

Stop Conditions

{
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:{region}:{account}:alarm:chaos-stop-cache-connections"
    }
  ]
}

Corresponding Alarm:

aws cloudwatch put-metric-alarm \
  --alarm-name "chaos-stop-cache-connections" \
  --namespace "AWS/ElastiCache" \
  --metric-name "CurrConnections" \
  --dimensions Name=CacheClusterId,Value={cache-cluster-id} \
  --statistic Average \
  --period 60 \
  --threshold 0 \
  --comparison-operator LessThanOrEqualToThreshold \
  --evaluation-periods 3 \
  --treat-missing-data notBreaching

FIS Experiment Template

{
  "description": "Interrupt ElastiCache AZ power to validate cache layer AZ resilience",
  "targets": {
    "elasticache-rg": {
      "resourceType": "aws:elasticache:replicationgroup",
      "resourceTags": {
        "AzImpairmentPower": "ElasticacheImpact"
      },
      "parameters": {
        "availabilityZoneIdentifier": "{target-az}"
      },
      "selectionMode": "ALL"
    }
  },
  "actions": {
    "interrupt-az-power": {
      "actionId": "aws:elasticache:replicationgroup-interrupt-az-power",
      "parameters": {
        "duration": "PT10M"
      },
      "targets": {
        "ReplicationGroups": "elasticache-rg"
      }
    }
  },
  "stopConditions": [
    {
      "source": "aws:cloudwatch:alarm",
      "value": "arn:aws:cloudwatch:{region}:{account}:alarm:chaos-stop-cache-connections"
    }
  ],
  "roleArn": "arn:aws:iam::{account}:role/FISExperimentRole",
  "tags": {
    "Purpose": "chaos-engineering",
    "RiskId": "R-009"
  }
}

Important notes:

• Target selection uses resourceTags ONLY — resourceArns and filters are NOT supported for ElastiCache replication groups
• The action was renamed from aws:elasticache:interrupt-cluster-az-power — always use the new name
• Requires Multi-AZ enabled on the replication group; NOT supported on ElastiCache Serverless

IAM Permissions

No AWS managed policy exists for ElastiCache FIS actions. Attach an inline policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ElastiCacheActions",
      "Effect": "Allow",
      "Action": [
        "elasticache:InterruptClusterAzPower",
        "elasticache:DescribeReplicationGroups"
      ],
      "Resource": "*"
    },
    {
      "Sid": "TagResolution",
      "Effect": "Allow",
      "Action": [
        "tag:GetResources"
      ],
      "Resource": "*"
    }
  ]
}

Execution Commands

# Tag the replication group
aws elasticache add-tags-to-resource \
  --resource-name "arn:aws:elasticache:{region}:{account}:replicationgroup:{rg-id}" \
  --tags Key=AzImpairmentPower,Value=ElasticacheImpact

# Check pre-experiment state
aws elasticache describe-replication-groups \
  --replication-group-id {rg-id} \
  --query 'ReplicationGroups[0].NodeGroups[].NodeGroupMembers[].[CacheClusterId,CurrentRole,PreferredAvailabilityZone]' \
  --output table

# Create and start experiment
aws fis create-experiment-template --cli-input-json file://examples/09-elasticache-az-power-template.json
aws fis start-experiment --experiment-template-id {TEMPLATE_ID}

# Monitor failover
watch -n 5 'aws elasticache describe-replication-groups \
  --replication-group-id {rg-id} \
  --query "ReplicationGroups[0].NodeGroups[].NodeGroupMembers[].[CacheClusterId,CurrentRole,PreferredAvailabilityZone]" \
  --output table'

Observation Metrics

Expected Results

If failed: Common causes — Multi-AZ not enabled, application using node-specific endpoints instead of primary endpoint, connection pool lacks retry logic, replication group is ElastiCache Serverless (not supported).

示例 10：ElastiCache 主节点重启 — 连接池韧性

架构模式：Application → ElastiCache Redis/Valkey（复制组、集群模式禁用） 注入方式：SSM Automation（RebootCacheCluster → 等待 available） 验证点：连接池韧性、重试逻辑、主节点短暂重启恢复

模板来源：基于 aws-samples/fis-template-library/elasticache-redis-primary-node-reboot

---

假设

当 ElastiCache 主节点被重启时：

• 应用连接经历短暂中断（1-3 分钟）
• 连接池检测到故障并自动重连
• 不会导致上游服务级联故障
• 主节点在 5 分钟内恢复到 available 状态
• 与 AZ 电源中断不同，副本替换不会被阻止

关键限制：RebootCacheCluster 不支持集群模式启用的集群（ClusterEnabled: true）。如果启用了集群模式，请使用示例 09（AZ 电源中断）或示例 11（故障转移）替代。

验证要点

• 主节点重启后 Redis/Valkey 客户端连接池的重连能力
• 缓存短暂不可用期间的应用重试逻辑
• 单节点故障影响（比 AZ 级别电源中断破坏性更小）
• 节点重启后的缓存预热行为
• 节点重启（同一节点保持主节点）与故障转移（角色互换）的区别

前置条件

• [ ] ElastiCache Redis/Valkey 复制组，ClusterEnabled: false
• [ ] 已启用 Multi-AZ 和自动故障转移
• [ ] 已确定主节点 CacheClusterId（参见执行命令）
• [ ] FIS IAM 角色，具有 AWSFaultInjectionSimulatorSSMAccess + iam:PassRole
• [ ] SSM Automation IAM 角色，具有 elasticache:RebootCacheCluster + elasticache:DescribeCacheClusters
• [ ] 应用连接池已配置重试逻辑

SSM 自动化 Runbook

description: 'FIS: Reboot ElastiCache primary node to test client resilience'
schemaVersion: '0.3'
assumeRole: '{{ AutomationAssumeRoleArn }}'
parameters:
  AutomationAssumeRoleArn:
    type: String
    description: IAM Role ARN for SSM Automation to assume
  CacheClusterId:
    type: String
    description: CacheClusterId of the primary node to reboot
mainSteps:
  - name: RebootPrimaryNode
    action: aws:executeAwsApi
    inputs:
      Service: elasticache
      Api: RebootCacheCluster
      CacheClusterId: '{{ CacheClusterId }}'
      CacheNodeIdsToReboot:
        - '0001'
    outputs:
      - Name: CacheClusterId
        Selector: $.CacheCluster.CacheClusterId
        Type: String

  - name: WaitForNodeAvailable
    action: aws:waitForAwsResourceProperty
    timeoutSeconds: 600
    inputs:
      Service: elasticache
      Api: DescribeCacheClusters
      CacheClusterId: '{{ CacheClusterId }}'
      PropertySelector: $.CacheClusters[0].CacheClusterStatus
      DesiredValues:
        - available

部署

1. 部署 SSM 自动化文档

aws ssm create-document \
  --name elasticache-primary-reboot \
  --document-type Automation \
  --content file://ssm-elasticache-reboot.yaml \
  --document-format YAML

2. 部署 IAM 角色

# FIS Role (trusts fis.amazonaws.com)
aws iam create-role \
  --role-name FIS-EC-Reboot \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Principal": {"Service": "fis.amazonaws.com"}, "Action": "sts:AssumeRole"}]
  }'

aws iam attach-role-policy \
  --role-name FIS-EC-Reboot \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSFaultInjectionSimulatorSSMAccess

aws iam put-role-policy \
  --role-name FIS-EC-Reboot \
  --policy-name PassSSMRole \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::{account}:role/SSM-EC-Reboot", "Condition": {"StringEquals": {"iam:PassedToService": "ssm.amazonaws.com"}}}]
  }'

# SSM Automation Role (trusts ssm.amazonaws.com)
aws iam create-role \
  --role-name SSM-EC-Reboot \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Principal": {"Service": "ssm.amazonaws.com"}, "Action": "sts:AssumeRole"}]
  }'

aws iam put-role-policy \
  --role-name SSM-EC-Reboot \
  --policy-name ElastiCacheAccess \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Action": ["elasticache:RebootCacheCluster", "elasticache:DescribeCacheClusters"], "Resource": "*"}]
  }'

3. 创建 FIS 实验

aws fis create-experiment-template --cli-input-json '{
  "description": "Reboot ElastiCache primary node to test connection pool resilience",
  "targets": {},
  "actions": {
    "reboot-primary": {
      "actionId": "aws:ssm:start-automation-execution",
      "parameters": {
        "documentArn": "arn:aws:ssm:{region}:{account}:document/elasticache-primary-reboot",
        "documentParameters": "{\"AutomationAssumeRoleArn\":\"arn:aws:iam::{account}:role/SSM-EC-Reboot\",\"CacheClusterId\":\"{primary-cache-cluster-id}\"}",
        "maxDuration": "PT15M"
      }
    }
  },
  "stopConditions": [{"source": "none"}],
  "roleArn": "arn:aws:iam::{account}:role/FIS-EC-Reboot",
  "tags": {"Purpose": "chaos-engineering", "RiskId": "R-010"}
}'

执行

# Identify the primary node
aws elasticache describe-replication-groups \
  --replication-group-id {rg-id} \
  --query 'ReplicationGroups[0].NodeGroups[].NodeGroupMembers[].[CacheClusterId,CurrentRole,PreferredAvailabilityZone]' \
  --output table

# Verify cluster mode is disabled
aws elasticache describe-replication-groups \
  --replication-group-id {rg-id} \
  --query 'ReplicationGroups[0].ClusterEnabled'

# Start the experiment
aws fis start-experiment --experiment-template-id {TEMPLATE_ID}

# Monitor node status
watch -n 10 'aws elasticache describe-cache-clusters \
  --cache-cluster-id {primary-cache-cluster-id} \
  --query "CacheClusters[0].CacheClusterStatus"'

观测指标

清理

SSM 自动化 Runbook 会等待节点恢复到 available 状态 — 无需手动清理。

# Verify node is back to available
aws elasticache describe-cache-clusters \
  --cache-cluster-id {primary-cache-cluster-id} \
  --query "CacheClusters[0].CacheClusterStatus"

# Delete experiment template when done
aws fis delete-experiment-template --id {TEMPLATE_ID}

Example 10: ElastiCache Primary Node Reboot — Connection Pool Resilience

Architecture pattern: Application → ElastiCache Redis/Valkey (replication group, cluster mode disabled) Injection method: SSM Automation (RebootCacheCluster → wait for available) Validates: Connection pool resilience, retry logic, brief primary node restart recovery

Template source: Based on aws-samples/fis-template-library/elasticache-redis-primary-node-reboot

---

Hypothesis

When the ElastiCache primary node is rebooted:

• Application connections experience a brief interruption (1-3 minutes)
• Connection pool detects the failure and reconnects automatically
• No cascading failures to upstream services
• Primary node returns to available status within 5 minutes
• Unlike AZ power interruption, replica replacements are NOT blocked

Critical limitation: RebootCacheCluster is NOT supported on cluster-mode-enabled clusters (ClusterEnabled: true). If cluster mode is enabled, use Example 09 (AZ Power Interruption) or Example 11 (Failover) instead.

What does this enable you to verify?

• Redis/Valkey client connection pool reconnection after primary node restart
• Application retry logic during brief cache unavailability
• Single-node failure impact (less disruptive than AZ-level power interruption)
• Cache warming behavior after node reboot
• Difference between node reboot (same node stays primary) vs. failover (role swap)

Prerequisites

• [ ] ElastiCache Redis/Valkey replication group with ClusterEnabled: false
• [ ] Multi-AZ and Automatic Failover enabled
• [ ] Primary node CacheClusterId identified (see Execution Commands)
• [ ] FIS IAM Role with AWSFaultInjectionSimulatorSSMAccess + iam:PassRole
• [ ] SSM Automation IAM Role with elasticache:RebootCacheCluster + elasticache:DescribeCacheClusters
• [ ] Application connection pool configured with retry logic

SSM Automation Runbook

description: 'FIS: Reboot ElastiCache primary node to test client resilience'
schemaVersion: '0.3'
assumeRole: '{{ AutomationAssumeRoleArn }}'
parameters:
  AutomationAssumeRoleArn:
    type: String
    description: IAM Role ARN for SSM Automation to assume
  CacheClusterId:
    type: String
    description: CacheClusterId of the primary node to reboot
mainSteps:
  - name: RebootPrimaryNode
    action: aws:executeAwsApi
    inputs:
      Service: elasticache
      Api: RebootCacheCluster
      CacheClusterId: '{{ CacheClusterId }}'
      CacheNodeIdsToReboot:
        - '0001'
    outputs:
      - Name: CacheClusterId
        Selector: $.CacheCluster.CacheClusterId
        Type: String

  - name: WaitForNodeAvailable
    action: aws:waitForAwsResourceProperty
    timeoutSeconds: 600
    inputs:
      Service: elasticache
      Api: DescribeCacheClusters
      CacheClusterId: '{{ CacheClusterId }}'
      PropertySelector: $.CacheClusters[0].CacheClusterStatus
      DesiredValues:
        - available

Setup

1. Deploy SSM Automation Document

aws ssm create-document \
  --name elasticache-primary-reboot \
  --document-type Automation \
  --content file://ssm-elasticache-reboot.yaml \
  --document-format YAML

2. Deploy IAM Roles

# FIS Role (trusts fis.amazonaws.com)
aws iam create-role \
  --role-name FIS-EC-Reboot \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Principal": {"Service": "fis.amazonaws.com"}, "Action": "sts:AssumeRole"}]
  }'

aws iam attach-role-policy \
  --role-name FIS-EC-Reboot \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSFaultInjectionSimulatorSSMAccess

aws iam put-role-policy \
  --role-name FIS-EC-Reboot \
  --policy-name PassSSMRole \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::{account}:role/SSM-EC-Reboot", "Condition": {"StringEquals": {"iam:PassedToService": "ssm.amazonaws.com"}}}]
  }'

# SSM Automation Role (trusts ssm.amazonaws.com)
aws iam create-role \
  --role-name SSM-EC-Reboot \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Principal": {"Service": "ssm.amazonaws.com"}, "Action": "sts:AssumeRole"}]
  }'

aws iam put-role-policy \
  --role-name SSM-EC-Reboot \
  --policy-name ElastiCacheAccess \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Action": ["elasticache:RebootCacheCluster", "elasticache:DescribeCacheClusters"], "Resource": "*"}]
  }'

3. Create FIS Experiment

aws fis create-experiment-template --cli-input-json '{
  "description": "Reboot ElastiCache primary node to test connection pool resilience",
  "targets": {},
  "actions": {
    "reboot-primary": {
      "actionId": "aws:ssm:start-automation-execution",
      "parameters": {
        "documentArn": "arn:aws:ssm:{region}:{account}:document/elasticache-primary-reboot",
        "documentParameters": "{\"AutomationAssumeRoleArn\":\"arn:aws:iam::{account}:role/SSM-EC-Reboot\",\"CacheClusterId\":\"{primary-cache-cluster-id}\"}",
        "maxDuration": "PT15M"
      }
    }
  },
  "stopConditions": [{"source": "none"}],
  "roleArn": "arn:aws:iam::{account}:role/FIS-EC-Reboot",
  "tags": {"Purpose": "chaos-engineering", "RiskId": "R-010"}
}'

Execution

# Identify the primary node
aws elasticache describe-replication-groups \
  --replication-group-id {rg-id} \
  --query 'ReplicationGroups[0].NodeGroups[].NodeGroupMembers[].[CacheClusterId,CurrentRole,PreferredAvailabilityZone]' \
  --output table

# Verify cluster mode is disabled
aws elasticache describe-replication-groups \
  --replication-group-id {rg-id} \
  --query 'ReplicationGroups[0].ClusterEnabled'

# Start the experiment
aws fis start-experiment --experiment-template-id {TEMPLATE_ID}

# Monitor node status
watch -n 10 'aws elasticache describe-cache-clusters \
  --cache-cluster-id {primary-cache-cluster-id} \
  --query "CacheClusters[0].CacheClusterStatus"'

Observation Metrics

Cleanup

The SSM Automation runbook waits for the node to return to available — no manual cleanup needed.

# Verify node is back to available
aws elasticache describe-cache-clusters \
  --cache-cluster-id {primary-cache-cluster-id} \
  --query "CacheClusters[0].CacheClusterStatus"

# Delete experiment template when done
aws fis delete-experiment-template --id {TEMPLATE_ID}

示例 11：ElastiCache 复制组故障转移 — 主副本角色切换

架构模式：应用程序 → ElastiCache Redis/Valkey（复制组，集群模式禁用或启用） 注入方式：SSM Automation（TestFailover → 等待 available） 验证点：自动故障转移、主副本角色切换、端点拓扑变更、连接恢复

模板来源：基于 ElastiCache TestFailover API

---

假设

当 ElastiCache 复制组触发自动故障转移时：

• 当前主节点变为副本，副本被提升为主节点
• 应用程序检测到端点拓扑变更并在 60 秒内重新连接
• 无数据丢失——提升后对新主节点的写入成功
• 复制组在 5 分钟内恢复到 available 状态
• 比重启（示例 10）更真实，因为端点拓扑实际发生了变化

与示例 10（主节点重启）的关键差异：

验证要点

• ElastiCache 自动故障转移机制和角色切换行为
• 应用程序对主端点拓扑变更的处理能力
• 主节点身份变更（而非仅重启）时的连接池行为
• 集群模式启用时的分片级故障转移（定向特定分片）
• 集群模式启用时的故障转移速率限制和顺序约束

前置条件

• [ ] 已启用自动故障转移的 ElastiCache Redis/Valkey 复制组
• [ ] 已启用多可用区
• [ ] 已确认 ReplicationGroupId 和目标 NodeGroupId
• [ ] 具有 AWSFaultInjectionSimulatorSSMAccess + iam:PassRole 权限的 FIS IAM 角色
• [ ] 具有 elasticache:TestFailover + elasticache:DescribeReplicationGroups 权限的 SSM Automation IAM 角色
• [ ] 集群模式启用时：已记录分片分布基线

SSM 自动化 Runbook

description: 'FIS: Test automatic failover on ElastiCache replication group'
schemaVersion: '0.3'
assumeRole: '{{ AutomationAssumeRoleArn }}'
parameters:
  AutomationAssumeRoleArn:
    type: String
    description: IAM Role ARN for SSM Automation to assume
  ReplicationGroupId:
    type: String
    description: Replication group ID to trigger failover on
  NodeGroupId:
    type: String
    description: >-
      Node group (shard) ID to failover. For cluster-mode-disabled,
      this is always '0001'. For cluster-mode-enabled, specify the
      target shard ID (e.g., '0001', '0002').
mainSteps:
  - name: TestFailover
    action: aws:executeAwsApi
    inputs:
      Service: elasticache
      Api: TestFailover
      ReplicationGroupId: '{{ ReplicationGroupId }}'
      NodeGroupId: '{{ NodeGroupId }}'
    outputs:
      - Name: ReplicationGroupId
        Selector: $.ReplicationGroup.ReplicationGroupId
        Type: String

  - name: WaitForReplicationGroupAvailable
    action: aws:waitForAwsResourceProperty
    timeoutSeconds: 600
    inputs:
      Service: elasticache
      Api: DescribeReplicationGroups
      ReplicationGroupId: '{{ ReplicationGroupId }}'
      PropertySelector: $.ReplicationGroups[0].Status
      DesiredValues:
        - available

部署

1. 部署 SSM Automation 文档

aws ssm create-document \
  --name elasticache-test-failover \
  --document-type Automation \
  --content file://ssm-elasticache-failover.yaml \
  --document-format YAML

2. 部署 IAM 角色

# FIS Role (trusts fis.amazonaws.com)
aws iam create-role \
  --role-name FIS-EC-Failover \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Principal": {"Service": "fis.amazonaws.com"}, "Action": "sts:AssumeRole"}]
  }'

aws iam attach-role-policy \
  --role-name FIS-EC-Failover \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSFaultInjectionSimulatorSSMAccess

aws iam put-role-policy \
  --role-name FIS-EC-Failover \
  --policy-name PassSSMRole \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::{account}:role/SSM-EC-Failover", "Condition": {"StringEquals": {"iam:PassedToService": "ssm.amazonaws.com"}}}]
  }'

# SSM Automation Role (trusts ssm.amazonaws.com)
aws iam create-role \
  --role-name SSM-EC-Failover \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Principal": {"Service": "ssm.amazonaws.com"}, "Action": "sts:AssumeRole"}]
  }'

aws iam put-role-policy \
  --role-name SSM-EC-Failover \
  --policy-name ElastiCacheAccess \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Action": ["elasticache:TestFailover", "elasticache:DescribeReplicationGroups"], "Resource": "*"}]
  }'

3. 创建 FIS 实验

aws fis create-experiment-template --cli-input-json '{
  "description": "Test ElastiCache automatic failover to validate primary-replica role swap",
  "targets": {},
  "actions": {
    "test-failover": {
      "actionId": "aws:ssm:start-automation-execution",
      "parameters": {
        "documentArn": "arn:aws:ssm:{region}:{account}:document/elasticache-test-failover",
        "documentParameters": "{\"AutomationAssumeRoleArn\":\"arn:aws:iam::{account}:role/SSM-EC-Failover\",\"ReplicationGroupId\":\"{rg-id}\",\"NodeGroupId\":\"0001\"}",
        "maxDuration": "PT15M"
      }
    }
  },
  "stopConditions": [{"source": "none"}],
  "roleArn": "arn:aws:iam::{account}:role/FIS-EC-Failover",
  "tags": {"Purpose": "chaos-engineering", "RiskId": "R-011"}
}'

执行

# Check current node roles (cluster-mode-disabled)
aws elasticache describe-replication-groups \
  --replication-group-id {rg-id} \
  --query 'ReplicationGroups[0].NodeGroups[].NodeGroupMembers[].[CacheClusterId,CurrentRole,PreferredAvailabilityZone]' \
  --output table

# For cluster-mode-enabled (CurrentRole is null), use CloudWatch IsMaster metric
# IsMaster = 1.0 → Primary, IsMaster = 0.0 → Replica

# Start the experiment
aws fis start-experiment --experiment-template-id {TEMPLATE_ID}

# Monitor replication group status
watch -n 10 'aws elasticache describe-replication-groups \
  --replication-group-id {rg-id} \
  --query "ReplicationGroups[0].{Status:Status,NodeGroups:NodeGroups[].NodeGroupMembers[].[CacheClusterId,CurrentRole]}" \
  --output table'

观测指标

清理

SSM Automation Runbook 会等待复制组恢复到 available 状态。故障转移是永久性的——角色已切换，无需回滚。

# Verify replication group is available
aws elasticache describe-replication-groups \
  --replication-group-id {rg-id} \
  --query "ReplicationGroups[0].Status"

# Verify new role assignment
aws elasticache describe-replication-groups \
  --replication-group-id {rg-id} \
  --query 'ReplicationGroups[0].NodeGroups[].NodeGroupMembers[].[CacheClusterId,CurrentRole]' \
  --output table

# Delete experiment template when done
aws fis delete-experiment-template --id {TEMPLATE_ID}

速率限制：每滚动 24 小时最多 15 个节点组。集群模式启用时，第一次故障转移必须完成后才能在同一复制组上触发另一次故障转移。

Example 11: ElastiCache Replication Group Failover — Primary-Replica Role Swap

Architecture pattern: Application → ElastiCache Redis/Valkey (replication group, cluster mode disabled or enabled) Injection method: SSM Automation (TestFailover → wait for available) Validates: Automatic failover, primary-replica role swap, endpoint topology change, connection recovery

Template source: Based on ElastiCache TestFailover API

---

Hypothesis

When automatic failover is triggered on an ElastiCache replication group:

• The current primary becomes a replica and a replica is promoted to primary
• Application detects the endpoint topology change and reconnects within 60 seconds
• No data loss — writes to the new primary succeed after promotion
• The replication group returns to available status within 5 minutes
• More realistic than a reboot (Example 10) because the endpoint topology actually changes

Key differences from Example 10 (Primary Reboot):

What does this enable you to verify?

• ElastiCache automatic failover mechanism and role swap behavior
• Application handling of primary endpoint topology changes
• Connection pool behavior when primary identity changes (not just restarts)
• Cluster-mode-enabled shard-level failover (target specific shards)
• Failover rate limits and sequential constraints for cluster-mode-enabled

Prerequisites

• [ ] ElastiCache Redis/Valkey replication group with Automatic Failover enabled
• [ ] Multi-AZ enabled
• [ ] ReplicationGroupId and target NodeGroupId identified
• [ ] FIS IAM Role with AWSFaultInjectionSimulatorSSMAccess + iam:PassRole
• [ ] SSM Automation IAM Role with elasticache:TestFailover + elasticache:DescribeReplicationGroups
• [ ] For cluster-mode-enabled: shard distribution baseline recorded

SSM Automation Runbook

description: 'FIS: Test automatic failover on ElastiCache replication group'
schemaVersion: '0.3'
assumeRole: '{{ AutomationAssumeRoleArn }}'
parameters:
  AutomationAssumeRoleArn:
    type: String
    description: IAM Role ARN for SSM Automation to assume
  ReplicationGroupId:
    type: String
    description: Replication group ID to trigger failover on
  NodeGroupId:
    type: String
    description: >-
      Node group (shard) ID to failover. For cluster-mode-disabled,
      this is always '0001'. For cluster-mode-enabled, specify the
      target shard ID (e.g., '0001', '0002').
mainSteps:
  - name: TestFailover
    action: aws:executeAwsApi
    inputs:
      Service: elasticache
      Api: TestFailover
      ReplicationGroupId: '{{ ReplicationGroupId }}'
      NodeGroupId: '{{ NodeGroupId }}'
    outputs:
      - Name: ReplicationGroupId
        Selector: $.ReplicationGroup.ReplicationGroupId
        Type: String

  - name: WaitForReplicationGroupAvailable
    action: aws:waitForAwsResourceProperty
    timeoutSeconds: 600
    inputs:
      Service: elasticache
      Api: DescribeReplicationGroups
      ReplicationGroupId: '{{ ReplicationGroupId }}'
      PropertySelector: $.ReplicationGroups[0].Status
      DesiredValues:
        - available

Setup

1. Deploy SSM Automation Document

aws ssm create-document \
  --name elasticache-test-failover \
  --document-type Automation \
  --content file://ssm-elasticache-failover.yaml \
  --document-format YAML

2. Deploy IAM Roles

# FIS Role (trusts fis.amazonaws.com)
aws iam create-role \
  --role-name FIS-EC-Failover \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Principal": {"Service": "fis.amazonaws.com"}, "Action": "sts:AssumeRole"}]
  }'

aws iam attach-role-policy \
  --role-name FIS-EC-Failover \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSFaultInjectionSimulatorSSMAccess

aws iam put-role-policy \
  --role-name FIS-EC-Failover \
  --policy-name PassSSMRole \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::{account}:role/SSM-EC-Failover", "Condition": {"StringEquals": {"iam:PassedToService": "ssm.amazonaws.com"}}}]
  }'

# SSM Automation Role (trusts ssm.amazonaws.com)
aws iam create-role \
  --role-name SSM-EC-Failover \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Principal": {"Service": "ssm.amazonaws.com"}, "Action": "sts:AssumeRole"}]
  }'

aws iam put-role-policy \
  --role-name SSM-EC-Failover \
  --policy-name ElastiCacheAccess \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Action": ["elasticache:TestFailover", "elasticache:DescribeReplicationGroups"], "Resource": "*"}]
  }'

3. Create FIS Experiment

aws fis create-experiment-template --cli-input-json '{
  "description": "Test ElastiCache automatic failover to validate primary-replica role swap",
  "targets": {},
  "actions": {
    "test-failover": {
      "actionId": "aws:ssm:start-automation-execution",
      "parameters": {
        "documentArn": "arn:aws:ssm:{region}:{account}:document/elasticache-test-failover",
        "documentParameters": "{\"AutomationAssumeRoleArn\":\"arn:aws:iam::{account}:role/SSM-EC-Failover\",\"ReplicationGroupId\":\"{rg-id}\",\"NodeGroupId\":\"0001\"}",
        "maxDuration": "PT15M"
      }
    }
  },
  "stopConditions": [{"source": "none"}],
  "roleArn": "arn:aws:iam::{account}:role/FIS-EC-Failover",
  "tags": {"Purpose": "chaos-engineering", "RiskId": "R-011"}
}'

Execution

# Check current node roles (cluster-mode-disabled)
aws elasticache describe-replication-groups \
  --replication-group-id {rg-id} \
  --query 'ReplicationGroups[0].NodeGroups[].NodeGroupMembers[].[CacheClusterId,CurrentRole,PreferredAvailabilityZone]' \
  --output table

# For cluster-mode-enabled (CurrentRole is null), use CloudWatch IsMaster metric
# IsMaster = 1.0 → Primary, IsMaster = 0.0 → Replica

# Start the experiment
aws fis start-experiment --experiment-template-id {TEMPLATE_ID}

# Monitor replication group status
watch -n 10 'aws elasticache describe-replication-groups \
  --replication-group-id {rg-id} \
  --query "ReplicationGroups[0].{Status:Status,NodeGroups:NodeGroups[].NodeGroupMembers[].[CacheClusterId,CurrentRole]}" \
  --output table'

Observation Metrics

Cleanup

The SSM Automation runbook waits for the replication group to return to available. The failover is permanent — roles are swapped. No rollback needed.

# Verify replication group is available
aws elasticache describe-replication-groups \
  --replication-group-id {rg-id} \
  --query "ReplicationGroups[0].Status"

# Verify new role assignment
aws elasticache describe-replication-groups \
  --replication-group-id {rg-id} \
  --query 'ReplicationGroups[0].NodeGroups[].NodeGroupMembers[].[CacheClusterId,CurrentRole]' \
  --output table

# Delete experiment template when done
aws fis delete-experiment-template --id {TEMPLATE_ID}

Rate limits: Up to 15 node groups per rolling 24-hour period. For cluster-mode-enabled, the first failover must complete before triggering another on the same replication group.

示例 12：MSK Broker 重启 — Kafka 客户端韧性

架构模式：Kafka 生产者/消费者 → Amazon MSK（托管 Kafka 集群） 注入方式：SSM Automation（kafka:RebootBroker → 等待 ACTIVE） 验证点：Kafka 生产者/消费者韧性、分区领导者选举、Broker 恢复

说明：MSK 没有原生 FIS 操作。故障注入使用 SSM Automation 调用 MSK API。

---

假设

当 MSK Broker 被重启时：

• Kafka 生产者切换到其他 Broker 并以最小消息丢失继续生产
• Kafka 消费者在 60 秒内完成分区再平衡并恢复消费
• 集群状态从 REBOOTING_BROKER 恢复到 ACTIVE
• 重启 Broker 上的分区领导者被重新选举到其他 Broker
• 无需人工干预——Broker 自动恢复上线

验证要点

• Kafka 生产者 acks 配置和 Broker 故障时的重试行为
• 消费者组再平衡速度和分区重新分配
• 单个 Broker 重启后的 MSK 集群恢复时间
• 应用程序对分区领导者变更的韧性
• 通过 FIS 执行 SSM Automation 的双角色 IAM 模式

前置条件

• [ ] 处于 ACTIVE 状态的 Amazon MSK 集群
• [ ] 已确认目标 Broker ID（使用 aws kafka list-nodes）
• [ ] 具有 AWSFaultInjectionSimulatorSSMAccess + iam:PassRole 权限的 FIS IAM 角色
• [ ] 具有 kafka:RebootBroker + kafka:DescribeCluster 权限的 SSM Automation IAM 角色
• [ ] Kafka 生产者已配置 acks=all 和重试逻辑
• [ ] Kafka 消费者已配置适当的 session.timeout.ms 和 heartbeat.interval.ms

SSM 自动化 Runbook

description: 'FIS: Reboot MSK broker to test Kafka client resilience'
schemaVersion: '0.3'
assumeRole: '{{ AutomationAssumeRoleArn }}'
parameters:
  AutomationAssumeRoleArn:
    type: String
    description: IAM Role ARN for SSM Automation to assume
  ClusterArn:
    type: String
    description: MSK cluster ARN
  BrokerId:
    type: String
    description: 'Broker ID to reboot (e.g., 1, 2, 3)'
mainSteps:
  - name: RebootMskBroker
    action: aws:executeAwsApi
    inputs:
      Service: kafka
      Api: RebootBroker
      ClusterArn: '{{ ClusterArn }}'
      BrokerIds:
        - '{{ BrokerId }}'
    outputs:
      - Name: ClusterArn
        Selector: $.ClusterArn
        Type: String
      - Name: ClusterOperationArn
        Selector: $.ClusterOperationArn
        Type: String

  - name: WaitForClusterActive
    action: aws:waitForAwsResourceProperty
    timeoutSeconds: 600
    inputs:
      Service: kafka
      Api: DescribeCluster
      ClusterArn: '{{ ClusterArn }}'
      PropertySelector: $.ClusterInfo.State
      DesiredValues:
        - ACTIVE

部署

1. 部署 SSM Automation 文档

aws ssm create-document \
  --name msk-broker-reboot \
  --document-type Automation \
  --content file://ssm-msk-broker-reboot.yaml \
  --document-format YAML

2. 部署 IAM 角色

# FIS Role (trusts fis.amazonaws.com)
aws iam create-role \
  --role-name FIS-MSK-Reboot \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Principal": {"Service": "fis.amazonaws.com"}, "Action": "sts:AssumeRole"}]
  }'

aws iam attach-role-policy \
  --role-name FIS-MSK-Reboot \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSFaultInjectionSimulatorSSMAccess

aws iam put-role-policy \
  --role-name FIS-MSK-Reboot \
  --policy-name PassSSMRole \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::{account}:role/SSM-MSK-Reboot", "Condition": {"StringEquals": {"iam:PassedToService": "ssm.amazonaws.com"}}}]
  }'

# SSM Automation Role (trusts ssm.amazonaws.com)
aws iam create-role \
  --role-name SSM-MSK-Reboot \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Principal": {"Service": "ssm.amazonaws.com"}, "Action": "sts:AssumeRole"}]
  }'

aws iam put-role-policy \
  --role-name SSM-MSK-Reboot \
  --policy-name MskAccess \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Action": ["kafka:RebootBroker", "kafka:DescribeCluster"], "Resource": "*"}]
  }'

3. 创建 FIS 实验

aws fis create-experiment-template --cli-input-json '{
  "description": "Reboot MSK broker to test Kafka producer and consumer resilience",
  "targets": {},
  "actions": {
    "reboot-broker": {
      "actionId": "aws:ssm:start-automation-execution",
      "parameters": {
        "documentArn": "arn:aws:ssm:{region}:{account}:document/msk-broker-reboot",
        "documentParameters": "{\"AutomationAssumeRoleArn\":\"arn:aws:iam::{account}:role/SSM-MSK-Reboot\",\"ClusterArn\":\"{cluster-arn}\",\"BrokerId\":\"{broker-id}\"}",
        "maxDuration": "PT15M"
      }
    }
  },
  "stopConditions": [{"source": "none"}],
  "roleArn": "arn:aws:iam::{account}:role/FIS-MSK-Reboot",
  "tags": {"Purpose": "chaos-engineering", "RiskId": "R-012"}
}'

执行

# List available broker IDs
aws kafka list-nodes --cluster-arn {CLUSTER_ARN} \
  --query 'NodeInfoList[].BrokerNodeInfo.BrokerId' --output json

# Verify cluster is ACTIVE
aws kafka describe-cluster --cluster-arn {CLUSTER_ARN} \
  --query 'ClusterInfo.State'

# Start the experiment
aws fis start-experiment --experiment-template-id {TEMPLATE_ID}

# Monitor cluster state
watch -n 10 'aws kafka describe-cluster \
  --cluster-arn {CLUSTER_ARN} \
  --query "ClusterInfo.State"'

观测指标

清理

kafka:RebootBroker 是自恢复的——Broker 自动恢复上线，无需手动清理。

# Verify cluster is back to ACTIVE
aws kafka describe-cluster --cluster-arn {CLUSTER_ARN} \
  --query 'ClusterInfo.State'

# Delete experiment template when done
aws fis delete-experiment-template --id {TEMPLATE_ID}

Example 12: MSK Broker Reboot — Kafka Client Resilience

Architecture pattern: Kafka Producers/Consumers → Amazon MSK (managed Kafka cluster) Injection method: SSM Automation (kafka:RebootBroker → wait for ACTIVE) Validates: Kafka producer/consumer resilience, partition leader election, broker recovery

Note: MSK has no native FIS action. Fault injection uses SSM Automation with the MSK API.

---

Hypothesis

When an MSK broker is rebooted:

• Kafka producers switch to other brokers and continue producing with minimal message loss
• Kafka consumers rebalance partitions and resume consuming within 60 seconds
• The cluster transitions to REBOOTING_BROKER then returns to ACTIVE
• Partition leaders on the rebooted broker are re-elected to other brokers
• No manual intervention required — the broker comes back online automatically

What does this enable you to verify?

• Kafka producer acks configuration and retry behavior during broker failure
• Consumer group rebalancing speed and partition reassignment
• MSK cluster recovery time after single broker reboot
• Application resilience to partition leader changes
• Two-role IAM pattern for SSM Automation via FIS

Prerequisites

• [ ] Amazon MSK cluster in ACTIVE state
• [ ] Target broker ID identified (use aws kafka list-nodes)
• [ ] FIS IAM Role with AWSFaultInjectionSimulatorSSMAccess + iam:PassRole
• [ ] SSM Automation IAM Role with kafka:RebootBroker + kafka:DescribeCluster
• [ ] Kafka producers configured with acks=all and retry logic
• [ ] Kafka consumers with appropriate session.timeout.ms and heartbeat.interval.ms

SSM Automation Runbook

description: 'FIS: Reboot MSK broker to test Kafka client resilience'
schemaVersion: '0.3'
assumeRole: '{{ AutomationAssumeRoleArn }}'
parameters:
  AutomationAssumeRoleArn:
    type: String
    description: IAM Role ARN for SSM Automation to assume
  ClusterArn:
    type: String
    description: MSK cluster ARN
  BrokerId:
    type: String
    description: 'Broker ID to reboot (e.g., 1, 2, 3)'
mainSteps:
  - name: RebootMskBroker
    action: aws:executeAwsApi
    inputs:
      Service: kafka
      Api: RebootBroker
      ClusterArn: '{{ ClusterArn }}'
      BrokerIds:
        - '{{ BrokerId }}'
    outputs:
      - Name: ClusterArn
        Selector: $.ClusterArn
        Type: String
      - Name: ClusterOperationArn
        Selector: $.ClusterOperationArn
        Type: String

  - name: WaitForClusterActive
    action: aws:waitForAwsResourceProperty
    timeoutSeconds: 600
    inputs:
      Service: kafka
      Api: DescribeCluster
      ClusterArn: '{{ ClusterArn }}'
      PropertySelector: $.ClusterInfo.State
      DesiredValues:
        - ACTIVE

Setup

1. Deploy SSM Automation Document

aws ssm create-document \
  --name msk-broker-reboot \
  --document-type Automation \
  --content file://ssm-msk-broker-reboot.yaml \
  --document-format YAML

2. Deploy IAM Roles

# FIS Role (trusts fis.amazonaws.com)
aws iam create-role \
  --role-name FIS-MSK-Reboot \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Principal": {"Service": "fis.amazonaws.com"}, "Action": "sts:AssumeRole"}]
  }'

aws iam attach-role-policy \
  --role-name FIS-MSK-Reboot \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSFaultInjectionSimulatorSSMAccess

aws iam put-role-policy \
  --role-name FIS-MSK-Reboot \
  --policy-name PassSSMRole \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::{account}:role/SSM-MSK-Reboot", "Condition": {"StringEquals": {"iam:PassedToService": "ssm.amazonaws.com"}}}]
  }'

# SSM Automation Role (trusts ssm.amazonaws.com)
aws iam create-role \
  --role-name SSM-MSK-Reboot \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Principal": {"Service": "ssm.amazonaws.com"}, "Action": "sts:AssumeRole"}]
  }'

aws iam put-role-policy \
  --role-name SSM-MSK-Reboot \
  --policy-name MskAccess \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{"Effect": "Allow", "Action": ["kafka:RebootBroker", "kafka:DescribeCluster"], "Resource": "*"}]
  }'

3. Create FIS Experiment

aws fis create-experiment-template --cli-input-json '{
  "description": "Reboot MSK broker to test Kafka producer and consumer resilience",
  "targets": {},
  "actions": {
    "reboot-broker": {
      "actionId": "aws:ssm:start-automation-execution",
      "parameters": {
        "documentArn": "arn:aws:ssm:{region}:{account}:document/msk-broker-reboot",
        "documentParameters": "{\"AutomationAssumeRoleArn\":\"arn:aws:iam::{account}:role/SSM-MSK-Reboot\",\"ClusterArn\":\"{cluster-arn}\",\"BrokerId\":\"{broker-id}\"}",
        "maxDuration": "PT15M"
      }
    }
  },
  "stopConditions": [{"source": "none"}],
  "roleArn": "arn:aws:iam::{account}:role/FIS-MSK-Reboot",
  "tags": {"Purpose": "chaos-engineering", "RiskId": "R-012"}
}'

Execution

# List available broker IDs
aws kafka list-nodes --cluster-arn {CLUSTER_ARN} \
  --query 'NodeInfoList[].BrokerNodeInfo.BrokerId' --output json

# Verify cluster is ACTIVE
aws kafka describe-cluster --cluster-arn {CLUSTER_ARN} \
  --query 'ClusterInfo.State'

# Start the experiment
aws fis start-experiment --experiment-template-id {TEMPLATE_ID}

# Monitor cluster state
watch -n 10 'aws kafka describe-cluster \
  --cluster-arn {CLUSTER_ARN} \
  --query "ClusterInfo.State"'

Observation Metrics

Cleanup

kafka:RebootBroker is self-recovering — the broker comes back online automatically. No manual cleanup required.

# Verify cluster is back to ACTIVE
aws kafka describe-cluster --cluster-arn {CLUSTER_ARN} \
  --query 'ClusterInfo.State'

# Delete experiment template when done
aws fis delete-experiment-template --id {TEMPLATE_ID}