Chaos Engineering

Chaos Engineering

Design experiments that surface real weaknesses in production systems — without becoming outages. Most "chaos engineering" attempts skip steady-state measurement, define no abort criteria, and have no blast-radius bound. This skill enforces the discipline that makes chaos experiments safe and useful.

When to use

• Planning a chaos experiment (what to break, where, when, how to abort)
• Calculating blast radius before running the experiment
• Reviewing an existing experiment plan for safety
• Choosing a chaos tool (Chaos Toolkit / Chaos Mesh / Litmus / Gremlin / AWS FIS)
• Writing a chaos experiment postmortem
• Running a Game Day exercise

When NOT to use

• General incident response (use incident-response)
• Threat hunting / red-team (use red-team, threat-detection)
• Performance load testing (different goal — chaos is about failure modes, not capacity)
• Production debugging (chaos discovers weaknesses preemptively, not after-the-fact)

Core principle: chaos without abort criteria is an outage

The 4 Principles of Chaos Engineering (Netflix, 2016):

1. Build a hypothesis around steady-state behavior. Not "what breaks?" but "X holds; will it still hold under fault Y?" 2. Vary real-world events. Inject realistic failures: kill nodes, slow networks, lose cache, throttle dependencies. 3. Run experiments in production. Staging never has the same failure modes. Start small. 4. Automate experiments to run continuously. One-off chaos is a press release; continuous chaos is engineering.

Add a fifth: Define abort criteria up front. A chaos experiment with no abort criteria is an outage by another name.

Quick start

SKILL=engineering/chaos-engineering/skills/chaos-engineering

# 1. Design an experiment
python "$SKILL/scripts/experiment_designer.py" --target "checkout-svc" --hypothesis "p99 latency stays <500ms" --attack latency --duration-min 15

# 2. Calculate blast radius
python "$SKILL/scripts/blast_radius_calculator.py" --traffic-share 0.05 --user-pop 1000000 --duration-min 15

# 3. Generate postmortem after the experiment
python "$SKILL/scripts/experiment_postmortem.py" --plan experiment.json --result-log results.txt

The 3 Python tools

All stdlib-only. Run with --help.

experiment_designer.py

Generates a structured experiment plan from inputs. Enforces the required sections (hypothesis, steady-state metric, blast radius, abort criteria, rollback).

python scripts/experiment_designer.py \
  --target "checkout-svc" \
  --hypothesis "p99 latency stays <500ms when payment-svc is slow" \
  --attack latency \
  --magnitude "+200ms" \
  --duration-min 15 \
  --blast-radius "5% of US traffic" \
  --abort-if "p99 > 1000ms OR error_rate > baseline + 1pp"

Outputs a markdown plan with: hypothesis, steady-state, attack, magnitude, duration, blast radius, abort criteria, rollback procedure, monitoring dashboards, and learning question.

blast_radius_calculator.py

Computes the blast radius of a planned experiment. Given traffic share + user population + duration, calculates expected affected users, expected error budget burn, and a risk score.

python scripts/blast_radius_calculator.py \
  --traffic-share 0.05 \
  --user-pop 1000000 \
  --duration-min 15 \
  --baseline-availability 0.999 \
  --expected-impact-availability 0.95

Outputs:

• Expected affected users
• Error budget consumed (in minutes of error budget)
• Risk score: GREEN / YELLOW / RED
• Recommendation: PROCEED / REDUCE / ABORT

GREEN = <1% error budget; YELLOW = 1-10%; RED = >10%.

experiment_postmortem.py

Produces a structured postmortem from an experiment plan + results. Catches the common postmortem failure modes: no learning recorded, no follow-up actions, blame-laden language.

python scripts/experiment_postmortem.py --plan experiment.json --result-log results.txt

Outputs markdown with: summary, hypothesis (was it confirmed/refuted?), what we learned, what surprised us, follow-up actions with owners, and link to next experiment.

The 7 attack types (taxonomy)

Different attacks reveal different weaknesses. See references/attack_taxonomy.md for full detail.

Pick the attack that matches the hypothesis. "What happens if X is slow?" → latency. "What happens if X loses network?" → partition.

Tooling chooser

Decision rules:

• k8s-only stack + OSS → Chaos Mesh or Litmus (Litmus has bigger experiment library)
• Multi-cloud + OSS → Chaos Toolkit
• AWS-heavy + simple needs → AWS FIS
• Enterprise + audit/compliance → Gremlin

See references/tooling_landscape.md for trade-offs.

Workflows

Workflow 1: Design and run a single experiment

1. State a hypothesis: "When [fault], steady-state metric X stays within Y."
2. Identify the steady-state metric — must be measurable BEFORE the experiment.
3. Run blast_radius_calculator.py — confirm GREEN before proceeding.
4. Run experiment_designer.py to produce the plan.
5. Get a peer review of the plan; confirm abort criteria are concrete.
6. Notify the on-call team in #incidents (or whatever channel).
7. Run the experiment with monitoring open.
8. If abort criteria are hit, abort immediately; record what happened.
9. Run experiment_postmortem.py to capture learnings.
10. File follow-up actions; link to next experiment.

Workflow 2: Game Day exercise

1. Pick a scenario (e.g., "primary database fails over").
2. Identify all dependent services that should keep working.
3. Build a multi-experiment plan covering each layer.
4. Schedule with stakeholders; on-call coverage required.
5. Run with a facilitator who manages the scenario.
6. Capture observations in a shared doc as they happen.
7. Single combined postmortem covering all observations.
8. Track follow-up actions in a board with owners.

Workflow 3: Continuous chaos (game days → daily)

1. Start: weekly Game Day in staging.
2. Move to: weekly Game Day in production with limited blast radius.
3. Mature to: continuous chaos via scheduled experiments (Litmus chaos schedule, Gremlin scenarios).
4. Wire to deployment: every prod deploy triggers a baseline chaos sweep.
5. Track: experiments per week, weaknesses discovered, MTTR trend.

Composition with other skills

This skill explicitly composes with two others in this library:

Anti-patterns

• No hypothesis — "let's break things" is sabotage, not engineering
• No steady-state metric — without a baseline, you can't tell if X broke
• No blast radius bound — full-prod experiment without limits = outage
• No abort criteria — see above; this is mandatory
• No on-call coverage — chaos without monitoring is unmonitored production
• Chaos in staging only — staging never has prod failure modes
• Chaos in dev — useless; dev has different failure modes from prod
• One-off chaos — single experiment is a press release; learning requires recurrence
• Blame-laden postmortem — record causes, not blame; teams stop running chaos otherwise

References

• references/chaos_principles.md — the 4 principles, history, when to start
• references/experiment_design.md — hypothesis structure, steady-state metrics, abort criteria
• references/attack_taxonomy.md — 7 attack types with examples and tooling
• references/tooling_landscape.md — Chaos Toolkit / Mesh / Litmus / Gremlin / FIS / DIY

Slash command

/chaos-experiment — interactive experiment design wizard that runs all 3 tools.

Asset templates

• assets/experiment_template.md — fill-in plan template
• assets/postmortem_template.md — structured postmortem template

Verifiable success

A team using this skill should achieve:

• 100% of chaos experiments have a written hypothesis, abort criteria, and blast-radius calculation
• Blast radius for any single experiment never exceeds 10% of error budget
• Mean time between chaos experiments <14 days (continuous, not one-off)
• Each experiment produces ≥1 follow-up action that gets shipped
• No chaos experiment escalates to a customer-impacting incident in trailing 90 days

Chaos Experiment

Fill in every section before running. Refuse to run if any section is empty.

Identity

• Experiment ID: <auto-generated; format: chaos-<target>-<attack>-<unix-ts>>
• Date: <YYYY-MM-DD>
• Owner: <your-handle@team>
• On-call team: <team channel / pager>
• Reviewer: <peer who reviewed this plan>

1. Hypothesis

When <fault>, <steady-state metric> stays <tolerance>.

Example: When payment-svc is +200ms slow, checkout p99 stays below 500ms.

2. Steady-state metric

• Metric: <e.g., p99 checkout latency>
• Baseline window: <e.g., 5 minutes pre-experiment>
• Tolerance: <e.g., within ±5% of baseline>
• Dashboard: <URL>

3. Attack

• Type: [ ] latency [ ] error [ ] cpu [ ] memory [ ] disk [ ] network-partition [ ] dependency-failure [ ] time-skew [ ] kill-instance
• Magnitude: <e.g., +200ms>
• Duration: <minutes>
• Target: <service / pod / instance / region>
• Tooling: <Chaos Toolkit / Chaos Mesh / Litmus / Gremlin / AWS FIS / Custom>

4. Blast radius

• Traffic share: <e.g., 5% of US>
• Expected affected users: <from blast_radius_calculator.py>
• Error budget consumed: <from blast_radius_calculator.py>
• Risk score: [ ] GREEN [ ] YELLOW [ ] RED

5. Abort criteria

Auto-trigger experiment termination if ANY of these hit.

• [ ] <signal 1, e.g., p99 > 1000ms>
• [ ] <signal 2, e.g., 5xx rate > baseline + 1pp>
• [ ] <signal 3, e.g., on-call paged SEV1/SEV2>

6. Rollback procedure

1. <step to disable fault, e.g., "kubectl delete chaos networkchaos/<name>"> 2. Verify steady state recovers within 2 minutes 3. If not recovering, escalate as incident; restore from backup if needed

7. Learning question

What do you expect NOT to learn? Force yourself to predict.

<your prediction>

Pre-flight checklist

• [ ] Hypothesis written
• [ ] Steady-state metric measured for ≥5 min
• [ ] Blast radius calculated (GREEN or YELLOW only)
• [ ] Abort criteria documented with thresholds
• [ ] Rollback procedure tested in staging
• [ ] On-call team notified
• [ ] Monitoring dashboards open
• [ ] Owner identified and reachable
• [ ] Time-box agreed
• [ ] Communication plan if abort triggers

Post-experiment

Run experiment_postmortem.py --plan <plan.json> --result-log <results> to generate the postmortem.

Chaos Experiment Postmortem

Identity

• Experiment: <experiment_id>
• Date: <YYYY-MM-DD>
• Target: <service>
• Owner: <handle@team>
• Postmortem facilitator: <handle@team>

Hypothesis

<hypothesis from the plan>

Outcome

• [ ] Held — hypothesis confirmed
• [ ] Refuted — hypothesis disproven
• [ ] Inconclusive — could not tell

Timeline

What we learned

<at least one concrete learning — required>

What surprised us

<unexpected observations; "nothing surprised us" is a signal that you didn't push hard enough>

What failed

<things that broke during the experiment that shouldn't have>

What held

<things that worked as expected — confidence-building data points>

Root causes (if any failures)

<technical analysis without blame>

Follow-up actions

Every experiment should produce ≥1 follow-up. If none — re-examine whether you tested anything new.

Next experiment

<what's the next experiment that builds on this learning?>

Stakeholder summary (1-2 sentences)

<for the team channel; describe outcome and biggest learning>

Attack taxonomy

7 categories of fault injection. Each tests a different system property. Pick the one whose failure mode matches your hypothesis.

1. Latency

What it tests: timeouts, retries, circuit breakers, fallback paths.

Inject: add N ms of delay to network responses to a target.

When to use:

• "What if dependency X is slow?"
• "Are timeouts configured correctly upstream?"
• "Does the retry budget kick in?"

Tools:

• Linux tc (traffic control) — direct kernel-level shaping
• Chaos Mesh NetworkChaos (delay)
• Toxiproxy — proxy-based, language-agnostic
• AWS FIS — aws:network:traffic-control action

Example magnitude: +200ms (90% of typical timeouts), +2000ms (test backoff), +30s (test giving-up logic).

2. Error injection

What it tests: error handling paths, fallback behavior, retry policies.

Inject: return errors (5xx, exceptions) for a fraction of requests.

When to use:

• "What happens when X starts failing?"
• "Does the fallback path actually work in prod?"
• "Are we logging errors correctly?"

Tools:

• Chaos Mesh HTTPChaos
• Service mesh (Istio, Linkerd) fault injection
• Toxiproxy with error toxic
• Application-level feature flag for synthetic errors

Example magnitude: 1% errors (test handler), 50% errors (test retry), 100% errors (test fallback path).

3. Resource exhaustion

What it tests: saturation handling, autoscaling, OOM behavior, disk-full handling.

Inject: consume CPU, memory, or disk on the target.

When to use:

• "What if memory leaks?"
• "Does the autoscaler kick in?"
• "What happens when disk fills?"

Sub-types:

• CPU pressure — peg cores at N% usage
• Memory pressure — allocate large blocks
• Disk fill — write large files until partition fills
• I/O saturation — high random read/write

Tools:

• stress-ng — CPU/memory/IO/disk
• Chaos Mesh StressChaos and IOChaos
• AWS FIS aws:ssm:send-command with stress-ng

Example magnitude: 80% CPU sustained, 90% memory, fill /var to 95%.

4. Network partition

What it tests: consensus protocols, leader election, split-brain prevention, region failover.

Inject: drop all packets between a set of hosts.

When to use:

• "What if AZ-A loses connectivity to AZ-B?"
• "Does the database elect a new primary?"
• "Does the cluster avoid split-brain?"

Tools:

• Chaos Mesh NetworkChaos (partition mode)
• tc with iptables drop rules
• AWS FIS aws:network:disrupt-connectivity

Example magnitude: drop 100% to peer X (full partition), drop 50% (degraded link).

5. Dependency failure

What it tests: graceful degradation, fallback to cache, fallback to default values.

Inject: make a downstream dependency unavailable (timeout, refuse connections).

When to use:

• "What if the rec engine goes down?"
• "Does Search degrade gracefully when ML models are unreachable?"
• "Is cache the fallback for the user-pref service?"

Tools:

• Service mesh fault injection (most flexible)
• Toxiproxy
• iptables rules to refuse connections
• Chaos Mesh NetworkChaos with corrupt or drop

Example magnitude: 100% requests to dep X timeout (full outage), 25% timeout (intermittent), 0% available for 5 min (sustained outage).

6. Time skew

What it tests: time-sensitive logic — token expiry, cron schedules, TTLs, retry backoff.

Inject: alter the wall clock seen by a process.

When to use:

• "What if NTP fails?"
• "What if a process clock drifts +5 minutes?"
• "Do tokens correctly fail validation when expired?"
• "Does cron skip or double-fire?"

Tools:

• libfaketime — preload library
• Chaos Mesh TimeChaos
• Custom: change container's /etc/localtime

Example magnitude: +1 minute (subtle), +5 minutes (TLS / token failures), +1 day (catastrophic for some logic).

Caution: time skew can cause cluster-wide consensus failures. Test in isolation first.

7. Infrastructure (kill instance / pod / container)

What it tests: auto-recovery, failover, replica count maintenance.

Inject: terminate an instance, pod, or container.

When to use:

• "Does Kubernetes restart the pod?"
• "Does the load balancer remove the instance from rotation?"
• "Is the replication factor maintained?"

Tools:

• Chaos Monkey (the original)
• Chaos Mesh PodChaos (kill, fail)
• AWS FIS aws:ec2:terminate-instances
• kubectl delete pod (manual, simplest)

Example magnitude: kill 1 of N pods (Chaos Monkey level), kill all pods of a deployment (test recreation), kill 1 of 3 replica DB nodes (test failover).

Choosing an attack

Combining attacks

Real outages often combine attacks (e.g., latency + saturation). Once basic experiments are stable, run combinations:

• Latency on dependency + CPU pressure on app → tests timeout + retry budget interaction
• Pod kill + network partition → tests recovery during a partition
• Disk fill + dependency failure → tests fallback path while disk is constrained

Combinations have higher risk; reduce blast radius accordingly.

Severity ladder

S1 — Latency (small)              ← start here
S2 — Error injection (low %)
S3 — Resource pressure (CPU/mem)
S4 — Latency (large) / errors (high %)
S5 — Single instance kill
S6 — Network partition (single peer)
S7 — Multiple instance kill
S8 — Region partition / time skew
S9 — Combinations of S5-S8        ← here be dragons

Don't skip levels. Earn confidence at S1-S3 before attempting S5+.

The principles of chaos engineering

Chaos engineering is the discipline of experimenting on a system in order to build confidence in its capability to withstand turbulent conditions in production. The phrase comes from Netflix's 2014-2016 work productizing what started as Chaos Monkey.

The 4 founding principles (Netflix, 2016)

1. Build a hypothesis around steady-state behavior

Steady state = a measurable, normal-operations metric (latency, throughput, conversion rate, error rate).

Bad: "What happens if the database goes down?" Good: "When the primary database fails over, p99 checkout latency stays below 800ms and conversion rate stays within 2% of baseline."

The hypothesis must be falsifiable — there must be a measurement that can disprove it.

2. Vary real-world events

Inject realistic failure modes:

• Servers crash
• Networks partition or slow
• Disks fill
• Dependencies time out or return errors
• Caches lose data
• Time skews

Don't inject implausible events (e.g., "what if all 50 zones in 5 regions go down simultaneously"). That's not chaos engineering, that's astronomy.

3. Run experiments in production

Staging never reproduces:

• Real traffic patterns
• Real cache hit rates
• Real cross-service dependencies
• Real data volumes
• Real user behavior

The only system that has prod failure modes is prod. Start with tiny blast radius (1%), grow as confidence grows.

4. Automate experiments to run continuously

A single chaos experiment is a press release. Continuous chaos is engineering.

Maturity progression: 1. Manual one-offs → 2. Weekly Game Days → 3. Scheduled experiments → 4. Continuous chaos in CI/CD

The 5th principle this skill adds:

5. Define abort criteria up front

A chaos experiment with no abort criteria is an outage. Every plan must include:

• A specific signal (metric, threshold)
• A specific action (auto-abort, manual abort, escalate)
• A timeline (within N seconds of breach)

If the threshold is hit, abort immediately. Investigate later.

When to start

You're ready for chaos engineering when:

• [ ] You have basic monitoring (you can detect a steady-state breach)
• [ ] You have on-call rotations (someone is watching when chaos runs)
• [ ] You have at least one tool to inject the desired fault
• [ ] You have an SLO/SLI defined (so you know what "good" looks like)
• [ ] You have postmortem culture that's blameless
• [ ] You have a leadership champion who'll defend the practice

If any of these are missing, fix them first. Premature chaos = outages with no learning.

When NOT to do chaos engineering

• During a release freeze
• During a known incident
• During peak traffic events without explicit approval
• On systems that don't have steady-state metrics
• On systems where you can't bound the blast radius
• On the day of a security disclosure
• When the team is already firefighting

Maturity model

Most teams should target L3 within 6-12 months of starting. L5 is rare and only justified for the largest distributed systems.

Common objections (and counters)

What a steady-state metric looks like

Good steady-state metrics:

• p99 request latency (objective, measurable per second)
• Error rate (objective, measurable)
• Conversion rate (business metric, slow but real)
• Successful logins per minute (business + tech signal)
• Queue depth (system health)

Bad metrics:

• "Things feel slow" (not measurable)
• CPU usage (a means, not an end)
• Number of pods running (not customer-facing)

Pick metrics that customers feel. CPU can spike without customer impact; latency and errors can't.

History

• 2010: Netflix launches Chaos Monkey (kills random EC2 instances)
• 2011: Simian Army expands (Latency Monkey, Conformity Monkey, etc.)
• 2014: Chaos engineering term coined; principles drafted
• 2016: principlesofchaos.org published
• 2018: Chaos Toolkit released as OSS
• 2019: Chaos Mesh and Litmus mature for Kubernetes
• 2020: AWS launches Fault Injection Simulator (FIS)
• 2023+: Chaos engineering becomes mainstream practice in SRE-heavy orgs

Further reading

• principlesofchaos.org — the foundational document
• Chaos Engineering (Casey Rosenthal, Nora Jones) — O'Reilly, 2020
• Learning Chaos Engineering (Russ Miles) — O'Reilly, 2019
• Netflix Tech Blog on Chaos Engineering posts (2016-2020)

Experiment design

A well-designed chaos experiment has 7 sections. Skip any of them and the experiment becomes either useless (no learning) or dangerous (no bounds).

The 7 sections

1. Hypothesis
2. Steady-state metric
3. Attack
4. Blast radius
5. Abort criteria
6. Rollback procedure
7. Learning question

1. Hypothesis

Format: When [fault], [steady-state metric] stays [tolerance].

Examples:

• "When the primary Postgres replica fails, checkout p99 latency stays below 500ms."
• "When 50% of payment-service requests are throttled to 1 RPS, conversion rate drops by less than 5% within 60 seconds of return-to-normal."
• "When us-east-1 is partitioned from us-west-2, Search continues to return results from us-west-2 within 200ms p99."

A good hypothesis:

• Names a specific fault (not "things break")
• Names a specific metric (not "everything")
• States a specific tolerance (not "good enough")
• Is measurable and falsifiable

2. Steady-state metric

The metric you'll measure before, during, and after the experiment.

Required properties:

• Quantitative — a number, not a feeling
• Customer-relevant — something users feel (latency, error rate, conversion)
• Measurable in <60s — slow metrics give you no time to abort
• Stable in normal operation — you need a baseline

3. Attack

The fault you're injecting. Must specify:

• Type — latency, error, resource, partition, dependency, time, infrastructure
• Magnitude — how much (e.g., "+200ms", "10% errors", "100% timeout to peer X")
• Duration — how long the attack runs (typically 5-30 minutes)
• Target — which subset of the system gets the attack

See attack_taxonomy.md for the 7 attack types.

4. Blast radius

The maximum scope of customer impact. Use blast_radius_calculator.py to compute:

• Affected users — traffic_share × user_population
• Error budget consumed — duration × traffic_share × availability_delta
• Risk score — GREEN (<1% budget) / YELLOW (1-10%) / RED (>10%)

Rule of thumb:

• Start at 1% traffic share
• Grow only after 3 successful experiments at the previous level
• Never exceed 10% of monthly error budget in a single experiment

5. Abort criteria

The signals that auto-trigger experiment termination. Each must be:

• Concrete — specific metric and threshold ("p99 > 1000ms" not "performance degrades")
• Detectable in <60s — latency, error rate, throughput
• Wired to action — manual abort link in the dashboard, automatic via alert webhook

Standard abort criteria:

6. Rollback procedure

How you'll revert the fault. Required because:

• Sometimes the chaos tool itself fails to revert
• Sometimes the fault has lingering effects (caches, connections)

Standard rollback: 1. Disable fault injection in tool 2. Verify steady-state recovers within 2 minutes 3. If not recovering, escalate as incident; restore from backup if needed

7. Learning question

What do you expect NOT to learn? Force yourself to predict the outcome.

Examples:

• "We expect the cache to absorb the latency. We'll learn whether the timeout configuration on the upstream is correct."
• "We expect failover to take 30s. We'll learn whether retry backoff is configured."

If you predicted the outcome correctly: confidence increased. If you didn't: there's an unknown — file a follow-up.

Pre-flight checklist

Before running the experiment, verify:

• [ ] Hypothesis written
• [ ] Steady-state metric measured for ≥5 minutes
• [ ] Blast radius calculated (GREEN or YELLOW)
• [ ] Abort criteria documented with thresholds
• [ ] Rollback procedure tested in staging
• [ ] On-call team notified in the team channel
• [ ] Monitoring dashboards open
• [ ] Owner identified and reachable
• [ ] Time-box agreed (max experiment duration)
• [ ] Communication plan if abort triggers

Time-boxing

Escalation

If abort criteria are hit:

1. Stop the experiment immediately (the obvious step many teams forget to script) 2. Verify steady-state recovery 3. If recovery doesn't happen in 5 min → declare an incident 4. Open a postmortem doc using experiment_postmortem.py 5. Notify stakeholders (whoever was promised "this won't impact anything") 6. Capture timeline while memory is fresh

Anti-patterns

• Hypothesis written after running — that's a postmortem, not chaos engineering
• Steady-state metric chosen during experiment — pick before
• Magnitude "small" — quantify; "small" varies by reader
• No abort criteria — never run without them
• Single owner of all chaos — culture problem; spread the practice
• Chaos that always succeeds — increase magnitude; you're not learning if everything passes
• Chaos that always fails — reduce magnitude; you can't learn if everything breaks
• Chaos with no follow-up actions — what was the point?

Tooling landscape

Six options. Pick by stack, license preference, and required attack types.

At-a-glance

Decision tree

Stack constraint?
├── Kubernetes-only ──┬── OSS preferred → Chaos Mesh OR Litmus
│                     │       (Litmus has the bigger experiment library;
│                     │        Chaos Mesh has cleaner CRD model)
│                     └── Enterprise budget → Gremlin
│
├── AWS-heavy ────────┬── Simple needs → AWS FIS
│                     ├── Multi-cloud + AWS → Chaos Toolkit + AWS plugin
│                     └── Enterprise → Gremlin
│
├── Multi-cloud ──────┬── OSS → Chaos Toolkit
│                     └── Enterprise → Gremlin
│
└── No infra constraint
    └── Just need fault injection → Toxiproxy (a single-purpose tool, not full chaos framework)

Chaos Toolkit

What it is: Python-based framework. You write experiments as JSON or YAML files; the CLI runs them.

Strengths:

• Lightweight; runs anywhere Python runs
• Plugin ecosystem for AWS, Azure, GCP, Kubernetes, etc.
• JSON experiments are version-controllable
• Apache 2 license

Weaknesses:

• No built-in scheduling (you bring cron / CI)
• Smaller experiment library than Litmus
• Plugin quality varies

Example experiment (JSON):

{
  "title": "Latency on payment-svc",
  "description": "p99 latency stays <500ms when payment is +200ms slow",
  "steady-state-hypothesis": {
    "title": "p99 < 500ms",
    "probes": [{ "type": "probe", "tolerance": [0, 500],
                 "provider": { "type": "http", "url": "https://my.dashboards/p99" } }]
  },
  "method": [{ "type": "action", "name": "add-latency",
               "provider": { "type": "process", "path": "tc", "arguments": [...] } }]
}

Chaos Mesh

What it is: Kubernetes operator + CRDs for chaos. Install in-cluster; kubectl apply an experiment.

Strengths:

• True k8s-native (no external orchestrator)
• Comprehensive coverage: network, pod, IO, stress, time, DNS, HTTP, kernel
• UI dashboard for running experiments
• CNCF Incubating project

Weaknesses:

• k8s-only
• CRD layout is opinionated; some types feel similar but aren't
• Setup requires cluster admin

Example experiment (CRD):

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payment-latency
spec:
  action: delay
  mode: one
  selector:
    namespaces: [default]
    labelSelectors:
      app: payment-svc
  delay:
    latency: 200ms
  duration: 5m

Litmus

What it is: Kubernetes chaos framework with a large experiment library. Argo-CD integration.

Strengths:

• 300+ pre-built experiments
• Strong Argo / GitOps integration
• ChaosHub community library
• Workflow capability for multi-step experiments

Weaknesses:

• More moving parts than Chaos Mesh
• Some pre-built experiments are thin wrappers; quality varies
• k8s-only

Gremlin

What it is: Commercial SaaS. Agents on hosts; central control plane.

Strengths:

• Polished UX
• Comprehensive attack library
• Audit logs (compliance)
• Multi-cloud, multi-OS
• Customer support

Weaknesses:

• Paid (per-host or per-MAU)
• Vendor lock-in
• Less control than OSS

When to choose: large enterprise, compliance/audit requirements, dedicated chaos team, budget exists.

AWS FIS (Fault Injection Simulator)

What it is: AWS-managed chaos service. Templates of "actions" (stop instance, throttle API) chained into experiments.

Strengths:

• IAM-integrated (proper auth/audit)
• Native to AWS services (RDS failover, ECS/EKS, Network Manager)
• Pay-per-experiment (no agents to maintain)

Weaknesses:

• AWS-only
• Smaller attack library than Chaos Mesh / Gremlin
• Multi-account is awkward

When to choose: AWS-heavy team that wants chaos without managing the chaos infra.

Custom (DIY)

When to choose:

• Single-cloud, single-stack, low complexity
• Budget = $0
• Have engineering capacity to maintain the tool
• Need a niche attack type that no tool covers

Implementation patterns:

• Bash scripts that wrap tc / iptables / kill / stress-ng
• Application-level chaos via feature flags + middleware
• Service mesh fault injection (Istio / Linkerd) — covers many cases without a chaos framework

Trade-offs:

• You build all the safety rails (abort, timeout, blast-radius)
• You build the scheduler
• You debug your own bugs

For most teams, this is a starter path; once chaos becomes regular, switch to a real tool.

Pricing rule of thumb

Migration paths

Selection checklist

Before committing:

• [ ] Stack matches (k8s vs multi-cloud vs AWS-only)
• [ ] Required attack types covered (cross-reference attack_taxonomy.md)
• [ ] Audit logging requirement met (Gremlin / AWS FIS only have full audit)
• [ ] Self-hosting requirement met (OSS only)
• [ ] Budget approved
• [ ] Run a 30-day proof-of-concept; verify abort path works