Rootcauseanalysis

Customization

Before executing, check for user customizations at: ~/.claude/PAI/USER/SKILLCUSTOMIZATIONS/RootCauseAnalysis/

If this directory exists, load and apply any PREFERENCES.md, configurations, or resources found there. These override default behavior. If the directory does not exist, proceed with skill defaults.

MANDATORY: Voice Notification (REQUIRED BEFORE ANY ACTION)

You MUST send this notification BEFORE doing anything else when this skill is invoked.

1. Send voice notification:

   curl -s -X POST http://localhost:31337/notify \
     -H "Content-Type: application/json" \
     -d '{"message": "Running the WORKFLOWNAME workflow in the RootCauseAnalysis skill to ACTION"}' \
     > /dev/null 2>&1 &

2. Output text notification:

   Running the **WorkflowName** workflow in the **RootCauseAnalysis** skill to ACTION...

This is not optional. Execute this curl command immediately upon skill invocation.

---

RootCauseAnalysis Skill

Structured investigation of why something failed — beyond the proximate cause, down to the contributing factors and latent conditions that actually made the failure possible. Grounded in Toyota Production System (Sakichi Toyoda), Kaoru Ishikawa, James Reason's Swiss Cheese model, Dean Gano's Apollo method, and Google SRE / Etsy blameless-postmortem culture.

The goal is not to find "the" root cause — that framing is almost always wrong. The goal is to identify contributing factors that are actionable. A good RCA ends with changes that prevent a class of failure, not just the specific incident.

Core Concept

Five axioms this skill operates on:

1. Proximate cause ≠ root cause. "The deploy failed because X crashed" is usually where real analysis starts, not where it ends. 2. There is rarely one cause. Incidents typically have multiple contributing factors — active failures (what a human did) and latent conditions (what the system allowed). James Reason's Swiss Cheese model. 3. Humans are not root causes. "Operator error" is a stop sign for analysis, not a conclusion. If a human could make the mistake, the system allowed it. Go deeper. 4. Actionability is the stop condition. A cause is "root enough" when it points to a change you can actually make. Go too shallow and you miss the fix; go too deep ("physics") and you can't act on it. 5. RCA is a bias-fight. Hindsight bias, confirmation bias, single-cause bias, and outcome bias all actively corrupt investigations. Structure exists to resist them.

Use / Win

When to use:

• Any incident or outage — production failure, security event, deploy gone bad.
• Recurring defects — bugs of the same shape keep appearing despite fixes.
• Quality problems — metrics drifting, users reporting the same class of issue.
• Postmortems — structured, blameless review of an incident's causal chain.
• Pre-launch risk analysis — inverting RCA with FMEA to catch failure modes before they happen.
• Security investigations — chain of events, contributing controls, latent conditions.
• Process failures — a person or team consistently missing a mark. Structure is probably the cause.

What you win:

• Actionable contributing factors (plural) rather than a single blame target.
• Latent conditions surfaced — the Swiss cheese holes lining up that nobody knew were there.
• Durable fixes — structural changes, not patches to the specific failure.
• Blame-free analysis — the team can be honest about what happened without self-protective omissions.
• Cross-incident pattern recognition — after a few RCAs, the repeated latent conditions become visible.
• Discipline against bias — structured methods force you past the first plausible story.

Default mental model: If the same failure class could happen again tomorrow, you haven't done RCA — you've done triage.

Workflow Routing

Route to the appropriate workflow based on the request.

Quick Reference

• 5 workflows — FiveWhys, Fishbone, Postmortem, FaultTree, KepnerTregoe
• 5 Whys: Linear/branching causal chain. Best for simple, single-thread incidents.
• Fishbone: 6 M's (Manpower, Machine, Method, Material, Measurement, Mother-Nature) for manufacturing; 4 P's (People, Process, Policies, Procedures) for service. Use when multiple category causes are suspected.
• Postmortem: Timeline + contributing factors + action items. Blameless framing mandatory.
• Fault Tree: AND/OR gate logic, deductive, top-down. Best for safety-critical and complex multi-path failures.
• Kepner-Tregoe IS/IS-NOT: Identify distinctions between where the problem occurred and where it did not. Best for subtle, hard-to-reproduce defects.

Context files (loaded on demand):

• Foundation.md — Toyoda, Ishikawa, Reason, Gano, Google SRE; canonical methods
• MethodSelection.md — decision flow for which workflow to use

Method Selection Guide

For non-trivial incidents: Postmortem wraps the others. Start with a Postmortem structure, use 5 Whys / Fishbone / FTA inside it as investigation tools.

Integration

Depends on: nothing — standalone analytical skill.

Works well with:

• SystemsThinking — RCA stops at contributing factors; SystemsThinking continues down to structure and mental models. Pair them when patterns repeat across incidents.
• FirstPrinciples — decompose a contributing factor to its fundamental truths before fixing.
• RedTeam — "how would we cause this again?" is adversarial RCA. Use RedTeam to stress-test remediations.
• Science — RCA is the scientific method applied to failures. Use Science for hypothesis generation during investigation.

Examples

Example 1: Production outage

User: "the payments service went down for 14 minutes last night"
→ Postmortem workflow
→ Timeline: deploy at 23:47 → health check passed → traffic shift 23:49 → p99 latency spike 23:51 → auto-rollback 00:01
→ 5 Whys inside: Why did p99 spike? Cold cache. Why cold? New pod group. Why no warm? No warm-up in deploy script. Why? Not in checklist. Why? Template predates the caching layer.
→ Contributing factors: deploy template stale (latent); no warm-up step (active); no cache-cold canary (latent)
→ Remediation: update deploy template, add warm-up step, add cold-cache canary gate

Example 2: Recurring defect

User: "users keep reporting the same kind of auth failure, we've fixed it 3 times"
→ Fishbone workflow
→ 6 M's expansion: People (ops auth rotates keys without notifying infra), Method (no key-rotation runbook), Machine (secret cache TTL exceeds rotation window), Material (shared key instead of per-service), Measurement (no key-expiry dashboard), Mother-Nature (none)
→ Root causes (multiple): Method + Material + Measurement all contribute. Single-point fix won't hold.

Example 3: Subtle defect

User: "this flaky test only fails in CI, not locally"
→ KepnerTregoe workflow
→ IS/IS-NOT table: fails on CI / passes locally; fails Tuesdays / not other days; fails on shared runners / not dedicated; fails with parallel test workers / not serial
→ Distinctions point to: time-zone + concurrency + shared file system
→ Hypothesis: test relies on local timezone assumption + race condition on shared /tmp — both only triggered in CI's environment.

Best Practices

1. Always blameless. The framing is "what system allowed this" not "who screwed up." Non-negotiable; corrupts the analysis otherwise. 2. Multiple causes, always. Single-root-cause conclusions are almost always wrong. Name at least three contributing factors before stopping. 3. Actionability test every cause. Can you change it? If no — go shallower. If yes — go one level deeper to make sure you've found the lever. 4. Timelines before theories. Reconstruct what happened before hypothesizing why. Hindsight bias compresses the timeline. 5. Ask "who else could make this mistake?" If the answer is "anyone on the team," it's a systemic cause, not individual error. 6. Separate investigation from judgment. Never let the incident review drift into performance conversations. Separate meeting.

Gotchas

• "Human error" is a starting point, not a root cause. It's where the investigation begins. Every human error sits on top of a system that made the error possible or probable.
• The first plausible cause is almost never the only one. Confirmation bias loves RCA. Keep going after you find one.
• Stopping at proximate cause is failure. "X crashed because Y returned null." Why did Y return null? Why wasn't null handled? Why wasn't that tested? Go down.
• Going too deep ≠ good RCA. "The fundamental cause is the second law of thermodynamics" is not actionable. Stop at the deepest actionable level.
• Asking "why" more than ~5 times often means you switched causal chains. Re-draw as a tree, not a line.
• Don't confuse correlation with cause. Two things happening together is a hypothesis to test, not a conclusion.
• Outcome bias is sneaky. Decisions that turn out badly get judged harshly even if they were right given the information at the time. Separate process quality from outcome.

---

Attribution: Frameworks drawn from Sakichi Toyoda (5 Whys, Toyota Production System), Kaoru Ishikawa (Guide to Quality Control, 1968; Fishbone diagram), James Reason (Human Error, 1990; Swiss Cheese model), Dean Gano (Apollo Root Cause Analysis, 2008), Charles Kepner & Benjamin Tregoe (The Rational Manager, 1965), Google SRE book, Etsy blameless postmortem culture (John Allspaw).

Execution Log

After completing any workflow, append a single JSONL entry:

echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","skill":"RootCauseAnalysis","workflow":"WORKFLOW_USED","input":"8_WORD_SUMMARY","status":"ok|error","duration_s":SECONDS}' >> ~/.claude/PAI/MEMORY/SKILLS/execution.jsonl

RootCauseAnalysis — Foundation

Canonical reference on the methods and theorists behind this skill. Load on demand when depth is needed beyond the workflows.

1. Five Whys

Origin. Sakichi Toyoda developed the technique in the 1930s as a manufacturing interrogation discipline. Taiichi Ohno embedded it in the Toyota Production System as "the basis of Toyota's scientific approach — by repeating why five times, the nature of the problem as well as its solution becomes clear." Published in Ohno's Toyota Production System: Beyond Large-Scale Production (1988).

Historical note: The technique was originally developed to understand why new features were needed, not as a formal RCA method — a distinction critics cite when evaluating its limits.

Key pitfalls:

• Stopping at blame. "Operator error" is never a root cause — always a symptom of a system that made the error possible or likely. If the chain ends with a human, ask one more why.
• Single-chain bias. Real systems have branching causal structures. Each "why" may have multiple valid answers. Treating the chain as strictly linear misses parallel contributing factors.
• Skipping levels. Impatient investigators jump from symptom to distant cause, constructing a chain that feels complete but skips intermediate mechanisms.
• Knowledge ceiling. The method cannot go beyond the investigator's existing knowledge. Different investigators consistently produce different chains for the same problem — the technique lacks repeatability.
• Shallow stop. Teruyuki Minoura, former Toyota managing director of global purchasing, criticized the method as "too basic" for problems requiring deep structural analysis.

Five Hows variant. After identifying the root cause, apply "How do we prevent this?" five times to construct an equally rigorous corrective action chain — ensuring the solution is as deep as the diagnosis.

2. Ishikawa / Fishbone Diagram

Origin. Kaoru Ishikawa first used the cause-and-effect diagram at Kawasaki Steel Works in 1943. Formally presented in 1945. Codified in Guide to Quality Control (JUSE Press, 1968), where he named it one of seven basic quality tools.

Category sets (adapt to context — Ishikawa recommended this):

• 6 M's (Manufacturing): Manpower, Machine, Method, Material, Measurement, Mother Nature
• 4 P's (Service): People, Process, Policies, Procedures
• 8 M's (Extended manufacturing): 6 M's + Management, Maintenance
• 8 P's (Business/marketing): Product, Price, Place, Promotion, People, Process, Physical Evidence, Partners

Canonical combination: Fishbone + Pareto + 5 Whys. Fishbone for breadth, Pareto for prioritization, 5 Whys for depth on the vital few.

3. Pareto Analysis

Origin. Vilfredo Pareto's 1896 observation that 80% of Italy's land was owned by 20% of the population. Joseph Juran applied it to quality management in the 1940s, coining "vital few and trivial many" as a quality prioritization principle.

Critical point. Pareto analysis identifies which causes matter most — it does not explain why those causes occur. It is a prioritization tool, not an investigation tool. Always follow with a depth method.

4. Fault Tree Analysis (FTA)

Origin. Developed at Bell Laboratories in 1961 by H.A. Watson for the U.S. Air Force Minuteman missile program. Standardized in IEC 61025, NRC NUREG-0492 (nuclear), SAE ARP4761 (aerospace). Widely used in aerospace, nuclear, chemical, and pharmaceutical industries.

Structure. Top-down, deductive. Start with undesired top event; decompose into contributing events using logical gates (AND/OR/Priority AND/Inhibit); terminate at basic events with known or estimable probabilities.

Key concept — minimal cut sets. The smallest sets of basic events whose joint occurrence causes the top event. Systems with many 1-event cut sets are fragile. Systems with all cut sets containing 3+ events have strong defense-in-depth.

Independence assumption caveat. Boolean propagation assumes statistical independence of basic events. In real systems, events are often correlated (common-mode failures). When events share a root cause, they are not independent, and the AND gate is far more probable than ∏ suggests. Flag common-mode possibilities explicitly.

5. Apollo Root Cause Analysis / RealityCharting

Origin. Dean Gano developed the Apollo method after working on the Three Mile Island nuclear incident analysis in the late 1970s. Published as Apollo Root Cause Analysis: A New Way of Thinking (Apollonian Publications, 1999; 3rd ed. 2007). RealityCharting software implements the method.

Core principle. Problems do not have a single root cause — they have an infinite continuum of causes connected by cause-and-effect relationships. The analyst's job is to build a causal graph that is comprehensive enough to support effective solutions AND bounded at the level where actionable interventions exist.

Key distinctions from 5 Whys:

• Evidence requirement — every cause node must have sensed evidence (observation, measurement, data). Causes without evidence are hypotheses, not findings.
• "What caused this?" framing instead of "Why?" — tends to produce mechanistic causes rather than blame assignments.
• Explicit branching instead of single-chain — forces completeness.

6. FMEA — Failure Mode and Effects Analysis

Origin. U.S. Military Procedure MIL-P-1629 (1949). Adopted by NASA (1960s), automotive industry (AIAG FMEA Manual), codified in IEC 60812. Current automotive standard: AIAG/VDA FMEA Handbook, 1st ed. 2019.

Relationship to RCA. FMEA is proactive — applied before failures occur to identify and mitigate potential failure modes. RCA is reactive — applied after failure. FMEA findings feed RCA prevention frameworks; RCA findings reveal failure modes that FMEA should have anticipated.

RPN formula: Risk Priority Number = Severity × Occurrence × Detection (each 1-10 scale).

Critical caveat: RPN is an ordinal product of three ordinal scales — it is not a ratio metric. The 2019 AIAG/VDA handbook replaced RPN as the primary prioritization metric with AP (Action Priority: H/M/L), weighting severity more heavily than the flat multiplicative RPN approach.

Variants:

• Design FMEA (DFMEA): product/system design; design engineers; during design phase
• Process FMEA (PFMEA): manufacturing or operational process; process engineers

7. Kepner-Tregoe Problem Analysis

Origin. Charles Kepner and Benjamin Tregoe, The Rational Manager (McGraw-Hill, 1965; updated as The New Rational Manager, 1981). Developed from behavioral research on how effective managers actually solve problems vs. how they think they solve them.

Core principle. Every problem is a deviation from expected performance, and every deviation has a specific cause that changed something. The IS/IS-NOT framework surfaces that change by defining exactly where and when the deviation exists and where it does not.

The critical test. A valid cause must explain all IS entries and be consistent with all IS NOT entries. Causes that can't explain both sides are eliminated. This "explain both sides" test is the heart of KT.

8. The Swiss Cheese Model

Origin. James T. Reason, University of Manchester. Human Error (Cambridge University Press, 1990) and Managing the Risks of Organizational Accidents (Ashgate, 1997). Applied in aviation (UK CAA), healthcare (IOM To Err Is Human, 2000), and nuclear safety.

Core model. Every system has multiple defensive layers — procedures, training, automation, monitoring, supervision, physical barriers. Each layer has holes (weaknesses, gaps, failures). Normally holes don't align across layers — a failure in one is caught by another. An accident occurs when holes in multiple layers align simultaneously.

Active failures vs. latent conditions:

• Active failures: Unsafe acts by front-line operators — errors, violations, misjudgments. Directly triggered the incident. Immediately visible.
• Latent conditions: Underlying weaknesses built into the system by designers, managers, maintainers — inadequate training, poor human-machine interface design, production pressure overriding safety culture, understaffing, ambiguous procedures. Present long before the incident. Often invisible until an accident occurs.

The reframe for RCA. Traditional RCA asks "Who made the error?" The Swiss Cheese model asks "Why did the system's defenses fail to catch that error?" This moves blame from front-line operators to systemic conditions.

Reason: "Unlike active failures whose effects are felt almost immediately, latent conditions may lie dormant within the system for many years before they combine with active failures and local triggers to create an accident opportunity."

Implication. Plugging holes in one layer is insufficient if the system relies on that single layer. Effective remediation adds layers or makes existing layers more robust. The most durable fixes address latent conditions — the holes that have been present for years — rather than only the active failure.

9. Postmortem Culture

Canonical sources. Site Reliability Engineering (Beyer, Jones, Petoff, Murphy — O'Reilly, 2016), Chapter 15: "Postmortem Culture." Google SRE Workbook (2018), Chapter 10. Etsy's Debriefing Facilitation Guide (John Allspaw, 2016), drawing on Sidney Dekker's The Field Guide to Understanding Human Error (2006).

Blameless definition (Google SRE). A written record of an incident, its impact, actions taken to resolve it, root causes, and follow-up actions — conducted without indicting any individual for bad behavior. Goal: systemic learning, not personal accountability.

Why blameless works. When engineers fear blame, they under-report, hide contributing information, and avoid honest analysis. Blameless makes it psychologically safe to expose the full causal picture.

Hindsight bias. Knowing the outcome makes the failure look obvious in retrospect. Investigators unconsciously reconstruct the decision sequence as if the outcome were predictable. Countermeasure: conduct timeline reconstruction forward from before the incident, not backward from the failure. Dekker calls this "studying the sharp end."

Etsy practice. "Once you welcome people into the room and set expectations about the mindset they should be in (blameless) and the outcome (learning), there's really only one thing to focus on: discovering the story behind the story."

10. Common RCA Mistakes

Stopping at proximate cause. The most frequent failure. Proximate cause is the immediate trigger. Root cause is the systemic condition that made the trigger possible. If your corrective action is "be more careful," you stopped at proximate cause.

Confirmation bias. Investigation teams form a hypothesis within the first hour and then collect confirming evidence. Countermeasure: require all evidence collected before hypotheses formally ranked. Designate "red team" member to argue against leading hypothesis.

Single root cause assumption. Most complex system failures have multiple necessary contributing causes — no single cause was sufficient alone. Use "contributing factors" language. Will Gallego's 2018 piece "No, Seriously. Root Cause is a Fallacy" argues this for software systems.

Mistaking correlation for causation. Change A and failure B occurred at the same time — therefore A caused B. Correlation is a candidate, not a conclusion. Require a mechanism.

Blaming humans as root cause. "Operator failed to follow procedure" closes the investigation prematurely. Correct next question: Why did the system make it possible to fail to follow that procedure?

Weak corrective actions. "Training" and "reminder emails" are the lowest-quality corrective actions. They rely on human memory under pressure — the same mechanism that failed.

11. The Actionability Test

A root cause is operationally defined as: the deepest cause in the chain at which you can implement a change that prevents recurrence. Two failure modes:

Stopped too shallow. "The deployment failed" — you can't change that. Go deeper until you reach a systemic condition you control.

Gone too deep. "Human beings make errors" — true, but not actionable at an organizational level. Back up one level.

Practical test. For each identified cause, ask: "Can I define a specific, implementable intervention that addresses this cause and is within our organization's authority to execute?" If yes — that is your actionable root cause. If not — continue searching.

12. RCA in Software Incidents

SRE practice. Google SRE defines RCA as identifying all contributing factors — plural — not a single root cause. Complex distributed systems fail through conjunctions of conditions.

Distributed systems specifics. Cause and effect frequently cross service boundaries. Tracing requires correlated logs across services (structured logging with trace IDs), distributed tracing (Jaeger, Zipkin, OpenTelemetry), and timeline reconstruction across system clocks (NTP sync issues are a real confounding variable).

Contributing factors framing. Modern SRE postmortem templates list: (1) proximate cause, (2) contributing factors, (3) detection failure (why didn't we know sooner?), (4) response failure (why did recovery take longer than expected?). Each category has independent corrective actions.

Five Whys in distributed systems. The depth-first, single-chain assumption breaks down completely in microservice architectures. Apollo/RealityCharting approach — explicit branching, evidence required at each node — is more appropriate than simple 5 Whys.

Observability as precondition. You cannot do RCA on a system you cannot observe. Structured logs, distributed traces, metrics with sufficient cardinality, correlation IDs are preconditions for meaningful RCA. First corrective action from any incident where observability was insufficient should be: instrument the system so the next incident of this type is diagnosable.

Canonical Citations

• Ohno, Taiichi. Toyota Production System: Beyond Large-Scale Production. Productivity Press, 1988.
• Ishikawa, Kaoru. Guide to Quality Control. JUSE Press, 1968.
• Kepner, Charles H. and Benjamin B. Tregoe. The New Rational Manager. Princeton Research Press, 1981.
• Reason, James T. Human Error. Cambridge University Press, 1990.
• Reason, James T. Managing the Risks of Organizational Accidents. Ashgate, 1997.
• Gano, Dean L. Apollo Root Cause Analysis: A New Way of Thinking. 3rd ed. Apollonian Publications, 2007.
• Beyer, Betsy, Chris Jones, Jennifer Petoff, and Niall Richard Murphy. Site Reliability Engineering. O'Reilly Media, 2016. Chapter 15.
• Allspaw, John. "Debriefing Facilitation Guide." Etsy, 2016.
• Dekker, Sidney. The Field Guide to Understanding Human Error. 3rd ed. Ashgate, 2014.
• Juran, Joseph M. Juran on Quality by Design. Free Press, 1992.
• Watson, H.A. "Launch Control Safety Study." Bell Telephone Laboratories / U.S. Air Force, 1961.
• IEC 61025: Fault Tree Analysis. International Electrotechnical Commission, 2006.
• AIAG/VDA FMEA Handbook. 1st ed. 2019.
• Gallego, Will. "No, Seriously. Root Cause is a Fallacy." willgallego.com, 2018.

RootCauseAnalysis — Method Selection Guide

When to use which workflow. The method matters — using 5 Whys on a safety-critical problem or FTA on a simple defect both produce bad outcomes.

Decision Flow

Is this an incident or a defect?
│
├─ Incident (production outage, security event, data loss)
│    │
│    └─ Use Postmortem as wrapper
│         ├─ Single thread, clear mechanism → 5 Whys inside
│         ├─ Multiple suspected categories → Fishbone inside
│         ├─ "Works here not there" subtle → Kepner-Tregoe inside
│         └─ Safety/security-critical → Fault Tree Analysis inside
│
└─ Defect (recurring bug, quality drift, process failure)
     │
     ├─ Simple, single-thread → 5 Whys
     ├─ Multi-category or brainstorm needed → Fishbone + Pareto
     ├─ Deviation from known-good → Kepner-Tregoe IS/IS-NOT
     ├─ Novel, never-happened-before → Apollo/RealityCharting
     └─ Complex interacting failures → Fault Tree

Quick Decision Table

Combining Methods

RCA methods nest and combine — they are not mutually exclusive.

The standard software-ops combination

Postmortem (wrapper) → 5 Whys (per thread) → Swiss Cheese (defensive layers review) → Action item strength ranking

The quality-investigation combination

Fishbone (breadth) → Pareto (prioritize vital few) → 5 Whys (depth on top causes) → Verification

The subtle-defect combination

Kepner-Tregoe (IS/IS-NOT identifies the change) → 5 Whys (go deeper into why the change wasn't caught) → Corrective action

The safety-critical combination

FTA (top-down deductive map) → FMEA (failure modes ranked by RPN/AP) → Postmortem (if incident occurred) → Action items at multiple layers

Anti-Patterns

Using 5 Whys when Fishbone is right.

• Signal: you keep getting stuck because the answer to "why?" has three valid parallel answers
• Switch to Fishbone so you can explore all branches

Using Fishbone when 5 Whys is right.

• Signal: you already know the category; you need depth, not breadth
• Use 5 Whys directly

Using FTA when you have no probability data.

• Signal: the quantitative benefit is lost; you're just drawing a tree
• Use Fishbone + 5 Whys instead

Using Kepner-Tregoe when there's no baseline.

• Signal: no "worked before" state exists; nothing to deviate from
• Use Apollo or Fishbone

Skipping the Postmortem wrapper for "small" incidents.

• Signal: you're making exceptions for "it wasn't a big one"
• Run the postmortem anyway — the discipline compounds; the exception never recovers the learning

Method-to-Domain Map

Speed vs. Thoroughness Tradeoff

Integration With Other Skills

• SystemsThinking — When multiple postmortems reveal the same structural cause, escalate to Iceberg / FindArchetype. RCA stops at contributing factors; SystemsThinking continues to structure and mental models.
• FirstPrinciples — Decompose a contributing factor to its fundamental truths before designing a fix.
• RedTeam — "How would we cause this again?" is adversarial RCA. Stress-test remediations.
• Science — RCA is the scientific method applied to failures. Use Science for hypothesis generation during investigation.

FaultTree Workflow — RootCauseAnalysis

Purpose

Construct a Fault Tree Analysis (FTA) — a top-down, deductive, Boolean-logic diagram that decomposes an undesired top event into its contributing events using logic gates. Developed at Bell Laboratories in 1961 for the Minuteman missile program; standardized in IEC 61025, NRC NUREG-0492 (nuclear), SAE ARP4761 (aerospace).

Unlike 5 Whys (linear) or Fishbone (categorical), FTA captures Boolean structure: which combinations of events must occur for the top event to happen. The key insight: incidents rarely have one cause; they have minimal cut sets — the smallest sets of basic events whose joint occurrence produces the top event.

Invocation

• "Fault tree," "FTA," "fault tree analysis"
• Safety-critical or security-critical system
• Complex multi-path failure where multiple defenses could each have prevented it
• Need quantitative probability estimate for a failure mode
• Systems with redundancy — need to reason about which defense combinations must all fail

Not for: fast operational triage. FTA is thorough but time-intensive; construction typically 2-8 hours.

The Structure

                       [ TOP EVENT ]
                            │
                          (Gate)
                   ┌────────┼────────┐
                   │        │        │
              [Event A] [Event B] [Event C]
                   │
                 (Gate)
              ┌────┴────┐
              │         │
         [Basic 1]  [Basic 2]

Terminology:

• Top event — the undesired outcome at the root of the tree
• Intermediate events — internal nodes, decomposed further
• Basic events — leaf nodes; cannot or need not be decomposed further; have assignable probabilities
• Gates — Boolean operators between levels

The Logic Gates

OR Gate

Symbol: Shape resembling a plus / pointed top.

Semantics: Top event occurs if any input occurs.

Top
 │
(OR)
 ├── A
 ├── B
 └── C

Top occurs if A OR B OR C.

Meaning: Each input alone is sufficient. No redundancy. Models single-point-of-failure structures.

AND Gate

Symbol: Flat-bottom, rounded top.

Semantics: Top event occurs only if all inputs occur (usually simultaneously).

Top
 │
(AND)
 ├── A
 ├── B
 └── C

Top occurs only if A AND B AND C (all three).

Meaning: All defenses must fail. Defense-in-depth. Models redundant protection.

Priority AND Gate

Semantics: All inputs must occur in a specific sequence. Order matters.

Useful when "A before B causes the problem" but "B before A does not."

Inhibit Gate

Semantics: Output occurs only if input occurs AND a conditioning event is present.

Models conditional failures — a failure mode that's only exposed under certain conditions.

Exclusive OR

Rare in FTA — used when top event occurs if exactly one (but not both) of two inputs occurs.

Execution

Step 1: Define the Top Event Precisely

The top event must be specific, observable, and measurable. Vague top events produce incoherent trees.

TOP EVENT: [specific undesired outcome]

Good: "User's payment is charged but order is not recorded." Bad: "Payment problems."

Step 2: Identify Immediate Causes and Logic

For the top event, ask: "What direct causes produce this event, and in what logical combination?"

• Are all causes necessary together? → AND
• Is any one cause sufficient alone? → OR
• Is there a priority sequence? → Priority AND
• Is there a conditioning factor? → Inhibit

Draw the gate and its inputs. Do not decompose further yet — just one level deep.

Step 3: Recursive Decomposition

For each non-basic intermediate event, repeat Step 2. Ask: what causes this event?

Continue until every leaf is a basic event:

• A basic event has a known or estimable probability
• OR a basic event is at the appropriate level of abstraction for this analysis

Stop conditions for a branch:

• Reach a physical component failure with a known probability
• Reach a human error with a statistical rate
• Reach a condition outside the system boundary (environmental, supply chain)
• Reach a level where further decomposition adds no analytical value

Step 4: Identify Minimal Cut Sets

A cut set is a set of basic events whose joint occurrence causes the top event. A minimal cut set is a cut set from which no member can be removed without breaking the causation.

Why it matters: Minimal cut sets identify the smallest combinations of failures that cause the top event. A system with many 1-event cut sets is fragile. A system with all cut sets containing 3+ events has strong defense-in-depth.

How to find them (for small trees, by hand):

• Under an OR gate, each input generates its own cut sets
• Under an AND gate, combine cut sets from each input (Cartesian product)
• For large trees, use MOCUS algorithm or FTA software

Step 5: Quantitative Analysis (if probabilities available)

For each basic event, assign a probability P(event). Propagate upward using Boolean algebra:

• OR gate: P(output) = 1 - ∏(1 - P(input_i))

(Approximate for small probabilities: P(output) ≈ Σ P(input_i))

• AND gate: P(output) = ∏ P(input_i) (assuming independence)

Critical caveat: Boolean propagation assumes statistical independence of basic events. In real systems, events are often correlated (common-mode failures). When events share a root cause, they are not independent, and the AND gate is far more probable than ∏ suggests. Flag common-mode possibilities explicitly.

Step 6: Prioritize Mitigation

For each minimal cut set:

• Quantify: what is P(cut set)?
• Rank: which cut sets have highest probability?
• Identify leverage: can any single basic event within the cut set be dramatically reduced?

Key insight: Eliminating one basic event from a 3-event cut set collapses its probability dramatically. Focus mitigation on the cheapest-to-eliminate basic event within the highest-probability cut set.

Output Format

🌲 FAULT TREE ANALYSIS: [top event]

TOP EVENT: [...]

TREE:
  [Top]
    │
   (OR)
    ├── Intermediate A
    │     │
    │    (AND)
    │     ├── Basic 1  (P = 0.01)
    │     └── Basic 2  (P = 0.05)
    │
    ├── Intermediate B
    │     │
    │    (OR)
    │     ├── Basic 3  (P = 0.02)
    │     └── Basic 4  (P = 0.03)
    │
    └── Basic 5 (P = 0.001)

MINIMAL CUT SETS:
1. {Basic 1, Basic 2} — P = 5.0e-4
2. {Basic 3} — P = 0.02
3. {Basic 4} — P = 0.03
4. {Basic 5} — P = 1.0e-3

TOP EVENT PROBABILITY (approx): ~0.054 per event exposure

PRIORITY MITIGATION:
1. Basic 4 — highest single-event cut set; eliminate or reduce
2. Basic 3 — second-highest single; same logic
3. {Basic 1, Basic 2} — 2-event AND, acceptable if independence holds
   ⚠ Common-mode check: are 1 and 2 truly independent?

COMMON-MODE FAILURES (flagged):
- [Potential correlated failure between events, explained]

RECOMMENDED ACTIONS:
- Eliminate single-event cut sets first
- Harden against common-mode failures
- Add defenses that raise smallest cut-set size

Worked Example — Payment Processing Unavailable

TOP EVENT: User payment processing unavailable for > 60 seconds.

TREE:
  [Payment unavailable]
     │
    (OR)
     ├── API gateway failure
     │     │
     │    (OR)
     │     ├── Gateway service down (P=0.0001)
     │     └── DNS resolution failure (P=0.00001)
     │
     ├── Payment service failure
     │     │
     │    (OR)
     │     ├── Service crash (P=0.0005)
     │     └── Config pushed wrong (P=0.0001)
     │
     └── Database unavailable
           │
          (AND)
           ├── Primary DB failure (P=0.001)
           ├── Replica promotion failure (P=0.01)
           └── Manual intervention not executed in SLA (P=0.1)

MINIMAL CUT SETS:
1. {Gateway service down} — P = 1.0e-4
2. {DNS resolution failure} — P = 1.0e-5
3. {Service crash} — P = 5.0e-4
4. {Config pushed wrong} — P = 1.0e-4
5. {Primary DB, Replica promotion, Manual intervention} — P = 1.0e-6

INSIGHT:
- The AND gate on DB failure collapses its probability to 1e-6 (defense-in-depth working)
- The OR gates on API gateway and payment service are weak (1-event cut sets)
- Payment service crash is the single highest-probability cut set (P = 5e-4)

PRIORITY:
1. Reduce P(payment service crash) — add crash watchdog, faster restart, circuit breaker
2. Reduce P(gateway service down) — redundant gateway instances
3. Replica promotion failure is high (P=0.01) — improve promotion automation, but AND gate makes this lower priority

Common Mistakes

• Confusing OR with AND. OR = single point of failure; AND = defense-in-depth. Getting this wrong inverts the analysis.
• Assuming independence. Real failures have common modes — shared deploys, shared dependencies, shared power. An AND gate is only as strong as its independence assumption.
• Decomposing too deep. "Electron did the wrong thing" is not a useful basic event. Stop at components with known failure rates.
• Using FTA for non-quantitative problems. If you have no probability data, the quantitative benefit is lost; 5 Whys or Fishbone may suffice.
• Neglecting human-error basic events. Software FTA often skips human errors (deploy mistakes, misconfigurations) — these are frequently the largest basic-event probabilities.
• Static tree. FTA must be updated as the system changes. An old tree analyzing an old architecture is misleading.

When to Use FTA

Tool Support

For non-trivial trees, use dedicated FTA software:

• SAPHIRE (free, NRC-distributed) — nuclear/aerospace
• FaultTree+ (commercial) — engineering
• EC FTA (open source) — research
• Graphviz/d2/Mermaid — sketching only; no cut set calculation

For simple trees (< 20 basic events), manual analysis is feasible.

Integration

• Entry from Postmortem — when the incident has redundant defenses that all failed
• Pairs with FMEA — FMEA enumerates failure modes; FTA traces consequences
• Feeds SystemsThinking — if common-mode failures keep appearing, the structure is generating them; escalate to CausalLoop / Archetypes

Attribution

H.A. Watson, Bell Telephone Laboratories (1961), original FTA development for U.S. Air Force Minuteman program. Formalized in IEC 61025 (Fault tree analysis), NRC NUREG-0492, SAE ARP4761 (aerospace safety). Canonical modern reference: W.E. Vesely et al., Fault Tree Handbook (NUREG-0492, NRC, 1981).

Fishbone Workflow — RootCauseAnalysis

Purpose

Construct a fishbone (Ishikawa) diagram — a cause-and-effect map that organizes contributing factors into named categories. Unlike 5 Whys, which follows a single linear chain, Fishbone is deliberately breadth-first: it forces you to consider all categories of cause before drilling down.

Best tool when multiple stakeholders contribute domain knowledge, the problem plausibly has causes in more than one category, or you need a structured brainstorm before narrowing.

Invocation

• "Fishbone," "Ishikawa," "cause-and-effect diagram"
• "What are all the things that could contribute to this?"
• Multiple stakeholders need to contribute
• Before running 5 Whys — to avoid fixating on one cause category prematurely
• Quality / defect investigation where the failure mode could come from several sources

The Structure

                  People              Process             Material
                     │                   │                   │
               ┌─────┴─────┐       ┌─────┴─────┐       ┌─────┴─────┐
               │           │       │           │       │           │
               │           │       │           │       │           │
        ─ ─ ─ ─┴─ ─ ─ ─ ─ ─┴─ ─ ─ ─┴─ ─ ─ ─ ─ ─┴─ ─ ─ ─┴─ ─ ─ ─ ─ ─┼─ ─ ─ ─ ▶ PROBLEM
                                                                    │
               │           │       │           │       │           │
               │           │       │           │       │           │
               └─────┬─────┘       └─────┬─────┘       └─────┬─────┘
                     │                   │                   │
                  Machine           Measurement         Environment

Problem at the "head" (right). Categories as major "bones." Specific causes as sub-bones within each category.

Choosing a Category Set

Ishikawa himself recommended adapting categories to context. Standard sets:

6 M's — Manufacturing / Technical Systems (default for software)

4 P's — Service / Customer-Facing Industries

8 M's — Extended Manufacturing

6 M's + Management (decisions, priorities, resource allocation) + Maintenance (upkeep, patching, lifecycle)

8 P's — Business / Marketing Strategy

Product/Service, Price, Place, Promotion, People, Process, Physical Evidence, Partners

For software incidents, 6 M's is almost always the right starting point. Adapt if the problem is clearly not fitting (e.g., pure process failure → use 4 P's).

Execution

Step 1: Write the Problem Statement

Place at the head of the fish. Must be measurable and specific.

PROBLEM: [Precise, observable statement]

Step 2: Select Category Set

Based on problem type. Document the choice in output.

Step 3: Brainstorm Causes Within Each Category

Rules:

• No evaluation during brainstorm. Every candidate is captured.
• Multiple stakeholders contribute. Different expertise surfaces different categories.
• Use sticky notes / whiteboard. Visual helps.
• Empty category is a signal. If People has zero causes, either this isn't a People problem or you need different experts in the room.

For each category, ask:

• What in this category could have contributed?
• What is unusual, stressed, or recently changed in this category?
• What would need to be true for this category to be a cause?

Step 4: Develop Sub-Bones (Nested 5 Whys)

For each cause, ask "Why does this happen?" 2-3 times. This is where 5 Whys nests inside Fishbone — each category branch gets its own small Whys chain.

Category: Process
  ├─ Cause: Deploy runbook missing
  │    └─ Why: Template predates current architecture
  │         └─ Why: No owner for runbook maintenance
  ├─ Cause: Rollback procedure untested
  │    └─ Why: Never run in prod
  │         └─ Why: Fear of causing secondary incident

Step 5: Prioritize Using Pareto

Not all identified causes matter equally. Apply Pareto analysis:

1. Quantify each cause — frequency from incident history, estimated impact, or expert estimate. 2. Sort descending by quantity. 3. Calculate cumulative percentage. 4. Identify the vital few (typically top 3-5 causes that account for 80% of impact). 5. Focus remediation on the vital few; park the rest.

Critical: Pareto identifies which causes to focus on; it does not tell you why those causes occur. Always follow with 5 Whys depth on the vital few.

Step 6: Verify the Top Causes

Do not assume causation. For each vital-few cause, plan verification:

CAUSE: [description]
VERIFICATION PLAN: [how we'll test whether this is actually a contributing cause]
- Evidence needed: [data/observation required]
- Test: [experiment or historical query]
- Expected outcome if cause is real: [prediction]

Output Format

🐟 FISHBONE ANALYSIS: [problem]

PROBLEM: [precise statement]
CATEGORY SET: [6M / 4P / 8M / 8P / custom]

CAUSE MAP:

People:
- [Cause A1]
  └─ Why: [depth]
- [Cause A2]

Machine:
- [Cause B1]
- [Cause B2]

Method:
- [Cause C1]
  └─ Why: [depth]
    └─ Why: [depth]

Material:
- [Cause D1]

Measurement:
- [Cause E1]

Environment:
- [Cause F1]

PARETO (top causes by impact):
| Cause | Impact | Cumulative % |
|-------|--------|--------------|
| [C1]  | 45%    | 45%          |
| [B2]  | 25%    | 70%          |
| [A1]  | 15%    | 85%          |  ← 80% threshold above this line
| [D1]  | 10%    | 95%          |
| ...   | ...    | ...          |

VITAL FEW (primary focus):
1. [Cause] — verification plan: [...]
2. [Cause] — verification plan: [...]
3. [Cause] — verification plan: [...]

CORRECTIVE ACTIONS: [after verification, each verified cause gets an action]

Worked Example — Elevated p99 Latency Post-Deploy

PROBLEM: Checkout service p99 latency spiked from 200ms to 3,200ms after deploy on 2026-04-10.

CATEGORY SET: 6 M's

CAUSE MAP:

Manpower (People):
- Deploying engineer was new to this service; unfamiliar with async-only pattern
- Reviewer approved the PR without running benchmarks

Machine:
- Instance type selected in Terraform was t3.small (underpowered for new workload)
- Connection pool sized for previous traffic pattern

Method:
- No canary release process — 100% of traffic shifted at once
- No pre-deploy performance smoke test

Material:
- New dependency introduced a synchronous external API call (was async in prior version)
- Shared library version upgraded; deprecated async API

Measurement:
- Percentile latency not in pre-deploy runbook check (only mean latency)
- No alert on p99 deviation during rollout

Environment:
- Deploy occurred during peak traffic window (14:00 UTC)
- External dependency had elevated latency that day (not factored in)

PARETO:
| Cause | Est. impact | Cumulative |
|-------|-------------|------------|
| Synchronous external call (Material) | 65% | 65% |
| No canary (Method) | 15% | 80% |
| t3.small instance (Machine) | 10% | 90% |
| Peak deploy window (Environment) | 5% | 95% |
| Others | 5% | 100% |

VITAL FEW:
1. Synchronous external call — verification: review diff; run local benchmark comparing sync vs. async
2. No canary process — verification: historical: have past full-traffic deploys also spiked?
3. Instance type — verification: redeploy to t3.medium; compare

CORRECTIVE ACTIONS:
- Convert external call back to async — owner: checkout — deadline: Apr 15
- Implement canary (10% → 50% → 100% with p99 gate) — owner: platform — deadline: Apr 30
- Instance sizing in Terraform per-service review — owner: infra — deadline: May 5

Common Mistakes

• Forcing causes into categories. If a cause doesn't fit any category, consider adding a new one — don't distort the cause.
• Empty categories ignored. An empty category often means you need a different expert. Don't just move on.
• Pareto skipped. Without prioritization, you try to fix everything and fix nothing.
• Verification skipped. A cause on the diagram is a hypothesis, not a conclusion. Test the top candidates before committing to remediation.
• Category choice wrong. 6 M's for a pure policy problem produces a distorted diagram. Pick (or adapt) the right categories.
• Single-session work. For complex incidents, the initial fishbone is draft 1. New evidence in subsequent sessions often reshapes the diagram.

Integration

• 5 Whys nests inside — each sub-bone can become its own 5 Whys chain
• Pareto nests inside — quantitative prioritization step
• Postmortem wraps Fishbone — Postmortem uses Fishbone for the "contributing factors" section
• Feeds SystemsThinking — if multiple fishbone causes point to structural issues, escalate to Iceberg

Attribution

Kaoru Ishikawa, first used at Kawasaki Steel Works (1943), formally presented 1945, codified in Guide to Quality Control (JUSE Press, 1968) as one of the seven basic quality tools. Category-set variants from American Society for Quality (ASQ) training literature and AIAG manufacturing standards.

FiveWhys Workflow — RootCauseAnalysis

Purpose

Walk a causal chain from symptom to systemic cause by repeatedly asking "Why?" The goal is not five questions — it is reaching an actionable systemic cause, which usually requires 4-7 iterations with at least one branch.

Originated with Sakichi Toyoda in the 1930s and embedded in the Toyota Production System by Taiichi Ohno as "the basis of Toyota's scientific approach." Simple, fast, low-overhead. Best first tool for most incidents.

Invocation

• "5 whys," "five whys," "do a 5 whys"
• Single-thread incident with known-proximate cause
• Quick operational triage when time is short
• As a sub-step inside Fishbone (each category bone gets a 5 Whys)

Canonical Procedure

Step 1: Write the Problem Statement Precisely

A vague statement produces a vague chain.

PROBLEM: [Specific, observable, measurable]

Bad: "Reliability is down." Good: "The payments service returned HTTP 500 for 14 minutes starting 2026-04-12 23:51 UTC, affecting 3,412 user checkouts."

Step 2: Ask "Why Did This Occur?"

Record the direct cause. Keep it concrete and mechanical.

WHY 1: [Direct mechanical cause]

Step 3: Ask "Why Did That Occur?"

Now ask why the previous answer is true. Again, mechanical.

WHY 2: [Cause of WHY 1]
WHY 3: [Cause of WHY 2]
WHY 4: [Cause of WHY 3]
...

Step 4: Stop When Actionable AND Systemic

Stop condition, both required:

• Actionable — you can define a concrete intervention that addresses this cause
• Systemic — the intervention prevents a class of failure, not just this instance

If you have one without the other, keep going.

• Actionable but not systemic ("patch this specific line"): you stopped too shallow
• Systemic but not actionable ("humans make mistakes"): you went too deep; back up one level

Step 5: Validate by Reading Forward

Read the chain from bottom to top as "Because X, therefore Y, ..., therefore problem."

If the forward read does not hold together, the chain has a logical jump. Fix it before concluding.

Step 6: Branch Where the Chain Forks

A single-chain 5 Whys is the most common failure mode. At each "Why?" ask: are there multiple valid answers?

When yes, record both branches:

WHY 3:
  ├─ Branch A: [Cause A]
  │    WHY 4A: [...]
  │    WHY 5A: [...]
  └─ Branch B: [Cause B]
       WHY 4B: [...]
       WHY 5B: [...]

Converging branches that share a common ancestor indicate a high-leverage systemic cause — one fix addresses multiple failure paths.

Step 7: Optional — Five Hows

After identifying the root cause, apply "How do we prevent this?" five times to the solution:

HOW 1: [First intervention]
HOW 2: [How do we make HOW 1 robust?]
HOW 3: [How do we prevent HOW 1 from decaying?]
...

Ensures the solution is as rigorous as the diagnosis.

Output Format

🔍 5 WHYS ANALYSIS: [problem, 12 words]

PROBLEM: [precise statement]

CHAIN:
- WHY 1: [cause]
- WHY 2: [cause of WHY 1]
- WHY 3: [cause of WHY 2]
- WHY 4: [cause of WHY 3]
- WHY 5: [cause of WHY 4 — root]

BRANCHES: [if any]
- At WHY N:
  ├─ Branch A: ...
  └─ Branch B: ...

ROOT CAUSE(S): [systemic, actionable]
- [Cause 1]
- [Cause 2]  ← if branches converged

CORRECTIVE ACTIONS:
- [Specific action — owner — deadline]
- [Specific action — owner — deadline]

VALIDATION (read forward):
Because [root], therefore [WHY 4], therefore [WHY 3], ..., therefore [problem].

Worked Example — Software Incident

PROBLEM: Production API returned HTTP 500 on 1,200 requests during 2026-04-12 14:00-14:14 UTC.

CHAIN:
- WHY 1: The payments database connection pool was exhausted.
- WHY 2: Query execution time spiked from 40ms p99 to 3,800ms p99.
- WHY 3: Queries were doing a full table scan on orders table.
- WHY 4: Missing index on `(customer_id, created_at)` — a frequently-joined column pair.
- WHY 5: Schema migration that added the new join pattern shipped without creating the index.
- (WHY 6 would be):  Migration PR template did not require EXPLAIN ANALYZE on new query patterns.

ROOT CAUSE: Migration review process has no query-plan analysis step.

CORRECTIVE ACTIONS:
- Add `EXPLAIN ANALYZE` output to migration PR template — owner: platform — deadline: Apr 18
- Backfill missing index now — owner: payments oncall — deadline: today
- Add runtime query-plan monitoring with alerting on new full scans — owner: observability — deadline: Apr 25

VALIDATION: Because migrations don't require query plans, the index was missed, which caused full scans, which spiked query time, which exhausted the pool, which returned 500s.

Common Mistakes

• Stopping at blame. If the chain ends at "engineer made a mistake," ask one more why. Systems allow mistakes; root cause is in the system.
• Skipping levels. Jumping from symptom to a distant conclusion ("deploys are bad!") sounds deep but skips mechanism. Each "why" must be the direct cause of the one above it.
• Single-chain bias. Most real incidents branch. A perfectly linear 5 Whys is suspicious — it usually means you picked the most obvious branch and ignored others.
• Stopping at "that's just how it is." If the answer doesn't suggest an action, it's not the root cause.
• Hindsight bias. You know the outcome. The engineer didn't. Ask "what would this person have reasonably believed at this moment?" not "why didn't they see it?"
• Generating corrective actions that rely on human vigilance. "Train people harder" / "remind everyone" are the weakest actions. Prefer automation, checks, or process changes that make the failure hard to repeat.

When NOT to Use 5 Whys

• Safety-critical systems. Use Fault Tree Analysis; 5 Whys cannot give probability estimates.
• Complex distributed systems with crossing service boundaries. Use Apollo/RealityCharting or Fishbone + Postmortem. Single-thread assumption breaks down.
• Subtle "works here but not there" defects. Use Kepner-Tregoe IS/IS-NOT.
• When you don't have the domain knowledge. The method cannot go deeper than the investigator's knowledge. Get experts in the room, or switch to Fishbone where multiple experts contribute.

Integration

• Nests inside Fishbone — each major category bone gets its own 5 Whys for depth
• Nests inside Postmortem — the "causes" section of a postmortem usually uses 5 Whys internally
• Feeds SystemsThinking/Iceberg — if the chain keeps branching into structural causes, escalate to Iceberg analysis

Attribution

Sakichi Toyoda (1930s, Toyota Industries). Formalized in Taiichi Ohno's Toyota Production System: Beyond Large-Scale Production (1988). Teruyuki Minoura's critiques on single-chain bias are canonical limits. "Five Hows" variant from lean manufacturing practice.

KepnerTregoe Workflow — RootCauseAnalysis

Purpose

Identify the cause of a deviation — something that was working and stopped, or something that works in one context but not another — by rigorously specifying what IS the problem and what IS NOT, then finding the distinction and the change.

From Charles Kepner and Benjamin Tregoe, The Rational Manager (1965). The method was derived from behavioral research on how effective managers actually solve problems vs. how they think they solve them — particularly powerful for subtle, intermittent, or environment-specific defects.

Core Principle

Every problem is a deviation from expected performance. Every deviation has a specific cause that changed something. The IS/IS-NOT framework surfaces that change by defining exactly where and when the deviation exists — and where it does not.

The critical question: what is different between where the problem exists and where it doesn't?

Invocation

• "Kepner-Tregoe," "KT analysis," "IS/IS-NOT"
• "This works here but not there"
• "It failed this time but worked yesterday"
• Intermittent defects, environment-specific failures
• Defects where "we didn't change anything" (you did — find what)
• Hard-to-reproduce bugs

The IS/IS-NOT Matrix

Four dimensions:

Execution

Step 1: State the Deviation

DEVIATION: [what was expected vs. what is happening]

Example:

• Expected: p99 latency < 300ms in all regions.
• Actual: p99 latency = 2,500ms in us-east-1a only, after 18:00 UTC, for requests hitting service version 3.4.2.

Step 2: Build the IS/IS-NOT Matrix

Fill the matrix methodically. The IS NOT column is equally important as IS — it defines the boundary of the problem.

Each cell should be specific and comparable to its opposite.

Step 3: Identify Distinctions

For each dimension, ask: what is different between IS and IS NOT?

• What's different about this service vs. similar services that are fine?
• What's different about this region vs. others?
• What's different about this time vs. times it didn't happen?
• What's different about this extent vs. baseline?

Distinctions are features unique to the problem's presence. Non-distinctions (things that are the same in both IS and IS NOT columns) cannot be causes.

Step 4: Identify Changes

For each distinction, ask: what changed recently in this area?

Changes are the mechanism by which the cause was introduced into the system.

Scope of "recently": usually the window between "last known good" and "first observed bad." Longer windows for slow-drift problems.

Changes to look for:

• Deploys, releases, configuration updates
• Schedule changes (cron jobs, batch processes)
• Environmental changes (load, weather, external dependencies)
• Personnel changes (new team member, shift change, vacation coverage)
• Data changes (schema migrations, new data volumes, new users)
• Hardware changes (instance type, network path, storage)
• Calendar events (month-end, quarter-end, holiday traffic)

Step 5: Generate Possible Causes

For each change, ask: how could this change have caused the observed deviation?

A possible cause must have a plausible mechanism. "We changed X" is correlation; "X causes Y because mechanism" is a hypothesis.

Step 6: Test Each Possible Cause

A valid cause must:

• Explain all entries in the IS column (why did it produce all these symptoms?)
• Be consistent with all entries in the IS NOT column (why didn't it affect these?)

Eliminate causes that can't explain both sides. A cause that explains IS but predicts problems in areas that are actually fine (IS NOT) is falsified.

This "explain both sides" test is the heart of KT. It is what distinguishes KT from weaker methods that accept the first plausible correlation.

Step 7: Verify the Surviving Cause

Confirm the identified cause produces the problem when present and doesn't when absent. Ideally: revert the change in a test environment and observe recovery.

Output Format

🔬 KEPNER-TREGOE ANALYSIS: [deviation]

DEVIATION:
- Expected: [...]
- Actual: [...]

IS/IS-NOT MATRIX:

| Dimension | IS | IS NOT | Distinction | Change |
|-----------|----|----|------|--------|
| What      | [specific]  | [similar OK] | [diff]      | [change in that area] |
| Where     | [location]  | [elsewhere OK] | [diff]      | [change in that location] |
| When      | [time]      | [other times OK] | [diff]   | [change at/before] |
| Extent    | [magnitude] | [baseline] | [diff]      | [change in scale] |

DISTINCTIONS:
- [D1]
- [D2]
- [D3]

CHANGES (candidates):
- [C1] — in area of [D1]
- [C2] — in area of [D2]

POSSIBLE CAUSES:
- [Cause A] — from change [C1] — mechanism: [...]
- [Cause B] — from change [C2] — mechanism: [...]

TESTING:
- Cause A:
  - Explains IS? ✓ / ✗ — [why]
  - Consistent with IS NOT? ✓ / ✗ — [why]
- Cause B: ...

SURVIVING CAUSE: [the one that explains all]
VERIFICATION PLAN: [how we'll confirm]

Worked Example — Memory Spike

DEVIATION:
- Expected: Node.js service holds stable ~400MB heap across all pods.
- Actual: Some pods spike to 2.8GB heap after 18:00 UTC, leading to OOM kill.

IS/IS-NOT MATRIX:

| Dimension | IS                       | IS NOT                          | Distinction                        | Change                                |
|-----------|--------------------------|--------------------------------|------------------------------------|---------------------------------------|
| What      | Heap spike to 2.8GB      | Normal 400MB heap              | Memory grows unboundedly           | Nothing new globally                 |
| Where     | us-east-1a pods          | us-east-1b, us-west-2          | 1a only                            | 1a got rolling deploy first          |
| When      | After 18:00 UTC          | Before 18:00 UTC               | Evening only                       | 18:00 = peak session creation        |
| Extent    | 7x normal heap           | 1x                             | Unbounded growth, not step-up      | — |

DISTINCTIONS:
- D1: Affects only 1a, not other regions
- D2: Only after 18:00 UTC
- D3: Unbounded growth (leak pattern), not one-time jump

CHANGES:
- C1: Service deploy 3.4.2 rolled to 1a first (partial rollout, not yet in other regions)
- C2: 3.4.2 introduced an in-memory session cache
- C3: Session creation peaks at 18:00 UTC (normal daily pattern; not a change, but interacts with C2)

POSSIBLE CAUSES:
- Cause A: 3.4.2's in-memory session cache has no eviction policy; at peak session creation, entries accumulate faster than they expire.

TESTING:
- Cause A:
  - Explains IS What (unbounded heap): ✓ — cache grows without bound
  - Explains IS Where (1a only): ✓ — 3.4.2 only deployed to 1a so far
  - Explains IS When (after 18:00): ✓ — peak session creation overwhelms
  - Explains IS Extent (7x): ✓ — cache can grow arbitrarily large
  - Consistent with IS NOT Where (1b, us-west-2 fine): ✓ — still on 3.4.1, no cache
  - Consistent with IS NOT When (before 18:00 fine): ✓ — lower session rate, growth slower; OOM not yet reached
  - ✓ All consistent.

SURVIVING CAUSE: 3.4.2 introduces session cache without eviction; unbounded at peak load.

VERIFICATION: Pull heap dump from affected pod; expect to see session objects dominating. Check 3.4.2 diff for session cache implementation.

REMEDIATION:
- Immediate: rollback 3.4.2 in 1a
- Structural: add LRU eviction + max-size limit to session cache in 3.4.3

Common Mistakes

• Sloppy IS NOT column. The IS NOT column defines the problem boundary. Empty or vague IS NOT = weak analysis.
• Skipping the "consistent with IS NOT" test. A cause that explains IS but predicts problems elsewhere that aren't happening is falsified. This test is the method's power.
• Missing changes. "Nothing changed" is always wrong in KT — something must have changed, or the deviation would not exist. Keep looking: deploys, cron jobs, data volume, schedule, personnel, external deps.
• Stopping at first plausible cause. Test against all dimensions of IS/IS-NOT. Premature conclusions are common.
• Environmental / temporal blind spots. People look at deploys but miss schedule-based changes (batch jobs, cron, end-of-month processes). Always scan calendar-triggered changes.
• Not verifying. The surviving cause is a hypothesis until tested in reality. Plan verification before acting.

When KT is the Right Tool

Integration

• Entry from Postmortem — when the incident is a deviation with unclear cause
• Feeds 5 Whys — once KT finds the change, 5 Whys can go deeper into why that change wasn't caught
• Pairs with Fishbone — if KT eliminates categories, Fishbone structures what remains
• Pairs with Observability — KT requires data about both IS and IS NOT. Poor observability → cannot run KT

Attribution

Charles H. Kepner and Benjamin B. Tregoe, The Rational Manager (McGraw-Hill, 1965); updated The New Rational Manager (Princeton Research Press, 1981). Grounded in behavioral research on effective managerial problem-solving vs. self-reported problem-solving. The methodology remains the canonical framework for deviation-based troubleshooting in engineering and operations.

Postmortem Workflow — RootCauseAnalysis

Purpose

Produce a structured, blameless written record of an incident — its impact, timeline, contributing factors, resolution, and follow-up actions. Grounded in Google SRE practice, Etsy's debriefing facilitation work (John Allspaw), and Sidney Dekker's Field Guide to Understanding Human Error.

The postmortem is the wrapper for other RCA tools. Inside the postmortem, use 5 Whys, Fishbone, Kepner-Tregoe as appropriate to investigate the causes.

The goal is systemic learning, not individual accountability. Postmortems that drift into blame produce worse postmortems, not better behavior.

Invocation

• "Postmortem," "incident review," "blameless postmortem"
• Production incident with user-visible impact
• Security incident
• Data loss or near-miss
• Novel failure mode (even without customer impact, if it could have been worse)
• Any incident where on-call had to intervene

Google SRE Trigger Criteria

Run a postmortem when any of:

• User-visible downtime or degradation above a threshold
• Data loss of any kind
• On-call engineer intervention required
• Novel failure mode (first occurrence of this type)
• A monitoring failure (incident should have been detected sooner)
• Anything that should not have happened by the system's own design expectations

Default to running one. The cost of skipping is high (missed learning); the cost of running is a few hours.

The Blameless Principle

Definition (Google SRE): A written record of an incident, its impact, the actions taken to resolve it, the root causes, and the follow-up actions — conducted without indicting any individual for bad behavior.

Why blameless works: When engineers fear blame, they under-report, hide contributing information, and avoid honest analysis. Blameless culture makes it psychologically safe to expose the full causal picture, including the engineer's own decision-making under incomplete information.

Blameless ≠ accountability-free. Action items have owners and deadlines. What blameless excludes: personal judgments, punitive framing, "why did you do that?" questions directed at individuals.

Etsy rule: "Once you welcome people into the room and set expectations about the mindset they should be in (blameless) and the outcome (learning), there's really only one thing to focus on: discovering the story behind the story."

Facilitation Rules

• No "why did you do that?" directed at individuals
• Instead: "What did you see? What did you know? What were you trying to accomplish?"
• Reconstruct the timeline forward, not backward — fight hindsight bias
• Study the "sharp end" (Dekker) — what was it like to be the on-call engineer, in that moment, with what they knew?
• Separate this meeting from any performance conversation. Never combine them.

Execution

Phase 1: Timeline Reconstruction (before investigation)

Reconstruct what happened in chronological order. Include:

• Every signal that was available to responders
• Every action taken
• Every communication (chat messages, phone calls, alerts)
• Decision points — what information was available at each?

Tools: Incident channel transcript, alert history, deploy logs, on-call notes, chat logs.

Critical: Reconstruct the timeline before hypothesizing about causes. If you start with a cause hypothesis, you will unconsciously organize the timeline to support it.

Phase 2: Investigate Causes (using 5 Whys, Fishbone, KT as appropriate)

For each contributing thread, run the appropriate sub-workflow:

• Clear single-thread cause → 5 Whys (FiveWhys.md)
• Multiple suspected categories → Fishbone (Fishbone.md)
• "Works on X, not Y" subtle defect → Kepner-Tregoe (KepnerTregoe.md)
• Complex multi-path failure → Fault Tree (FaultTree.md)

Multiple contributing factors is the default. Modern SRE postmortems list contributing factors (plural), not a single root cause. Will Gallego's 2018 piece "No, Seriously. Root Cause is a Fallacy" codified this.

Phase 3: Classify the Failure Modes

For each contributing factor, classify:

• Proximate cause — the immediate trigger
• Contributing factor — conditions that enabled the trigger or worsened its impact
• Detection failure — why wasn't this caught sooner?
• Response failure — why did recovery take longer than expected?

Each category gets independent corrective actions. You might fix the proximate cause, improve detection, and improve runbook — separately.

Phase 4: Apply Swiss Cheese Model (James Reason)

Most incidents aren't caused by a single failure — they occur when holes in multiple layers of defense align simultaneously.

Map the defenses that existed:

SWISS CHEESE MAP:

Defensive layer 1: [what was supposed to prevent this]
- Hole: [why it didn't]

Defensive layer 2: [next defense]
- Hole: [why it didn't]

Defensive layer 3: [next defense]
- Hole: [why it didn't]

[Incident occurred when holes aligned]

Active failures (what a human did) + latent conditions (what the system allowed) both contribute. Fix latent conditions — they've been present for years and caused this only when activated by a particular active failure. Fixing them prevents a whole class of future incidents.

Phase 5: Generate Action Items

Each action item gets:

• Owner — a specific person (not a team)
• Deadline — a specific date
• Verification — how we'll know it's done
• Strength rating — how much it reduces recurrence probability

Action strength hierarchy (strongest to weakest):

Rule: If your top action items are "training" and "documentation," go back. You have weak actions. They rely on the same human vigilance that failed this time.

Phase 6: Write the Document

# Postmortem: [Incident title]

**Date:** YYYY-MM-DD
**Authors:** [list]
**Status:** Draft / Final
**Classification:** Internal / Confidential

## Summary

[1-2 paragraph executive summary. Non-technical stakeholders should be able to read this.]

## Impact

- User-facing: [N users affected, M% of traffic, X minutes]
- Revenue: [$ impact if calculable]
- Data: [any loss, corruption, or exposure]
- On-call: [who paged, for how long]

## Timeline

**All times UTC.**

- `23:47` — Deploy D-1234 pushed to production
- `23:49` — Traffic shift to new version
- `23:51` — First p99 latency alert fires
- `23:52` — On-call paged (Alice)
- `23:53` — Alice acknowledges; begins investigation
- `23:55` — Alice identifies elevated 500s in payments service
- `23:58` — Alice initiates rollback
- `00:01` — Rollback complete; metrics recover
- `00:05` — Incident closed

## Contributing Factors

### Proximate Cause
[What immediately triggered the incident]

### Systemic Factors
[The conditions that made the proximate cause possible or likely]

1. **[Factor 1]**
   - Evidence: ...
   - Why it existed: ...

2. **[Factor 2]**
   - ...

### Detection Gaps
[Why didn't we know sooner?]

### Response Gaps
[Why did recovery take as long as it did?]

## Swiss Cheese Analysis

Defensive layers that existed and the holes in each:

- **Layer 1 — CI gate:** Tests passed; hole — no load tests, didn't cover this query pattern
- **Layer 2 — Canary:** Not used for this deploy; hole — deploy process doesn't require canary for payments service
- **Layer 3 — Pre-deploy runbook:** Runbook doesn't include p99 check; hole — runbook predates p99 monitoring
- **Layer 4 — Monitoring:** p99 alerting did fire but took 2 minutes; hole — alert evaluation window too long

## What Went Well

[Always include. Practices that worked. Morale fuel.]

- Rollback procedure ran cleanly
- On-call response time was under SLA
- Cross-team communication in incident channel was clear

## What Went Poorly

[Blameless. Process failures, not person failures.]

- Deploy process allowed full-traffic cutover without canary
- Runbook was out of date
- Alert evaluation window was too long for this class of problem

## Action Items

| # | Action | Strength | Owner | Deadline | Verification |
|---|--------|----------|-------|----------|--------------|
| 1 | Require canary (10/50/100 with p99 gate) for payments deploys | Automation (Strong) | Platform-eng | Apr 30 | PR merged + canary used on next 3 deploys |
| 2 | Update pre-deploy runbook with p99 check | Standardize (Medium) | Payments-oncall | Apr 18 | Runbook diff merged; team review confirms |
| 3 | Reduce p99 alert evaluation window from 5m to 1m for payments | Simplify (Medium) | SRE | Apr 22 | PagerDuty config diff merged |
| 4 | Backfill missing index | Eliminate (Strongest) | Payments | Today | Index exists; query plan confirms use |
| 5 | Add EXPLAIN ANALYZE to migration PR template | Force function (Strong) | Platform-eng | Apr 25 | PR template updated; next 3 migrations include it |

## Lessons Learned

[What do we know now that we didn't know before?]

- Deploy tests don't cover database query plan regression — we were relying on a latent assumption.
- p99 alerts fired but not fast enough to prevent user impact. Alert latency matters as much as alert accuracy.
- The migration review process was the structural hole — not the engineer, not the deploy.

## Follow-Up

- [ ] Action items tracked in [ticket system]
- [ ] Re-review in 30 days to confirm action completion
- [ ] Pattern check: are similar incidents occurring across other services? (escalate to SystemsThinking if yes)

Phase 7: Distribute and Track

• Publish internally (visibility drives learning)
• Tag in incident-management system
• Track action items to completion — postmortems without follow-through are documentation theater
• Revisit at 30 days: did the actions actually prevent the class of failure?

Common Mistakes

• Single root cause framing. Use "contributing factors" (plural). Single-root is almost always wrong.
• Drifting into blame. The moment "why did Alice do X?" becomes the focus, the postmortem is compromised. Rewind.
• Weak action items. "Training" and "reminders" rely on human vigilance. Rank actions by strength; prefer automation and force functions.
• Skipping the "what went well" section. It's not decoration; it reinforces the practices that saved the incident from being worse.
• No owner or deadline on action items. Un-owned = un-done.
• Combining postmortem with performance review. Never. Different meetings, different participants, different psychological contracts.
• Hindsight bias in timeline. "Alice should have noticed X" — at 23:55, with the signals she had, would you have? Reconstruct what was known at each moment, not what was discoverable in retrospect.

Integration

• Wraps: 5 Whys, Fishbone, Kepner-Tregoe, Fault Tree — use whichever fits each contributing thread
• Feeds SystemsThinking — if the same postmortem-worthy pattern keeps recurring, escalate to Iceberg / FindArchetype
• Feeds FMEA — proactive sister tool; postmortem findings reveal failure modes that FMEA should anticipate going forward
• Tracked in incident system — Google SRE pattern: postmortem linked to every incident

Attribution

Google SRE Book (Beyer, Jones, Petoff, Murphy, Site Reliability Engineering, O'Reilly 2016), Chapter 15 — canonical modern reference. Etsy debriefing practice (John Allspaw, 2016). Sidney Dekker, Field Guide to Understanding Human Error (Ashgate, 2014 — 3rd ed.). Blameless framing from Reason's Human Error (1990). Action strength hierarchy adapted from healthcare safety literature (Institute for Safe Medication Practices). "Root cause is a fallacy" framing: Will Gallego, willgallego.com, 2018.