Root Cause Analysis

Root Cause Analysis Orchestration Skill

This skill helps you systematically identify the root cause of any problem using proven methodologies from the Toyota Production System and other industry-standard techniques.

Quick Reference: When to Load Which Resource

Core Principle

Do not treat symptoms—find and fix the root cause. As Taiichi Ohno, architect of the Toyota Production System, said: "By repeating why five times, the nature of the problem as well as its solution becomes clear."

Orchestration Protocol

Phase 1: Problem Classification

Quickly identify your problem domain and complexity:

Problem Domain:

• Software: Code bugs, system failures, performance, deployment
• Hardware: Equipment, mechanical, electrical, maintenance
• Process: Workflow, procedures, organizational, communication
• Personal: Life challenges, productivity, habits, wellbeing

Complexity Level:

• Simple: Clear failure chain, 1-2 likely causes → Use 5 Whys
• Complex: Multiple possible causes, unknown scope → Start with Fishbone
• Critical/Safety: High stakes, needs rigor → Use Fault Tree
• Multiple Issues: Many competing problems → Use Pareto first

Action: Load appropriate resource file(s) based on classification.

Phase 2: Methodology Selection

Based on problem type, select your approach:

Phase 3: Execution & Verification

During Analysis: 1. Define problem clearly (What/Where/When/Impact) 2. Gather evidence systematically 3. Apply selected methodology 4. Document reasoning at each step 5. Verify root cause with Forward/Backward tests

Before Finalizing:

• Validate conclusion against evidence
• Check for red flags (see common-root-causes.md)
• Confirm actionability (can you fix this?)
• Develop solutions addressing root cause

Problem Definition Framework

Create a clear problem statement before analysis:

Essential Elements:

• What: Observable symptom (not assumed cause)
• Where: Location/system/component affected
• When: Timeline, frequency, pattern
• Impact: Users/systems affected, severity

Example: "Users in EU region experience 3-5 second dashboard load delays during 9-11 AM UTC peak hours, affecting ~2,000 daily active users. Started after v2.4 deployment on Nov 18th."

Evidence Gathering (Go and See)

Follow Toyota's principle—collect facts, not opinions:

Key Evidence Sources:

• Logs, metrics, monitoring data
• Timeline of events and changes
• System/code/configuration changes before problem
• Environmental factors (load, traffic, season)
• User reports and reproduction steps
• System state before/during/after

RCA Methodologies

See resources/rca-methodologies.md for complete methodology guide.

Resource Files Summary

resources/rca-methodologies.md

Comprehensive methodology guide covering:

• 5 Whys: Step-by-step process with software examples
• Fishbone Diagram: Structure, 6 M's categories, process
• Pareto Analysis: Prioritization using 80/20 rule
• Fault Tree Analysis: Top-down formal analysis
• Barrier Analysis: Control failure examination
• Structured 6-phase RCA process, domain-specific guidance, templates

resources/common-root-causes.md

Pattern reference catalog by domain:

• Software Engineering: Code defects, configuration, dependencies, deployment
• Hardware & Equipment: Mechanical, electrical, operational, maintenance
• Process & Operations: Workflow, design, resources
• Personal/Life: Health, habits, environment, skills
• Red flags, recurring themes, pattern recognition

resources/example-analyses.md

Four worked examples with full analysis: 1. Software Bug: JWT authentication (5 Whys) 2. Vehicle Maintenance: Overheating (5 Whys) 3. System Failure: E-commerce checkout (Fishbone + 5 Whys) 4. Personal Productivity: Missed deadlines (Fishbone + 5 Whys)

resources/advanced-techniques.md

Formal methods for complex problems:

• Fault Tree Analysis: Boolean logic, safety systems
• Barrier Analysis: Control failures
• Multi-Methodology Chains: Complex orchestration
• Verification Frameworks: Comprehensive testing

How This Skill Works

1. Clarify your situation: Domain, observations, evidence, time 2. Recommend approach: Complexity analysis, methodology, resources 3. Guide through analysis: Problem statement, evidence, methodology, verification 4. Deliver output: Analysis, root cause, solutions, implementation

Quick Start: 5-Minute RCA

1. State problem (What/Where/When/Impact) 2. First Why: fact-based answer 3. Second Why: dig deeper 4. Third Why: dig deeper again 5. Verify: would fixing this prevent it?

Templates & Examples

• 5 Whys Template in resources/rca-methodologies.md
• Fishbone Template in resources/rca-methodologies.md
• Worked Examples in resources/example-analyses.md
• Solution Structures in resources/example-analyses.md

Next Steps

1. Identify problem domain (software/hardware/process/personal) 2. Load appropriate resource from table above 3. Select methodology based on complexity 4. Follow step-by-step process in resource 5. Verify root cause (Forward/Backward tests) 6. Develop actionable solutions

---

Remember: Goal is systematic investigation—disciplined questioning until you reach a cause you can actually fix.

Advanced RCA Techniques: Complex Problem Orchestration

This resource covers advanced methodologies and strategies for high-complexity root cause analyses, formal methods, and multi-methodology orchestration patterns.

Advanced Fault Tree Analysis

Quantitative FTA

Extends basic FTA with probability calculations to determine system reliability.

Key Metrics:

• Failure Rate (λ): Failures per unit time
• Mean Time Between Failures (MTBF): Average time until next failure
• Mean Time To Repair (MTTR): Average restoration time
• System Reliability (R): Probability system functions at time t

Boolean Algebra for FTA

For complex systems:

Minimal Cut Sets: Smallest combinations of basic events that cause top event

• Identify by removing non-critical events
• Used to calculate probability

Probability Calculation (for independent events):

• OR Gate: P(Top) = 1 - ∏(1 - P(input))
• AND Gate: P(Top) = ∏P(input)

When to Use Quantitative FTA

• Safety-critical systems (aviation, medical, nuclear)
• Regulatory compliance requirements (ISO, IEC, DO-178)
• High-cost failure analysis
• Need to calculate system reliability targets
• Risk assessment with probability thresholds

Example: Aircraft Engine Failure (Simplified)

Top Event: Engine Failure
└─ OR Gate
   ├─ Fuel System Failure
   │  └─ AND Gate
   │     ├─ Fuel tank rupture (λ=1e-6/hr)
   │     ├─ Fuel pump failure (λ=2e-6/hr)
   │     └─ Valve failure (λ=1e-6/hr)
   │
   ├─ Combustion Failure
   │  └─ OR Gate
   │     ├─ Ignition failure (λ=5e-7/hr)
   │     ├─ Combustor damage (λ=1e-7/hr)
   │
   └─ Mechanical Failure
      └─ OR Gate
         ├─ Bearing seizure (λ=3e-7/hr)
         ├─ Blade fracture (λ=2e-8/hr)

Calculated failure rates identify highest-risk paths for mitigation.

---

Extended Barrier Analysis

Barrier Framework Expansion

Move beyond simple prevention/detection/mitigation to comprehensive layers:

Layer 1: Risk Prevention

• Eliminate hazard at source
• Simplify design to reduce failure modes
• Add redundancy
• Example: Input validation to prevent invalid data

Layer 2: Access Control

• Prevent exposure to hazard
• Example: Role-based access controls
• Example: Physical barriers

Layer 3: Immediate Defense

• Detect and respond quickly
• Example: Automated alerts and failovers
• Example: Circuit breakers

Layer 4: Physical Safeguards

• Contain/mitigate damage if failure occurs
• Example: Bulkheads in ships
• Example: Firewall between systems

Layer 5: Emergency Response

• Procedures and tools to respond
• Example: Rollback procedures
• Example: Incident response runbooks

Layer 6: Recovery

• Restore system after incident
• Example: Backup/restore procedures
• Example: Data reconstruction

Layer 7: Learning

• Prevent recurrence through organizational learning
• Example: Post-mortems and documentation
• Example: Training updates

Barrier Effectiveness Assessment

For each barrier:

Barrier Assessment Template
├─ Exists? (Yes/No/Partial)
├─ Active? (Always/Conditional/Manual)
├─ Functioning? (Yes/No/Degraded)
├─ Maintenance? (Current/Overdue/Never)
├─ Independence? (Dependent on other barriers? Yes/No)
└─ Probability of Success (when needed)

Interaction Analysis

Common barrier interaction patterns:

• Common Cause Failure: Single event defeats multiple barriers
• Example: Power outage defeats both primary and backup systems
• Mitigation: Add independent power sources

• Functional Dependency: Barrier depends on another barrier
• Example: Alert system depends on monitoring which depends on network
• Mitigation: Identify and eliminate dependencies

• Cascading Failure: One barrier failure leads to another
• Example: Primary circuit breaker fails → load transferred → backup overloaded
• Mitigation: Design for graceful degradation

---

Multi-Methodology Orchestration

For complex problems, chain multiple methodologies in sequence:

Orchestration Pattern 1: Explore → Prioritize → Analyze

Best for: Complex systems with multiple possible causes

Step 1: Fishbone (Explore)

• Brainstorm across all categories
• Identify 5-7 possible root cause paths

Step 2: Pareto (Prioritize)

• Estimate frequency/impact of each path
• Identify vital few (top 2-3)

Step 3: 5 Whys (Analyze)

• Deep dive on each priority path
• Reach actionable root causes

Step 4: Barrier Analysis (Strengthen)

• Identify what defenses failed
• Add layers of protection

Orchestration Pattern 2: Categorize → Structure → Verify

Best for: Process failures and organizational issues

Step 1: Fishbone (Categorize)

• Organize issues by domain (people, process, tools, etc.)
• Create mental model of system

Step 2: Barrier Analysis (Structure)

• Map what should prevent each category of problem
• Identify missing or failed controls

Step 3: 5 Whys (Verify)

• Deep dive on barrier failures
• Understand why controls broke

Step 4: Solution Design

• Add/strengthen barriers based on findings

Orchestration Pattern 3: Prioritize → Deep Dive → Structure

Best for: High-cost, high-impact problems with resource constraints

Step 1: Pareto (Prioritize)

• Identify highest-impact contributing factors
• Focus resources on vital few

Step 2: Fault Tree (Structure)

• Map failure paths for high-impact issues
• Quantify probability if possible

Step 3: 5 Whys (Deep Dive)

• Understand why each path exists
• Identify controllable causes

Orchestration Pattern 4: Compare → Categorize → Analyze

Best for: Issues appearing in multiple contexts

Step 1: Compare Cases

• Collect similar incidents
• Identify patterns and differences

Step 2: Fishbone (Categorize)

• Separate domain-specific from common causes
• Identify what's universal

Step 3: 5 Whys (Analyze)

• Deep dive on common factors
• Find systemic root cause

---

Enterprise-Scale RCA Orchestration

For large organizations analyzing systemic issues:

Phase 1: Incident Triage

Severity Assessment:

• Critical: Causes complete service outage or safety risk
• High: Significant impact to users or systems
• Medium: Noticeable but non-critical impact
• Low: Minimal user impact, mostly visibility

Complexity Assessment:

• Simple: Clear cause apparent, single domain
• Complex: Multiple possible causes, cross-domain
• Systemic: Organizational or process-level issues

Methodology Selection Matrix:

Phase 2: Evidence Collection & Analysis

Time-Boxed Exploration:

• Critical: 4 hours max exploration before initiating fix
• High: 2 hours exploration
• Medium: 1 hour focused analysis
• Low: Opportunistic analysis during maintenance

Evidence Hierarchy: 1. Direct observation and logs 2. Metrics and monitoring data 3. Timestamps and change logs 4. Expert knowledge (interviews) 5. Assumptions and estimates

Phase 3: Root Cause Documentation

Structured Report Template:

INCIDENT REPORT
├─ Executive Summary (1-2 sentences)
├─ Timeline (what happened and when)
├─ Impact (users/systems affected, severity)
├─ Detection & Response (who detected, when fixed)
├─ Analysis Method Used
├─ Root Cause (include verification tests)
├─ Contributing Factors
├─ Immediate Fixes (what was done)
├─ Permanent Fixes (what prevents recurrence)
├─ Systemic Improvements (strengthen overall system)
├─ Monitoring Enhancements
├─ Learning & Follow-up
└─ Sign-offs & Approvals

Phase 4: Blameless Post-Mortem

Key Principles:

• Focus on systems and processes, not individuals
• Assume everyone did right thing with information they had
• Psychological safety essential for honest analysis
• Goal is learning, not punishment

Process:

1. Gather Participants: People involved in incident and relevant stakeholders 2. Timeline Construction: Build objective timeline of events 3. Diverge: Explore multiple perspectives freely 4. Converge: Synthesize findings into coherent narrative 5. Identify Patterns: What system design enabled this? 6. Action Items: Concrete improvements with ownership and due dates 7. Follow-up: Verify improvements actually reduce risk

Phase 5: Organizational Learning

Knowledge Transfer:

• Document root cause and solution in searchable system
• Share at team meetings and cross-team forums
• Update runbooks and procedures
• Include in training for onboarding

Pattern Tracking:

• Categorize root causes by type
• Identify recurring patterns
• Prioritize systemic fixes by frequency
• Measure impact of improvements

---

Verification Frameworks

Comprehensive Root Cause Testing

Before declaring root cause verified, pass all tests:

Test 1: Forward Test

• If root cause existed, would it create observed problem?
• Must answer: Yes, unambiguously
• Evidence required: Logic or simulation

Test 2: Backward Test

• If root cause eliminated, would problem not occur?
• Must answer: Yes, with high confidence
• Evidence: Fix implemented and verified working

Test 3: Scope Test

• Does this cause explain ALL instances of problem?
• Account for variations in manifestation
• If not universal, identify conditional factors

Test 4: Evidence Test

• What data supports this causal chain?
• Can provide:
• Logs showing cause precedes effect
• Metrics correlating with problem
• Code changes introducing condition
• Configuration changes enabling it

Test 5: Alternative Test

• Could any other cause produce same problem?
• Eliminate competing hypotheses
• Use process of elimination if needed

Test 6: Actionability Test

• Can this cause actually be addressed?
• Not too abstract, not too specific
• Leads to concrete preventive action
• Organization has authority to fix

Failing Test Interpretation

---

Special Situations

Recurring Problems (Repeated Root Causes)

When same problem happens multiple times after "fix":

Likely Causes: 1. Root cause wrong (restart analysis) 2. Fix incomplete or temporary 3. Root cause actually systemic (requires bigger fix) 4. Workaround implemented instead of real fix 5. New instance of problem (different root cause)

Prevention:

• Monitor after fix
• Verify fix reaches all affected systems
• Root cause analysis of each recurrence
• Implement systemic fixes vs. point fixes

Invisible Root Causes (No Clear Event)

When problem has no obvious triggering event:

Approach: 1. Look for cumulative conditions (load, time, count) 2. Check for race conditions or concurrency 3. Examine state transitions 4. Look for slow degradation 5. Review change history for recent modifications

Multiple Simultaneous Root Causes

When multiple independent causes converged:

Analysis:

• Identify each independent cause path
• Map interactions between them
• Determine if single-cause fixes adequate
• Consider systemic changes to reduce vulnerability

Example: System crash caused by: 1. Memory leak gradual accumulation (Software) 2. More traffic than usual (External) 3. Rebalancing scheduled today (Operations)

All three needed to occur for crash. Fix any one would have prevented.

---

Tool Support for Advanced RCA

Digital Fishbone Diagramming

Tools: Miro, Lucidchart, Visio, Draw.io

Advantages:

• Easy collaboration
• Can capture and organize brainstorm
• Export and share
• Template available

Fault Tree Analysis Software

Tools: FaultTree+, WindChill, ReliaSoft

Features:

• Formal notation
• Probability calculation
• Cut set analysis
• Reliability modeling

Evidence Documentation

Tools: Confluence, Notion, DocumentDB

Structure:

• Central repository
• Searchable by problem type
• Linked to solutions
• Accessible to all teams

Incident Management

Tools: Atlassian Jira, ServiceNow, PagerDuty

Integration:

• Incident creation to RCA workflow
• Automated evidence collection
• Action tracking
• Post-mortem documentation

---

Common Pitfalls in Advanced RCA

Pitfall 1: Analysis Paralysis

Spending too long in analysis phase, delaying fixes.

Prevention:

• Time-box exploration
• Accept good-enough analysis under time pressure
• Implement immediate fix while analyzing root cause
• Can improve later

Pitfall 2: Over-Complexity

Applying too formal methodology to simple problems.

Prevention:

• Match methodology to complexity
• Start simple, escalate if needed
• 5 Whys for simple, reserve advanced methods for truly complex

Pitfall 3: Missing Context

Failing to understand organizational/political context.

Prevention:

• Involve stakeholders early
• Understand incentives and constraints
• Consider implementation feasibility
• Get buy-in on root cause before fixes

Pitfall 4: No Follow-Through

Identifying root cause but failing to implement prevention.

Prevention:

• Assign owners to action items
• Track completion with deadlines
• Measure impact of fixes
• Follow up on systemic improvements

Pitfall 5: Knowledge Loss

RCA findings not shared or documented.

Prevention:

• Require documentation
• Conduct post-mortems with team
• Share in learning forums
• Update procedures and runbooks
• Include in training

---

Integration with Systems Thinking

Connect RCA to broader system understanding:

System Archetypes

Recognize recurring system patterns:

• Balancing Loop: System self-corrects (thermostat)
• Reinforcing Loop: Problem amplifies (success builds success or failure builds failure)
• Delayed Feedback: Effects not immediate (poor diet→weight gain delayed weeks)
• Policy Resistance: System resists change (fix creates new problem)
• Shifting Burden: Quick fix prevents real fix (treating symptoms)

Leverage Points (Donella Meadows)

For systemic change, target highest-leverage interventions:

1. Paradigm Shifts: Change how system thinks (most powerful) 2. Rules/Incentives: Formal rules and metrics 3. Information Flows: Who knows what 4. Power/Authority: Decision-making authority 5. Material/Energy: Physical changes 6. System Parameters: Adjusting boundaries

RCA should identify not just immediate cause but system structure enabling it.

---

Continuous Improvement Integration

Connect RCA findings to improvement initiatives:

Kaizen Cycles: 1. Identify problem through RCA 2. Plan improvement based on root cause 3. Implement small improvement 4. Check effectiveness 5. Act to standardize or escalate 6. Repeat with next problem

Lean Principles:

• Eliminate waste (unnecessary processes, steps)
• RCA identifies process waste
• Value stream mapping shows where improvements happen
• Continuous flow reduces failure opportunities

Six Sigma DMAIC:

• Define: Problem identified through symptoms
• Measure: RCA gathering phase
• Analyze: RCA analysis methodologies
• Improve: Based on root cause understanding
• Control: Prevention measures

---

Remember: Advanced methodologies provide rigor for high-stakes problems. Don't let formality prevent action—use as much rigor as situation demands, not more.

Common Root Causes by Domain

This reference guide catalogs frequently encountered root causes across different problem domains to help pattern recognition during root cause analysis.

Software Engineering

Code Defects

Logic Errors

• Off-by-one errors in loops or array indexing
• Incorrect conditional logic (AND vs OR, negation errors)
• Edge case handling missing (null, empty, boundary values)
• Race conditions in concurrent code
• Integer overflow/underflow
• Floating-point precision issues

Resource Management

• Memory leaks (unreleased resources)
• Connection pool exhaustion
• File descriptor leaks
• Thread pool exhaustion
• Deadlocks or livelocks

Data Handling

• Incorrect data validation
• SQL injection vulnerabilities
• XSS vulnerabilities
• Deserialization errors
• Character encoding issues (UTF-8, ASCII)
• Timezone handling errors

Configuration Issues

Environment Mismatches

• Development vs production configuration differences
• Missing environment variables
• Incorrect feature flags
• Wrong API endpoints or URLs
• Certificate/credential mismatches

Resource Limits

• Insufficient memory allocation
• CPU throttling or limits
• Disk space exhaustion
• Network bandwidth limits
• Connection timeout settings too low/high

Dependencies

Version Conflicts

• Incompatible library versions
• Breaking changes in dependencies
• Transitive dependency conflicts
• Missing dependencies

External Service Failures

• Third-party API changes
• Authentication/authorization issues
• Rate limiting
• Network connectivity problems
• DNS resolution failures

Deployment Issues

Incomplete Rollouts

• Database migrations not applied
• Configuration not updated
• Cache not invalidated
• CDN not purged
• Partial deployment (some instances old version)

Rollback Problems

• Database schema incompatible with old code
• Data migrations irreversible
• Configuration state not restored

Process Failures

Testing Gaps

• Missing test coverage for specific scenarios
• Tests not running in CI/CD
• Test environment differs from production
• Performance testing not conducted
• Integration testing insufficient

Review Process

• Code review checklist incomplete
• Security review skipped
• Performance impact not assessed
• Database changes not reviewed by DBA
• No architecture review for major changes

Communication Breakdowns

• Requirements misunderstood
• Assumptions not validated
• Changes not communicated to stakeholders
• Documentation outdated
• Tribal knowledge not shared

Hardware & Equipment

Mechanical Failures

Wear and Tear

• Normal component end-of-life
• Preventive maintenance schedule not followed
• Lubrication insufficient
• Corrosion from environmental exposure
• Fatigue from cyclic loading

Material Defects

• Manufacturing defect
• Substandard materials used
• Material degradation (UV, chemical, thermal)
• Contamination during production

Installation Errors

• Incorrect assembly
• Improper torque specifications
• Misalignment
• Missing or wrong components
• Inadequate sealing or fastening

Electrical/Electronic

Power Issues

• Voltage spikes or sags
• Insufficient power supply capacity
• Grounding problems
• EMI/RFI interference
• Power supply component failure

Thermal Problems

• Inadequate cooling/ventilation
• Thermal cycling stress
• Overheating due to dust/debris
• Ambient temperature outside specifications

Connection Failures

• Loose connections
• Corrosion on contacts
• Cable damage or degradation
• Connector wear
• Improper crimping or soldering

Operational Issues

Improper Use

• Operating outside design parameters
• Overloading
• Wrong operating mode
• Inadequate warm-up or cool-down
• Using wrong consumables (fuel, oil, etc.)

Maintenance Gaps

• Scheduled maintenance missed
• Wrong maintenance procedures
• Incorrect parts used in repairs
• Inadequate cleaning
• Calibration not performed

Process & Operations

Workflow Failures

Communication

• Information not reaching the right people
• Handoff documentation incomplete
• Language or terminology barriers
• Conflicting instructions
• Assumptions not verified

Training

• Insufficient initial training
• No refresher training
• Undocumented procedures
• Training materials outdated
• Skills gap not identified

Documentation

• Procedures not documented
• Documentation not accessible
• Steps unclear or ambiguous
• Documentation not updated
• Version control issues

Design Flaws

Process Design

• Single points of failure
• No error checking/validation steps
• Manual steps prone to error
• No feedback loops
• Conflicting requirements

Incentive Misalignment

• Metrics encourage wrong behaviors
• Pressure to cut corners
• Blame culture discourages reporting
• Speed prioritized over quality
• Individual vs team optimization

Resource Constraints

Insufficient Capacity

• Understaffing
• Equipment capacity inadequate
• Budget limitations
• Time pressure
• Competing priorities

Resource Quality

• Inadequate tools
• Substandard materials
• Insufficient expertise
• Outdated equipment
• Poor working conditions

Personal/Life Issues

Health & Energy

Physical Health

• Insufficient sleep
• Poor nutrition
• Lack of exercise
• Undiagnosed medical condition
• Chronic pain or discomfort
• Medication side effects

Mental Health

• Chronic stress
• Anxiety or depression
• Burnout
• Cognitive overload
• Emotional exhaustion

Habits & Behaviors

Time Management

• No prioritization system
• Overcommitment
• Procrastination patterns
• Interruption-driven work
• No boundaries between work/life

Decision Making

• Analysis paralysis
• Impulsive decisions
• Avoiding difficult choices
• Following others' priorities
• Not saying "no"

Environment & Context

Physical Environment

• Cluttered workspace
• Poor ergonomics
• Noise and distractions
• Inadequate tools/equipment
• Uncomfortable temperature/lighting

Social Environment

• Toxic relationships
• Lack of support system
• Negative peer influence
• Isolation
• Communication issues

Systems & Structure

• No routines or systems
• Conflicting commitments
• Lack of automation
• No tracking mechanisms
• Poor organization

Knowledge & Skills

Information Gaps

• Missing key information
• Don't know what you don't know
• Misinformation or outdated knowledge
• No access to expertise
• Learning resources unavailable

Skill Deficits

• Never learned the skill
• Skill degradation from disuse
• Technology/field evolved
• No opportunity to practice
• Insufficient feedback

Red Flag "Root Causes" (Go Deeper!)

If your analysis ends with any of these, you haven't reached the root cause yet:

❌ "Human error" → Why did the human make that error? What in the system allowed it?

❌ "User made a mistake" → Why was the mistake possible? What prevented detection?

❌ "Someone forgot" → Why wasn't there a reminder/checklist/automation?

❌ "Bad luck" → What made the system vulnerable to this "luck"?

❌ "It just broke" → What caused the failure? Why did it fail now?

❌ "Not enough time" → Why was time insufficient? What prioritization led to this?

❌ "Lack of communication" → What process or tool failure enabled miscommunication?

❌ "Budget constraints" → What drove budget allocation? Why was this not funded?

❌ "That's just how it is" → What systemic issue perpetuates this?

❌ "Third-party failure" → Why are we vulnerable to this failure? What's missing?

Pattern Recognition

Recurring Theme: Missing Safeguards

When you see:

• No validation
• No testing
• No monitoring
• No redundancy
• No fallback

Common root cause: Process doesn't include defensive measures

Recurring Theme: Knowledge Gaps

When you see:

• Didn't know
• Assumed
• Misunderstood
• Not documented
• First time

Common root cause: Learning/knowledge sharing system failure

Recurring Theme: Resource/Capacity

When you see:

• Exhausted
• Overwhelmed
• Insufficient
• Too slow
• Overloaded

Common root cause: Capacity planning or scaling issue

Recurring Theme: Change Management

When you see:

• After deployment
• After update
• After migration
• Recent change
• New version

Common root cause: Change control or testing process gap

Using This Reference

1. During Analysis: Review relevant categories for ideas 2. Pattern Matching: Compare your problem to common causes 3. Verification: Check if identified cause matches known patterns 4. Prevention: Use to identify similar vulnerabilities in your systems 5. Training: Share with teams for collective knowledge

Contributing to This List

As you encounter new root causes: 1. Document the pattern 2. Categorize appropriately 3. Note distinctive characteristics 4. Share with the team 5. Update this reference

This living document improves with use and sharing.

Root Cause Analysis Examples

This document contains complete RCA examples across different domains to illustrate the methodologies in practice.

Example 1: Software Bug (5 Whys)

Problem Statement

What: User authentication fails intermittently with "Invalid token" error Where: Production API, /auth/verify endpoint When: Started November 20, 2024, around 14:00 UTC Impact: ~15% of authentication attempts failing, affecting 500+ users per hour

Evidence Gathered

• Error logs show "JWT signature verification failed"
• Only affects users authenticated after 14:00 UTC
• Code deployment occurred at 13:45 UTC
• Environment variable JWT_SECRET unchanged
• Multiple API instances running (scaled to 5 instances at 13:50 UTC)

5 Whys Analysis

WHY #1: Why are JWT signature verifications failing?

• Answer: Different instances use different signing keys
• Evidence: Logged the key fingerprints—2 different values across instances
• Verified by: Connected to each instance and checked environment

WHY #2: Why do different instances have different signing keys?

• Answer: New instances generated a new random secret on startup
• Evidence: Code review shows JWT_SECRET = os.environ.get('JWT_SECRET', generate_random_secret())
• Verified by: Checked application code in auth/config.py:23

WHY #3: Why did new instances generate a random secret instead of using the configured one?

• Answer: Environment variable JWT_SECRET wasn't set in the container orchestration
• Evidence: Kubernetes deployment YAML missing JWT_SECRET in env section
• Verified by: Checked deployment.yaml in k8s/production/

WHY #4: Why wasn't JWT_SECRET in the deployment configuration?

• Answer: When we migrated from Docker Compose to Kubernetes, secrets weren't migrated
• Evidence: Docker Compose file has JWT_SECRET, Kubernetes manifests don't
• Verified by: Git history shows K8s migration commit didn't include secrets

WHY #5: Why weren't secrets migrated during the Kubernetes migration?

• Answer: Migration checklist didn't include environment variable audit
• Evidence: Migration guide document doesn't mention checking all env vars
• Verified by: Reviewed docs/kubernetes-migration.md

Root Cause

Migration checklist incomplete—doesn't include environment variable verification

Solutions

Immediate Fix:

• Manually added JWT_SECRET to Kubernetes secrets and deployment
• Restarted all pods to load the correct secret
• Verified all instances now use same key
• Status: ✅ Deployed and verified in 20 minutes

Root Cause Fix:

• Update Kubernetes migration checklist to include env var audit
• Status: ✅ Completed

Systemic Improvements: 1. Pre-deployment validation: Add startup check that fails if critical env vars missing

• Implementation: auth/config.py validates all required vars or exits
• Status: ✅ Implemented

2. Configuration parity tests: CI/CD checks Docker Compose and K8s configs match

• Implementation: Script compares env vars across config files
• Status: ✅ Added to CI pipeline

3. Monitoring: Alert if multiple instances have different key fingerprints

• Implementation: Health check endpoint reports key fingerprint, monitor checks consistency
• Status: 🔄 In progress

Lessons Learned

• Migration checklists are critical and must be comprehensive
• Fallback defaults (random secret generation) can mask configuration problems
• Multi-instance deployments need consistency validation
• Early detection (startup validation) prevents runtime failures

---

Example 2: Car Maintenance (5 Whys)

Problem Statement

What: Car engine overheating Where: 2018 Honda Civic, occurred during highway driving When: Started last week, happens after 30 minutes of driving Impact: Unsafe to drive, risk of engine damage

Evidence Gathered

• Temperature gauge rises to red zone after 30 min
• Coolant level appears normal when cold
• No visible leaks under the car
• Recent oil change 2 weeks ago
• No warning lights before overheating started

5 Whys Analysis

WHY #1: Why is the engine overheating?

• Answer: Coolant isn't circulating properly
• Evidence: Lower radiator hose stays cool while upper hose gets very hot
• Verified by: Touched both hoses (carefully) while engine running

WHY #2: Why isn't coolant circulating properly?

• Answer: Water pump isn't functioning
• Evidence: Removed belt and tested pump—no resistance when spinning pulley
• Verified by: Mechanic inspection confirmed internal pump failure

WHY #3: Why did the water pump fail?

• Answer: Pump bearings seized due to contaminated coolant
• Evidence: Drained coolant—looks rusty and contains debris
• Verified by: Coolant test shows high contamination, wrong type mixed in

WHY #4: Why was the coolant contaminated with the wrong type?

• Answer: During recent oil change, shop topped off coolant with wrong type
• Evidence: Receipt shows "fluids topped off," shop confirmed they used universal coolant mixed with OEM coolant
• Verified by: Called shop, they acknowledged mixing coolant types

WHY #5: Why did the shop use the wrong coolant type?

• Answer: Shop policy is to use universal coolant for all vehicles to save costs
• Evidence: Shop manager confirmed this is standard practice
• Verified by: Discussion with shop manager

Root Cause

Shop's cost-saving policy of using universal coolant instead of manufacturer-specified coolant, which led to chemical incompatibility and contamination

Solutions

Immediate Fix:

• Replaced water pump
• Flushed entire cooling system
• Filled with correct Honda OEM coolant
• Status: ✅ Completed, car running normally

Root Cause Fix:

• Switch to a shop that uses manufacturer-specified fluids
• Status: ✅ Found new shop with better practices

Preventive Measures: 1. Service verification: Always verify fluids used match manufacturer specs 2. Regular inspection: Check coolant condition during oil changes 3. Documentation: Keep records of all fluids used and brands 4. Awareness: Learned that "universal" doesn't mean "compatible with anything"

Lessons Learned

• Cheap service can be expensive in the long run
• Trust but verify—even reputable shops make poor choices
• Manufacturer specifications exist for good reasons
• Small maintenance decisions can have large consequences

---

Example 3: Production System Failure (Fishbone + 5 Whys)

Problem Statement

What: E-commerce site experiencing 60% increase in checkout failures Where: Production payment processing service When: Started Friday 3 PM, coinciding with Black Friday traffic spike Impact: ~$50K revenue loss per hour, customer complaints surging

Fishbone Analysis

METHODS/PROCESS:

• Retry logic insufficient
• Error handling incomplete
• Circuit breaker not implemented

MACHINES/EQUIPMENT:

• Payment gateway API
• Database connection pool
• Load balancer configuration

MATERIALS/INPUTS:

• Traffic volume (3x normal)
• Payment data validation
• Third-party API responses

MEASUREMENTS/MONITORING:

• Limited observability into payment flow
• No alerting on payment gateway errors
• Missing SLA monitoring

MANPOWER:

• On-call engineer unfamiliar with payment code
• Payment processor contact info outdated
• No runbook for payment failures

ENVIRONMENT:

• Black Friday traffic surge
• Third-party payment gateway under load
• Database experiencing high connection count

Top Candidates from Fishbone

1. Database connection pool exhausted (Equipment) 2. Payment gateway timeout during high load (Environment + Measurement) 3. No circuit breaker causing retry storms (Methods)

5 Whys Analysis (for top candidate)

WHY #1: Why are payment requests failing?

• Answer: Payment service timing out waiting for database responses
• Evidence: Logs show "connection timeout" errors, P95 latency 30s (normally <1s)
• Verified by: Database monitoring shows all connections in use

WHY #2: Why is the database connection pool exhausted?

• Answer: Payment queries are taking 30+ seconds instead of <1 second
• Evidence: Query logs show full table scans on transactions table
• Verified by: EXPLAIN shows query not using indexes

WHY #3: Why is the query doing full table scans?

• Answer: New query joins on customer_id but column isn't indexed
• Evidence: Schema shows index on transaction_id but not customer_id
• Verified by: Checked database schema and query plan

WHY #4: Why was the query added without an index?

• Answer: Recent feature added customer transaction history, didn't anticipate load
• Evidence: Git history shows feature added Nov 15, tested with <1000 records
• Verified by: Code review of commit ab3c92f, load tests done with small dataset

WHY #5: Why didn't load testing catch this?

• Answer: Load tests use synthetic data (1000 customers), production has 500K+ customers
• Evidence: Load test configuration shows DATA_SIZE=small
• Verified by: Reviewed CI/CD test configuration

Root Cause

Load testing performed with non-representative dataset size, failing to identify performance issues that manifest only at production scale

Solutions

Immediate Fix (5:30 PM):

• Added index on transactions.customer_id
• Increased connection pool from 20 to 50
• Restarted payment service instances
• Result: Failure rate dropped to <1%, latency normalized

Root Cause Fix: 1. Production-scale load testing:

• Update load tests to use production-representative data volumes
• Test with 1M+ customer records
• Status: ✅ Implemented

2. Query performance review:

• Add database query analysis to code review checklist
• Require EXPLAIN plans for new queries
• Status: ✅ Added to PR template

Systemic Improvements: 1. Observability:

• Add query performance monitoring
• Alert on slow queries (>1s)
• Dashboard showing connection pool utilization
• Status: ✅ Deployed

2. Resilience:

• Implement circuit breaker for database calls
• Add graceful degradation (show cached transaction history)
• Status: 🔄 In progress

3. Capacity planning:

• Regular review of query performance as data grows
• Automated index suggestions based on query patterns
• Status: 📋 Planned

Lessons Learned

• Load tests must use production-representative data volumes
• Performance characteristics change non-linearly with scale
• Circuit breakers are essential for preventing cascade failures
• Observability gaps delayed identification by 2+ hours
• Business events (Black Friday) are high-risk deployment periods

---

Example 4: Personal Productivity (Fishbone + 5 Whys)

Problem Statement

What: Consistently missing project deadlines despite working long hours Where: Work projects and personal goals When: Ongoing for 3+ months Impact: Stress, poor work quality, work-life imbalance, reputation damage

Fishbone Analysis

PHYSICAL:

• Sleeping 5-6 hours (need 7-8)
• Skipping meals
• No exercise routine
• Afternoon energy crashes

MENTAL:

• Difficulty focusing for >30 minutes
• Decision fatigue by mid-day
• Anxiety about deadlines
• Procrastination on difficult tasks

SOCIAL:

• Saying yes to every request
• Helping others at expense of own work
• Checking Slack constantly
• Unscheduled interruptions

ENVIRONMENTAL:

• Open office environment (noisy)
• Notifications always on
• Cluttered workspace
• No designated deep work time

RESOURCES:

• Calendar overbooked
• No time for planning
• Limited tools for task management
• Unclear priorities from management

HABITS:

• Multitasking constantly
• No morning routine
• Working on whatever seems urgent
• Not blocking time for focused work
• Checking email first thing

Top Candidates from Fishbone

1. No protected time for focused work (Resources + Environment) 2. Saying yes to every request (Social + Habits) 3. Poor sleep affecting focus and decision-making (Physical + Mental)

5 Whys Analysis (for top candidate)

WHY #1: Why am I missing deadlines?

• Answer: Not making meaningful progress on important projects
• Evidence: Time tracking shows <2 hours/day on priority projects
• Verified by: Reviewed last 2 weeks of time logs

WHY #2: Why am I only spending 2 hours/day on priority projects?

• Answer: Constantly interrupted by meetings, messages, and requests
• Evidence: Calendar shows 25+ meetings/week, Slack shows 100+ messages/day
• Verified by: Counted calendar events and Slack analytics

WHY #3: Why do I accept so many meetings and requests?

• Answer: Don't want to disappoint people or seem unhelpful
• Evidence: Journal entries show guilt when saying no, people-pleasing pattern
• Verified by: Self-reflection and discussion with therapist

WHY #4: Why is saying no associated with guilt?

• Answer: Belief that my value comes from being available and helpful
• Evidence: Identified pattern of self-worth tied to others' approval
• Verified by: Therapy sessions revealed this core belief

WHY #5: Why do I tie self-worth to being available for others?

• Answer: Learned pattern from childhood—approval came from being helpful
• Evidence: Family dynamics rewarded being the "helpful one"
• Verified by: Therapy exploration of family patterns

Root Cause

Core belief that self-worth depends on availability and helping others, leading to inability to protect time for own priorities

Solutions

Immediate Changes: 1. Calendar blocking:

• Blocked 9-11 AM daily for deep work (no meetings)
• Set Slack to DND during deep work blocks
• Status: ✅ Implemented, following for 3 weeks

2. Response templates:

• Created polite ways to say no or defer
• "I'm at capacity but can help next week"
• Status: ✅ Using regularly

Root Cause Work: 1. Therapy:

• Working on separating self-worth from productivity/helping
• Building healthier boundaries
• Status: 🔄 Ongoing

2. Values clarification:

• Identified core values beyond being helpful
• Using values to prioritize decisions
• Status: 🔄 In progress

Systemic Changes: 1. Weekly planning:

• Sunday evening: review priorities for week
• Identify top 3 must-do items
• Status: ✅ Habit established

2. Communication:

• Discussed boundaries with manager
• Got explicit permission to decline certain requests
• Status: ✅ Completed

3. Health foundations:

• Sleep: 7 hours minimum, tracked
• Exercise: 3x/week scheduled
• Status: ✅ Following for 1 month

Results After 6 Weeks

• Deep work time increased from 2 hrs/day to 5 hrs/day
• Met last 4 deadlines successfully
• Sleep improved to 7+ hours
• Reduced stress and anxiety
• Better work quality with focused time

Lessons Learned

• Personal issues often have deep psychological roots
• Systemic change requires both external (calendar) and internal (beliefs) work
• Saying no is a skill that can be learned
• Self-awareness is the first step to change
• Small habit changes compound over time

---

Key Takeaways Across Examples

Common Patterns

1. Cascade Effects:

• Small issues (wrong coolant, missing index) cascade to large problems
• Prevention at the earliest point is most effective

2. Process Gaps:

• Missing checklists, incomplete tests, inadequate review
• Process improvements prevent entire classes of problems

3. Measurement Matters:

• What gets measured gets managed
• Observability gaps delay problem identification

4. Root Causes Are Often Systemic:

• Not just technical failures but process, culture, beliefs
• Sustainable fixes require system-level changes

Methodology Selection

• Simple/linear problems: 5 Whys is fast and effective
• Complex/unclear problems: Fishbone first to explore, then 5 Whys to drill down
• Multiple concurrent issues: Pareto analysis to prioritize
• High-stakes problems: Multiple methods + formal documentation

Documentation Value

All examples show value of:

• Clear problem statements
• Evidence-based reasoning
• Verification of assumptions
• Systematic solution implementation
• Lessons learned capture

Using These Examples

1. Learning: Study the methodology application 2. Templates: Use as starting points for your own RCA 3. Pattern matching: Compare your problems to these examples 4. Teaching: Share with team to build RCA skills 5. Reference: Consult when conducting your own analyses

Each problem domain has unique characteristics, but the systematic approach applies universally.

RCA Methodologies: Comprehensive Guide

This resource covers the five core root cause analysis methodologies with step-by-step guidance, when to use each, and practical examples.

1. The 5 Whys Method

The 5 Whys is the foundational systematic root cause analysis method, developed by Sakichi Toyoda for the Toyota Production System.

Key Principles

• Go and See: Base answers on direct observation, not assumptions
• Data Validation: Support each cause-effect link with evidence
• People Over Blame: Focus on process failures, not individual fault
• Actionable Root Cause: Stop when you reach a cause you can fix

Process

1. State the problem clearly (What, Where, When, Impact) 2. Ask "Why did this happen?" and answer with facts 3. For each answer, ask "Why?" again 4. Continue until you reach the root cause (usually 3-7 iterations) 5. Verify the root cause leads back to the problem

When to Use

• Single clear failure chains
• Need quick analysis under time pressure
• Problem has obvious initial cause but needs deeper investigation
• Immediate action required

Example: Production API Failure

Problem: Production API returning 500 errors

Why #1: Why is the API returning 500 errors?
→ Database connection pool is exhausted

Why #2: Why is the connection pool exhausted?
→ Connections not being released after queries

Why #3: Why aren't connections being released?
→ Connection timeout set too high, queries hanging

Why #4: Why are queries hanging?
→ Recent code change added unindexed query on large table

Why #5: Why was unindexed query added?
→ Code review process doesn't include database performance checks

Root Cause: Missing database performance validation in code review
Solution: Add DB query analysis to CI/CD pipeline and review checklist

Best Practices

• Support each answer with evidence (logs, data, observations)
• Dig deeper when answers are vague or blame-oriented
• Consider multiple pathways if answer could lead to multiple causes
• Stop at actionable causes, not abstract ones

---

2. Fishbone Diagram (Ishikawa)

Created by Kaoru Ishikawa, this visual tool organizes potential causes into categories for comprehensive systematic analysis.

Structure

                    Methods          Machines
                      /                 /
                     /                 /
        ___________/_________________/__________ PROBLEM/EFFECT
                   \                 \
                    \                 \
                  Materials      Measurements
                  (+ Manpower, Mother Nature/Environment)

The 6 M's Categories

• Manpower/People: Skills, training, staffing, communication, procedures
• Methods/Process: Workflows, documentation, standardization, procedures
• Machines/Equipment: Tools, hardware, software systems, infrastructure
• Materials/Inputs: Components, supplies, dependencies, data quality
• Measurements/Monitoring: Metrics, monitoring, detection, feedback systems
• Mother Nature/Environment: External factors, conditions, seasonal effects

Process

1. Draw the fishbone with problem at the head 2. Add main category branches (6 M's or domain-specific) 3. Brainstorm causes for each category 4. Add sub-causes as smaller branches 5. Identify most likely root causes for investigation 6. Use 5 Whys to drill deeper into top 2-3 candidates

When to Use

• Complex problems with multiple potential causes
• Team brainstorming sessions
• Problems where category analysis adds clarity
• Need to explore all possible contributing factors
• Want systematic coverage across all dimensions

Example: E-Commerce Checkout Failures

METHODS/PROCESS:

• Retry logic insufficient
• Error handling incomplete
• Circuit breaker not implemented

MACHINES/EQUIPMENT:

• Payment gateway API
• Database connection pool
• Load balancer configuration

MATERIALS/INPUTS:

• Traffic volume (3x normal)
• Payment data validation
• Third-party API responses

MEASUREMENTS/MONITORING:

• Limited observability into payment flow
• No alerting on payment gateway errors
• Missing SLA monitoring

MANPOWER:

• On-call engineer unfamiliar with payment code
• Payment processor contact info outdated
• No runbook for payment failures

ENVIRONMENT:

• Black Friday traffic surge
• Third-party payment gateway under load
• Database experiencing high connection count

Advantages

• Visual representation helps team communication
• Ensures comprehensive exploration
• Good for cross-functional analysis
• Helps identify contributing vs. root causes
• Tracks multiple cause paths

Disadvantages

• Can be overwhelming for simple problems
• May identify causes that don't actually contribute
• Requires domain knowledge for meaningful categories

---

3. Pareto Analysis (80/20 Rule)

Prioritize problems by identifying which causes contribute most to the effect.

Principle

Typically, 20% of causes account for 80% of problems. Focus resources on the vital few.

Process

1. List all potential causes 2. Measure or estimate frequency/impact of each 3. Sort by impact (highest to lowest) 4. Calculate cumulative percentage 5. Identify the vital few causing 80% of impact 6. Focus investigation on high-impact causes

When to Use

• Multiple problems competing for attention
• Need to prioritize limited resources
• Data-driven decision required
• Want to maximize impact of corrective actions
• Resource constraints necessitate triage

Example: Software Bugs by Category

Category              Count    Cumulative %
---------------------------------------------
Missing input validation    45      45%
Insufficient error handling 25      70%
Resource leaks              15      85%
Concurrency issues          8       93%
Other                       7      100%

Focus on input validation and error handling (70% of issues)

Implementation Tips

• Use historical data when available
• Be objective about measurements
• Don't ignore the "other" 20% (might contain critical issues)
• Recalculate regularly as patterns change
• Combine with other methods for complete analysis

---

4. Fault Tree Analysis (FTA)

Top-down deductive approach using Boolean logic to analyze failure modes, primarily for safety-critical and high-stakes systems.

When to Use

• Safety-critical systems (aerospace, medical, automotive)
• High-cost failures requiring rigorous analysis
• Complex systems with multiple failure paths
• Regulatory or compliance requirements
• Need to identify all possible failure combinations

Basic Structure

                         ╔═══════════════╗
                         ║  TOP EVENT    ║
                         ║ (Undesired)   ║
                         ╚═══════════════╝
                                ▲
                         ┌──────┴──────┐
                         │             │
                     ╔═══╩═══╗     ╔═══╩═══╗
                     ║ OR Gate║     ║ AND Gate║
                     ╚═══╤═══╝     ╚═══╤═══╝
                    ┌────┴────┐       │    │
                    │         │       │    │
                 ┌──▼──┐  ┌───▼──┐ ┌─▼─┐ ┌─▼─┐
                 │Basic│  │Basic │ │ B │ │ B │
                 │Event│  │Event │ │ E │ │ E │
                 └─────┘  └──────┘ └───┘ └───┘

Logic Gates

• OR Gate: Any single input can cause the output
• AND Gate: All inputs must occur for the output
• XOR: Exactly one input must occur

Process

1. Define the undesired top event 2. Identify immediate causes (first level) 3. Decompose each cause recursively 4. Continue until reaching basic events 5. Apply Boolean logic to gates 6. Calculate probability if data available 7. Identify critical failure combinations

Advantages

• Rigorous, formal analysis
• Identifies all failure paths
• Quantifiable risk assessment
• Good for complex systems
• Regulatory compliant

Disadvantages

• Complex and time-consuming
• Requires specialized knowledge
• Can become very large
• Best with quantitative failure data
• Not suitable for quick analysis

---

5. Barrier Analysis

Examines what controls or barriers failed to prevent or detect the problem.

Key Questions

• What barriers existed to prevent this problem?
• Which barriers failed? Why?
• What barriers were missing?
• How did the problem bypass existing controls?

Categories of Barriers

Preventive Barriers:

• Controls designed to stop the problem before it starts
• Example: Input validation, access controls, safety interlocks

Detective Barriers:

• Controls designed to catch the problem early
• Example: Monitoring, alerts, status checks, reviews

Mitigating Barriers:

• Controls designed to reduce impact once problem occurs
• Example: Failovers, circuit breakers, rollback procedures

Process

1. Map all barriers that should have prevented the problem 2. For each barrier, determine:

• Did it exist? (Yes/No/Partial)
• Was it active? (Yes/No/Conditional)
• Did it function? (Yes/No/Partially)
• Why did it fail? (If applicable)

3. Identify gaps (missing barriers) 4. Analyze barrier interactions 5. Develop improvements

When to Use

• Process breakdowns
• Multiple failures cascading
• Systematic failures in systems
• Safety or security incidents
• Want to strengthen system resilience

Example: Payment Processing Failure

Preventive Barriers:

• ❌ Query performance monitoring (Missing)
• ✓ Code review (Existed but incomplete)
• ❌ Load testing with production volume (Failed)

Detective Barriers:

• ✓ Error logging (Worked but slow alerting)
• ❌ Performance alerts on latency (Missing)
• ✓ Health checks (Existed but inadequate)

Mitigating Barriers:

• ❌ Circuit breaker (Not implemented)
• ✓ Manual failover (Existed, took 45 minutes)

Root improvement focus: Implement all missing barriers, especially query monitoring and performance alerts.

---

6-Phase Structured RCA Process

Integrating all methodologies into a complete process:

Phase 1: Define the Problem

Create clear problem statement with What/Where/When/Impact:

• What: Observable symptom (not assumed cause)
• Where: Location, system, component
• When: Timeline, frequency, pattern
• Impact: Users affected, severity, business impact

Phase 2: Gather Evidence

Follow "Go and See" principle—collect facts, not opinions:

• Logs, metrics, monitoring data
• Timeline of events and changes
• Recent system/code/configuration changes
• Environmental factors (load, traffic, season)
• User reports and reproduction steps

Phase 3: Select & Apply Methodology

Choose based on problem complexity:

• Simple: 5 Whys alone
• Complex: Fishbone + 5 Whys
• Multiple: Pareto + 5 Whys
• Safety-critical: Fault Tree
• Process failures: Barrier Analysis

Phase 4: Verify Root Cause

Test conclusions:

• Forward Test: Would this root cause create the observed problem?
• Backward Test: Would eliminating this prevent the problem?
• Evidence Test: Data supporting causal chain?
• Scope Test: Explains all problem instances?

Phase 5: Develop Solutions

Address root cause with:

• Eliminate: Remove cause entirely
• Control: Add safeguards
• Detect: Improve monitoring
• Mitigate: Reduce impact

Phase 6: Implement & Verify

• Execute solution
• Monitor for side effects
• Measure effectiveness
• Document and share learning
• Follow up on recurrence

---

Red Flags: Signs You Haven't Found Root Cause

❌ "Human error" → Why did human make that error? What in system allowed it?

❌ "User made a mistake" → Why was mistake possible? What prevented detection?

❌ "Someone forgot" → Why no reminder/checklist/automation?

❌ "Bad luck" → What made system vulnerable?

❌ "It just broke" → What caused failure? Why now?

❌ "Not enough time" → Why insufficient? What prioritization led to this?

❌ "Lack of communication" → What process/tool failure enabled miscommunication?

❌ "Budget constraints" → Why not funded? What drove allocation?

❌ "That's just how it is" → What systemic issue perpetuates this?

Keep digging until you reach a controllable, actionable cause.

---

Methodology Selection Heuristic

---

Domain-Specific Guidance

Software Debugging Focus Areas

Key 5 Whys questions:

• Why did the code allow this condition?
• Why wasn't this caught in testing?
• Why didn't monitoring detect earlier?
• Why didn't code review catch it?
• Why doesn't our process prevent this class of error?

Fishbone categories for software:

• Code (logic errors, edge cases, concurrency)
• Configuration (environment, feature flags, secrets)
• Dependencies (versions, API changes, compatibility)
• Deployment (rollout completeness, migrations, caching)
• Testing (coverage, environment differences)
• Monitoring (blind spots, alert delays)

Hardware/Mechanical Focus Areas

Fishbone categories (adapted):

• Design (engineering, tolerances, specifications)
• Materials (quality, wear, fatigue, corrosion)
• Assembly (installation, alignment, torque)
• Operation (usage patterns, load, stress)
• Maintenance (schedules, procedures, parts)
• Environment (temperature, humidity, contamination)

Process/Organizational Focus Areas

Barrier Analysis focus:

• Communication channels and effectiveness
• Approval/review checkpoints and rigor
• Training programs and knowledge sharing
• Documentation accuracy and accessibility
• Incentive alignment with desired outcomes
• Escalation procedures and timeliness

---

Templates

5 Whys Template

PROBLEM STATEMENT:
[What, Where, When, Impact]

WHY #1: Why did [problem] occur?
Answer: [Evidence-based]
Evidence: [Data/logs/observations]

WHY #2: Why did [answer #1] occur?
Answer: [Evidence-based]
Evidence: [Data/logs/observations]

WHY #3: Why did [answer #2] occur?
Answer: [Evidence-based]
Evidence: [Data/logs/observations]

WHY #4: Why did [answer #3] occur?
Answer: [Evidence-based]
Evidence: [Data/logs/observations]

WHY #5: Why did [answer #4] occur?
Answer: [Evidence-based]
Evidence: [Data/logs/observations]

ROOT CAUSE:
[Deepest actionable cause]

VERIFICATION:
- Forward test: [Would this cause the problem?]
- Backward test: [Would fixing this prevent it?]
- Evidence: [What supports this?]

SOLUTION:
- Immediate: [Stop current problem]
- Root cause: [Prevent recurrence]
- Systemic: [Strengthen system]
- Monitoring: [Detect if recurs]

Fishbone Template

PROBLEM/EFFECT: [The problem]

MANPOWER/PEOPLE:
- [Cause]
  - [Sub-cause]

METHODS/PROCESS:
- [Cause]
  - [Sub-cause]

MACHINES/EQUIPMENT:
- [Cause]
  - [Sub-cause]

MATERIALS/INPUTS:
- [Cause]
  - [Sub-cause]

MEASUREMENTS/MONITORING:
- [Cause]
  - [Sub-cause]

ENVIRONMENT:
- [Cause]
  - [Sub-cause]

TOP CANDIDATES:
1. [Most likely based on analysis]
2. [Second most likely]
3. [Third most likely]

NEXT STEPS:
[Apply 5 Whys to top candidates]

---

Best Practices

Do's ✓

• Base analysis on facts and evidence, not assumptions
• Use "Go and See"—observe directly
• Focus on process/system failures, not blame
• Document each step and reasoning
• Verify root causes before implementing solutions
• Involve people with direct problem knowledge
• Consider multiple perspectives and hypotheses
• Stop at actionable root cause
• Share learning to prevent similar issues
• Combine methodologies for complex problems

Don'ts ✗

• Stop at symptoms or proximate causes
• Accept "human error" as root cause
• Skip evidence gathering
• Blame individuals—fix systems
• Implement solutions without verification
• Rush to solutions before understanding
• Ignore contradictory evidence
• Forget follow-up on effectiveness
• Use only one methodology for complex problems
• Make root cause analysis about punishment

---

Continuous Improvement

After solving immediate problem:

1. Pattern Recognition: Is this part of larger pattern? 2. Process Improvement: How prevent this class of problems? 3. Knowledge Sharing: Who else should learn from this? 4. Monitoring Enhancement: Can detect earlier next time? 5. Documentation: Capture solution for future reference?

This embodies Toyota's Kaizen philosophy of continuous improvement.

---

Remember: The goal is disciplined systematic investigation until reaching a root cause you can actually fix. Sometimes that's three whys, sometimes seven. The number matters less than the rigor of the process.

Software Debugging Root Cause Analysis Template

Problem Statement

Date/Time Discovered: [YYYY-MM-DD HH:MM UTC] Reporter: [Name/Team] Severity: [Critical/High/Medium/Low] Environment: [Production/Staging/Development]

Description: [What happened? Be specific about the symptom, not the assumed cause]

Impact:

• Users affected: [Number/percentage/specific users]
• Systems affected: [Which services/components]
• Business impact: [Revenue, reputation, functionality]

First Observed: [When did this start? Any triggering events?]

Frequency: [Always/Intermittent/Under specific conditions]

Evidence Gathered

Error Messages/Stack Traces

[Paste relevant error messages and stack traces]

Logs

[Include relevant log entries with timestamps]

Metrics/Monitoring Data

• [CPU usage, memory, latency, error rates, etc.]
• [Include graphs or data points]

Recent Changes

• [ ] Code deployments: [List with timestamps]
• [ ] Configuration changes: [List with timestamps]
• [ ] Infrastructure changes: [List with timestamps]
• [ ] Dependency updates: [List with timestamps]

Reproduction Steps

1. [Step 1] 2. [Step 2] 3. [Step 3] Expected: [What should happen] Actual: [What actually happens]

System State

• Code version: [commit hash/version]
• Dependencies: [relevant library versions]
• Environment config: [relevant settings]
• Resource usage: [CPU/memory/disk/network]

5 Whys Analysis

WHY #1: Why did [the problem] occur?

• Answer:
• Evidence:
• Verified by: [How did you confirm this?]

WHY #2: Why did [answer to WHY #1] happen?

• Answer:
• Evidence:
• Verified by:

WHY #3: Why did [answer to WHY #2] happen?

• Answer:
• Evidence:
• Verified by:

WHY #4: Why did [answer to WHY #3] happen?

• Answer:
• Evidence:
• Verified by:

WHY #5: Why did [answer to WHY #4] happen?

• Answer:
• Evidence:
• Verified by:

[Continue if needed...]

Root Cause Identified

Root Cause: [The deepest actionable cause you identified]

Category: [Code Defect/Configuration/Dependencies/Resources/Data/Deployment/Process]

Verification

Forward Test: [If this root cause exists, would it create the observed problem? Yes/No + explanation]

Backward Test: [If we fix this root cause, will the problem be prevented? Yes/No + explanation]

Evidence Support: [What data/logs/tests support this conclusion?]

Completeness: [Does this explain all instances of the problem? Any outliers?]

Contributing Factors

[Other factors that made the problem possible or worse, even if not the root cause]

1. [Factor 1] 2. [Factor 2]

Solutions

Immediate Fix (Stop the Bleeding)

Action: [What to do right now] Timeline: [How quickly] Risks: [Any risks of the quick fix] Status: [ ] Implemented [ ] Verified [ ] Rolled back

Root Cause Fix (Prevent Recurrence)

Action: [Permanent fix for the root cause] Implementation plan: [Steps to implement] Timeline: [Estimated time] Testing plan: [How to verify the fix] Status: [ ] Designed [ ] In Progress [ ] Testing [ ] Deployed

Systemic Improvements (Strengthen the System)

[Improvements to prevent similar issues in the future]

1. [Improvement area]: [Description]

• Action: [Specific action]
• Owner: [Who]
• Timeline: [When]

2. [Improvement area]: [Description]

• Action: [Specific action]
• Owner: [Who]
• Timeline: [When]

Detection/Monitoring Enhancements

[How to catch this earlier or prevent it from reaching production]

• [ ] Add alert for [specific condition]
• [ ] Add test for [scenario]
• [ ] Add monitoring for [metric]
• [ ] Update dashboard to show [indicator]

Prevention Checklist

What could have prevented this?

• [ ] Better testing: [What type? Unit/Integration/E2E/Performance]
• [ ] Code review focus: [What should reviewers look for?]
• [ ] Monitoring: [What should be monitored?]
• [ ] Documentation: [What should be documented?]
• [ ] Process change: [What process should change?]
• [ ] Tooling: [What tools would help?]
• [ ] Training: [What knowledge gap exists?]

Lessons Learned

What Went Well

• [Positive aspects of detection, response, or resolution]

What Could Be Improved

• [Areas for improvement in process, tools, or skills]

Knowledge Sharing

• [ ] Document in knowledge base
• [ ] Share in team meeting
• [ ] Update runbooks/playbooks
• [ ] Create/update tests
• [ ] Update coding guidelines

Timeline

Follow-Up

• [ ] Monitor for recurrence (Duration: [timeframe])
• [ ] Verify metrics returned to normal
• [ ] Complete systemic improvements
• [ ] Share lessons learned
• [ ] Update documentation
• [ ] Schedule review of prevention measures

---

Prepared by: [Name] Reviewed by: [Name(s)] Date: [YYYY-MM-DD]