Incident Commander

Incident Commander

The agent classifies incident severity, reconstructs timelines from heterogeneous event sources, and generates structured post-incident reviews with root cause analysis and action items.

---

Quick Start

# Classify an incident (JSON or stdin)
echo '{"description": "Database connections timing out", "affected_users": "80%", "business_impact": "high"}' \
  | python scripts/incident_classifier.py --format text

# Multi-dimensional severity scoring
python scripts/severity_classifier.py incident.json --format markdown

# Reconstruct timeline with phase detection and gap analysis
python scripts/timeline_reconstructor.py --input events.json --detect-phases --gap-analysis --format markdown

# Build structured timeline with MTTD/MTTR metrics
python scripts/incident_timeline_builder.py incident_data.json --format markdown

# Generate Post-Incident Review
python scripts/pir_generator.py --incident incident.json --rca-method fishbone --action-items --format markdown

# Generate postmortem with benchmark comparisons
python scripts/postmortem_generator.py incident_data.json --format markdown

Tools Overview

---

Workflow 1: Incident Response (Detection to Resolution)

Step 1 -- Classify severity.

python scripts/severity_classifier.py incident.json --format json

The agent scores across five dimensions: revenue impact (25%), user scope (25%), data/security risk (20%), service criticality (15%), blast radius (15%).

Validation checkpoint: Severity classification includes confidence score and recommended escalation path.

Step 2 -- Establish command.

The Incident Commander:

• Assigns within 5 min (SEV-1) or 30 min (SEV-2)
• Creates war room and incident tracking ticket
• Sends initial notification using generated template
• Coordinates between technical teams and stakeholders
• Shields responders from external distractions

Step 3 -- Investigate and mitigate.

The agent generates targeted investigation commands based on the affected service:

kubectl get pods -n production -l app=<service>
kubectl logs -l app=<service> --tail=100
helm history <service> -n production

Decision framework for SEV-1/SEV-2:

• Bias toward action over analysis
• Prefer rollbacks to risky fixes under pressure
• Document every decision for later review
• Consult SMEs but do not block on them

Step 4 -- Communicate.

The agent generates three communication templates per severity: 1. Internal notification -- technical details, response team, war room link 2. Executive summary -- business impact, ETA, leadership actions required 3. Customer communication -- impact scope, what is being done, next update time

Validation checkpoint: All stakeholders notified within committed timeframes.

---

Workflow 2: Post-Incident Review

Step 1 -- Reconstruct the timeline.

python scripts/timeline_reconstructor.py --input events.json --detect-phases --gap-analysis --format markdown

The agent accepts events from logs, alerts, Slack messages, and deployment systems. Each event needs a timestamp and description. Optional fields: source, type, actor, severity.

Supported phases: detection, declaration, escalation, investigation, mitigation, communication, resolution.

Step 2 -- Perform root cause analysis.

python scripts/pir_generator.py --incident incident.json --timeline timeline.json --rca-method five_whys --action-items

Available RCA methods:

Step 3 -- Generate action items.

The agent categorizes action items as: immediate_fix, process_improvement, monitoring_alerting, documentation, training, architectural, tooling.

Each action item includes: title, owner, priority, deadline, success criteria, and dependencies.

Step 4 -- Validate postmortem quality.

python scripts/postmortem_generator.py incident_data.json --format json

The agent checks:

• Every contributing factor has at least one action item (coverage gap detection)
• Action items have quality scores (0-100) based on specificity
• MTTD/MTTR benchmarked against industry standards
• Missing actions suggested for uncovered themes

Validation checkpoint: Zero coverage gaps. All P0 action items have owners and deadlines within 48 hours.

---

Workflow 3: Escalation Management

Technical escalation path:

Business escalation:

---

Anti-Patterns

1. Individual blame in postmortems -- focus on system failures. "Why did the process allow this?" not "Why did Alice do this?" 2. Skipping PIR for SEV-2 -- every SEV-1 and SEV-2 gets a postmortem within 3 business days. 3. Action items without owners -- every item needs a specific person and deadline. 4. Deploying fixes under pressure without validation -- validate fixes before declaring resolution; plan for secondary failures. 5. Communication gaps -- provide updates even when there is no new information.

---

Troubleshooting

---

References

---

Integration Points

---

Last Updated: April 2026 Version: 1.1.0

Incident Report: [INC-YYYY-NNNN] [Title]

Severity: SEV[1-4] Status: [Active | Mitigated | Resolved] Incident Commander: [Name] Date: [YYYY-MM-DD]

---

Executive Summary

[2-3 sentence summary of the incident: what happened, impact scope, resolution status. Written for executive audience — no jargon, focus on business impact.]

---

Impact Statement

Customer-Facing Impact

[Describe what customers experienced: error messages, degraded functionality, complete outage. Be specific about which user journeys were affected.]

---

Timeline

Key Decision Points

1. [HH:MM] [Decision] — [Rationale and outcome] 2. [HH:MM] [Decision] — [Rationale and outcome]

Timeline Gaps

[Note any periods >15 minutes without logged events. These represent potential blind spots in the response.]

---

Root Cause Analysis

Root Cause

[Clear, specific statement of the root cause. Not "human error" — describe the systemic failure.]

Contributing Factors

1. [Factor Category: Process/Tooling/Human/Environment] — [Description] 2. [Factor Category] — [Description] 3. [Factor Category] — [Description]

5-Whys Analysis

Why did the service degrade? → [Answer]

Why did [answer above] happen? → [Answer]

Why did [answer above] happen? → [Answer]

Why did [answer above] happen? → [Answer]

Why did [answer above] happen? → [Root systemic cause]

---

Response Metrics

---

Action Items

Action Item Types

• Detection: Improve ability to detect this class of issue faster
• Prevention: Prevent this class of issue from occurring
• Mitigation: Reduce impact when this class of issue occurs
• Process: Improve response process and coordination

---

Lessons Learned

What Went Well

• [Specific positive outcome from the response]
• [Specific positive outcome]

What Didn't Go Well

• [Specific area for improvement]
• [Specific area for improvement]

Where We Got Lucky

• [Things that could have made this worse but didn't]

---

Communication Log

---

Participants

---

Appendix

Related Incidents

• [INC-YYYY-NNNN] — [Brief description of related incident]

Reference Links

• [Link to monitoring dashboard]
• [Link to deployment logs]
• [Link to incident channel archive]

---

This report follows the blameless postmortem principle. The goal is systemic improvement, not individual accountability. All contributing factors should trace to process, tooling, or environmental gaps that can be addressed with concrete action items.

Runbook: [Service/Component Name]

Owner: [Team Name] Last Updated: [YYYY-MM-DD] Reviewed By: [Name] Review Cadence: Quarterly

---

Service Overview

Architecture Summary

[2-3 sentence description of the service architecture. Include key components, data stores, and external dependencies.]

---

Alert Response Decision Tree

High Error Rate (>5%)

Error Rate Alert Fired
├── Check: Is this a deployment-related issue?
│   ├── YES → Go to "Recent Deployment Rollback" section
│   └── NO → Continue
├── Check: Is a downstream dependency failing?
│   ├── YES → Go to "Dependency Failure" section
│   └── NO → Continue
├── Check: Is there unusual traffic volume?
│   ├── YES → Go to "Traffic Spike" section
│   └── NO → Continue
└── Escalate: Engage on-call secondary + service owner

High Latency (p99 > [threshold]ms)

Latency Alert Fired
├── Check: Database query latency elevated?
│   ├── YES → Go to "Database Performance" section
│   └── NO → Continue
├── Check: Connection pool utilization >80%?
│   ├── YES → Go to "Connection Pool Exhaustion" section
│   └── NO → Continue
├── Check: Memory/CPU pressure on service instances?
│   ├── YES → Go to "Resource Exhaustion" section
│   └── NO → Continue
└── Escalate: Engage on-call secondary + service owner

Service Unavailable (Health Check Failing)

Health Check Alert Fired
├── Check: Are all instances down?
│   ├── YES → Go to "Complete Outage" section
│   └── NO → Continue
├── Check: Is only one AZ affected?
│   ├── YES → Go to "AZ Failure" section
│   └── NO → Continue
├── Check: Can instances be restarted?
│   ├── YES → Go to "Instance Restart" section
│   └── NO → Continue
└── Escalate: Declare incident, engage IC

---

Common Scenarios

Recent Deployment Rollback

Symptoms: Error rate spike or latency increase within 60 minutes of a deployment.

Diagnosis: 1. Check deployment history: kubectl rollout history deployment/[service-name] 2. Compare error rate timing with deployment timestamp 3. Review deployment diff for risky changes

Mitigation: 1. Initiate rollback: kubectl rollout undo deployment/[service-name] 2. Verify rollback: kubectl rollout status deployment/[service-name] 3. Confirm error rate returns to baseline (allow 5 minutes) 4. If rollback fails: escalate immediately

Communication: If customer-impacting, update status page within 5 minutes of confirming impact.

---

Database Performance

Symptoms: Elevated query latency, connection pool saturation, timeout errors.

Diagnosis: 1. Check active queries: SELECT * FROM pg_stat_activity WHERE state = 'active'; 2. Check for long-running queries: SELECT pid, now() - pg_stat_activity.query_start AS duration, query FROM pg_stat_activity WHERE state != 'idle' ORDER BY duration DESC; 3. Check connection count: SELECT count(*) FROM pg_stat_activity; 4. Check table bloat and vacuum status

Mitigation: 1. Kill long-running queries if identified: SELECT pg_terminate_backend([pid]); 2. If connection pool exhausted: increase pool size via config (requires restart) 3. If read replica available: redirect read traffic 4. If write-heavy: identify and defer non-critical writes

Escalation Trigger: If query latency >10s for >5 minutes, escalate to DBA on-call.

---

Connection Pool Exhaustion

Symptoms: Connection timeout errors, pool utilization >90%, requests queuing.

Diagnosis: 1. Check pool metrics: current size, active connections, waiting requests 2. Check for connection leaks: connections held >30s without activity 3. Review recent config changes or deployments

Mitigation: 1. Increase pool size (if infrastructure allows): update config, rolling restart 2. Kill idle connections exceeding timeout 3. If caused by leak: identify and restart affected instances 4. Enable connection pool auto-scaling if available

Prevention: Pool utilization alerting at 70% (warning) and 85% (critical).

---

Dependency Failure

Symptoms: Errors correlated with downstream service failures, circuit breakers tripping.

Diagnosis: 1. Check dependency status dashboards 2. Verify circuit breaker state: open/half-open/closed 3. Check for correlation with dependency deployments or incidents 4. Test dependency health endpoints directly

Mitigation: 1. If circuit breaker not tripping: verify timeout/threshold configuration 2. Enable graceful degradation (serve cached/default responses) 3. If critical path: engage dependency team via incident process 4. If non-critical path: disable feature flag for affected functionality

Communication: Coordinate with dependency team IC if both services have active incidents.

---

Traffic Spike

Symptoms: Sudden traffic increase beyond normal patterns, resource saturation.

Diagnosis: 1. Check traffic source: organic growth vs. bot traffic vs. DDoS 2. Review rate limiting effectiveness 3. Check auto-scaling status and capacity

Mitigation: 1. If bot/DDoS: enable rate limiting, engage security team 2. If organic: trigger manual scale-up, increase auto-scaling limits 3. Enable request queuing or load shedding if at capacity 4. Consider feature flag toggles to reduce per-request cost

---

Complete Outage

Symptoms: All instances unreachable, health checks failing across AZs.

Diagnosis: 1. Check infrastructure status (AWS/GCP status page) 2. Verify network connectivity and DNS resolution 3. Check for infrastructure-level incidents (region outage) 4. Review recent infrastructure changes (Terraform, network config)

Mitigation: 1. If infra provider issue: activate disaster recovery plan 2. If DNS issue: update DNS records, reduce TTL 3. If deployment corruption: redeploy last known good version 4. If data corruption: engage data recovery procedures

Escalation: Immediately declare SEV1 incident. Engage infrastructure team and management.

---

Instance Restart

Symptoms: Individual instances unhealthy, OOM kills, process crashes.

Diagnosis: 1. Check instance logs for crash reason 2. Review memory/CPU usage patterns before crash 3. Check for memory leaks or resource exhaustion 4. Verify configuration consistency across instances

Mitigation: 1. Restart unhealthy instances: kubectl delete pod [pod-name] 2. If recurring: cordon node and migrate workloads 3. If memory leak: schedule immediate patch with increased memory limit 4. Monitor for recurrence after restart

---

AZ Failure

Symptoms: All instances in one availability zone failing, others healthy.

Diagnosis: 1. Confirm AZ-specific failure vs. instance-specific issues 2. Check cloud provider AZ status 3. Verify load balancer is routing around failed AZ

Mitigation: 1. Ensure load balancer marks AZ instances as unhealthy 2. Scale up remaining AZs to handle redirected traffic 3. If auto-scaling: verify it's responding to increased load 4. Monitor remaining AZs for cascade effects

---

Key Metrics & Dashboards

---

Escalation Contacts

---

Maintenance Procedures

Planned Maintenance Checklist

• [ ] Maintenance window scheduled and communicated (72 hours advance for Tier 1)
• [ ] Status page updated with planned maintenance notice
• [ ] Rollback plan documented and tested
• [ ] On-call notified of maintenance window
• [ ] Customer notification sent (if SLA-impacting)
• [ ] Post-maintenance verification plan ready

Health Verification After Changes

1. Check all health endpoints return 200 2. Verify error rate returns to baseline within 5 minutes 3. Confirm latency within normal range 4. Run synthetic transaction test 5. Monitor for 15 minutes before declaring success

---

Revision History

---

This runbook should be reviewed quarterly and updated after every incident that reveals missing procedures. The on-call engineer should be able to follow this document without prior context about the service. If any section requires tribal knowledge to execute, it needs to be expanded.

Post-Incident Review: Payment API Database Connection Pool Exhaustion

Executive Summary

On March 15, 2024, we experienced a sev2 incident affecting ['payment-api', 'checkout-service', 'subscription-billing']. The incident lasted 1h 5m and had the following impact: 80% of users unable to complete payments or checkout. Approximately 2,400 failed payment attempts during the incident. Users experienced immediate 500 errors when attempting to pay. The incident has been resolved and we have identified specific actions to prevent recurrence.

Incident Overview

• Incident ID: INC-2024-0315-001
• Date & Time: 2024-03-15 14:30:00 UTC
• Duration: 1h 5m
• Severity: SEV2
• Status: Resolved
• Incident Commander: Mike Rodriguez
• Responders: Sarah Chen - On-call Engineer, Primary Responder, Tom Wilson - Database Team Lead, Lisa Park - Database Engineer, Mike Rodriguez - Incident Commander, David Kumar - DevOps Engineer

Customer Impact

80% of users unable to complete payments or checkout. Approximately 2,400 failed payment attempts during the incident. Users experienced immediate 500 errors when attempting to pay.

Business Impact

Estimated revenue loss of $45,000 during outage period. No SLA breaches as resolution was within 2-hour window. 12 customer escalations through support channels.

Timeline

No detailed timeline available.

Root Cause Analysis

Analysis Method: 5 Whys Analysis

Why Analysis

Why 1: Why did Database connection pool exhaustion caused widespread 500 errors in payment processing API, preventing users from completing purchases. Root cause was an inefficient database query introduced in deployment v2.3.1.? Answer: New deployment introduced a regression

Why 2: Why wasn't this detected earlier? Answer: Code review process missed the issue

Why 3: Why didn't existing safeguards prevent this? Answer: Testing environment didn't match production

Why 4: Why wasn't there a backup mechanism? Answer: Further investigation needed

Why 5: Why wasn't this scenario anticipated? Answer: Further investigation needed

What Went Well

• The incident was successfully resolved
• Incident command was established
• Multiple team members collaborated on resolution

What Didn't Go Well

• Analysis in progress

Lessons Learned

Lessons learned to be documented following detailed analysis.

Action Items

Action items to be defined.

Follow-up and Prevention

Prevention Measures

Based on the root cause analysis, the following preventive measures have been identified:

• Implement comprehensive testing for similar scenarios
• Improve monitoring and alerting coverage
• Enhance error handling and resilience patterns

Follow-up Schedule

• 1 week: Review action item progress
• 1 month: Evaluate effectiveness of implemented changes
• 3 months: Conduct follow-up assessment and update preventive measures

Appendix

Additional Information

• Incident ID: INC-2024-0315-001
• Severity Classification: sev2
• Affected Services: payment-api, checkout-service, subscription-billing

References

• Incident tracking ticket: [Link TBD]
• Monitoring dashboards: [Link TBD]
• Communication thread: [Link TBD]

--- Generated on 2026-02-16 by PIR Generator

Incident Commander Skill

A comprehensive incident response framework providing structured tools for managing technology incidents from detection through resolution and post-incident review.

Overview

This skill implements battle-tested practices from SRE and DevOps teams at scale, providing:

• Automated Severity Classification - Intelligent incident triage
• Timeline Reconstruction - Transform scattered events into coherent narratives
• Post-Incident Review Generation - Structured PIRs with RCA frameworks
• Communication Templates - Pre-built stakeholder communication
• Comprehensive Documentation - Reference guides for incident response

Quick Start

Classify an Incident

# From JSON file
python scripts/incident_classifier.py --input incident.json --format text

# From stdin text
echo "Database is down affecting all users" | python scripts/incident_classifier.py --format text

# Interactive mode
python scripts/incident_classifier.py --interactive

Reconstruct Timeline

# Analyze event timeline
python scripts/timeline_reconstructor.py --input events.json --format text

# With gap analysis
python scripts/timeline_reconstructor.py --input events.json --gap-analysis --format markdown

Generate PIR Document

# Basic PIR
python scripts/pir_generator.py --incident incident.json --format markdown

# Comprehensive PIR with timeline
python scripts/pir_generator.py --incident incident.json --timeline timeline.json --rca-method fishbone

Scripts

incident_classifier.py

Purpose: Analyzes incident descriptions and provides severity classification, team recommendations, and response templates.

Input: JSON object with incident details or plain text description Output: JSON + human-readable classification report

Example Input:

{
  "description": "Database connection timeouts causing 500 errors",
  "service": "payment-api",
  "affected_users": "80%",
  "business_impact": "high"
}

Key Features:

• SEV1-4 severity classification
• Recommended response teams
• Initial action prioritization
• Communication templates
• Response timelines

timeline_reconstructor.py

Purpose: Reconstructs incident timelines from timestamped events, identifies phases, and performs gap analysis.

Input: JSON array of timestamped events Output: Formatted timeline with phase analysis and metrics

Example Input:

[
  {
    "timestamp": "2024-01-01T12:00:00Z",
    "source": "monitoring",
    "message": "High error rate detected",
    "severity": "critical",
    "actor": "system"
  }
]

Key Features:

• Phase detection (detection → triage → mitigation → resolution)
• Duration analysis
• Gap identification
• Communication effectiveness analysis
• Response metrics

pir_generator.py

Purpose: Generates comprehensive Post-Incident Review documents with multiple RCA frameworks.

Input: Incident data JSON, optional timeline data Output: Structured PIR document with RCA analysis

Key Features:

• Multiple RCA methods (5 Whys, Fishbone, Timeline, Bow Tie)
• Automated action item generation
• Lessons learned categorization
• Follow-up planning
• Completeness assessment

Sample Data

The assets/ directory contains sample data files for testing:

• sample_incident_classification.json - Database connection pool exhaustion incident
• sample_timeline_events.json - Complete timeline with 21 events across phases
• sample_incident_pir_data.json - Comprehensive incident data for PIR generation
• simple_incident.json - Minimal incident for basic testing
• simple_timeline_events.json - Simple 4-event timeline

Expected Outputs

The expected_outputs/ directory contains reference outputs showing what each script produces:

• incident_classification_text_output.txt - Detailed classification report
• timeline_reconstruction_text_output.txt - Complete timeline analysis
• pir_markdown_output.md - Full PIR document
• simple_incident_classification.txt - Basic classification example

Reference Documentation

references/incident_severity_matrix.md

Complete severity classification system with:

• SEV1-4 definitions and criteria
• Response requirements and timelines
• Escalation paths
• Communication requirements
• Decision trees and examples

references/rca_frameworks_guide.md

Detailed guide for root cause analysis:

• 5 Whys methodology
• Fishbone (Ishikawa) diagram analysis
• Timeline analysis techniques
• Bow Tie analysis for high-risk incidents
• Framework selection guidelines

references/communication_templates.md

Standardized communication templates:

• Severity-specific notification templates
• Stakeholder-specific messaging
• Escalation communications
• Resolution notifications
• Customer communication guidelines

Usage Patterns

End-to-End Incident Workflow

1. Initial Classification

echo "Payment API returning 500 errors for 70% of requests" | \
  python scripts/incident_classifier.py --format text

2. Timeline Reconstruction (after collecting events)

python scripts/timeline_reconstructor.py \
  --input events.json \
  --gap-analysis \
  --format markdown \
  --output timeline.md

3. PIR Generation (after incident resolution)

python scripts/pir_generator.py \
  --incident incident.json \
  --timeline timeline.md \
  --rca-method fishbone \
  --output pir.md

Integration Examples

CI/CD Pipeline Integration:

# Classify deployment issues
cat deployment_error.log | python scripts/incident_classifier.py --format json

Monitoring Integration:

# Process alert events
curl -s "monitoring-api/events" | python scripts/timeline_reconstructor.py --format text

Runbook Generation: Use classification output to automatically select appropriate runbooks and escalation procedures.

Quality Standards

• Zero External Dependencies - All scripts use only Python standard library
• Dual Output Format - Both JSON (machine-readable) and text (human-readable)
• Robust Input Handling - Graceful handling of missing or malformed data
• Professional Defaults - Opinionated, battle-tested configurations
• Comprehensive Testing - Sample data and expected outputs included

Technical Requirements

• Python 3.6+
• No external dependencies required
• Works with standard Unix tools (pipes, redirection)
• Cross-platform compatible

Severity Classification Reference

Getting Help

Each script includes comprehensive help:

python scripts/incident_classifier.py --help
python scripts/timeline_reconstructor.py --help  
python scripts/pir_generator.py --help

For methodology questions, refer to the reference documentation in the references/ directory.

Contributing

When adding new features: 1. Maintain zero external dependencies 2. Add comprehensive examples to assets/ 3. Update expected outputs in expected_outputs/ 4. Follow the established patterns for argument parsing and output formatting

License

This skill is part of the claude-skills repository. See the main repository LICENSE for details.

Incident Communication Templates

Overview

This document provides standardized communication templates for incident response. These templates ensure consistent, clear communication across different severity levels and stakeholder groups.

Template Usage Guidelines

General Principles

1. Be Clear and Concise - Use simple language, avoid jargon 2. Be Factual - Only state what is known, avoid speculation 3. Be Timely - Send updates at committed intervals 4. Be Actionable - Include next steps and expected timelines 5. Be Accountable - Include contact information for follow-up

Template Selection

• Choose templates based on incident severity and audience
• Customize templates with specific incident details
• Always include next update time and contact information
• Escalate template types as severity increases

---

SEV1 Templates

Initial Alert - Internal Teams

Subject: 🚨 [SEV1] CRITICAL: {Service} Complete Outage - Immediate Response Required

CRITICAL INCIDENT ALERT - IMMEDIATE ATTENTION REQUIRED

Incident Summary:
- Service: {Service Name}
- Status: Complete Outage
- Start Time: {Timestamp}
- Customer Impact: {Impact Description}
- Estimated Affected Users: {Number/Percentage}

Immediate Actions Needed:
✓ Incident Commander: {Name} - ASSIGNED
✓ War Room: {Bridge/Chat Link} - JOIN NOW
✓ On-Call Response: {Team} - PAGED
⏳ Executive Notification: In progress
⏳ Status Page Update: Within 15 minutes

Current Situation:
{Brief description of what we know}

What We're Doing:
{Immediate response actions being taken}

Next Update: {Timestamp - 15 minutes from now}

Incident Commander: {Name}
Contact: {Phone/Slack}

THIS IS A CUSTOMER-IMPACTING INCIDENT REQUIRING IMMEDIATE ATTENTION

Executive Notification - SEV1

Subject: 🚨 URGENT: Customer-Impacting Outage - {Service}

EXECUTIVE ALERT: Critical customer-facing incident

Service: {Service Name}
Impact: {Customer impact description}
Duration: {Current duration} (started {start time})
Business Impact: {Revenue/SLA/compliance implications}

Customer Impact Summary:
- Affected Users: {Number/percentage}
- Revenue Impact: {$ amount if known}
- SLA Status: {Breach status}
- Customer Escalations: {Number if any}

Response Status:
- Incident Commander: {Name} ({contact})
- Response Team Size: {Number of engineers}
- Root Cause: {If known, otherwise "Under investigation"}
- ETA to Resolution: {If known, otherwise "Investigating"}

Executive Actions Required:
- [ ] Customer communication approval needed
- [ ] Legal/compliance notification: {If applicable}
- [ ] PR/Media response preparation: {If needed}
- [ ] Resource allocation decisions: {If escalation needed}

War Room: {Link}
Next Update: {15 minutes from now}

This incident meets SEV1 criteria and requires executive oversight.

{Incident Commander contact information}

Customer Communication - SEV1

Subject: Service Disruption - Immediate Action Being Taken

We are currently experiencing a service disruption affecting {service description}.

What's Happening:
{Clear, customer-friendly description of the issue}

Impact:
{What customers are experiencing - be specific}

What We're Doing:
We detected this issue at {time} and immediately mobilized our engineering team. We are actively working to resolve this issue and will provide updates every 15 minutes.

Current Actions:
• {Action 1 - customer-friendly description}
• {Action 2 - customer-friendly description}
• {Action 3 - customer-friendly description}

Workaround:
{If available, provide clear steps}
{If not available: "We are working on alternative solutions and will share them as soon as available."}

Next Update: {Timestamp}
Status Page: {Link}
Support: {Contact information if different from usual}

We sincerely apologize for the inconvenience and are committed to resolving this as quickly as possible.

{Company Name} Team

Status Page Update - SEV1

Status: Major Outage

{Timestamp} - Investigating

We are currently investigating reports of {service} being unavailable. Our team has been alerted and is actively investigating the cause.

Affected Services: {List of affected services}
Impact: {Customer-facing impact description}

We will provide an update within 15 minutes.

{Timestamp} - Identified

We have identified the cause of the {service} outage. Our engineering team is implementing a fix.

Root Cause: {Brief, customer-friendly explanation}
Expected Resolution: {Timeline if known}

Next update in 15 minutes.

{Timestamp} - Monitoring

The fix has been implemented and we are monitoring the service recovery. 

Current Status: {Recovery progress}
Next Steps: {What we're monitoring}

We expect full service restoration within {timeframe}.

{Timestamp} - Resolved

{Service} is now fully operational. We have confirmed that all functionality is working as expected.

Total Duration: {Duration}
Root Cause: {Brief summary}

We apologize for the inconvenience. A full post-incident review will be conducted and shared within 24 hours.

---

SEV2 Templates

Team Notification - SEV2

Subject: ⚠️ [SEV2] {Service} Performance Issues - Response Team Mobilizing

SEV2 INCIDENT: Performance degradation requiring active response

Incident Details:
- Service: {Service Name}
- Issue: {Description of performance issue}
- Start Time: {Timestamp}
- Affected Users: {Percentage/description}
- Business Impact: {Impact on business operations}

Current Status:
{What we know about the issue}

Response Team:
- Incident Commander: {Name} ({contact})
- Primary Responder: {Name} ({team})
- Supporting Teams: {List of engaged teams}

Immediate Actions:
✓ {Action 1 - completed}
⏳ {Action 2 - in progress}
⏳ {Action 3 - next step}

Metrics:
- Error Rate: {Current vs normal}
- Response Time: {Current vs normal}  
- Throughput: {Current vs normal}

Communication Plan:
- Internal Updates: Every 30 minutes
- Stakeholder Notification: {If needed}
- Status Page Update: {Planned/not needed}

Coordination Channel: {Slack channel}
Next Update: {30 minutes from now}

Incident Commander: {Name} | {Contact}

Stakeholder Update - SEV2

Subject: [SEV2] Service Performance Update - {Service}

Service Performance Incident Update

Service: {Service Name}
Duration: {Current duration}
Impact: {Description of user impact}

Current Status:
{Brief status of the incident and response efforts}

What We Know:
• {Key finding 1}
• {Key finding 2}
• {Key finding 3}

What We're Doing:
• {Response action 1}
• {Response action 2}
• {Monitoring/verification steps}

Customer Impact:
{Realistic assessment of what users are experiencing}

Workaround:
{If available, provide steps}

Expected Resolution:
{Timeline if known, otherwise "Continuing investigation"}

Next Update: {30 minutes}
Contact: {Incident Commander information}

This incident is being actively managed and does not currently require escalation.

Customer Communication - SEV2 (Optional)

Subject: Temporary Service Performance Issues

We are currently experiencing performance issues with {service name} that may affect your experience.

What You Might Notice:
{Specific symptoms users might experience}

What We're Doing:
Our team identified this issue at {time} and is actively working on a resolution. We expect to have this resolved within {timeframe}.

Workaround:
{If applicable, provide simple workaround steps}

We will update our status page at {link} with progress information.

Thank you for your patience as we work to resolve this issue quickly.

{Company Name} Support Team

---

SEV3 Templates

Team Assignment - SEV3

Subject: [SEV3] Issue Assignment - {Component} Issue

SEV3 Issue Assignment

Service/Component: {Affected component}
Issue: {Description}
Reported: {Timestamp}
Reporter: {Person/system that reported}

Issue Details:
{Detailed description of the problem}

Impact Assessment:
- Affected Users: {Scope}
- Business Impact: {Assessment}
- Urgency: {Business hours response appropriate}

Assignment:
- Primary: {Engineer name}
- Team: {Responsible team}
- Expected Response: {Within 2-4 hours}

Investigation Plan:
1. {Investigation step 1}
2. {Investigation step 2}
3. {Communication checkpoint}

Workaround:
{If known, otherwise "Investigating alternatives"}

This issue will be tracked in {ticket system} as {ticket number}.

Team Lead: {Name} | {Contact}

Status Update - SEV3

Subject: [SEV3] Progress Update - {Component}

SEV3 Issue Progress Update

Issue: {Brief description}
Assigned to: {Engineer/Team}
Investigation Status: {Current progress}

Findings So Far:
{What has been discovered during investigation}

Next Steps:
{Planned actions and timeline}

Impact Update:
{Any changes to scope or urgency}

Expected Resolution:
{Timeline if known}

This issue continues to be tracked as SEV3 with no escalation required.

Contact: {Assigned engineer} | {Team lead}

---

SEV4 Templates

Issue Documentation - SEV4

Subject: [SEV4] Issue Documented - {Description}

SEV4 Issue Logged

Description: {Clear description of the issue}
Reporter: {Name/system}
Date: {Date reported}

Impact:
{Minimal impact description}

Priority Assessment:
This issue has been classified as SEV4 and will be addressed in the normal development cycle.

Assignment:
- Team: {Responsible team}
- Sprint: {Target sprint}
- Estimated Effort: {Story points/hours}

This issue is tracked as {ticket number} in {system}.

Product Owner: {Name}

---

Escalation Templates

Severity Escalation

Subject: ESCALATION: {Original Severity} → {New Severity} - {Service}

SEVERITY ESCALATION NOTIFICATION

Original Classification: {Original severity}
New Classification: {New severity}  
Escalation Time: {Timestamp}
Escalated By: {Name and role}

Escalation Reasons:
• {Reason 1 - scope expansion/duration/impact}
• {Reason 2}
• {Reason 3}

Updated Impact:
{New assessment of customer/business impact}

Updated Response Requirements:
{New response team, communication frequency, etc.}

Previous Response Actions:
{Summary of actions taken under previous severity}

New Incident Commander: {If changed}
Updated Communication Plan: {New frequency/recipients}

All stakeholders should adjust response according to {new severity} protocols.

Incident Commander: {Name} | {Contact}

Management Escalation

Subject: MANAGEMENT ESCALATION: Extended {Severity} Incident - {Service}

Management Escalation Required

Incident: {Service} {brief description}
Original Severity: {Severity}
Duration: {Current duration}
Escalation Trigger: {Duration threshold/scope change/customer escalation}

Current Status:
{Brief status of incident response}

Challenges Encountered:
• {Challenge 1}
• {Challenge 2}
• {Resource/expertise needs}

Business Impact:
{Updated assessment of business implications}

Management Decision Required:
• {Decision 1 - resource allocation/external expertise/communication}
• {Decision 2}

Recommended Actions:
{Incident Commander's recommendations}

This escalation follows standard procedures for {trigger type}.

Incident Commander: {Name}
Contact: {Phone/Slack}
War Room: {Link}

---

Resolution Templates

Resolution Confirmation - All Severities

Subject: RESOLVED: [{Severity}] {Service} Incident - {Brief Description}

INCIDENT RESOLVED

Service: {Service Name}
Issue: {Brief description}
Duration: {Total duration}
Resolution Time: {Timestamp}

Resolution Summary:
{Brief description of how the issue was resolved}

Root Cause:
{Brief explanation - detailed PIR to follow}

Impact Summary:
- Users Affected: {Final count/percentage}
- Business Impact: {Final assessment}
- Services Affected: {List}

Resolution Actions Taken:
• {Action 1}
• {Action 2}
• {Verification steps}

Monitoring:
We will continue monitoring {service} for {duration} to ensure stability.

Next Steps:
• Post-incident review scheduled for {date}
• Action items to be tracked in {system}
• Follow-up communication: {If needed}

Thank you to everyone who participated in the incident response.

Incident Commander: {Name}

Customer Resolution Communication

Subject: Service Restored - Thank You for Your Patience

Service Update: Issue Resolved

We're pleased to report that the {service} issues have been fully resolved as of {timestamp}.

What Was Fixed:
{Customer-friendly explanation of the resolution}

Duration:
The issue lasted {duration} from {start time} to {end time}.

What We Learned:
{Brief, high-level takeaway}

Our Commitment:
We are conducting a thorough review of this incident and will implement improvements to prevent similar issues in the future. A summary of our findings and improvements will be shared {timeframe}.

We sincerely apologize for any inconvenience this may have caused and appreciate your patience while we worked to resolve the issue.

If you continue to experience any problems, please contact our support team at {contact information}.

Thank you,
{Company Name} Team

---

Template Customization Guidelines

Placeholders to Always Replace

• {Service} / {Service Name} - Specific service or component
• {Timestamp} - Specific date/time in consistent format
• {Name} / {Contact} - Actual names and contact information
• {Duration} - Actual time durations
• {Link} - Real URLs to war rooms, status pages, etc.

Language Guidelines

• Use active voice ("We are investigating" not "The issue is being investigated")
• Be specific about timelines ("within 30 minutes" not "soon")
• Avoid technical jargon in customer communications
• Include empathy in customer-facing messages
• Use consistent terminology throughout incident lifecycle

Timing Guidelines

Audience-Specific Considerations

Engineering Teams

• Include technical details
• Provide specific metrics and logs
• Include coordination channels
• List specific actions and owners

Executive/Business

• Focus on business impact
• Include customer and revenue implications
• Provide clear timeline and resource needs
• Highlight any external factors (PR, legal, compliance)

Customers

• Use plain language
• Focus on customer impact and workarounds
• Provide realistic timelines
• Include support contact information
• Show empathy and accountability

---

Last Updated: February 2026 Next Review: May 2026 Owner: Incident Management Team

Incident Severity Classification Matrix

Overview

This document defines the severity classification system used for incident response. The classification determines response requirements, escalation paths, and communication frequency.

Severity Levels

SEV1 - Critical Outage

Definition: Complete service failure affecting all users or critical business functions

Impact Criteria

• Customer-facing services completely unavailable
• Data loss or corruption affecting users
• Security breaches with customer data exposure
• Revenue-generating systems down
• SLA violations with financial penalties
• > 75% of users affected

Response Requirements

Escalation Path

1. Immediate: On-call Engineer → Incident Commander 2. 15 minutes: VP Engineering + Customer Success VP 3. 30 minutes: CTO 4. 60 minutes: CEO + Full Executive Team

Communication Requirements

• Frequency: Every 15 minutes until resolution
• Channels: PagerDuty, Phone, Slack, Email, Status Page
• Recipients: All engineering, executives, customer success
• Template: SEV1 Executive Alert Template

---

SEV2 - Major Impact

Definition: Significant degradation affecting subset of users or non-critical functions

Impact Criteria

• Partial service degradation (25-75% of users affected)
• Performance issues causing user frustration
• Non-critical features unavailable
• Internal tools impacting productivity
• Data inconsistencies not affecting user experience
• API errors affecting integrations

Response Requirements

Escalation Path

1. Immediate: On-call Engineer → Team Lead 2. 30 minutes: Engineering Manager 3. 2 hours: VP Engineering 4. 4 hours: CTO (if unresolved)

Communication Requirements

• Frequency: Every 30 minutes during active response
• Channels: PagerDuty, Slack, Email
• Recipients: Engineering team, product team, relevant stakeholders
• Template: SEV2 Major Impact Template

---

SEV3 - Minor Impact

Definition: Limited impact with workarounds available

Impact Criteria

• Single feature or component affected
• < 25% of users impacted
• Workarounds available
• Performance degradation not significantly impacting UX
• Non-urgent monitoring alerts
• Development/test environment issues

Response Requirements

Escalation Path

1. Immediate: Assigned Engineer 2. 4 hours: Team Lead 3. 1 business day: Engineering Manager (if needed)

Communication Requirements

• Frequency: At key milestones only
• Channels: Slack, Email
• Recipients: Assigned team, team lead
• Template: SEV3 Minor Impact Template

---

SEV4 - Low Impact

Definition: Minimal impact, cosmetic issues, or planned maintenance

Impact Criteria

• Cosmetic bugs
• Documentation issues
• Logging or monitoring gaps
• Performance issues with no user impact
• Development/test environment issues
• Feature requests or enhancements

Response Requirements

Communication Requirements

• Frequency: Standard development cycle updates
• Channels: Ticket system
• Recipients: Product owner, assigned developer
• Template: Standard issue template

Classification Guidelines

User Impact Assessment

Business Impact Assessment

Duration Considerations

Decision Tree

1. Is this a security incident with data exposure?
   → YES: SEV1 (regardless of user count)
   → NO: Continue to step 2

2. Are revenue-generating services completely down?
   → YES: SEV1
   → NO: Continue to step 3

3. What percentage of users are affected?
   → > 75%: SEV1
   → 25-75%: SEV2
   → 5-25%: SEV3
   → < 5%: SEV4

4. Apply business impact modifiers
5. Consider duration factors
6. When in doubt, err on higher severity

Examples

SEV1 Examples

• Payment processing system completely down
• All user authentication failing
• Database corruption causing data loss
• Security breach with customer data exposed
• Website returning 500 errors for all users

SEV2 Examples

• Payment processing slow (30-second delays)
• Search functionality returning incomplete results
• API rate limits causing partner integration issues
• Dashboard displaying stale data (> 1 hour old)
• Mobile app crashing for 40% of users

SEV3 Examples

• Single feature in admin panel not working
• Email notifications delayed by 1 hour
• Non-critical API endpoint returning errors
• Cosmetic UI bug in settings page
• Development environment deployment failing

SEV4 Examples

• Typo in help documentation
• Log format change needed for analysis
• Non-critical performance optimization
• Internal tool enhancement request
• Test data cleanup needed

Escalation Triggers

Automatic Escalation

• SEV1 incidents automatically escalate every 30 minutes if unresolved
• SEV2 incidents escalate after 2 hours without significant progress
• Any incident with expanding scope increases severity
• Customer escalation to support triggers severity review

Manual Escalation

• Incident Commander can escalate at any time
• Technical leads can request escalation
• Business stakeholders can request severity review
• External factors (media attention, regulatory) trigger escalation

Communication Templates

SEV1 Executive Alert

Subject: 🚨 CRITICAL INCIDENT - [Service] Complete Outage

URGENT: Customer-facing service outage requiring immediate attention

Service: [Service Name]
Start Time: [Timestamp]
Impact: [Description of customer impact]
Estimated Affected Users: [Number/Percentage]
Business Impact: [Revenue/SLA/Brand implications]

Incident Commander: [Name] ([Contact])
Response Team: [Team members engaged]

Current Status: [Brief status update]
Next Update: [Timestamp - 15 minutes from now]
War Room: [Bridge/Chat link]

This is a customer-impacting incident requiring executive awareness.

SEV2 Major Impact

Subject: ⚠️ [SEV2] [Service] - Major Performance Impact

Major service degradation affecting user experience

Service: [Service Name]
Start Time: [Timestamp] 
Impact: [Description of user impact]
Scope: [Affected functionality/users]

Response Team: [Team Lead] + [Team members]
Status: [Current mitigation efforts]
Workaround: [If available]

Next Update: 30 minutes
Status Page: [Link if updated]

Review and Updates

This severity matrix should be reviewed quarterly and updated based on:

• Incident response learnings
• Business priority changes
• Service architecture evolution
• Regulatory requirement changes
• Customer feedback and SLA updates

Last Updated: February 2026 Next Review: May 2026 Owner: Engineering Leadership

Incident Response Framework Reference

Production-grade incident management knowledge base synthesizing PagerDuty, Google SRE, and Atlassian methodologies into a unified, opinionated framework. This document is the source of truth for incident commanders operating under pressure.

---

1. Industry Framework Comparison

PagerDuty Incident Response Model

PagerDuty's open-source incident response process defines four core roles and five process phases. The model prioritizes speed of mobilization over process perfection.

Roles:

• Incident Commander (IC): Owns the incident end-to-end. Does NOT perform technical investigation. Delegates, coordinates, and makes final escalation decisions. The IC is the single point of authority; conflicting opinions are resolved by the IC, not by committee.
• Scribe: Captures timestamped decisions, actions, and findings in the incident channel. The scribe never participates in technical work. A good scribe reduces postmortem preparation time by 70%.
• Subject Matter Expert (SME): Pulled in on-demand for specific subsystems. SMEs report findings to the IC, not to each other. Parallel SME investigations must be coordinated through the IC to avoid duplicated effort.
• Customer Liaison: Owns all outbound customer communication. Drafts status page updates for IC approval. Shields the technical team from inbound customer inquiries during active incidents.

Process Phases: Detect, Triage, Mobilize, Mitigate, Resolve, Postmortem.

Communication Protocol: PagerDuty mandates a dedicated Slack channel per incident, a bridge call for SEV1/SEV2, and status updates at fixed cadences (every 15 min for SEV1, every 30 min for SEV2). All decisions are announced in the channel, never in DMs or side threads.

Google SRE: Managing Incidents (Chapter 14)

Google's SRE model, documented in Site Reliability Engineering (O'Reilly, 2016), emphasizes role separation and clear handoffs as the primary mechanisms for preventing incident chaos.

Key Principles:

• Operational vs. Communication Tracks: Google splits incident work into two parallel tracks. The operational track handles technical mitigation. The communication track handles stakeholder updates, executive briefings, and customer notifications. These tracks run independently with the IC bridging them.
• Role Separation is Non-Negotiable: The person debugging the system must never be the person updating stakeholders. Cognitive load from context-switching between technical work and communication degrades both outputs. Google measured a 40% increase in mean-time-to-resolution (MTTR) when a single person attempted both.
• Clear Handoffs: When an IC rotates out (recommended every 60-90 minutes for SEV1), the handoff includes: current status summary, active hypotheses, pending actions, and escalation state. Handoffs happen on the bridge call, not asynchronously.
• Defined Command Post: All communication flows through a single channel. Google uses the term "command post" -- a virtual or physical location where all incident participants converge.

Atlassian Incident Management Model

Atlassian's model, published in their Incident Management Handbook, is severity-driven and template-heavy. It favors structured playbooks over improvisation.

Key Characteristics:

• Severity Levels Drive Everything: The assigned severity determines who gets paged, what communication templates are used, response time SLAs, and postmortem requirements. Severity is assigned at triage and reassessed every 30 minutes.
• Handbook-Driven Approach: Atlassian maintains runbooks for every known failure mode. During incidents, responders follow documented playbooks before improvising. This reduces MTTR for known issues by 50-60% but requires significant upfront investment in documentation.
• Communication Templates: Pre-written templates for status page updates, customer emails, and executive summaries. Templates include severity-specific language and are reviewed quarterly. This eliminates wordsmithing during active incidents.
• Values-Based Decisions: When runbooks do not cover the situation, Atlassian defaults to a decision hierarchy: (1) protect customer data, (2) restore service, (3) preserve evidence for root cause analysis.

Framework Comparison Table

When to Use Which Framework

• Teams under 20 engineers: Start with PagerDuty's model. It is lightweight and prescriptive enough to work without heavy process investment. Add Atlassian-style runbooks as you identify recurring failure modes.
• Teams running 50+ microservices: Adopt Google SRE's dual-track model. The operational/communication split becomes critical when incidents span multiple teams and subsystems.
• Regulated industries (finance, healthcare, government): Use Atlassian's handbook-driven approach as the foundation. Regulatory auditors expect documented procedures, and templates satisfy compliance requirements for incident communication records.
• Hybrid (recommended for most teams at scale): Use PagerDuty's role definitions, Google's track separation, and Atlassian's template library. This is the approach codified in the rest of this document.

---

2. Severity Definitions

Severity Classification Matrix

Customer-Facing Signals by Severity

SEV1 Signals: Support ticket volume spikes >500% of baseline within 15 minutes. Social media mentions of outage trend upward. Revenue dashboards show >95% drop in transaction volume. Multiple monitoring systems alarm simultaneously.

SEV2 Signals: Support ticket volume spikes 100-500% of baseline. Specific feature-related complaints cluster in support channels. Partial transaction failures visible in payment dashboards. Single monitoring system shows sustained alerting.

SEV3 Signals: Sporadic support tickets with a common pattern (under 20/hour). Users report intermittent issues with workarounds. Monitoring shows degraded but not critical metrics.

SEV4 Signals: Internal team notices issue during routine work. Occasional user mention with no pattern or urgency. Monitoring shows minor anomaly within acceptable thresholds.

Severity Upgrade and Downgrade Criteria

Upgrade from SEV2 to SEV1: Impact expands to >80% of users, revenue impact confirmed above $10K/hour, data integrity compromise confirmed, or mitigation attempt fails after 45 minutes.

Downgrade from SEV1 to SEV2: Partial mitigation restores service for >70% of users, revenue impact drops below $10K/hour, and no ongoing data integrity concern.

Downgrade from SEV2 to SEV3: Workaround deployed and communicated, impact limited to <10% of users, and no revenue impact.

Severity changes must be announced by the IC in the incident channel with justification. The scribe logs the timestamp and rationale.

---

3. Role Definitions

Incident Commander (IC)

The IC is the single decision-maker during an incident. This role exists to eliminate decision-by-committee, which adds 20-40 minutes to MTTR in measured studies.

Responsibilities:

• Assign severity level at triage (reassess every 30 minutes)
• Assign all other incident roles
• Approve status page updates before publication
• Make go/no-go decisions on mitigation strategies (rollback, feature flag, scaling)
• Decide when to escalate to executive leadership
• Declare incident resolved and initiate postmortem scheduling

Decision Authority: The IC can authorize rollbacks, page any team member regardless of org chart, approve customer communications, and override objections from individual contributors during active mitigation. The IC cannot approve financial expenditures above $50K or public press statements -- those require VP/C-level approval.

What the IC Must NOT Do: Debug code, write queries, SSH into production servers, or perform any hands-on technical work. The moment an IC starts debugging, incident coordination degrades. If the IC is the only person with domain expertise, they must hand off IC duties before engaging technically.

Communications Lead

Responsibilities:

• Draft all status page updates using severity-appropriate templates
• Coordinate with Customer Liaison on outbound customer messaging
• Maintain the executive summary document (updated every 30 min for SEV1/SEV2)
• Manage the stakeholder notification list and delivery
• Post scheduled updates even when there is no new information ("We are continuing to investigate" is a valid update)

Operations Lead

Responsibilities:

• Coordinate technical investigation across SMEs
• Maintain the running hypothesis list and assign investigation tasks
• Report technical findings to the IC in plain language
• Execute mitigation actions approved by the IC
• Track parallel workstreams and prevent duplicated effort

Scribe

Responsibilities:

• Maintain a timestamped log of all decisions, actions, and findings
• Document who said what and when in the incident channel
• Capture rollback decisions, hypothesis changes, and escalation triggers
• Produce the initial postmortem timeline (saves 2-4 hours of postmortem prep)

Subject Matter Experts (SMEs)

SMEs are paged on-demand by the IC for specific subsystems. They report findings to the Operations Lead, not directly to stakeholders. An SME who identifies a potential fix proposes it to the IC for approval before executing. SMEs are released from the incident explicitly by the IC when their subsystem is cleared.

Customer Liaison

Owns the customer-facing voice during the incident. Monitors support channels for inbound customer reports. Drafts customer notification emails. Updates the public status page (after IC approval). Shields the technical team from direct customer inquiries during active mitigation.

---

4. Communication Protocols

Incident Channel Naming Convention

Format: #inc-YYYYMMDD-brief-desc

Examples:

• #inc-20260216-payment-api-timeout
• #inc-20260216-db-primary-failover
• #inc-20260216-auth-service-degraded

Channel topic must include: severity, IC name, bridge call link, status page link. Example topic: SEV1 | IC: @jane.smith | Bridge: https://meet.example.com/inc-20260216 | Status: https://status.example.com

Internal Status Update Templates

SEV1/SEV2 Update Template (posted in incident channel and executive Slack channel):

INCIDENT UPDATE - [SEV1/SEV2] - [HH:MM UTC]
Status: [Investigating | Identified | Mitigating | Resolved]
Impact: [Specific user-facing impact in plain language]
Current Action: [What is actively being done right now]
Next Update: [HH:MM UTC]
IC: @[name]

Executive Summary Template (for SEV1, updated every 30 min):

EXECUTIVE SUMMARY - [Incident Title] - [HH:MM UTC]
Severity: SEV1
Duration: [X hours Y minutes]
Customer Impact: [Number of affected users/transactions]
Revenue Impact: [Estimated $ if known, "assessing" if not]
Current Status: [One sentence]
Mitigation ETA: [Estimated time or "unknown"]
Next Escalation Point: [What triggers executive action]

Status Page Update Templates

SEV1 Initial Post:

Title: [Service Name] - Service Disruption
Body: We are currently experiencing a disruption affecting [service/feature].
Users may encounter [specific symptom: errors, timeouts, inability to access].
Our engineering team has been mobilized and is actively investigating.
We will provide an update within 15 minutes.

SEV1 Update (mitigation in progress):

Title: [Service Name] - Service Disruption (Update)
Body: We have identified the cause of the disruption affecting [service/feature]
and are implementing a fix. Some users may continue to experience [symptom].
We expect to have an update on resolution within [X] minutes.

SEV1 Resolution:

Title: [Service Name] - Resolved
Body: The disruption affecting [service/feature] has been resolved as of [HH:MM UTC].
Service has been restored to normal operation. Users should no longer experience
[symptom]. We will publish a full incident report within 48 hours.
We apologize for the inconvenience.

SEV2 Initial Post:

Title: [Service Name] - Degraded Performance
Body: We are investigating reports of degraded performance affecting [feature].
Some users may experience [specific symptom]. A workaround is [available/not yet available].
Our team is actively investigating and we will provide an update within 30 minutes.

Bridge Call / War Room Etiquette

1. Mute by default. Unmute only when speaking to the IC or Operations Lead. 2. Identify yourself before speaking. "This is [name] from [team]." Every time. 3. State findings, then recommendations. "Database replication lag is 45 seconds and climbing. I recommend we fail over to the secondary cluster." 4. IC confirms before action. No unilateral action on production systems during an incident. The IC says "approved" or "hold" before anyone executes. 5. No side conversations. If two SMEs need to discuss a hypothesis, they take it to a breakout channel and report back findings to the main bridge. 6. Time-box debugging. The IC sets 15-minute timers for investigation threads. If a hypothesis is not confirmed or denied in 15 minutes, pivot to the next hypothesis or escalate.

Customer Notification Templates

SEV1 Customer Email (B2B, enterprise accounts):

Subject: [Company Name] Service Incident - [Date]

Dear [Customer Name],

We are writing to inform you of a service incident affecting [product/service]
that began at [HH:MM UTC] on [date].

Impact: [Specific impact to this customer's usage]
Current Status: [Brief status]
Expected Resolution: [ETA if known, or "We are working to resolve this as quickly as possible"]

We will continue to provide updates every [15/30] minutes until resolution.
Your dedicated account team is available at [contact info] for any questions.

Sincerely,
[Name], [Title]

---

5. Escalation Matrix

Escalation Tiers

Tier 1 - Within Team (0-15 minutes): On-call engineer investigates. If the issue is within the team's domain and matches a known runbook, resolve without escalation. Page the IC if severity is SEV2 or higher, or if the issue is not resolved within 15 minutes.

Tier 2 - Cross-Team (15-45 minutes): IC pages SMEs from adjacent teams. Common cross-team escalations: database team for replication issues, networking team for connectivity failures, security team for suspicious activity. Cross-team SMEs join the incident channel and bridge call.

Tier 3 - Executive (45+ minutes or immediate for SEV1): VP of Engineering notified for all SEV1 incidents immediately. CTO notified if SEV1 exceeds 1 hour without mitigation progress. CEO notified if SEV1 involves data breach or regulatory implications. Executive involvement is for resource allocation and external communication decisions, not technical direction.

Time-Based Escalation Triggers

Escalation Path Examples

Database failover failure: On-call DBA (Tier 1, 0-15 min) -> IC + DBA team lead (Tier 2, 15 min) -> Infrastructure VP + cloud provider support (Tier 3, 45 min)

Payment processing outage: On-call payments engineer (Tier 1, 0-5 min) -> IC + payments team lead + payment provider liaison (Tier 2, 5 min, immediate due to revenue impact) -> CFO + VP Eng (Tier 3, 15 min if provider-side issue confirmed)

Security incident (suspected breach): Security on-call (Tier 1, 0-5 min) -> CISO + IC + legal counsel (Tier 2, immediate) -> CEO + external incident response firm (Tier 3, within 1 hour if breach confirmed)

On-Call Rotation Best Practices

• Primary + secondary on-call for every critical service. Secondary is paged automatically if primary does not acknowledge within 5 minutes.
• On-call shifts are 7 days maximum. Longer rotations degrade alertness and response quality.
• Handoff checklist: Current open issues, recent deploys in the last 48 hours, known risks or maintenance windows, escalation contacts for dependent services.
• On-call load budget: No more than 2 pages per night on average, measured weekly. Exceeding this indicates systemic reliability issues that must be addressed with engineering investment, not heroic on-call effort.

---

6. Incident Lifecycle Phases

Phase 1: Detection

Detection comes from three sources, in order of preference:

1. Automated monitoring (preferred): Alerting rules on latency (p99 > 2x baseline), error rates (5xx > 1% of requests), saturation (CPU > 85%, memory > 90%, disk > 80%), and business metrics (transaction volume drops > 20% from 15-minute rolling average). Alerts should fire within 60 seconds of threshold breach. 2. Internal reports: An engineer notices anomalous behavior during routine work. Internal detection typically adds 5-15 minutes to response time compared to automated monitoring. 3. Customer reports: Customers contact support about issues. This is the worst detection source. If customers detect incidents before monitoring, the monitoring coverage has a gap that must be closed in the postmortem.

Detection SLA: SEV1 incidents must be detected within 5 minutes of impact onset. If detection latency exceeds this, the postmortem must include a monitoring improvement action item.

Phase 2: Triage

The first responder performs initial triage within 5 minutes of detection:

1. Scope assessment: How many users, services, or regions are affected? Check dashboards, not assumptions. 2. Severity assignment: Use the severity matrix in Section 2. When in doubt, assign higher severity. Downgrading is cheap; delayed escalation is expensive. 3. IC assignment: For SEV1/SEV2, page the on-call IC immediately. For SEV3, the first responder may self-assign IC duties. 4. Initial hypothesis: What changed in the last 2 hours? Check deploy logs, config changes, upstream dependency status, and traffic patterns. 70% of incidents correlate with a change deployed in the prior 2 hours.

Phase 3: Mobilization

The IC executes mobilization within 10 minutes of assignment:

1. Create incident channel: #inc-YYYYMMDD-brief-desc. Set topic with severity, IC name, bridge link. 2. Assign roles: Communications Lead, Operations Lead, Scribe. For SEV3/SEV4, the IC may cover multiple roles. 3. Open bridge call (SEV1/SEV2): Share link in incident channel. All responders join within 5 minutes. 4. Post initial summary: Current understanding, affected services, assigned roles, first actions. 5. Notify stakeholders: Page dependent teams. Notify customer support leadership. For SEV1, notify executive chain per escalation matrix.

Phase 4: Investigation

Investigation runs as parallel workstreams coordinated by the Operations Lead:

• Workstream discipline: Each SME investigates one hypothesis at a time. The Operations Lead tracks active hypotheses on a shared list. Completed investigations report: confirmed, denied, or inconclusive.
• Hypothesis testing priority: (1) Recent changes (deploys, configs, feature flags), (2) Upstream dependency failures, (3) Capacity exhaustion, (4) Data corruption, (5) Security compromise.
• 15-minute rule: If a hypothesis is not confirmed or denied within 15 minutes, the IC decides whether to continue, pivot, or escalate. Unbounded investigation is the leading cause of extended MTTR.
• Evidence collection: Screenshots, log snippets, metric graphs, and query results are posted in the incident channel, not described verbally. The scribe tags evidence with timestamps.

Phase 5: Mitigation

Mitigation prioritizes restoring service over finding root cause:

• Rollback first: If a deploy correlates with the incident, roll it back before investigating further. A 5-minute rollback beats a 45-minute investigation. Rollback authority rests with the IC.
• Feature flags: Disable the suspected feature via feature flag if available. This is faster and less risky than a full rollback.
• Scaling: If the issue is capacity-related, scale horizontally before investigating the traffic source.
• Failover: If a primary system is unrecoverable, fail over to the secondary. Test failover procedures quarterly so this is a routine, not a gamble.
• Customer workaround: If mitigation will take time, publish a workaround for customers (e.g., "Use the mobile app while we restore web access").

Mitigation verification: After applying mitigation, monitor key metrics for 15 minutes before declaring the issue mitigated. Premature declarations that the issue is mitigated followed by recurrence damage team credibility and customer trust.

Phase 6: Resolution

Resolution is declared when the root cause is addressed and service is operating normally:

• Verification checklist: Error rates returned to baseline, latency returned to baseline, no ongoing customer reports, monitoring confirms stability for 30+ minutes.
• Incident channel update: IC posts final status with resolution summary, total duration, and next steps.
• Status page update: Post resolution notice within 15 minutes of declaring resolved.
• Stand down: IC explicitly releases all responders. SMEs return to normal work. Bridge call is closed.

Phase 7: Postmortem

Postmortem is mandatory for SEV1 and SEV2. Optional but recommended for SEV3. Never conducted for SEV4.

• Timeline: Postmortem document drafted within 24 hours. Postmortem meeting held within 72 hours (3 business days). Action items assigned and tracked in the team's issue tracker.
• Blameless standard: The postmortem examines systems, processes, and tools -- not individual performance. "Why did the system allow this?" not "Why did [person] do this?"
• Required sections: Timeline (from scribe's log), root cause analysis (using 5 Whys or fault tree), impact summary (users, revenue, duration), what went well, what went poorly, action items with owners and due dates.
• Action items and recurrence: Every postmortem produces 3-7 concrete action items. Items without owners and due dates are not action items. Teams should close 80%+ within 30 days. If the same root cause appears in two postmortems within 6 months, escalate to engineering leadership as a systemic reliability investment area.

Root Cause Analysis (RCA) Frameworks Guide

Overview

This guide provides detailed instructions for applying various Root Cause Analysis frameworks during Post-Incident Reviews. Each framework offers a different perspective and approach to identifying underlying causes of incidents.

Framework Selection Guidelines

---

5 Whys Analysis Framework

Purpose

Iteratively drill down through cause-effect relationships to identify root causes.

When to Use

• Simple, linear cause-effect chains
• Time-pressured analysis
• Process-related failures
• Individual component failures

Process Steps

Step 1: Problem Statement

Write a clear, specific problem statement.

Good Example:

"The payment API returned 500 errors for 2 hours on March 15, affecting 80% of checkout attempts."

Poor Example:

"The system was broken."

Step 2: First Why

Ask why the problem occurred. Focus on immediate, observable causes.

Example:

• Why 1: Why did the payment API return 500 errors?
• Answer: The database connection pool was exhausted.

Step 3: Subsequent Whys

For each answer, ask "why" again. Continue until you reach a root cause.

Example Chain:

• Why 2: Why was the database connection pool exhausted?
• Answer: The application was creating more connections than usual.

• Why 3: Why was the application creating more connections?
• Answer: A new feature wasn't properly closing connections.

• Why 4: Why wasn't the feature properly closing connections?
• Answer: Code review missed the connection leak pattern.

• Why 5: Why did code review miss this pattern?
• Answer: We don't have automated checks for connection pooling best practices.

Step 4: Validation

Verify that addressing the root cause would prevent the original problem.

Best Practices

1. Ask at least 3 "whys" - Surface causes are rarely root causes 2. Focus on process failures, not people - Avoid blame, focus on system improvements 3. Use evidence - Support each answer with data or observations 4. Consider multiple paths - Some problems have multiple root causes 5. Test the logic - Work backwards from root cause to problem

Common Pitfalls

• Stopping too early - First few whys often reveal symptoms, not causes
• Single-cause assumption - Complex systems often have multiple contributing factors
• Blame focus - Focusing on individual mistakes rather than system failures
• Vague answers - Use specific, actionable answers

5 Whys Template

## 5 Whys Analysis

**Problem Statement:** [Clear description of the incident]

**Why 1:** [First why question]
**Answer:** [Specific, evidence-based answer]
**Evidence:** [Supporting data, logs, observations]

**Why 2:** [Second why question]
**Answer:** [Specific answer based on Why 1]
**Evidence:** [Supporting evidence]

[Continue for 3-7 iterations]

**Root Cause(s) Identified:**
1. [Primary root cause]
2. [Secondary root cause if applicable]

**Validation:** [Confirm that addressing root causes would prevent recurrence]

---

Fishbone (Ishikawa) Diagram Framework

Purpose

Systematically analyze potential causes across multiple categories to identify contributing factors.

When to Use

• Complex incidents with multiple potential causes
• When human factors are suspected
• Systemic or organizational issues
• When 5 Whys doesn't reveal clear root causes

Categories

People (Human Factors)

• Training and Skills
• Insufficient training on new systems
• Lack of domain expertise
• Skill gaps in team
• Knowledge not shared across team

• Communication
• Poor communication between teams
• Unclear responsibilities
• Information not reaching right people
• Language/cultural barriers

• Decision Making
• Decisions made under pressure
• Insufficient information for decisions
• Risk assessment inadequate
• Approval processes bypassed

Process (Procedures and Workflows)

• Documentation
• Outdated procedures
• Missing runbooks
• Unclear instructions
• Process not documented

• Change Management
• Inadequate change review
• Rushed deployments
• Insufficient testing
• Rollback procedures unclear

• Review and Approval
• Code review gaps
• Architecture review skipped
• Security review insufficient
• Performance review missing

Technology (Systems and Tools)

• Architecture
• Single points of failure
• Insufficient redundancy
• Scalability limitations
• Tight coupling between systems

• Monitoring and Alerting
• Missing monitoring
• Alert fatigue
• Inadequate thresholds
• Poor alert routing

• Tools and Automation
• Manual processes prone to error
• Tool limitations
• Automation gaps
• Integration issues

Environment (External Factors)

• Infrastructure
• Hardware failures
• Network issues
• Capacity limitations
• Geographic dependencies

• Dependencies
• Third-party service failures
• External API changes
• Vendor issues
• Supply chain problems

• External Pressure
• Time pressure from business
• Resource constraints
• Regulatory changes
• Market conditions

Process Steps

Step 1: Define the Problem

Place the incident at the "head" of the fishbone diagram.

Step 2: Brainstorm Causes

For each category, brainstorm potential contributing factors.

Step 3: Drill Down

For each factor, ask what caused that factor (sub-causes).

Step 4: Identify Primary Causes

Mark the most likely contributing factors based on evidence.

Step 5: Validate

Gather evidence to support or refute each suspected cause.

Fishbone Template

## Fishbone Analysis

**Problem:** [Incident description]

### People
**Training/Skills:**
- [Factor 1]: [Evidence/likelihood]
- [Factor 2]: [Evidence/likelihood]

**Communication:**
- [Factor 1]: [Evidence/likelihood]

**Decision Making:**
- [Factor 1]: [Evidence/likelihood]

### Process
**Documentation:**
- [Factor 1]: [Evidence/likelihood]

**Change Management:**
- [Factor 1]: [Evidence/likelihood]

**Review/Approval:**
- [Factor 1]: [Evidence/likelihood]

### Technology
**Architecture:**
- [Factor 1]: [Evidence/likelihood]

**Monitoring:**
- [Factor 1]: [Evidence/likelihood]

**Tools:**
- [Factor 1]: [Evidence/likelihood]

### Environment
**Infrastructure:**
- [Factor 1]: [Evidence/likelihood]

**Dependencies:**
- [Factor 1]: [Evidence/likelihood]

**External Factors:**
- [Factor 1]: [Evidence/likelihood]

### Primary Contributing Factors
1. [Factor with highest evidence/impact]
2. [Second most significant factor]
3. [Third most significant factor]

### Root Cause Hypothesis
[Synthesized explanation of how factors combined to cause incident]

---

Timeline Analysis Framework

Purpose

Analyze the chronological sequence of events to identify decision points, missed opportunities, and process gaps.

When to Use

• Extended incidents (> 1 hour)
• Complex multi-phase incidents
• When response effectiveness is questioned
• Communication or coordination failures

Analysis Dimensions

Detection Analysis

• Time to Detection: How long from onset to first alert?
• Detection Method: How was the incident first identified?
• Alert Effectiveness: Were the right people notified quickly?
• False Negatives: What signals were missed?

Response Analysis

• Time to Response: How long from detection to first response action?
• Escalation Timing: Were escalations timely and appropriate?
• Resource Mobilization: How quickly were the right people engaged?
• Decision Points: What key decisions were made and when?

Communication Analysis

• Internal Communication: How effective was team coordination?
• External Communication: Were stakeholders informed appropriately?
• Communication Gaps: Where did information flow break down?
• Update Frequency: Were updates provided at appropriate intervals?

Resolution Analysis

• Mitigation Strategy: Was the chosen approach optimal?
• Alternative Paths: What other options were considered?
• Resource Allocation: Were resources used effectively?
• Verification: How was resolution confirmed?

Process Steps

Step 1: Event Reconstruction

Create comprehensive timeline with all available events.

Step 2: Phase Identification

Identify distinct phases (detection, triage, escalation, mitigation, resolution).

Step 3: Gap Analysis

Identify time gaps and analyze their causes.

Step 4: Decision Point Analysis

Examine key decision points and alternative paths.

Step 5: Effectiveness Assessment

Evaluate the overall effectiveness of the response.

Timeline Template

## Timeline Analysis

### Incident Phases
1. **Detection** ([start] - [end], [duration])
2. **Triage** ([start] - [end], [duration])
3. **Escalation** ([start] - [end], [duration])
4. **Mitigation** ([start] - [end], [duration])
5. **Resolution** ([start] - [end], [duration])

### Key Decision Points
**[Timestamp]:** [Decision made]
- **Context:** [Situation at time of decision]
- **Alternatives:** [Other options considered]
- **Outcome:** [Result of decision]
- **Assessment:** [Was this optimal?]

### Communication Timeline
**[Timestamp]:** [Communication event]
- **Channel:** [Slack/Email/Phone/etc.]
- **Audience:** [Who was informed]
- **Content:** [What was communicated]
- **Effectiveness:** [Assessment]

### Gaps and Delays
**[Time Period]:** [Description of gap]
- **Duration:** [Length of gap]
- **Cause:** [Why did gap occur]
- **Impact:** [Effect on incident response]

### Response Effectiveness
**Strengths:**
- [What went well]
- [Effective decisions/actions]

**Weaknesses:**
- [What could be improved]
- [Missed opportunities]

### Root Causes from Timeline
1. [Process-based root cause]
2. [Communication-based root cause]
3. [Decision-making root cause]

---

Bow Tie Analysis Framework

Purpose

Analyze both preventive measures (left side) and protective measures (right side) around an incident.

When to Use

• High-severity incidents (SEV1)
• Security incidents
• Safety-critical systems
• When comprehensive barrier analysis is needed

Components

Hazards

What conditions create the potential for incidents?

Examples:

• High traffic loads
• Software deployments
• Human interactions with critical systems
• Third-party dependencies

Top Event

What actually went wrong? This is the center of the bow tie.

Examples:

• "Database became unresponsive"
• "Payment processing failed"
• "User authentication service crashed"

Threats (Left Side)

What specific causes could lead to the top event?

Examples:

• Code defects in new deployment
• Database connection pool exhaustion
• Network connectivity issues
• DDoS attack

Consequences (Right Side)

What are the potential impacts of the top event?

Examples:

• Revenue loss
• Customer churn
• Regulatory violations
• Brand damage
• Data loss

Barriers

What controls exist (or could exist) to prevent threats or mitigate consequences?

Preventive Barriers (Left Side):

• Code reviews
• Automated testing
• Load testing
• Input validation
• Rate limiting

Protective Barriers (Right Side):

• Circuit breakers
• Failover systems
• Backup procedures
• Customer communication
• Rollback capabilities

Process Steps

Step 1: Define the Top Event

Clearly state what went wrong.

Step 2: Identify Threats

Brainstorm all possible causes that could lead to the top event.

Step 3: Identify Consequences

List all potential impacts of the top event.

Step 4: Map Existing Barriers

Identify current controls for each threat and consequence.

Step 5: Assess Barrier Effectiveness

Evaluate how well each barrier worked (or failed).

Step 6: Recommend Additional Barriers

Identify new controls needed to prevent recurrence.

Bow Tie Template

## Bow Tie Analysis

**Top Event:** [What went wrong]

### Threats (Potential Causes)
1. **[Threat 1]**
   - Likelihood: [High/Medium/Low]
   - Current Barriers: [Preventive controls]
   - Barrier Effectiveness: [Assessment]

2. **[Threat 2]**
   - Likelihood: [High/Medium/Low]
   - Current Barriers: [Preventive controls]
   - Barrier Effectiveness: [Assessment]

### Consequences (Potential Impacts)
1. **[Consequence 1]**
   - Severity: [High/Medium/Low]
   - Current Barriers: [Protective controls]
   - Barrier Effectiveness: [Assessment]

2. **[Consequence 2]**
   - Severity: [High/Medium/Low]
   - Current Barriers: [Protective controls]
   - Barrier Effectiveness: [Assessment]

### Barrier Analysis
**Effective Barriers:**
- [Barrier that worked well]
- [Why it was effective]

**Failed Barriers:**
- [Barrier that failed]
- [Why it failed]
- [How to improve]

**Missing Barriers:**
- [Needed preventive control]
- [Needed protective control]

### Recommendations
**Preventive Measures:**
1. [New barrier to prevent threat]
2. [Improvement to existing barrier]

**Protective Measures:**
1. [New barrier to mitigate consequence]
2. [Improvement to existing barrier]

---

Framework Comparison

Combining Frameworks

5 Whys + Fishbone

Use 5 Whys for initial analysis, then Fishbone to explore contributing factors.

Timeline + 5 Whys

Use Timeline to identify key decision points, then 5 Whys on critical failures.

Fishbone + Bow Tie

Use Fishbone to identify causes, then Bow Tie to develop comprehensive prevention strategy.

Quality Checklist

• [ ] Root causes address systemic issues, not symptoms
• [ ] Analysis is backed by evidence, not assumptions
• [ ] Multiple perspectives considered (technical, process, human)
• [ ] Recommendations are specific and actionable
• [ ] Analysis focuses on prevention, not blame
• [ ] Findings are validated against incident timeline
• [ ] Contributing factors are prioritized by impact
• [ ] Root causes link clearly to preventive actions

Common Anti-Patterns

• Human Error as Root Cause - Dig deeper into why human error occurred
• Single Root Cause - Complex systems usually have multiple contributing factors
• Technology-Only Focus - Consider process and organizational factors
• Blame Assignment - Focus on system improvements, not individual fault
• Generic Recommendations - Provide specific, measurable actions
• Surface-Level Analysis - Ensure you've reached true root causes

---

Last Updated: February 2026 Next Review: August 2026 Owner: SRE Team + Engineering Leadership