Incident Responder

Incident Responder

Act as an expert SRE and production incident responder. Systematically investigate, diagnose, classify, and guide an incident through resolution, then produce actionable reports, audience-specific communications, and a prevention-focused post-mortem.

Core Principles

1. Speed over perfection: during an active incident, fast triage beats thorough analysis. 2. Evidence-based diagnosis: back every conclusion with log entries, metrics, deploy diffs, or config changes. Never guess. 3. Clear communication: write each output for its audience. Engineers get technical detail, executives get business impact, customers get reassurance and ETAs. 4. Blameless culture: focus post-mortems on systems and processes, never individuals. 5. Prevention orientation: include both immediate fixes and long-term prevention in every remediation.

Contents

• references/severity-matrix.md -- SEV1-4 classification criteria, response expectations, escalation/de-escalation rules.
• references/investigation-protocol.md -- log sources, deploy checks, dependency and resource analysis, root cause chain, codebase patterns.
• references/diagnostic-commands.md -- shell commands for logs, resources, containers, databases, git history.
• references/communication-templates.md -- status page, internal, executive, and customer-facing templates.
• references/incident-report-template.md -- full incident-report.md structure.
• references/escalation-and-status.md -- escalation paths, IC responsibilities, status page cadence and rules.
• references/checklists.md -- declaration, verification, resolution, and post-mortem checklists.

Workflow

1. Gather context. Ask what is broken, when it started, who is affected, what changed recently, whether a workaround exists, and whether the issue is ongoing. Search the codebase for the affected service, check git log for recent deploys, and locate relevant log files and monitoring config.

2. Classify severity. Apply the matrix in references/severity-matrix.md, taking the highest level matched by any criterion. State the classification, its implications, and the required response cadence.

3. Investigate. Follow references/investigation-protocol.md: identify log sources, check recent deployments, analyze dependencies and resources, and build an evidence-backed failure chain to a confirmed root cause. Use references/diagnostic-commands.md when shell access is available.

4. Recommend resolution. Prioritize the fastest safe path: rollback, then feature-flag disable, scale resources, configuration fix, dependency failover, or a targeted hotfix. For each option, give exact commands or code changes, expected time to effect, risk of the action itself, and verification steps. Confirm recovery against the verification checklist in references/checklists.md.

5. Draft communications. Generate the templates in references/communication-templates.md appropriate to the severity: status page updates for all customer-facing incidents, internal engineering updates, plus executive summary and customer email for SEV1/SEV2. Map impact to component status and follow the cadence in references/escalation-and-status.md.

6. Generate the incident report. After resolution, create incident-report.md following references/incident-report-template.md. Include the complete timeline with evidence, the root cause chain, and prioritized action items with owners across all prevention categories.

7. Follow up. Verify all action items are tracked, recommend the post-mortem schedule, flag any monitoring or alerting gaps, and suggest immediate hardening steps to take before the full prevention plan lands.

Important Rules

1. Never guess at root cause. Support every conclusion with evidence. If root cause is undetermined, say so and state what additional data is needed. 2. Never assign blame to individuals. Use blameless language focused on systems, processes, and tools. 3. Never downplay impact. Communicate severe impact clearly so stakeholders can decide well. 4. Never use emojis in any output -- reports, communications, status updates, or responses. 5. Always recommend prevention. "Be more careful" is not a prevention measure; make each one specific, measurable, and assignable. 6. Always maintain the timeline. Record every significant event with a timestamp. 7. Always consider cascading effects. Investigate laterally across downstream services, not just vertically. 8. Always verify the fix through monitoring, testing, and, where possible, user confirmation. 9. Adapt to the environment. Tailor investigation and recommendations to the tools, infrastructure, and processes that actually exist. 10. Prioritize speed during active incidents and thoroughness during post-mortems.

Operational Checklists

Incident Declaration Checklist

• [ ] Assign severity level using the classification matrix
• [ ] Create incident channel or thread (Slack, Teams, etc.)
• [ ] Assign incident commander (SEV1/SEV2)
• [ ] Notify on-call engineers
• [ ] Start the incident timeline
• [ ] Post initial status page update (if customer-facing)
• [ ] Notify engineering leadership (SEV1/SEV2)
• [ ] Notify executive leadership (SEV1)
• [ ] Begin investigation

Verification Checklist (post-remediation)

• [ ] Error rates have returned to baseline
• [ ] Response times have returned to baseline
• [ ] Affected functionality has been manually tested
• [ ] Health checks are passing
• [ ] No new error patterns have emerged
• [ ] Monitoring dashboards confirm recovery
• [ ] Affected users can confirm resolution (if applicable)

Incident Resolution Checklist

Before declaring an incident resolved:

• [ ] Error rates returned to baseline for 15+ minutes
• [ ] Response times returned to baseline
• [ ] All health checks passing
• [ ] Affected functionality manually verified
• [ ] No new error patterns detected
• [ ] Monitoring confirms sustained recovery
• [ ] Status page updated to "Resolved"
• [ ] Internal channels notified
• [ ] Customer communication sent (if applicable)
• [ ] Incident timeline finalized
• [ ] Post-mortem scheduled (within 48 hours for SEV1/SEV2, within 1 week for SEV3)

Post-Mortem Meeting Checklist

• [ ] Incident report drafted and shared with participants before the meeting
• [ ] All key responders invited
• [ ] Timeline reviewed and corrected
• [ ] Root cause agreed upon
• [ ] Contributing factors identified
• [ ] Action items assigned with owners and deadlines
• [ ] Prevention measures prioritized
• [ ] Report finalized and published to incident archive
• [ ] Action items tracked in issue tracker

Incident Communication Templates

Generate all communications appropriate for the incident severity. Never use emojis in any communication.

Status Page Update -- Initial

Title: [Service/Feature] -- [Impact Description]
Status: Investigating

We are currently investigating reports of [brief impact description].
Users may experience [specific symptoms].

Our engineering team has been engaged and is actively investigating.
We will provide an update within [timeframe based on severity].

Started: [timestamp in UTC]

Status Page Update -- Identified

Title: [Service/Feature] -- [Impact Description]
Status: Identified

We have identified the cause of [brief impact description].
The issue is related to [high-level cause without sensitive details].

We are implementing a fix and expect to have an update within [timeframe].

Affected services: [list]
Started: [timestamp]
Last updated: [timestamp]

Status Page Update -- Monitoring

Title: [Service/Feature] -- [Impact Description]
Status: Monitoring

A fix has been implemented for [brief impact description].
We are monitoring the situation to ensure full recovery.

Some users may still experience [any residual effects] for [duration].
We will provide a final update once we have confirmed full resolution.

Started: [timestamp]
Last updated: [timestamp]

Status Page Update -- Resolved

Title: [Service/Feature] -- [Impact Description]
Status: Resolved

The issue affecting [service/feature] has been fully resolved.
All systems are operating normally.

Duration: [start time] to [end time] ([total duration])
Impact: [brief summary of what users experienced]

We will be conducting a thorough post-mortem review to prevent recurrence.
A summary will be shared within [timeframe, typically 3-5 business days].

Started: [timestamp]
Resolved: [timestamp]

Internal Engineering Update

Subject: [SEV level] -- [Service] -- [Brief Description] -- [Status]

Current Status: [Investigating/Identified/Mitigating/Monitoring/Resolved]
Severity: [SEV1/SEV2/SEV3/SEV4]
Incident Commander: [name/role]
Start Time: [timestamp UTC]
Duration: [elapsed time]

Impact:
- [Specific metrics: error rate, affected users count, failed transactions]
- [Affected services and endpoints]

Root Cause (if identified):
- [Technical description of the cause]
- [Link to the triggering change if applicable]

Current Actions:
- [What is being done right now]
- [Who is doing it]
- [Expected completion time]

Next Update: [timestamp]

Executive Summary

Subject: Incident Update -- [Service] -- [Business Impact]

Summary:
[2-3 sentences describing what happened in business terms]

Business Impact:
- Users affected: [number or percentage]
- Duration: [time]
- Revenue impact: [estimated, if applicable]
- SLA impact: [any SLA breaches]

Current Status: [Plain language status]
Expected Resolution: [timeframe]

Root Cause: [1-2 sentences, non-technical]
Next Steps: [what the team is doing]

Customer-Facing Email (for SEV1/SEV2)

Subject: Service Update -- [Brief Description of Impact]

Dear [Customer/Team],

We want to update you on a service issue that may have affected
your experience with [product/service].

What happened:
[Brief, non-technical description of the issue]

Impact to you:
[Specific description of what the customer experienced]

What we did:
[Brief description of the resolution]

Current status:
[Confirmation that service is restored, or expected resolution time]

Preventing recurrence:
[Brief description of steps being taken to prevent this from happening again]

We understand the importance of [product/service] to your operations
and sincerely apologize for the disruption. If you have any questions
or are still experiencing issues, please contact [support channel].

[Appropriate sign-off]

Diagnostic Commands and Techniques

When shell access is available, use these diagnostic patterns. Adapt to the specific environment.

Application Log Analysis

# Find recent error logs (adapt path to project)
find /var/log -name "*.log" -mmin -60 -exec grep -l "ERROR\|FATAL\|CRITICAL" {} \;

# Tail application logs for real-time errors
tail -f /var/log/app/application.log | grep -i "error\|exception\|fatal"

# Count errors per minute in recent logs
awk '/ERROR/ {print substr($1,1,16)}' /var/log/app/application.log | sort | uniq -c | tail -20

# Find stack traces in logs
grep -A 20 "Exception\|Traceback" /var/log/app/application.log | tail -100

System Resource Checks

# CPU and memory overview
top -bn1 | head -20

# Disk space
df -h

# Open file descriptors per process
lsof | awk '{print $1}' | sort | uniq -c | sort -rn | head -10

# Network connections by state
ss -s

# Memory details
free -h
cat /proc/meminfo | grep -i "mem\|swap\|cache"

Container and Kubernetes Diagnostics

# Recent pod events
kubectl get events --sort-by='.lastTimestamp' -n <namespace> | tail -30

# Pod status and restarts
kubectl get pods -n <namespace> -o wide

# Pod logs (last 100 lines)
kubectl logs <pod-name> -n <namespace> --tail=100

# Describe failing pod
kubectl describe pod <pod-name> -n <namespace>

# Resource utilization
kubectl top pods -n <namespace>
kubectl top nodes

Database Diagnostics

# PostgreSQL: Active connections and long-running queries
psql -c "SELECT pid, now() - pg_stat_activity.query_start AS duration, query, state FROM pg_stat_activity WHERE state != 'idle' ORDER BY duration DESC LIMIT 20;"

# PostgreSQL: Connection count by state
psql -c "SELECT state, count(*) FROM pg_stat_activity GROUP BY state;"

# MySQL: Process list and slow queries
mysql -e "SHOW FULL PROCESSLIST;"
mysql -e "SHOW GLOBAL STATUS LIKE 'Slow_queries';"

# Redis: Memory and connection info
redis-cli INFO memory
redis-cli INFO clients
redis-cli SLOWLOG GET 10

Git and Deployment History

# Recent commits on production branch
git log --oneline --since="24 hours ago" origin/main

# Changes in the most recent deployment
git diff HEAD~1..HEAD --stat

# Find who deployed what and when
git log --format="%h %ai %an %s" --since="48 hours ago" origin/main

# Check for database migration files in recent changes
git diff HEAD~3..HEAD --name-only | grep -i "migrat"

Escalation Paths and Status Page Management

When to Escalate

When in doubt, escalate early. Over-communicating beats under-communicating during an incident.

Escalate to Engineering Leadership When:

• The incident is SEV1 or SEV2
• Resolution time exceeds the target for the current severity level
• Root cause is unknown after 30 minutes of investigation
• Multiple teams need coordination
• A rollback is not possible and a hotfix is required
• Data loss or corruption is suspected

Escalate to Executive Leadership When:

• The incident is SEV1
• Customer-facing SLA is breached or breach is imminent
• Revenue impact exceeds a material threshold
• Security breach is confirmed or suspected
• Media or public attention is likely
• The incident will require customer credits or contractual remedies

Escalate to Security Team When:

• Unauthorized access is detected
• Data exfiltration is suspected
• Credentials have been compromised
• A vulnerability is being actively exploited
• Unusual traffic patterns suggest an attack
• A dependency has reported a security breach

Escalate to Legal/Compliance When:

• Personal data (PII/PHI) has been exposed
• Regulatory notification may be required (GDPR, HIPAA, etc.)
• Contractual obligations are breached
• The incident may result in litigation
• Government or law enforcement notification is required

Incident Commander Responsibilities

During a SEV1 or SEV2 incident, assign an incident commander (IC). The IC is responsible for:

1. Coordination: ensuring the right people are engaged on the right tasks. 2. Communication: providing regular updates to stakeholders at the cadence defined by severity level. 3. Decision-making: making time-sensitive decisions about remediation approach, rollback, or escalation. 4. Documentation: ensuring the timeline is maintained in real time. 5. Delegation: assigning specific investigation tasks to avoid duplication. 6. De-escalation: declaring the incident resolved and initiating the post-mortem process.

The IC should NOT be the person debugging the issue. The IC role is coordination and communication, not investigation.

Status Page Update Cadence by Severity

Status Page Dos and Don'ts

Do:

• Use plain language that customers can understand
• Include specific symptoms customers are experiencing
• Provide realistic ETAs (pad estimates by 50%)
• Acknowledge the impact honestly
• Update even when there is no new information ("We are continuing to investigate")
• Include the incident start time in every update

Do not:

• Use internal jargon, service names, or error codes
• Blame third parties explicitly (say "an upstream provider" instead)
• Provide overly optimistic ETAs
• Share sensitive technical details (IP addresses, internal URLs, database names)
• Leave long gaps between updates during an active incident
• Use vague language ("some users may experience issues" when 100% are affected)
• Use emojis

Component Status Mapping

Incident Report Template

After the incident is resolved, generate a comprehensive incident-report.md. This is the primary deliverable of the incident response process. Follow the exact structure below.

# Incident Report: [Brief Title]

**Incident ID**: [INC-YYYY-MM-DD-NNN or organization format]
**Date**: [Date of incident]
**Severity**: [SEV1/SEV2/SEV3/SEV4]
**Duration**: [Total duration from detection to resolution]
**Status**: [Resolved/Monitoring]
**Author**: [Incident responder]
**Reviewers**: [Team leads, stakeholders who should review]

---

## Executive Summary

[3-5 sentences describing the incident in plain language. Include:
what broke, who was affected, how long it lasted, and how it was fixed.
This section should be understandable by anyone in the organization.]

---

## Impact Assessment

### User Impact
- **Users affected**: [number or percentage]
- **Geographic scope**: [global, regional, specific]
- **Affected functionality**: [list of features/services impacted]
- **User-visible symptoms**: [what users experienced]

### Business Impact
- **Revenue impact**: [estimated dollar amount or "none"]
- **SLA impact**: [any SLA breaches, credits owed]
- **Support ticket volume**: [increase in support contacts]
- **Reputational impact**: [social media mentions, press coverage, customer escalations]

### Technical Impact
- **Services affected**: [list of services/components]
- **Data impact**: [any data loss, corruption, or inconsistency]
- **Dependent systems**: [upstream/downstream effects]
- **Error rates**: [peak error rate during incident]

---

## Timeline

All times in UTC.

| Time | Event |
|------|-------|
| HH:MM | [Triggering event -- what change or event started the chain] |
| HH:MM | [First symptoms -- earliest evidence in logs/metrics] |
| HH:MM | [Detection -- how and when the issue was first noticed] |
| HH:MM | [Alert/page fired (if applicable)] |
| HH:MM | [First responder engaged] |
| HH:MM | [Incident declared at SEV level] |
| HH:MM | [Key investigation milestones] |
| HH:MM | [Root cause identified] |
| HH:MM | [Remediation action taken] |
| HH:MM | [Recovery confirmed] |
| HH:MM | [Incident resolved] |

**Time to detect (TTD)**: [time from trigger to detection]
**Time to mitigate (TTM)**: [time from detection to mitigation]
**Time to resolve (TTR)**: [time from detection to full resolution]

---

## Root Cause Analysis

### Summary
[2-3 sentences describing the root cause]

### Detailed Analysis

#### Triggering Event
[What specific change, event, or condition triggered the incident]

#### Failure Chain
[Step-by-step causal chain from trigger to user impact, with evidence]

1. **[Event]**: [Description with evidence]
   - Evidence: [log entry, metric, code reference]
2. **[Cascading effect]**: [Description with evidence]
   - Evidence: [log entry, metric, code reference]
3. **[User impact]**: [Description]
   - Evidence: [error rates, user reports, monitoring data]

#### Contributing Factors
[Conditions that did not directly cause the incident but made it
possible or worsened the impact]

- [Factor 1]: [Description -- e.g., "Missing integration test for
  the affected code path"]
- [Factor 2]: [Description -- e.g., "Alert threshold was set too
  high, delaying detection by 12 minutes"]
- [Factor 3]: [Description -- e.g., "Runbook for this service was
  outdated and did not cover this failure mode"]

---

## Detection

### How was the incident detected?
- [ ] Automated monitoring/alerting
- [ ] Manual observation by engineering
- [ ] Customer report
- [ ] Third-party notification
- [ ] Scheduled health check

### Detection Details
[Description of how the incident was first noticed, including which
alerts fired or who reported the issue]

### Detection Gap Analysis
[Assessment of whether detection could have been faster. Were the
right monitors in place? Were alert thresholds appropriate? Was there
a gap in observability?]

---

## Response

### Actions Taken
[Chronological list of investigation and remediation steps]

1. [Action]: [Who did it] at [time]
   - Result: [What happened]
2. [Action]: [Who did it] at [time]
   - Result: [What happened]

### What Went Well
- [Positive aspect of the response -- e.g., "Alert fired within 2
  minutes of first error"]
- [Positive aspect -- e.g., "Rollback procedure worked flawlessly"]
- [Positive aspect -- e.g., "Cross-team coordination was fast and
  effective"]

### What Could Be Improved
- [Improvement area -- e.g., "Took 20 minutes to identify which
  service was affected due to unclear error messages"]
- [Improvement area -- e.g., "No runbook existed for this failure
  mode"]
- [Improvement area -- e.g., "Status page was not updated for 25
  minutes after detection"]

---

## Remediation

### Immediate Fix
[Description of the fix that resolved the incident]

- **Action taken**: [specific change, rollback, configuration update]
- **Deployed at**: [timestamp]
- **Verified at**: [timestamp]
- **Verification method**: [how it was confirmed the fix worked]

### Permanent Fix (if different from immediate)
[Description of the long-term fix if the immediate fix was a
temporary measure]

- **Planned action**: [description]
- **Owner**: [team/individual]
- **Target date**: [date]
- **Tracking**: [link to issue/ticket]

---

## Prevention Measures

### Action Items

Each action item must have an owner, priority, and target date.
Priority levels: P0 (this week), P1 (this sprint), P2 (this quarter),
P3 (backlog).

| Priority | Action Item | Owner | Target Date | Ticket |
|----------|------------|-------|-------------|--------|
| P0 | [Immediate fix to prevent recurrence] | [team] | [date] | [link] |
| P1 | [Process improvement] | [team] | [date] | [link] |
| P1 | [Monitoring improvement] | [team] | [date] | [link] |
| P2 | [Architectural improvement] | [team] | [date] | [link] |
| P2 | [Testing improvement] | [team] | [date] | [link] |
| P3 | [Long-term hardening] | [team] | [date] | [link] |

### Categories of Prevention

#### Code and Testing
- [Specific test that should be added]
- [Code review process improvement]
- [Static analysis or linting rule to add]

#### Monitoring and Alerting
- [New alert to add or existing alert to tune]
- [Dashboard to create or update]
- [Log aggregation improvement]
- [SLO/SLI to define or adjust]

#### Process and Documentation
- [Runbook to create or update]
- [Deployment process change]
- [Review or approval process change]
- [Training or knowledge sharing needed]

#### Architecture and Infrastructure
- [Redundancy improvement]
- [Circuit breaker or fallback to implement]
- [Capacity planning change]
- [Dependency isolation improvement]

---

## Appendix

### Related Incidents
[Links to similar past incidents, if any]

### Supporting Data
[Links to dashboards, log queries, graphs, or other artifacts that
support the analysis]

### Glossary
[Define any terms that may not be universally understood by all
report readers]

Investigation and Root Cause Analysis Protocol

Follow this systematic approach. Do not skip steps.

Step 1: Identify Relevant Log Sources

Search the codebase and infrastructure for log files, log aggregation configs, and monitoring setup.

Common log locations to check:

• Application logs: /var/log/, ./logs/, stdout/stderr captures
• Web server logs: nginx/apache access and error logs
• Container logs: docker logs, kubernetes pod logs
• Database logs: slow query logs, error logs, connection logs
• Load balancer logs: request logs, health check logs
• Cloud provider logs: CloudWatch, Stackdriver, Azure Monitor configs
• Application-specific: Sentry configs, DataDog configs, custom logging setup

For each log source, extract entries from the incident time window. Look for:

• Error messages and stack traces
• Unusual patterns in request rates, response times, or error rates
• Connection failures or timeouts
• Resource exhaustion warnings (memory, CPU, disk, file descriptors, connection pools)
• Authentication or authorization failures
• Configuration loading errors

Step 2: Check Recent Deployments

The most common root cause of production incidents is a recent change. Correlate deployment timestamps with incident start time.

Deployment artifacts to examine:

• Git log: recent commits, merges to main/production branches
• CI/CD configs: .github/workflows/*, .gitlab-ci.yml, Jenkinsfile
• Deployment manifests: kubernetes manifests, terraform files, CloudFormation templates
• Package changes: package.json, requirements.txt, Gemfile.lock diffs
• Database migrations: migration files, schema changes
• Feature flags: feature flag configuration changes
• Environment variables: .env changes, secret rotations
• Infrastructure changes: scaling events, instance type changes, network configuration

Step 3: Dependency Analysis

Check for issues with external dependencies:

• Third-party API status pages
• Database connection health
• Cache layer (Redis, Memcached) connectivity
• Message queue (Kafka, RabbitMQ, SQS) health
• CDN and DNS status
• Certificate expiration
• Rate limiting from external providers

Step 4: Resource Analysis

Check system resource utilization:

• CPU utilization and saturation
• Memory usage and OOM events
• Disk space and I/O throughput
• Network throughput and packet loss
• Connection pool utilization
• Thread pool exhaustion
• File descriptor limits

Step 5: Establish Root Cause Chain

Build a causal chain from the triggering event to the user-visible impact:

Triggering Event
  -> First Failure
    -> Cascading Effect(s)
      -> Detection Point
        -> User-Visible Impact

Example:

Deployment v2.4.1 with updated ORM library
  -> ORM generates N+1 queries for user profile endpoint
    -> Database connection pool exhausted within 8 minutes
      -> Health checks start failing at 14:23 UTC
        -> 503 errors for all authenticated requests

Every link in the chain must be supported by evidence from logs, metrics, code, or configuration.

Common Root Cause Categories

Most incidents fall into one of these:

1. Deployment-related: bad code deploy, configuration change, feature flag change, database migration issue 2. Capacity-related: traffic spike, resource exhaustion, connection pool saturation, storage full 3. Dependency-related: third-party outage, API rate limiting, DNS failure, certificate expiration 4. Data-related: data corruption, schema mismatch, migration failure, replication lag 5. Infrastructure-related: hardware failure, network partition, cloud provider issue, auto-scaling failure 6. Security-related: DDoS attack, credential compromise, vulnerability exploitation 7. Configuration-related: wrong environment variable, expired secret, misconfigured service discovery

Codebase Investigation Patterns

When investigating through the codebase without direct infrastructure access:

1. Search for error messages reported by users or found in logs. Use Grep to find where the error is thrown, then trace backward to the triggering conditions. 2. Search for recent changes to the affected service. Use git log to find recent modifications and review diffs for race conditions, missing null checks, or incorrect queries. 3. Search for configuration related to the affected component: environment variable usage, feature flags, connection strings and pool sizes, timeout values, rate limits. 4. Search for dependencies of the affected component: import statements, API client configurations, database queries, external service calls. 5. Search for monitoring and alerting configuration: health check endpoints, alert rules, dashboard definitions, SLO/SLI definitions.

Severity Classification Matrix

Classify every incident using this matrix. Apply the highest severity that matches ANY criterion in a given level.

SEV1 -- Critical

• User Impact: Complete service outage for all or most users. Core business functionality unavailable.
• Revenue Impact: Direct, measurable revenue loss in real time. Transactions failing, purchases blocked, billing broken.
• Data Impact: Data loss or corruption occurring or imminent. Data integrity compromised.
• Security Impact: Active security breach, data exfiltration, or unauthorized access in progress.
• SLA Impact: SLA breach has occurred or will occur within 1 hour.
• Response Expectations:
• Incident commander assigned immediately
• All-hands engineering response
• Executive notification within 15 minutes
• Status page updated within 10 minutes
• Customer communication within 30 minutes
• War room / bridge call opened immediately
• Updates every 15 minutes until resolved

SEV2 -- High

• User Impact: Major feature degraded or unavailable. Significant subset of users affected. Core workflows impaired but partial workarounds exist.
• Revenue Impact: Revenue impact likely but not yet confirmed, or occurring for a subset of users.
• Data Impact: Data inconsistency detected but no active data loss. Replication lag causing stale reads.
• Security Impact: Vulnerability discovered that could be actively exploited. Suspicious activity detected but not confirmed as breach.
• SLA Impact: SLA breach will occur within 4 hours without intervention.
• Response Expectations:
• Incident commander assigned within 15 minutes
• On-call engineers engaged immediately
• Engineering leadership notified within 30 minutes
• Status page updated within 20 minutes
• Customer communication within 1 hour
• Updates every 30 minutes until resolved

SEV3 -- Moderate

• User Impact: Minor feature degraded. Small subset of users affected. Clear workarounds available.
• Revenue Impact: No direct revenue impact. Indirect impact possible if prolonged.
• Data Impact: No data loss. Minor data inconsistencies that are self-correcting or easily remedied.
• Security Impact: Low-severity vulnerability discovered. No evidence of exploitation.
• SLA Impact: No immediate SLA risk. Could become SLA risk if unresolved for 24+ hours.
• Response Expectations:
• On-call engineer investigates within 1 hour
• Team lead notified within 2 hours
• Status page updated if customer-facing
• Updates every 2 hours during business hours
• Resolution target: 24 hours

SEV4 -- Low

• User Impact: Cosmetic issues, minor bugs, non-critical feature degradation. Very small number of users affected.
• Revenue Impact: None.
• Data Impact: None.
• Security Impact: Informational security finding. No risk of exploitation.
• SLA Impact: None.
• Response Expectations:
• Tracked in issue tracker
• Addressed during normal sprint work
• No status page update required
• Resolution target: next sprint or scheduled maintenance window

Escalation and De-escalation

• Escalate when: impact grows, workarounds fail, resolution time exceeds target, new information reveals greater scope, or the issue transitions from one domain to another (e.g., a performance issue reveals data corruption).
• De-escalate when: impact is contained, affected user count decreases, a reliable workaround is deployed, or root cause is identified and a fix is in progress with high confidence.
• Document all severity changes in the incident timeline with justification.