Managing Incidents

Incident Management

Provide end-to-end incident management guidance covering detection, response, communication, and learning. Emphasizes SRE culture, blameless post-mortems, and structured processes for high-reliability operations.

When to Use This Skill

Apply this skill when:

• Setting up incident response processes for a team
• Designing on-call rotations and escalation policies
• Creating runbooks for common failure scenarios
• Conducting blameless post-mortems after incidents
• Implementing incident communication protocols (internal and external)
• Choosing incident management tooling and platforms
• Improving MTTR and incident frequency metrics

Core Principles

Incident Management Philosophy

Declare Early and Often: Do not wait for certainty. Declaring an incident enables coordination, can be downgraded if needed, and prevents delayed response.

Mitigation First, Root Cause Later: Stop customer impact immediately (rollback, disable feature, failover). Debug and fix root cause after stability restored.

Blameless Culture: Assume good intentions. Focus on how systems failed, not who failed. Create psychological safety for honest learning.

Clear Command Structure: Assign Incident Commander (IC) to own coordination. IC delegates tasks but does not do hands-on debugging.

Communication is Critical: Internal coordination via dedicated channels, external transparency via status pages. Update stakeholders every 15-30 minutes during critical incidents.

Severity Classification

Standard severity levels with response times:

SEV0 (P0) - Critical Outage:

• Impact: Complete service outage, critical data loss, payment processing down
• Response: Page immediately 24/7, all hands on deck, executive notification
• Example: API completely down, entire customer base affected

SEV1 (P1) - Major Degradation:

• Impact: Major functionality degraded, significant customer subset affected
• Response: Page during business hours, escalate off-hours, IC assigned
• Example: 15% error rate, critical feature unavailable

SEV2 (P2) - Minor Issues:

• Impact: Minor functionality impaired, edge case bug, small user subset
• Response: Email/Slack alert, next business day response
• Example: UI glitch, non-critical feature slow

SEV3 (P3) - Low Impact:

• Impact: Cosmetic issues, no customer functionality affected
• Response: Ticket queue, planned sprint
• Example: Visual inconsistency, documentation error

For detailed severity decision framework and interactive classifier, see references/severity-classification.md.

Incident Roles

Incident Commander (IC):

• Owns overall incident response and coordination
• Makes strategic decisions (rollback vs. debug, when to escalate)
• Delegates tasks to responders (does NOT do hands-on debugging)
• Declares incident resolved when stability confirmed

Communications Lead:

• Posts status updates to internal and external channels
• Coordinates with stakeholders (executives, product, support)
• Drafts post-incident customer communication
• Cadence: Every 15-30 minutes for SEV0/SEV1

Subject Matter Experts (SMEs):

• Hands-on debugging and mitigation
• Execute runbooks and implement fixes
• Provide technical context to IC

Scribe:

• Documents timeline, actions, decisions in real-time
• Records incident notes for post-mortem reconstruction

Assign roles based on severity:

• SEV2/SEV3: Single responder
• SEV1: IC + SME(s)
• SEV0: IC + Communications Lead + SME(s) + Scribe

For detailed role responsibilities, see references/incident-roles.md.

On-Call Management

Rotation Patterns

Primary + Secondary:

• Primary: First responder
• Secondary: Backup if primary doesn't ack within 5 minutes
• Rotation length: 1 week (optimal balance)

Follow-the-Sun (24/7):

• Team A: US hours, Team B: Europe hours, Team C: Asia hours
• Benefit: No night shifts, improved work-life balance
• Requires: Multiple global teams

Tiered Escalation:

• Tier 1: Junior on-call (common issues, runbook-driven)
• Tier 2: Senior on-call (complex troubleshooting)
• Tier 3: Team lead/architect (critical decisions)

Best Practices

• Rotation length: 1 week per rotation
• Handoff ceremony: 30-minute call to discuss active issues
• Compensation: On-call stipend + time off after major incidents
• Tooling: PagerDuty, Opsgenie, or incident.io
• Limits: Max 2-3 pages per night; escalate if exceeded

Incident Response Workflow

Standard incident lifecycle:

Detection → Triage → Declaration → Investigation
  ↓
Mitigation → Resolution → Monitoring → Closure
  ↓
Post-Mortem (within 48 hours)

Key Decision Points

When to Declare: When in doubt, declare (can always downgrade severity)

When to Escalate:

• No progress after 30 minutes
• Severity increases (SEV2 → SEV1)
• Specialized expertise needed

When to Close:

• Issue resolved and stable for 30+ minutes
• Monitoring shows all metrics at baseline
• No customer-reported issues

For complete workflow details, see references/incident-workflow.md.

Communication Protocols

Internal Communication

Incident Slack Channel:

• Format: #incident-YYYY-MM-DD-topic-description
• Pin: Severity, IC name, status update template, runbook links

War Room: Video call for SEV0/SEV1 requiring real-time voice coordination

Status Update Cadence:

• SEV0: Every 15 minutes
• SEV1: Every 30 minutes
• SEV2: Every 1-2 hours or at major milestones

External Communication

Status Page:

• Tools: Statuspage.io, Instatus, custom
• Stages: Investigating → Identified → Monitoring → Resolved
• Transparency: Acknowledge issue publicly, provide ETAs when possible

Customer Email:

• When: SEV0/SEV1 affecting customers
• Timing: Within 1 hour (acknowledge), post-resolution (full details)
• Tone: Apologetic, transparent, action-oriented

Regulatory Notifications:

• Data Breach: GDPR requires notification within 72 hours
• Financial Services: Immediate notification to regulators
• Healthcare: HIPAA breach notification rules

For communication templates, see examples/communication-templates.md.

Runbooks and Playbooks

Runbook Structure

Every runbook should include: 1. Trigger: Alert conditions that activate this runbook 2. Severity: Expected severity level 3. Prerequisites: System state requirements 4. Steps: Numbered, executable commands (copy-pasteable) 5. Verification: How to confirm fix worked 6. Rollback: How to undo if steps fail 7. Owner: Team/person responsible 8. Last Updated: Date of last revision

Best Practices

• Executable: Commands copy-pasteable, not just descriptions
• Tested: Run during disaster recovery drills
• Versioned: Track changes in Git
• Linked: Reference from alert definitions
• Automated: Convert manual steps to scripts over time

For runbook templates, see examples/runbooks/ directory.

Blameless Post-Mortems

Blameless Culture Tenets

Assume Good Intentions: Everyone made the best decision with information available.

Focus on Systems: Investigate how processes failed, not who failed.

Psychological Safety: Create environment where honesty is rewarded.

Learning Opportunity: Incidents are gifts of organizational knowledge.

Post-Mortem Process

1. Schedule Review (Within 48 Hours): While memory is fresh

2. Pre-Work: Reconstruct timeline, gather metrics/logs, draft document

3. Meeting Facilitation:

• Timeline walkthrough
• 5 Whys Analysis to identify systemic root causes
• What Went Well / What Went Wrong
• Define action items with owners and due dates

4. Post-Mortem Document:

• Sections: Summary, Timeline, Root Cause, Impact, What Went Well/Wrong, Action Items
• Distribution: Engineering, product, support, leadership
• Storage: Archive in searchable knowledge base

5. Follow-Up: Track action items in sprint planning

For detailed facilitation guide and template, see references/blameless-postmortems.md and examples/postmortem-template.md.

Alert Design Principles

Actionable Alerts Only:

• Every alert requires human action
• Include graphs, runbook links, recent changes
• Deduplicate related alerts
• Route to appropriate team based on service ownership

Preventing Alert Fatigue:

• Audit alerts quarterly: Remove non-actionable alerts
• Increase thresholds for noisy metrics
• Use anomaly detection instead of static thresholds
• Limit: Max 2-3 pages per night

Tool Selection

Incident Management Platforms

PagerDuty:

• Best for: Established enterprises, complex escalation policies
• Cost: $19-41/user/month
• When: Team size 10+, budget $500+/month

Opsgenie:

• Best for: Atlassian ecosystem users, flexible routing
• Cost: $9-29/user/month
• When: Using Atlassian products, budget $200-500/month

incident.io:

• Best for: Modern teams, AI-powered response, Slack-native
• When: Team size 5-50, Slack-centric culture

For detailed tool comparison, see references/tool-comparison.md.

Status Page Solutions

Statuspage.io: Most trusted, easy setup ($29-399/month) Instatus: Budget-friendly, modern design ($19-99/month)

Metrics and Continuous Improvement

Key Incident Metrics

MTTA (Mean Time To Acknowledge):

• Target: < 5 minutes for SEV1
• Improvement: Better on-call coverage

MTTR (Mean Time To Recovery):

• Target: < 1 hour for SEV1
• Improvement: Runbooks, automation

MTBF (Mean Time Between Failures):

• Target: > 30 days for critical services
• Improvement: Root cause fixes

Incident Frequency:

• Track: SEV0, SEV1, SEV2 counts per month
• Target: Downward trend

Action Item Completion Rate:

• Target: > 90%
• Improvement: Sprint integration, ownership clarity

Continuous Improvement Loop

Incident → Post-Mortem → Action Items → Prevention
   ↑                                          ↓
   └──────────── Fewer Incidents ─────────────┘

Decision Frameworks

Severity Classification Decision Tree

Is production completely down or critical data at risk?
├─ YES → SEV0
└─ NO  → Is major functionality degraded?
          ├─ YES → Is there a workaround?
          │        ├─ YES → SEV1
          │        └─ NO  → SEV0
          └─ NO  → Are customers impacted?
                   ├─ YES → SEV2
                   └─ NO  → SEV3

Use interactive classifier: python scripts/classify-severity.py

Escalation Matrix

For detailed escalation guidance, see references/escalation-matrix.md.

Mitigation vs. Root Cause

Prioritize Mitigation When:

• Active customer impact ongoing
• Quick fix available (rollback, disable feature)

Prioritize Root Cause When:

• Customer impact already mitigated
• Fix requires careful analysis

Default: Mitigation first (99% of cases)

Anti-Patterns to Avoid

• Delayed Declaration: Waiting for certainty before declaring incident
• Skipping Post-Mortems: "Small" incidents still provide learning
• Blame Culture: Punishing individuals prevents systemic learning
• Ignoring Action Items: Post-mortems without follow-through waste time
• No Clear IC: Multiple people leading creates confusion
• Alert Fatigue: Noisy, non-actionable alerts cause on-call to ignore pages
• Hands-On IC: IC should delegate debugging, not do it themselves

Implementation Checklist

Phase 1: Foundation (Week 1)

• [ ] Define severity levels (SEV0-SEV3)
• [ ] Choose incident management platform
• [ ] Set up basic on-call rotation
• [ ] Create incident Slack channel template

Phase 2: Processes (Weeks 2-3)

• [ ] Create first 5 runbooks for common incidents
• [ ] Set up status page
• [ ] Train team on incident response
• [ ] Conduct tabletop exercise

Phase 3: Culture (Weeks 4+)

• [ ] Conduct first blameless post-mortem
• [ ] Establish post-mortem cadence
• [ ] Implement MTTA/MTTR dashboards
• [ ] Track action items in sprint planning

Phase 4: Optimization (Months 3-6)

• [ ] Automate incident declaration
• [ ] Implement runbook automation
• [ ] Monthly disaster recovery drills
• [ ] Quarterly incident trend reviews

Integration with Other Skills

Observability: Monitoring alerts trigger incidents → Use incident-management for response

Disaster Recovery: DR provides recovery procedures → Incident-management provides operational response

Security Incident Response: Similar process with added compliance/forensics

Infrastructure-as-Code: IaC enables fast recovery via automated rebuild

Performance Engineering: Performance incidents trigger response → Performance team investigates post-mitigation

Examples and Templates

Runbook Templates:

• examples/runbooks/database-failover.md
• examples/runbooks/cache-invalidation.md
• examples/runbooks/ddos-mitigation.md

Post-Mortem Template:

• examples/postmortem-template.md - Complete blameless post-mortem structure

Communication Templates:

• examples/communication-templates.md - Status updates, customer emails

On-Call Handoff:

• examples/oncall-handoff-template.md - Weekly handoff format

Integration Scripts:

• examples/integrations/pagerduty-slack.py
• examples/integrations/statuspage-auto-update.py
• examples/integrations/postmortem-generator.py

Scripts

Interactive Severity Classifier:

python scripts/classify-severity.py

Asks questions to determine appropriate severity level based on impact and urgency.

Further Reading

Books:

• Google SRE Book: "Postmortem Culture" (Chapter 15)
• "The Phoenix Project" by Gene Kim
• "Site Reliability Engineering" (Full book)

Online Resources:

• Atlassian: "How to Run a Blameless Postmortem"
• PagerDuty: "Incident Response Guide"
• Google SRE: "Postmortem Culture: Learning from Failure"

Standards:

• Incident Command System (ICS) - FEMA standard adapted for tech
• ITIL Incident Management - Traditional IT service management

Incident Communication Templates

Overview

Communication templates for internal coordination and external customer updates during incidents. Customize for your organization while maintaining clarity and consistency.

---

Internal Status Update Template (Slack)

Use: Post in incident Slack channel every 15-30 minutes during active incidents

**[TIMESTAMP] Update #[N] - [Status]**

**Current Status:** [Investigating | Identified | Implementing Fix | Monitoring | Resolved]

**Issue:** [One-sentence description of what's broken]

**Impact:** [Who/what is affected, e.g., "~5,000 customers unable to complete checkout"]

**Progress:** [What we've learned/done since last update]

**Next Steps:** [What we're doing next]

**ETA:** [Expected resolution time or "Unknown - investigating"]

**Next Update:** [TIMESTAMP + 15/30 min]

Example:

14:45 PST Update #3 - Implementing Fix

Current Status: Implementing Fix

Issue: API experiencing elevated error rates (15% of requests failing)

Impact: ~5,000 customers unable to complete checkout

Progress: Root cause identified - database connection pool exhausted. Mitigation deployed: increased pool size from 100 to 200 connections. Error rate dropped from 15% to 5%.

Next Steps: Monitoring error rate trend. Investigating secondary issue with retry logic causing thundering herd.

ETA: Expect full resolution within 30 minutes

Next Update: 15:00 PST

---

External Status Page Update Template

Use: Post to public status page (Statuspage.io, Instatus, etc.)

Status: Investigating

**[Timestamp] Investigating - [Component Name]**

We are investigating reports of [brief issue description]. Customers may experience [specific impact, e.g., "slow page loads" or "checkout errors"]. We are working to identify the root cause and will provide updates as we learn more.

Example:

Dec 3, 14:20 PST Investigating - API

We are investigating reports of elevated API error rates. Customers may experience errors when attempting to complete checkout. We are working to identify the root cause and will provide updates every 15 minutes.

---

Status: Identified

**[Timestamp] Identified - [Component Name]**

We have identified the issue: [brief technical description in plain language]. We are implementing a fix and expect resolution within [timeframe]. Customers are still experiencing [impact].

Example:

Dec 3, 14:30 PST Identified - API

We have identified the issue: our database connection pool reached capacity during higher-than-expected traffic. We are increasing the pool size and expect resolution within 30 minutes. Customers are still experiencing occasional checkout errors (~5% failure rate, down from 15%).

---

Status: Monitoring

**[Timestamp] Monitoring - [Component Name]**

The fix has been applied and [metric] has returned to normal levels. We are monitoring the situation to ensure stability before marking this incident as resolved.

Example:

Dec 3, 15:00 PST Monitoring - API

The fix has been applied and API error rates have returned to normal levels (<0.1%). We are monitoring the situation for the next 30 minutes to ensure stability before marking this incident as resolved.

---

Status: Resolved

**[Timestamp] Resolved - [Component Name]**

This incident has been resolved. [Component] is operating normally. Customers should no longer experience [impact]. We will be conducting a post-mortem to prevent recurrence and will share findings soon.

Example:

Dec 3, 15:45 PST Resolved - API

This incident has been resolved. The API is operating normally. Customers should no longer experience checkout errors. We will be conducting a post-mortem to prevent recurrence and will share a summary with affected customers within 24 hours.

---

Customer Email Template (Post-Incident)

Use: Send to affected customers after SEV0/SEV1 incidents

Timing: Within 24 hours of resolution

Subject Line: [Resolved] [Service Name] Incident on [Date]

Subject: [Resolved] API Outage on December 3, 2025

Dear [Customer Name / Customers],

We experienced a service disruption on [Date] from [Start Time] to [End Time] [Timezone]. During this time, [describe customer-facing impact in plain language].

**What Happened:**
[2-3 sentences explaining what broke in non-technical terms]

**Impact:**
- Duration: [X hours/minutes]
- Affected Users: [Approximately X customers or "All users"]
- Affected Functionality: [Specific features that were unavailable]
- Data: [Confirm no data loss or describe scope if any]

**How We Resolved It:**
[1-2 sentences describing the fix applied]

**What We're Doing to Prevent This:**
We take incidents seriously and are implementing the following improvements:
1. [Prevention measure 1]
2. [Prevention measure 2]
3. [Prevention measure 3]

**We Sincerely Apologize:**
We understand how disruptive service interruptions are and apologize for the inconvenience. If you have any questions or concerns, please contact our support team at [support email/phone].

Thank you for your patience and continued trust.

Sincerely,
[Name]
[Title]
[Company]

Example:

Subject: [Resolved] API Outage on December 3, 2025

Dear Customers,

We experienced an API outage on December 3, 2025, from 2:15 PM to 3:45 PM PST. During this time, approximately 5,000 customers were unable to complete checkout transactions.

What Happened:
Our database connection pool reached capacity during higher-than-expected holiday traffic. This caused our API to reject requests and display error messages during checkout.

Impact:
- Duration: 90 minutes (2:15 PM - 3:45 PM PST)
- Affected Users: Approximately 5,000 customers
- Affected Functionality: Checkout and payment processing
- Data: No customer data was lost or compromised

How We Resolved It:
We increased our database connection pool capacity and implemented circuit breaker logic to prevent similar cascading failures.

What We're Doing to Prevent This:
1. Adding proactive monitoring alerts for connection pool saturation (before errors occur)
2. Implementing load testing for 2x traffic scenarios before all releases
3. Auto-scaling our database connection pool to handle traffic spikes

We Sincerely Apologize:
We understand how frustrating this disruption was, especially during the holiday shopping season. We apologize for the inconvenience and appreciate your patience. If you have any questions, please contact support@example.com or call 1-800-XXX-XXXX.

Thank you for your continued trust.

Sincerely,
Jane Smith
VP Engineering
Example Company

---

Executive Brief Template (Internal)

Use: Brief C-level stakeholders during/after SEV0 incidents

Format: Email or Slack DM

Subject: Executive Brief: [Incident Title]

**Incident:** [One-line description]
**Status:** [Ongoing | Resolved]
**Severity:** [SEV0 | SEV1]
**Duration:** [X hours/minutes]

**Business Impact:**
- Customers Affected: [Number or percentage]
- Revenue Impact: [$ estimate or "Investigating"]
- SLA Breach: [Yes/No]
- PR/Media Risk: [High/Medium/Low]

**Current Status:**
[2-3 sentences on where we are in incident response]

**Root Cause:**
[If identified: brief explanation | If not: "Still investigating"]

**Resolution:**
[If ongoing: "Expected resolution: [timeframe]" | If resolved: "Resolved at [time]"]

**Customer Communication:**
[Status page updated? Customer email sent? Planned?]

**Next Steps:**
1. [Action 1]
2. [Action 2]
3. [Post-mortem scheduled for [date]]

**Contact:** [IC or Incident Manager name and Slack handle]

Example:

Executive Brief: API Outage - Dec 3, 2025

Incident: API error rates reached 15%, preventing checkout
Status: Resolved
Severity: SEV1
Duration: 90 minutes (2:15 PM - 3:45 PM PST)

Business Impact:
- Customers Affected: ~5,000 (15% of active users during incident)
- Revenue Impact: Estimated $12,000 in lost transactions
- SLA Breach: Yes - 99.9% monthly SLA now at 99.85%
- PR/Media Risk: Low - proactive customer communication sent

Root Cause:
Database connection pool exhausted during 2x traffic spike (holiday shopping). Contributing factors: no proactive monitoring, aggressive retry logic amplified impact.

Resolution:
Resolved at 3:45 PM PST. Connection pool size increased, circuit breaker implemented.

Customer Communication:
- Status page updated throughout (6 updates)
- Post-incident email sent to all affected customers
- Support team briefed on expected follow-up questions

Next Steps:
1. Add proactive connection pool monitoring (by Dec 10)
2. Load test all releases for 2x traffic (checklist updated)
3. Post-mortem scheduled for Dec 5 at 10am

Contact: @bob (IC) in #incident-2025-12-03-api-outage

---

On-Call Handoff Template

Use: Weekly on-call rotation handoff (e.g., every Monday)

Format: Slack post in #on-call channel or shared Google Doc

# On-Call Handoff - Week of [Date]

**Outgoing On-Call:** @[name]
**Incoming On-Call:** @[name]
**Handoff Date:** [Date and Time]

---

## Active Issues

### [Issue 1 Title]
- **Status:** [Monitoring | Investigating | Resolved]
- **Severity:** [SEV2/SEV3 typically for active issues]
- **Description:** [1-2 sentences]
- **Next Steps:** [What incoming on-call should do]
- **Resources:** [Link to ticket, dashboard, runbook]

### [Issue 2 Title]
[Same format as above]

**No Active Issues:** All systems normal ✅

---

## Upcoming Changes

### [Deploy/Maintenance 1]
- **Date/Time:** [When it's happening]
- **Impact:** [Expected customer impact or "No downtime expected"]
- **Risk Level:** [High/Medium/Low]
- **IC Assigned:** [Yes/No - who?]
- **Runbook:** [Link if applicable]

### [Deploy/Maintenance 2]
[Same format as above]

---

## New Runbooks / Updates

- **Added:** [Runbook name and link]
- **Updated:** [Runbook name and what changed]

**No Changes:** All runbooks current ✅

---

## System Health Notes

**Overall:** [Green/Yellow/Red - brief assessment]

- **[System 1]:** [Status and any notes]
- **[System 2]:** [Status and any notes]
- **[System 3]:** [Status and any notes]

**Monitoring Issues:**
- [Any dashboards down, alert issues, blind spots]

---

## Questions / Clarifications

**Q:** [Any questions from incoming on-call]
**A:** [Answers from outgoing on-call]

---

## Access Verification

- [ ] VPN connected
- [ ] PagerDuty app configured
- [ ] AWS/GCP console access verified
- [ ] Database access verified
- [ ] Monitoring dashboards accessible
- [ ] Slack notifications enabled

---

## Contact Info

**Outgoing On-Call:** @[handle] | [phone if needed]
**Incoming On-Call:** @[handle] | [phone if needed]
**Escalation:** @[team-lead-handle] | @[senior-oncall-handle]

**Handoff Complete:** ✅ [Timestamp when both parties confirm]

Example:

# On-Call Handoff - Week of Dec 3-10, 2025

Outgoing On-Call: @alice
Incoming On-Call: @bob
Handoff Date: Monday, Dec 3, 10:00 AM PST

---

## Active Issues

### Database Replica Lag Spike
- Status: Monitoring
- Severity: SEV3
- Description: Replica lag spiking to 30 seconds every few hours, cause unknown. Primary unaffected.
- Next Steps: Monitor replica lag dashboard, escalate to @db-team if lag > 60 seconds
- Resources: https://monitoring.example.com/dashboard/db-replication

No other active issues. All systems normal ✅

---

## Upcoming Changes

### API Deploy v2.3.0
- Date/Time: Wednesday, Dec 5, 2:00 PM PST
- Impact: No downtime expected (rolling deploy)
- Risk Level: Medium (includes database migration)
- IC Assigned: Yes - @alice on standby
- Runbook: https://wiki.example.com/deploys/api-v2.3.0

### Database Maintenance Window
- Date/Time: Saturday, Dec 7, 2:00 AM PST
- Impact: Read-only mode for 30 minutes
- Risk Level: Low (routine maintenance)
- IC Assigned: No (automated)
- Runbook: RB-DB-003 (maintenance procedures)

---

## New Runbooks / Updates

- Updated: RB-DB-001 (Database Failover) - New DNS process added
- Added: RB-CACHE-001 (Redis Cache Invalidation) - New runbook for cache issues

---

## System Health Notes

Overall: Green (replica lag being monitored)

- API: Green - 99.99% uptime this week
- Database: Yellow - Replica lag spikes (under investigation)
- Cache: Green - All metrics normal
- CDN: Green - No issues

Monitoring Issues:
- None - all dashboards and alerts operational

---

## Questions / Clarifications

Q: What should I do if replica lag hits 60 seconds?
A: Page @db-team immediately. If they don't respond in 5 min, consider failing over to replica (use RB-DB-001).

Q: Is the v2.3.0 deploy high risk?
A: Medium risk due to DB migration. I'll be on Slack Wednesday afternoon as backup IC if needed.

---

## Access Verification

- [X] VPN connected
- [X] PagerDuty app configured
- [X] AWS console access verified
- [X] Database access verified
- [X] Monitoring dashboards accessible
- [X] Slack notifications enabled

---

## Contact Info

Outgoing On-Call: @alice | 555-0123 (cell)
Incoming On-Call: @bob | 555-0456 (cell)
Escalation: @team-lead | @senior-oncall

Handoff Complete: ✅ Dec 3, 10:15 AM PST

---

Regulatory Notification Template

Use: Required for data breaches, security incidents (GDPR, HIPAA, PCI)

Consult Legal Team Before Sending

GDPR Data Breach Notification (72-hour requirement)

Subject: Data Breach Notification - [Company Name]

[Regulatory Authority Name]
[Address]

Date: [Date]

Re: Personal Data Breach Notification

Dear [Authority Name],

[Company Name] is writing to notify you of a personal data breach pursuant to Article 33 of the General Data Protection Regulation (GDPR).

1. Description of the Breach:
[Date and time breach discovered]
[Nature of the breach - unauthorized access, data loss, etc.]

2. Categories and Approximate Number of Data Subjects Concerned:
[E.g., "Approximately 5,000 EU residents"]

3. Categories and Approximate Number of Personal Data Records Concerned:
[E.g., "Names, email addresses, hashed passwords"]

4. Likely Consequences of the Breach:
[Describe potential impact on individuals]

5. Measures Taken to Address the Breach:
[Actions taken to stop breach and mitigate impact]

6. Measures Taken to Mitigate Adverse Effects:
[E.g., "Password reset required for all affected accounts"]

7. Contact Point:
[Name, title, email, phone of Data Protection Officer]

We will provide further information as our investigation progresses.

Sincerely,
[Name]
[Title]
[Company]
[Contact Information]

---

Incident Closure Announcement

Use: Post in incident Slack channel when IC declares incident resolved

**🎉 INCIDENT RESOLVED 🎉**

**Incident:** [Title]
**Duration:** [X hours/minutes]
**Resolved At:** [Timestamp]

**Summary:**
[1-2 sentences on what happened and how it was fixed]

**Final Metrics:**
- MTTA: [X minutes]
- MTTR: [X minutes]
- Customer Impact: [Brief summary]

**Next Steps:**
1. Post-mortem scheduled: [Date/Time]
2. Action items will be tracked in [Jira/GitHub/etc.]
3. Customer communication: [Sent/Scheduled]

**Thank You:**
Thank you to everyone who responded! Special thanks to:
- IC: @[name]
- SMEs: @[name1], @[name2]
- Communications: @[name]

**Post-Mortem Doc:** [Link to post-mortem draft]

---

This incident channel will remain open for 24 hours, then will be archived.

---

Tips for Effective Communication

Dos

✅ Be Clear: Use simple language, avoid jargon ✅ Be Honest: Don't over-promise on ETAs ✅ Be Timely: Update on schedule even if "no new information" ✅ Be Empathetic: Acknowledge customer frustration ✅ Be Specific: "15% error rate" vs. "some errors"

Don'ts

❌ Don't Speculate: Stick to confirmed facts ❌ Don't Blame: External communication should never blame individuals or teams ❌ Don't Over-Commit: Avoid "This will never happen again" ❌ Don't Ghost: Even "no update" is an update ❌ Don't Use Acronyms: Customers don't know "SEV1" or "MTTR"

---

Further Reading

• Atlassian: "How to Communicate During an Incident"
• PagerDuty: "Incident Communication Best Practices"
• Statuspage.io: "Writing Great Incident Updates"

On-Call Handoff Template

Overview

Structure weekly on-call handoffs to ensure smooth transition of responsibilities, context transfer, and continuity of operational awareness. Conduct handoff ceremony at consistent time (e.g., Monday 10am) with 30-minute dedicated meeting.

---

Pre-Handoff Checklist

Outgoing On-Call:

• [ ] Review all incidents from past week
• [ ] Identify ongoing issues requiring attention
• [ ] Check monitoring for degraded services
• [ ] Review upcoming maintenance windows
• [ ] Prepare notes on known issues
• [ ] Update runbooks if needed

Incoming On-Call:

• [ ] Verify PagerDuty/Opsgenie account active
• [ ] Test alert delivery (phone, SMS, push)
• [ ] Review on-call procedures and escalation paths
• [ ] Have laptop and VPN ready
• [ ] Review recent incident post-mortems

---

Handoff Meeting Template

Meeting Details

Date: [YYYY-MM-DD] Outgoing On-Call: [@outgoing-person] Incoming On-Call: [@incoming-person] Duration: 30 minutes Video Call: [Zoom/Meet link]

---

Section 1: Week in Review

Incident Summary

Total Incidents: [N]

Key Incidents to Discuss:

Incident 1: [Brief Title]

• When: [Date/Time]
• Severity: [SEV0/SEV1/SEV2/SEV3]
• Impact: [Customer/service impact]
• Root Cause: [Technical cause]
• Resolution: [How resolved]
• Follow-Up Actions: [Open action items]
• Post-Mortem: [Link to doc if applicable]

Incident 2: [Brief Title]

• When: [Date/Time]
• Severity: [SEV0/SEV1/SEV2/SEV3]
• Impact: [Customer/service impact]
• Root Cause: [Technical cause]
• Resolution: [How resolved]
• Follow-Up Actions: [Open action items]

---

Section 2: Ongoing Issues

Active Monitoring Items

Issue 1: [Brief Description]

• Status: [Monitoring | Under Investigation | Escalated]
• Since: [Date issue started]
• Impact: [Current impact, if any]
• Context: [Background information]
• Next Steps: [What to watch for or do]
• Escalation: [Who to contact if worsens]

Example:

Issue: Database Replication Lag Elevated

Status: Monitoring
Since: Nov 30, 2025
Impact: No customer impact, but lag 2-3 seconds vs normal 0.5s
Context: Traffic increased 20% due to holiday shopping. Replication keeping up but slower than usual. DBA team aware.
Next Steps: Monitor lag metric. Escalate if > 5 seconds or query errors appear.
Escalation: @db-sre-oncall
Alert: database-replication-lag-warning
Dashboard: https://monitoring.example.com/dashboard/database-health

---

Known Issues (Non-Urgent)

Issue 1: [Brief Description]

• Impact: [Minor/cosmetic/low priority]
• Ticket: [JIRA-123]
• Notes: [Any context]

Example:

Issue: Slow Dashboard Load Times (EU Region)

Impact: Dashboards load in 3-5 seconds vs normal 1-2 seconds. Non-critical feature.
Ticket: PERF-456
Notes: Performance team investigating. No customer complaints. CDN optimization planned for Sprint 42.

---

Section 3: Scheduled Maintenance

Upcoming Maintenance Windows

Maintenance 1: [Brief Description]

• When: [Date/Time with timezone]
• Duration: [Expected duration]
• Impact: [Customer impact, if any]
• Team Responsible: [@team]
• Your Role: [What on-call needs to do]
• Runbook: [Link to maintenance procedure]

Example:

Maintenance: Database Upgrade (PostgreSQL 14 → 15)

When: Dec 7, 2025 02:00-04:00 PST (Sunday night)
Duration: 2 hours (max)
Impact: 5-minute read-only mode during cutover. No downtime expected.
Team Responsible: @db-sre-team
Your Role: Monitor alerts during window. Be available for escalation if issues occur.
Runbook: https://wiki.example.com/database-upgrade-procedure
Slack Channel: #maintenance-dec7-db-upgrade
Rollback Plan: Available in runbook (revert to snapshot if critical issues)

---

Section 4: System Health Overview

Service Status Dashboard

Core Services:

• API: ✅ Healthy (error rate < 0.1%, latency p99 < 200ms)
• Database: ⚠️ Monitoring (replication lag 2-3s, see "Ongoing Issues")
• Cache (Redis): ✅ Healthy (hit rate 94%)
• Job Queue: ✅ Healthy (processing rate normal)
• CDN: ✅ Healthy (cache hit rate 97%)

Dashboard: https://monitoring.example.com/dashboard/overview

---

Recent Configuration Changes

Change 1: [Description]

• When: [Date/Time]
• Who: [@person]
• What Changed: [Brief description]
• Monitoring: [What to watch]

Example:

Change: Increased API Rate Limits for Enterprise Tier

When: Dec 2, 2025 14:00 PST
Who: @backend-team
What Changed: Increased rate limit from 1000 → 5000 req/min for enterprise customers
Monitoring: Watch API error rate and database connection pool utilization
Risk: Potential for database overload if multiple enterprise customers spike simultaneously
Rollback: Feature flag "enterprise_rate_limit_v2" can be disabled instantly

---

Section 5: Key Contacts and Resources

Escalation Paths

By Service Area:

Executive Escalation (SEV0 Only):

• VP Engineering: @vp-eng (Slack + Phone: +1-555-0100)
• CTO: @cto (Phone: +1-555-0101)
• CEO: @ceo (Phone: +1-555-0102) - Only for business-critical SEV0

---

Key Resources

Runbooks:

• Master Index: https://wiki.example.com/runbooks
• Database Failover: runbooks/database-failover.md
• Cache Invalidation: runbooks/cache-invalidation.md
• DDoS Mitigation: runbooks/ddos-mitigation.md

Dashboards:

• Overall Health: https://monitoring.example.com/dashboard/overview
• API Metrics: https://monitoring.example.com/dashboard/api
• Database Health: https://monitoring.example.com/dashboard/database
• Infrastructure: https://monitoring.example.com/dashboard/infra

Documentation:

• Incident Response Guide: https://wiki.example.com/incident-response
• Severity Classification: https://wiki.example.com/severity-levels
• On-Call Procedures: https://wiki.example.com/oncall-guide

Slack Channels:

• #incidents (automated incident creation)
• #oncall-general (on-call team coordination)
• #oncall-handoff (handoff notes archive)
• #database-operations (database team)
• #platform-engineering (infrastructure team)

---

Section 6: Tips and Lessons Learned

What Went Well This Week

Example:

- Quick response to API incident (MTTA 3 minutes, MTTR 45 minutes)
- Database team proactive about replication lag, no surprises
- Runbook for cache invalidation worked perfectly, no escalation needed

Challenges and Learnings

Example:

- Alert fatigue: Received 12 non-actionable "warning" alerts for disk space that were false positives
  → Action: Opened ticket to adjust thresholds (INFRA-789)

- Cache invalidation runbook missing step for verifying propagation to all regions
  → Action: Updated runbook with verification step

- Unclear when to escalate for replication lag - waited longer than needed
  → Learning: Escalate to @db-sre-oncall if lag > 5 seconds, don't wait

Advice for Incoming On-Call

Example:

- Keep an eye on database replication lag metric - it's been elevated all week
- EU region performance slower than usual, but no customer complaints yet
- Database maintenance Sunday night - be available 2am-4am PST
- New runbook for DDoS mitigation just added - review before your shift
- Coffee machine on floor 3 is broken, use floor 5 :)

---

Section 7: Action Items

Follow-Up from Last Week

• [ ] [ITEM-1] - [Description] - Owner: [@person] - Due: [Date]
• [ ] [ITEM-2] - [Description] - Owner: [@person] - Due: [Date]

Example:

- [x] Update cache-invalidation runbook with verification step - Owner: @alice - Due: Dec 4
- [ ] Adjust disk space alert thresholds to reduce false positives - Owner: @bob - Due: Dec 10
- [ ] Schedule disaster recovery drill for Q1 2026 - Owner: @oncall-lead - Due: Dec 15

---

New Action Items from Handoff

• [ ] [NEW-ITEM-1] - [Description] - Owner: [@person] - Due: [Date]
• [ ] [NEW-ITEM-2] - [Description] - Owner: [@person] - Due: [Date]

---

Section 8: Questions and Clarifications

Incoming On-Call Questions:

Q: [Question from incoming on-call] A: [Answer from outgoing on-call]

Example:

Q: What's the typical response time for database team during off-hours?
A: Usually 5-10 minutes. They're very responsive. Use @db-sre-oncall in Slack first, escalate to phone if no response in 10 min.

Q: Is there a runbook for the replication lag issue?
A: Yes - RB-DB-003 (Database Replication Repair). But consult @db-sre-oncall before executing since they're already monitoring it.

Q: What's the threshold for escalating to VP Engineering?
A: SEV0 incidents lasting > 30 minutes, or any data breach/security incident, or if customer-facing impact is severe and we need executive communication.

---

Post-Handoff Actions

Outgoing On-Call:

• [ ] Post handoff notes in #oncall-handoff Slack channel
• [ ] Update on-call schedule if any coverage gaps
• [ ] Archive handoff document in wiki: https://wiki.example.com/oncall-handoffs/YYYY-MM-DD
• [ ] Complete on-call feedback survey (if applicable)

Incoming On-Call:

• [ ] Acknowledge receipt of handoff
• [ ] Review all linked dashboards and runbooks
• [ ] Test alert delivery immediately after handoff
• [ ] Introduce yourself in #oncall-general Slack channel
• [ ] Review action items and clarify ownership

---

Handoff Confirmation

Outgoing On-Call Confirmation:

I confirm that I have:
- Reviewed all incidents from the past week
- Briefed @[incoming] on ongoing issues
- Provided context on upcoming maintenance
- Shared key contacts and escalation paths
- Answered all questions to the best of my ability

Signed: [@outgoing-person], [Date]

Incoming On-Call Confirmation:

I confirm that I have:
- Reviewed all handoff materials
- Asked clarifying questions
- Tested alert delivery (PagerDuty/phone/SMS)
- Reviewed key dashboards and runbooks
- Understand escalation procedures

Ready to assume on-call responsibility.

Signed: [@incoming-person], [Date]

---

Template Usage Notes

Customization

Adapt sections based on organization size:

• Small teams (< 10 engineers): Focus on Sections 1-3, 5-6
• Medium teams (10-50 engineers): Use all sections
• Large teams (50+ engineers): Add service-specific handoffs

Frequency:

• Weekly rotation: Full handoff every week
• Daily rotation: Abbreviated handoff (Sections 1-2, 6 only)
• Follow-the-sun: Twice daily brief handoff (15 minutes)

---

Handoff Best Practices

Timing:

• Schedule at consistent time (e.g., Monday 10am)
• Allow 30 minutes minimum
• No multitasking - give full attention

Documentation:

• Archive handoff notes for future reference
• Update runbooks based on learnings
• Track common issues for process improvement

Communication:

• Use video call, not just Slack
• Screen share dashboards and runbooks
• Encourage questions - no question is too basic

Cultural:

• Recognize outgoing on-call's efforts
• Share positive incidents (fast resolution, good teamwork)
• Acknowledge challenges without blame

---

Related Templates

• Incident Response: communication-templates.md
• Post-Mortem: postmortem-template.md
• Runbooks: runbooks/ directory

---

Changelog

---

Contact

Owner: SRE Team Slack: #oncall-general Documentation: https://wiki.example.com/oncall-guide

For template improvements: Open PR or post in #sre-team

Post-Mortem: [Incident Title]

Incident ID: INC-YYYY-MM-DD-### Severity: [SEV0 | SEV1 | SEV2] Duration: [X hours/minutes] ([Start Time] - [End Time] [Timezone]) Date: [Month Day, Year] Author: [Name] Attendees: [List all post-mortem meeting participants] Status: [Draft | Final | Archived]

---

Incident Summary

In 2-3 sentences, describe what happened in plain language accessible to non-technical readers.

Example:

On December 3, 2025, our API experienced elevated error rates (15% failure) due to database connection pool exhaustion. Approximately 5,000 customers were unable to complete checkout during the 90-minute incident. The issue was resolved by increasing the connection pool size and implementing circuit breaker logic.

---

Impact Assessment

Customer Impact

• Users Affected: [Number or percentage] ([X% of active users])
• Duration: [Hours/minutes of customer-facing impact]
• Affected Functionality: [Specific features unavailable]
• Data Loss: [None | Describe scope if any]
• Geographic Spread: [Regions affected: US-East, EU, Global, etc.]

Business Impact

• Revenue Impact: [Estimated $ amount or "Negligible"]
• SLA Breach: [Yes/No - If yes, describe SLA violation]
• Support Load: [Number of tickets filed during incident]
• Reputation: [Social media mentions, news coverage if any]
• Contractual: [Customer SLA violations, penalties if applicable]

Technical Impact

• Services Affected: [List all affected services/components]
• Infrastructure: [Database, API servers, network, etc.]
• Dependencies: [Third-party services impacted, upstream/downstream]
• Data Integrity: [Data loss, corruption, or inconsistency]

---

Timeline

All times in [Timezone]

Provide a chronological record of events from detection to full recovery.

Key Milestones:

• Detection: [HH:MM] (Time from incident start to first alert)
• Acknowledgment: [HH:MM] (MTTA: [X minutes])
• Mitigation: [HH:MM] (Time to stop customer impact)
• Resolution: [HH:MM] (MTTR: [X minutes])

---

Root Cause Analysis

Primary Root Cause

What was the direct trigger or failure point that caused the incident?

Example:

Database connection pool size (100 connections) was insufficient for peak traffic load (2x normal due to holiday shopping season).

Contributing Factors

What conditions worsened the impact or prevented faster resolution?

1. [Factor 1: e.g., Aggressive retry logic caused "thundering herd" when database slowed] 2. [Factor 2: e.g., No monitoring/alerting on connection pool saturation] 3. [Factor 3: e.g., Recent code deploy increased database query frequency by 20%] 4. [Factor 4: e.g., Runbook for database scaling was outdated]

5 Whys Analysis

Dig deeper to identify systemic root causes:

1. Why did [incident occur]?

• [Answer]

2. Why [answer to #1]?

• [Answer]

3. Why [answer to #2]?

• [Answer]

4. Why [answer to #3]?

• [Answer]

5. Why [answer to #4]?

• [Answer]

Systemic Root Cause: [The process, tool, or cultural issue revealed by 5 Whys]

Example:

Release process lacks load testing requirement for traffic spike scenarios, allowing performance-impacting changes to reach production without validation.

---

What Went Well

Celebrate effective incident response behaviors to reinforce positive actions.

1. [Positive aspect]: [Description]

• Example: "Fast detection: Alert fired within 2 minutes of error rate increase"

2. [Positive aspect]: [Description]

• Example: "Clear IC leadership: @bob immediately took IC role and delegated tasks"

3. [Positive aspect]: [Description]

• Example: "Effective communication: Status updates posted every 15 minutes"

4. [Positive aspect]: [Description]

• Example: "Quick mitigation: Connection pool increase deployed within 10 minutes"

5. [Positive aspect]: [Description]

• Example: "No data loss: All customer data remained intact"

---

What Went Wrong

Identify improvement areas, focusing on process, tools, or training gaps (NOT individuals).

Important: Frame as learning opportunities, not blame. Use "How did the system fail?" not "Who failed?"

1. [Improvement area]: [Description]

• Example: "No proactive monitoring: Connection pool saturation not monitored, only reactive error alerts"

2. [Improvement area]: [Description]

• Example: "Untested retry logic: Circuit breaker not tested under database degradation scenarios"

3. [Improvement area]: [Description]

• Example: "Missing load testing: Recent deploy not load tested for 2x traffic spike"

4. [Improvement area]: [Description]

• Example: "Communication delay: First status page update 30 minutes into incident"

5. [Improvement area]: [Description]

• Example: "Outdated runbook: Database scaling runbook referenced deprecated configuration"

---

Action Items

Define specific, actionable improvements with clear owners and due dates.

Requirements for each action item:

• Specific: Clear task description, not vague ("Improve monitoring" ❌ | "Add connection pool saturation alert" ✅)
• Owned: Single person responsible (not "team")
• Dated: Realistic due date
• Prioritized: High/Medium/Low based on impact

Example Action Items:

---

Lessons Learned

Key takeaways for organizational knowledge and future prevention.

1. [Lesson]: [Description of insight gained]

• Example: "Connection pool metrics are critical leading indicators: Monitor saturation before errors occur"

2. [Lesson]: [Description of insight gained]

• Example: "Retry logic needs circuit breakers: Prevent thundering herd in database degradation scenarios"

3. [Lesson]: [Description of insight gained]

• Example: "Load testing must include traffic spike scenarios: 2x traffic should be standard test case"

4. [Lesson]: [Description of insight gained]

• Example: "Proactive monitoring > Reactive alerting: Leading indicators (saturation) better than lagging (errors)"

5. [Lesson]: [Description of insight gained]

• Example: "Runbooks are living documents: Quarterly review and update cycle needed"

---

Supporting Data

Attach relevant logs, metrics, screenshots for reference.

Graphs and Metrics

• API Error Rate: [Link to Grafana dashboard or screenshot]
• Database Connection Pool: [Link to metrics]
• Traffic Patterns: [Link to traffic graphs]
• CPU/Memory/Disk: [Link to infrastructure metrics]

Log Excerpts

[Paste relevant log entries showing error states, warnings, or key events]

Example:
2025-12-03 14:15:32 ERROR: Database connection pool exhausted (max: 100, active: 100)
2025-12-03 14:15:33 ERROR: Checkout API failed: connection timeout

Configuration Changes

[Show before/after configuration if relevant]

Example:
Before: connection_pool_size = 100
After:  connection_pool_size = 200

Alerts Fired

• 14:15:32 PST: API-Error-Rate-High (Datadog)
• 14:17:45 PST: Database-Connection-Pool-Low (manual check, not automated)
• 14:30:15 PST: API-Error-Rate-Normal (recovery)

---

Prevention Measures

Summarize how action items will prevent recurrence.

Short-Term (Completed within 2 weeks): 1. [Immediate fix applied, e.g., "Connection pool size increased to 200"] 2. [Monitoring added, e.g., "Alert added for 80% pool saturation"]

Medium-Term (Completed within 1-2 months): 1. [Process improvement, e.g., "Load testing requirement added to release checklist"] 2. [Tool improvement, e.g., "Circuit breaker library implemented"]

Long-Term (Completed within 3-6 months): 1. [Architectural change, e.g., "Auto-scaling connection pool implementation"] 2. [Training, e.g., "Team training on database performance tuning"]

---

Communication

Internal Communication

Incident Channel: [#incident-YYYY-MM-DD-topic] Status Updates: [Number of updates posted, cadence] Stakeholders Notified: [Engineering, product, support, leadership]

External Communication

Status Page: [Link to public status page updates] Customer Email: [Sent? Yes/No - If yes, link to email draft] Social Media: [Twitter, LinkedIn updates if applicable] Press/Media: [Any media coverage or press releases]

Regulatory Notifications

Required: [Yes/No - GDPR, HIPAA, PCI, etc.] Notification Sent: [Yes/No - If yes, date and recipient] Documentation: [Link to regulatory notification records]

---

Follow-Up Actions

Action Item Tracking

• Review Cadence: [Weekly in sprint planning]
• Tracking Tool: [Jira, GitHub Issues, Linear, etc.]
• Completion Target: [Date by which all high-priority items should be complete]

Post-Mortem Distribution

• [X] Engineering team (via Slack #engineering)
• [X] Product team (via email)
• [X] Support team (via Slack #support)
• [X] Leadership (via email)
• [X] Knowledge base (archived in Confluence/Wiki)

Post-Mortem Review

Next Post-Mortem Review: [Quarterly incident review meeting] Discussion: [Will this post-mortem be discussed in team meeting? When?]

---

Related Incidents

Previous Similar Incidents:

• [INC-YYYY-MM-DD-###: Brief description, link]
• [INC-YYYY-MM-DD-###: Brief description, link]

Pattern Detection: [If this is a recurring issue, note the pattern and systemic cause]

Example:

This is the third database connection pool exhaustion incident in 6 months. Indicates need for auto-scaling connection pool implementation (action item added).

---

Appendix

Glossary

[Term 1]: [Definition for non-technical readers] [Term 2]: [Definition for non-technical readers]

Example:

• Connection Pool: A cache of database connections maintained by the application to avoid creating a new connection for each query
• Circuit Breaker: A design pattern that prevents cascading failures by stopping requests to a failing service

References

• Runbook: [Link to relevant runbook, e.g., RB-DB-001 Database Failover]
• Monitoring Dashboard: [Link to Grafana/Datadog dashboard]
• Incident Channel: [Link to Slack incident channel archive]
• Related Documentation: [Links to architecture docs, configuration guides, etc.]

---

Sign-Off

Approved By:

• Incident Commander: @[IC name] - [Date]
• Engineering Manager: @[Manager name] - [Date]
• VP Engineering: @[VP name] - [Date] (for SEV0 only)

Post-Mortem Meeting Held: [Yes/No - Date and attendees] Document Status: [Draft → Final → Archived] Archive Location: [Link to knowledge base location]

---

Document Changelog

---

Remember:

• Blameless: Focus on systems, not people
• Specific: Action items must be clear and actionable
• Honest: Psychological safety requires honesty
• Timely: Complete within 48 hours while memory fresh
• Follow-Through: Track action items to completion

Questions? Contact @[IC or post-mortem facilitator]

Runbook: Cache Invalidation

Metadata

• Runbook ID: RB-CACHE-001
• Owner: Platform Engineering Team (@platform-team)
• Last Updated: 2025-12-05
• Version: 1.3
• Estimated Duration: 5-10 minutes

---

Trigger Conditions

Execute this runbook when:

• Stale data detected in production (users seeing outdated content)
• Code deployment requires cache clear (API schema changes, pricing updates)
• Configuration changes not reflecting (feature flags, A/B test variants)
• Corrupted cache entries causing application errors
• Emergency data correction requiring immediate propagation

Alert Names:

• Cache-Stale-Data-Detected
• Cache-Hit-Rate-Anomaly
• API-Serving-Old-Data

Manual Triggers:

• Post-deployment cache clear requested by engineering team
• Customer support escalation reporting stale data
• Product team requesting immediate content update propagation

---

Severity Classification

Expected Severity: SEV2 (Minor Issues) - SEV1 (Major Degradation)

Reasoning:

• SEV2: Limited data staleness, affects subset of users, workaround available (hard refresh)
• SEV1: Critical data incorrect (pricing, inventory, permissions), widespread user impact
• SEV0: Escalate only if incorrect cached data causing financial loss or security vulnerability

Severity Decision:

Is cached data causing security vulnerability or financial loss?
├─ YES → SEV0 (Page immediately, declare incident)
└─ NO  → Is critical business data incorrect (pricing, inventory, auth)?
          ├─ YES → SEV1 (Page during business hours, IC assigned)
          └─ NO  → SEV2 (Slack alert, execute runbook during business hours)

---

Prerequisites

Before executing, verify:

1. Cache System Health:

   # Check Redis cluster health
   redis-cli -h cache-cluster.example.com PING
   # Expected: PONG

   # Check cluster info
   redis-cli -h cache-cluster.example.com CLUSTER INFO
   # Expected: cluster_state:ok

2. Identify Cache Scope: Determine what needs invalidation:

• Global: Entire cache (all keys) - RARE, high impact
• Namespace: Specific service or feature (user:*, product:*)
• Pattern: Keys matching pattern (product:123:*, session:abc*)
• Single Key: Specific cache entry (product:456, user:789:profile)

3. Impact Assessment:

   # Check current cache hit rate
   redis-cli -h cache-cluster.example.com INFO stats | grep keyspace_hits

   # Estimate keys affected
   redis-cli -h cache-cluster.example.com --scan --pattern "product:*" | wc -l

4. Required Access:

• Redis CLI access or admin credentials
• Kubernetes access (if invalidating via API)
• Monitoring dashboard access
• Slack channel: #platform-engineering

5. Database Load Check:

   # Verify database can handle cache miss traffic
   curl https://monitoring.example.com/api/v1/query?query=database_cpu_percent
   # Expected: < 60% (buffer for cache miss spike)

WARNING: Invalidating large cache volumes causes database load spike. If DB CPU > 70%, consult DBA before proceeding.

---

Steps

Step 1: Announce Cache Invalidation

Purpose: Notify team and prepare for potential brief performance impact.

Post in #platform-engineering (or incident channel if SEV1):

"@here Executing cache invalidation for [scope: global/namespace/pattern].
Expected impact: Potential brief latency increase (1-2 minutes) as cache rebuilds.
Reason: [Brief reason: stale data, deployment, config change]
ETA for completion: [TIMESTAMP + 5-10 minutes]"

If SEV1/SEV0: Post status page update (see communication-templates.md).

---

Step 2: Verify Database Can Handle Load

Purpose: Prevent cascading failure from database overload.

# Check database connection pool
curl https://monitoring.example.com/api/v1/query?query=database_connection_pool_usage

# Check database CPU
curl https://monitoring.example.com/api/v1/query?query=database_cpu_percent

# Check query queue depth
curl https://monitoring.example.com/api/v1/query?query=database_query_queue_length

Health Criteria:

• Database CPU < 60%
• Connection pool usage < 70%
• Query queue depth < 100

If Database Unhealthy:

• Option A: Partial invalidation (invalidate keys in batches)
• Option B: Scale database read replicas first
• Option C: Escalate to @db-sre-oncall for guidance

---

Step 3: Choose Invalidation Method

Purpose: Select appropriate method based on scope and urgency.

Option A: Pattern-Based Invalidation (RECOMMENDED)

Use when: Need to invalidate specific namespace or pattern (most common).

# Connect to Redis
redis-cli -h cache-cluster.example.com

# Scan and delete keys matching pattern
redis-cli -h cache-cluster.example.com --scan --pattern "product:*" | xargs redis-cli -h cache-cluster.example.com DEL

# For large datasets (> 10,000 keys), use batching
redis-cli -h cache-cluster.example.com --scan --pattern "product:*" | while read key; do
  redis-cli -h cache-cluster.example.com DEL "$key"
  sleep 0.01  # Rate limit to avoid overwhelming Redis
done

Verification:

# Confirm keys deleted
redis-cli -h cache-cluster.example.com --scan --pattern "product:*" | wc -l
# Expected: 0 (or significantly reduced count)

---

Option B: Single Key Invalidation

Use when: Need to invalidate specific cache entry.

# Delete single key
redis-cli -h cache-cluster.example.com DEL "product:456"

# Verify deletion
redis-cli -h cache-cluster.example.com EXISTS "product:456"
# Expected: (integer) 0

---

Option C: Namespace Invalidation via API

Use when: Application provides cache invalidation API endpoint.

# POST to invalidation endpoint
curl -X POST https://api.example.com/admin/cache/invalidate \
  -H "Authorization: Bearer $ADMIN_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "namespace": "product",
    "reason": "Product pricing update",
    "force": true
  }'

# Expected Response
# {"status": "success", "keys_invalidated": 1234, "duration_ms": 543}

---

Option D: Time-Based Expiration Update (GENTLE)

Use when: Not urgent, prefer gradual invalidation.

# Reduce TTL to 60 seconds for gradual invalidation
redis-cli -h cache-cluster.example.com --scan --pattern "product:*" | while read key; do
  redis-cli -h cache-cluster.example.com EXPIRE "$key" 60
done

Benefit: Avoids sudden cache miss spike, cache naturally expires over 1 minute.

---

Option E: Global Cache Flush (NUCLEAR OPTION)

Use when: Complete cache corruption, all data stale, no other option.

WARNING: Causes massive database load spike. Only use in emergencies after consulting IC and DBA.

# DANGER: Flushes entire Redis database
redis-cli -h cache-cluster.example.com FLUSHDB

# For multi-node cluster, flush all nodes
for node in cache-node-1 cache-node-2 cache-node-3; do
  redis-cli -h $node.example.com FLUSHDB
done

Pre-Flush Requirements:

• [ ] IC approval obtained
• [ ] Database team notified and standing by
• [ ] Monitoring dashboard open
• [ ] Rollback plan ready (restore from Redis persistence if available)

---

Step 4: Verify Cache Invalidation

Purpose: Confirm keys deleted and cache rebuilding correctly.

# Check keys deleted
redis-cli -h cache-cluster.example.com --scan --pattern "product:*" | wc -l
# Expected: 0 or significantly reduced

# Monitor cache miss rate (should spike briefly)
curl https://monitoring.example.com/api/v1/query?query=rate(cache_misses_total[1m])

# Monitor cache hit rate (should recover within 2-5 minutes)
curl https://monitoring.example.com/api/v1/query?query=cache_hit_rate

# Check API latency (may increase briefly)
curl https://monitoring.example.com/api/v1/query?query=http_request_duration_seconds{quantile="0.99"}

Expected Behavior:

• Cache miss rate spikes immediately (2-10x normal)
• API latency increases briefly (1.5-2x normal)
• Cache hit rate recovers within 2-5 minutes
• Database CPU increases temporarily (10-20% increase)

---

Step 5: Verify Data Freshness

Purpose: Confirm new data correctly propagating to users.

Test with API Request

# Fetch data via API (should trigger cache rebuild)
curl -v https://api.example.com/v1/products/456 | jq '.updated_at'

# Verify response contains fresh data
# Expected: Timestamp shows recent update (not stale value)

Test from User Perspective

# Test from multiple regions (if multi-region cache)
for region in us-east us-west eu-west; do
  echo "Testing region: $region"
  curl -H "X-Region: $region" https://api.example.com/v1/products/456 | jq '.updated_at'
done

# Expected: All regions return fresh data

Manual Verification Checklist

• [ ] Load affected page/API endpoint in browser/Postman
• [ ] Verify data shows expected values (not stale)
• [ ] Check multiple users/sessions if user-specific cache
• [ ] Verify across all regions if multi-region deployment

---

Step 6: Monitor for Stability

Purpose: Ensure cache invalidation didn't cause secondary issues.

Monitor for 10-15 minutes:

# Watch cache hit rate recovery
watch -n 5 'redis-cli -h cache-cluster.example.com INFO stats | grep keyspace_hits'

# Monitor API error rate
curl https://monitoring.example.com/api/v1/query?query=rate(api_errors_total[5m])

# Monitor database CPU
curl https://monitoring.example.com/api/v1/query?query=database_cpu_percent

# Monitor API latency
curl https://monitoring.example.com/api/v1/query?query=http_request_duration_seconds{quantile="0.99"}

Monitoring Dashboard: https://monitoring.example.com/dashboard/cache-health

Stability Criteria:

• Cache hit rate > 85% (recovered to normal)
• API error rate < 0.5% (normal baseline)
• Database CPU < 70% (normal range)
• API p99 latency < 500ms (acceptable)

If Unstable: See "Rollback" section below.

---

Step 7: Update Team and Close

Purpose: Notify team of completion and document outcome.

Post in #platform-engineering (or incident channel):

"Cache invalidation complete.
- Keys invalidated: [N keys or pattern]
- Duration: [X minutes]
- Cache hit rate recovered: [Y%]
- API latency returned to normal: p99 [Z]ms
- Database CPU stable: [N%]
- No errors detected.

Monitoring for next 30 minutes. Will close if stable."

If SEV1/SEV0: Post status page update: "Monitoring - Issue resolved, watching for stability."

---

Step 8: Document in Incident Timeline (If SEV1+)

Purpose: Record actions for post-mortem.

Post in incident channel:

"[TIMESTAMP] Cache invalidation executed
- Scope: [Pattern/namespace]
- Method: [Pattern-based / API / Manual]
- Keys invalidated: [N]
- Impact: Brief latency increase (1-2 minutes), cache rebuilt successfully
- Verification: Fresh data confirmed across all regions
- Monitoring: Stable"

---

Rollback Procedure

When to Rollback:

• Database overload (CPU > 90%, query queue growing)
• API error rate spike (> 5%)
• Cache not rebuilding (hit rate stuck < 50% after 10 minutes)
• Application errors from malformed cache data

Rollback Steps:

Option A: Restore from Redis Persistence (If Available)

# Stop Redis
redis-cli -h cache-cluster.example.com SHUTDOWN SAVE

# Restore from last RDB snapshot (if persistence enabled)
# SSH to Redis server
ssh admin@cache-cluster.example.com
sudo cp /var/lib/redis/dump.rdb.backup /var/lib/redis/dump.rdb
sudo systemctl start redis

# Verify restoration
redis-cli -h cache-cluster.example.com DBSIZE
# Expected: Previous key count restored

---

Option B: Disable Cache Temporarily

Use when: Cache causing more problems than solving.

# Set feature flag to bypass cache
curl -X POST https://api.example.com/admin/feature-flags \
  -H "Authorization: Bearer $ADMIN_API_TOKEN" \
  -d '{"flag": "cache_enabled", "value": false}'

# Verify flag set
curl https://api.example.com/admin/feature-flags/cache_enabled
# Expected: {"flag": "cache_enabled", "value": false}

Impact: API latency increases 2-3x, database load increases 3-5x. Only sustainable short-term.

Recovery: Fix underlying issue, re-enable cache gradually:

# Re-enable cache
curl -X POST https://api.example.com/admin/feature-flags \
  -d '{"flag": "cache_enabled", "value": true}'

---

Option C: Rate-Limit Cache Rebuilds

Use when: Cache rebuilding too aggressively, overwhelming database.

# Implement application-level rate limiting for cache writes
# (Requires code change or runtime config if supported)

# Temporary: Reduce Redis connection pool to slow cache writes
kubectl set env deployment/api REDIS_POOL_SIZE=10 -n production

---

Escalation Criteria

Escalate to Senior Engineer (@platform-lead) if:

• Database CPU > 90% after invalidation
• API error rate > 5%
• Cache hit rate not recovering (< 50% after 15 minutes)
• Unclear which keys to invalidate
• Global flush being considered

Escalate to VP Engineering if:

• SEV0 incident (financial/security impact)
• Customer data integrity concerns
• Multi-region cache synchronization failure

---

Common Issues and Solutions

Issue 1: Cache Not Rebuilding

Symptom: Cache hit rate remains low (< 50%) 10+ minutes after invalidation.

Solutions:

• Check Application Health: Verify API pods running: kubectl get pods -n production
• Check Redis Connection: Verify app can connect to Redis: kubectl logs deployment/api -n production | grep redis
• Check Database: Verify DB responding: pg_isready -h primary-db.example.com
• Manual Cache Warm: Trigger cache population via script or API calls

Manual Cache Warm:

# Warm cache with top 100 product pages
for id in {1..100}; do
  curl https://api.example.com/v1/products/$id > /dev/null
  sleep 0.1
done

---

Issue 2: Database Overload

Symptom: Database CPU > 80%, query queue growing, API timeouts.

Solutions:

• Pause Invalidation: Stop if using batched approach
• Scale Read Replicas: Add temporary read replicas (AWS RDS/GCP Cloud SQL)
• Enable Query Caching: Database-level query cache if available
• Rate Limit Traffic: Temporarily rate limit API requests

Emergency Database Relief:

# Scale RDS read replicas (AWS)
aws rds create-db-instance-read-replica \
  --db-instance-identifier mydb-replica-temp \
  --source-db-instance-identifier mydb-primary \
  --db-instance-class db.r5.large

# Takes 5-10 minutes to provision

---

Issue 3: Multi-Region Cache Desync

Symptom: Some regions showing fresh data, others showing stale data.

Solutions:

• Invalidate Each Region Separately:

  for region in us-east us-west eu-west; do
    redis-cli -h cache-$region.example.com --scan --pattern "product:*" | xargs redis-cli -h cache-$region.example.com DEL
  done

• Check Replication Lag: If using Redis replication between regions
• Verify CDN Purge: If using CDN, purge CDN cache separately:

  # Cloudflare purge
  curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/purge_cache" \
    -H "Authorization: Bearer $CF_API_TOKEN" \
    -d '{"tags": ["product"]}'

---

Issue 4: Accidentally Deleted Wrong Keys

Symptom: Unintended cache keys deleted, unrelated functionality broken.

Solutions:

• Restore from Backup: If Redis persistence enabled, restore from RDB snapshot
• Rebuild Specific Namespace: Re-populate accidentally deleted keys:

  # Trigger cache rebuild for specific namespace
  curl -X POST https://api.example.com/admin/cache/rebuild \
    -d '{"namespace": "user", "async": true}'

• Wait for Natural Rebuild: Cache will rebuild on-demand as users access data

---

Post-Execution Checklist

After completing cache invalidation, verify:

• [ ] Targeted keys successfully deleted
• [ ] Fresh data verified via API/browser
• [ ] Cache hit rate recovered (> 85%)
• [ ] API latency returned to normal (< 200ms p99)
• [ ] Database CPU returned to baseline (< 60%)
• [ ] No application errors detected
• [ ] Multi-region consistency verified (if applicable)
• [ ] Team notified of completion
• [ ] Incident timeline updated (if SEV1+)
• [ ] Runbook updated if steps changed

---

Related Runbooks

• RB-DB-001: Database Failover (if cache overload causes DB issues)
• RB-PERF-001: Performance Troubleshooting (if latency remains high)
• RB-CDN-001: CDN Cache Purge (if CDN also caching stale data)

---

Cache Invalidation Patterns

Pattern 1: Deployment-Triggered Invalidation

Use when: Code deployment changes API schema or data format.

Automation:

# GitHub Actions workflow
- name: Invalidate cache post-deployment
  run: |
    curl -X POST https://api.example.com/admin/cache/invalidate \
      -H "Authorization: Bearer ${{ secrets.ADMIN_API_TOKEN }}" \
      -d '{"namespace": "api_v1", "reason": "Deployment ${{ github.sha }}"}'

---

Pattern 2: Time-Based Auto-Expiration

Use when: Data naturally becomes stale over time (news, prices).

Implementation:

# Set TTL when writing to cache (application code)
redis-cli SET "product:456:price" "29.99" EX 3600  # 1 hour TTL

# Or update TTLs in bulk
redis-cli --scan --pattern "product:*" | while read key; do
  redis-cli EXPIRE "$key" 3600
done

---

Pattern 3: Event-Driven Invalidation

Use when: Backend data changes should immediately invalidate cache.

Implementation (Pseudocode):

# After database write
def update_product(product_id, data):
    db.update(product_id, data)
    cache.delete(f"product:{product_id}")
    cache.delete(f"product:{product_id}:*")  # Invalidate all related keys

---

Pattern 4: Gradual Rollout Invalidation

Use when: Want to test cache invalidation impact on subset of traffic.

Implementation:

# Invalidate 10% of keys (testing)
redis-cli --scan --pattern "product:*" | head -n 100 | xargs redis-cli DEL

# Monitor impact for 15 minutes

# Invalidate remaining 90%
redis-cli --scan --pattern "product:*" | xargs redis-cli DEL

---

Testing and Validation

Last Tested: 2025-11-20 (disaster recovery drill) Test Frequency: Monthly (simulate cache invalidation) Next Test: 2026-01-05

Test Procedure: 1. Announce test in #platform-engineering (off-hours) 2. Invalidate non-critical cache namespace (e.g., "test:*") 3. Monitor cache hit rate recovery 4. Verify no impact to production traffic 5. Document any runbook issues discovered 6. Update runbook if needed

---

Changelog

---

Contact

Owner: Platform Engineering Team Slack: #platform-engineering On-Call: @platform-oncall Escalation: @platform-lead

For questions or improvements: Open PR against runbooks/cache-invalidation.md

Runbook: Database Failover

Metadata

• Runbook ID: RB-DB-001
• Owner: Database SRE Team (@db-sre-team)
• Last Updated: 2025-12-03
• Version: 2.1
• Estimated Duration: 10-15 minutes

---

Trigger Conditions

Execute this runbook when:

• Primary database becomes unreachable (alert: "PostgreSQL Primary Down")
• Primary database performance severely degraded (> 10 seconds query latency)
• Planned maintenance requiring failover
• Database corruption detected on primary

Alert Names:

• PostgreSQL-Primary-Down
• Database-Failover-Required
• Primary-DB-Health-Critical

---

Severity Classification

Expected Severity: SEV1 (Major Degradation)

Reasoning: Application remains functional with degraded performance until failover completes. Becomes SEV0 if secondary unavailable or replication lag > 1 minute.

---

Prerequisites

Before executing, verify:

1. Secondary Database Health:

   pg_isready -h secondary-db.example.com -p 5432

Expected: secondary-db.example.com:5432 - accepting connections

2. Replication Lag:

   -- Run on secondary
   SELECT EXTRACT(EPOCH FROM (now() - pg_last_xact_replay_timestamp())) AS lag_seconds;

Expected: < 5 seconds

If lag > 5 seconds: Wait for replication to catch up or accept potential data loss (consult IC).

3. Disk Space on Secondary:

   df -h /var/lib/postgresql

Expected: > 20% free space

4. Required Access:

• SSH access to database servers
• Sudo privileges for PostgreSQL commands
• AWS/GCP console access for DNS updates
• Slack channel: #database-operations

---

Steps

Step 1: Verify Primary is Truly Down

Purpose: Prevent unnecessary failover due to transient network issues.

# From monitoring server
ping -c 5 primary-db.example.com

# Check PostgreSQL connectivity
pg_isready -h primary-db.example.com -p 5432

# Check last heartbeat in monitoring
curl https://monitoring.example.com/api/v1/query?query=up{job="postgresql-primary"}

Expected Result: Ping fails OR PostgreSQL not accepting connections

If Primary Responds: Do NOT proceed with failover. Investigate performance issue instead (see RB-DB-002).

---

Step 2: Announce Failover in Incident Channel

Purpose: Notify team and stakeholders of impending failover.

Post in #incident-YYYY-MM-DD-database:

"@here Starting database failover from primary-db to secondary-db.
Expected downtime: 2-5 minutes.
ETA for completion: 15:30 PST (in 10 minutes)."

---

Step 3: Check Replication Status on Secondary

Purpose: Ensure secondary is up-to-date before promotion.

# SSH to secondary database
ssh admin@secondary-db.example.com

# Check replication status
sudo -u postgres psql -c "SELECT * FROM pg_stat_wal_receiver;"

Expected Output:

status    | streaming
received  | <recent timestamp>

If NOT streaming: Do NOT promote. Investigate replication failure first.

---

Step 4: Promote Secondary to Primary

Purpose: Make secondary the new writable database.

# On secondary database server
sudo -u postgres pg_ctl promote -D /var/lib/postgresql/data

# Verify promotion
sudo -u postgres psql -c "SELECT pg_is_in_recovery();"

Expected Result: f (false = not in recovery = primary mode)

If Promotion Fails:

• Check PostgreSQL logs: tail -f /var/log/postgresql/postgresql.log
• Verify no standby.signal file blocking promotion
• Escalate to senior DBA if unclear

---

Step 5: Update DNS to Point to New Primary

Purpose: Route application traffic to new primary database.

Option A: Route53 (AWS)

# Update DNS record via AWS CLI
aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890ABC \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "primary-db.example.com",
        "Type": "A",
        "TTL": 60,
        "ResourceRecords": [{"Value": "10.0.2.20"}]
      }
    }]
  }'

Option B: Terraform

# Update terraform variable
cd infrastructure/terraform/database
echo 'primary_db_ip = "10.0.2.20"' >> terraform.tfvars

# Apply change
terraform plan -target=aws_route53_record.db_primary
terraform apply -target=aws_route53_record.db_primary

Verification:

# Verify DNS propagation
dig primary-db.example.com +short
# Expected: 10.0.2.20 (new primary IP)

# Wait for DNS TTL (60 seconds)
sleep 60

---

Step 6: Update Application Configuration

Purpose: Ensure application connects to correct database endpoint.

Option A: No Action Required (if using DNS)

If application uses primary-db.example.com hostname, DNS update is sufficient.

Option B: Update Environment Variables

# Update Kubernetes configmap
kubectl edit configmap app-config -n production

# Update DATABASE_HOST value
# DATABASE_HOST: secondary-db.example.com → primary-db.example.com

# Restart application pods to pick up new config
kubectl rollout restart deployment/api -n production

---

Step 7: Verify Application Connectivity

Purpose: Confirm application successfully connecting to new primary.

# Check application error rates in monitoring
curl https://monitoring.example.com/api/v1/query?query=rate(api_errors_total[5m])

# Check database connection pool
curl https://monitoring.example.com/api/v1/query?query=database_connections_active

# Test write operation
curl -X POST https://api.example.com/health/write-test

Expected Result:

• Error rate returns to baseline (< 0.1%)
• Database connections active (> 0)
• Write test succeeds: {"status": "ok", "write_success": true}

If Errors Persist:

• Check application logs for connection errors
• Verify DNS propagation: nslookup primary-db.example.com
• Verify firewall rules allow app → new primary
• Escalate to application team if needed

---

Step 8: Monitor for Stability

Purpose: Ensure failover was successful and system is stable.

# Monitor for 10 minutes
# - Database CPU, memory, disk I/O
# - Application error rates
# - Query latency

Monitoring Dashboard: https://monitoring.example.com/dashboard/database-health

Stability Criteria:

• Query latency < 100ms (p99)
• Database CPU < 70%
• Application error rate < 0.1%
• No new database alerts

If Unstable: See "Rollback" section below.

---

Step 9: Document Failover in Incident Timeline

Purpose: Record actions for post-mortem.

Post in #incident-YYYY-MM-DD-database:

"Failover complete. New primary: secondary-db (10.0.2.20).
- Promotion: 15:20 PST
- DNS Update: 15:22 PST
- App connectivity verified: 15:25 PST
- Monitoring for stability."

---

Step 10: Update Runbook (If Steps Changed)

Purpose: Keep runbook accurate for future use.

# If any steps failed or needed modification
# Update this runbook in Git

git checkout -b update-db-failover-runbook
# Edit runbook
git add runbooks/database-failover.md
git commit -m "Update database failover runbook based on 2025-12-03 incident"
git push origin update-db-failover-runbook
# Create PR for review

---

Rollback Procedure

When to Rollback:

• New primary experiencing high error rates (> 5%)
• New primary performance worse than before failover
• Data inconsistency detected

Rollback Steps:

1. Announce Rollback:

   Post in incident channel:
   "@here Rollback initiated. Reverting to original primary."

2. Revert DNS Change:

   # Point DNS back to original primary IP
   aws route53 change-resource-record-sets \
     --hosted-zone-id Z1234567890ABC \
     --change-batch '{...}' # Original IP

3. Demote Current Primary Back to Secondary:

   # On current primary (being demoted)
   sudo -u postgres pg_ctl stop -D /var/lib/postgresql/data
   # Reconfigure as replica (create standby.signal)
   touch /var/lib/postgresql/data/standby.signal
   sudo -u postgres pg_ctl start -D /var/lib/postgresql/data

4. Verify Rollback:

• Check application error rates
• Verify database connectivity
• Monitor for 10 minutes

5. Post-Incident Action:

• Schedule post-mortem to understand why rollback was needed
• Investigate secondary database issues before retry

---

Escalation Criteria

Escalate to Senior DBA (@senior-dba-oncall) if:

• Replication lag > 1 minute (data loss risk)
• Secondary database not healthy
• Promotion fails with unclear error
• Application connectivity issues after failover
• Rollback required

Escalate to VP Engineering if:

• Estimated downtime > 30 minutes
• Data loss occurred
• Customer-facing impact severe (SEV0)

---

Common Issues and Solutions

Issue 1: Replication Lag High

Symptom: Replication lag > 5 seconds

Solutions:

• Wait: Allow replication to catch up (if lag increasing, investigate network)
• Accept Data Loss: Consult IC, promote anyway if critical
• Investigate: Check network latency, disk I/O on secondary

---

Issue 2: Secondary Not Responding

Symptom: pg_isready fails on secondary

Solutions:

• Check PostgreSQL Status: systemctl status postgresql
• Restart PostgreSQL: sudo systemctl restart postgresql
• Check Disk Space: df -h /var/lib/postgresql
• Escalate: If restart fails, escalate to senior DBA

---

Issue 3: DNS Not Propagating

Symptom: Application still connecting to old primary after DNS update

Solutions:

• Verify DNS Change: dig primary-db.example.com +short
• Wait for TTL: 60 seconds for DNS cache expiration
• Flush Application DNS Cache: Restart application pods
• Temporary Fix: Update application config directly with new IP

---

Issue 4: Application Connection Errors

Symptom: API returning database connection errors after failover

Solutions:

• Check Connection String: Verify hostname/IP correct
• Check Firewall Rules: telnet primary-db.example.com 5432
• Verify PostgreSQL Accepting Connections: sudo -u postgres psql -c "SHOW listen_addresses;"
• Restart Application: kubectl rollout restart deployment/api

---

Post-Execution Checklist

After completing failover, verify:

• [ ] Secondary promoted to primary successfully
• [ ] DNS updated to point to new primary
• [ ] Application connectivity verified
• [ ] Error rates returned to baseline
• [ ] Monitoring stable for 10+ minutes
• [ ] Incident timeline documented
• [ ] Old primary (now secondary) replication configured
• [ ] Runbook updated if steps changed
• [ ] Post-mortem scheduled (within 48 hours)

---

Related Runbooks

• RB-DB-002: Database Performance Troubleshooting
• RB-DB-003: Database Replication Repair
• RB-DB-004: Database Backup and Restore

---

Testing and Validation

Last Tested: 2025-11-15 (disaster recovery drill) Test Frequency: Quarterly (every 3 months) Next Test: 2026-02-15

Test Procedure: 1. Announce planned test in #database-operations 2. Execute failover during low-traffic window (2am PST) 3. Verify application functionality 4. Fail back to original primary 5. Document any runbook issues discovered

---

Changelog

---

Contact

Owner: Database SRE Team Slack: #database-operations On-Call: @db-sre-oncall Escalation: @senior-dba-oncall

For questions or improvements: Open PR against runbooks/database-failover.md

Escalation Matrix

Table of Contents

• Overview
• Escalation Decision Matrix
• Escalation Triggers
• Time-Based Escalation
• Severity-Based Escalation
• Expertise-Based Escalation
• Escalation Contacts
• On-Call Hierarchy
• Specialized Escalation Paths
• Escalation Communication
• Internal Notification
• Executive Notification
• Avoiding Over-Escalation
• Further Reading

Overview

Escalation ensures incidents get appropriate resources and attention based on duration, complexity, and impact. This matrix defines when and how to escalate incidents.

Escalation Decision Matrix

Escalation Triggers

Time-Based Escalation

15 Minutes - No Acknowledgment:

• Primary on-call hasn't acknowledged alert
• Auto-escalate to secondary on-call via PagerDuty/Opsgenie

30 Minutes - No Progress:

• Investigation ongoing but no mitigation path identified
• Escalate to team lead or senior engineer for guidance

60 Minutes - Extended Duration:

• Incident unresolved after 1 hour
• Escalate to engineering manager or director
• Consider declaring SEV0 if customer impact severe

Severity-Based Escalation

SEV2 → SEV1:

• Impact expands (5% → 25% of customers)
• Escalate to team lead, assign IC

SEV1 → SEV0:

• Complete outage or critical functionality lost
• Escalate to VP Engineering, notify executives

Expertise-Based Escalation

Database Issues:

• Escalate to @db-team within 15 minutes if database-related

Network/Infrastructure:

• Escalate to @infra-team if network or infrastructure issue

Security Incident:

• Escalate to @security-team immediately for security issues

Third-Party Vendor:

• Open vendor support ticket within 15 minutes
• Escalate through TAM (Technical Account Manager) if available

Escalation Contacts

On-Call Hierarchy

Primary On-Call
    ↓ (5 min no ack)
Secondary On-Call
    ↓ (15 min no resolution)
Team Lead / Senior Engineer
    ↓ (30 min no resolution OR SEV0)
Engineering Manager
    ↓ (60 min OR high business impact)
Director / VP Engineering

Specialized Escalation Paths

Database:

• Primary: @db-oncall
• Secondary: @senior-dba
• Escalation: @db-team-lead

Security:

• Primary: @security-oncall
• Immediate: @security-team-lead (for all security incidents)
• Legal: @legal-team (for data breaches)

Infrastructure:

• Primary: @infra-oncall
• Secondary: @sre-team
• Escalation: @infra-lead

Escalation Communication

Internal Notification

When escalating, post in incident channel:

@team-lead Escalating to you - no progress after 30 minutes.
Current status: [Brief summary]
Need: [Specific help needed]

Executive Notification

For SEV0 incidents, notify executives within 15 minutes:

• Who to notify: CTO, VP Engineering, CEO (for major outages)
• How: Slack DM + brief email
• What to include: Business impact, ETA, current actions

Executive Brief Template:

SEV0 Incident: [Title]
Duration: [X minutes]
Impact: [Y% of customers unable to [action]]
Current Status: [Investigating/Mitigating]
ETA: [Best estimate or "Unknown"]
IC: @[name]

Avoiding Over-Escalation

Do NOT escalate if:

• Issue being actively mitigated
• Clear path to resolution identified
• Escalation would distract without adding value

Example: Database slow, root cause identified, mitigation in progress → Do NOT escalate unless mitigation fails.

Further Reading

• PagerDuty: "Escalation Policies Best Practices"
• Google SRE: "Incident Management"