Incident Management Engineer

Incident Management Engineer

When to Use

• Define or revise severity levels and escalation policies
• Design on-call rotations, schedules, and handoffs
• Integrate alerting → paging → incident channel → ticket timeline
• Run blameless postmortem process and action-item tracking
• Report incident metrics and improve MTTR/MTTD
• Configure status page and customer comms workflows for outages

When NOT to Use

• Fix pipelines, deploys, or service code during outage → devops, fullstack-software-engineer
• Investigate malware, phishing, or SOC alerts → soc-analyst (deep hunts → defensive-security-analyst)
• Enterprise security IR and legal/compliance program → cybersecurity
• Cross-team launch programs and RAID → technical-program-manager
• Data platform-specific ops → data-system-ops-lead
• Write customer-facing runbooks only → tech-writer-researcher
• Single-account repro and support escalations → support-engineer

Related skills

Core Workflows

1. Severity and escalation

1. Align severity to customer impact, not alert noise 2. Map each level: response time, who pages, comms required 3. Document escalation ladder (primary → secondary → manager → exec) 4. Review quarterly with recent incident data

See `references/severity_escalation.md` for matrix template.

2. On-call program

• Primary + secondary coverage; no single point of failure
• Rotation length: prefer weekly over daily for sustainability
• Fairness: track pages per person; cap repeat pages
• Handoff ritual with open incidents and deploy context

See `references/on_call_design.md` for rotation and handoff patterns.

3. Incident lifecycle tooling

Standard flow:

Alert → page → incident declared → comms channel → roles assigned → mitigate → resolve → postmortem

• Auto-create incident record with timeline (who/when)
• Integrate chat, tickets, and paging in one timeline
• Reserve manual steps for role assignment and customer comms approval

See `references/incident_tooling.md` for integration checklist.

4. Active incident (commander-lite)

During SEV1–2:

• Time-box updates (e.g., every 30 min until stable)
• Log decisions in incident timeline
• Defer root-cause deep dive until mitigated

See `references/incident_lifecycle.md` for phases.

5. Postmortem program

• Blameless; focus on systems and process
• Within 48h for SEV1–2; required before closing incident
• Action items: owner, due date, tracked to completion
• Share learnings broadly; link detection gaps to monitoring (devops)

See `references/postmortem_process.md` for template and metrics.

6. Metrics and improvement

Track monthly:

• Incident count by severity
• MTTD, MTTR (mitigation and full resolution)
• Repeat incidents (same root cause class)
• Postmortem action item closure rate
• On-call load (pages per engineer)

When to load references

• SEV matrix and escalation → references/severity_escalation.md
• Rotations and handoffs → references/on_call_design.md
• Lifecycle phases → references/incident_lifecycle.md
• PagerDuty/Slack/ticket wiring → references/incident_tooling.md
• Postmortems and metrics → references/postmortem_process.md

Incident lifecycle

Table of contents

1. Phases 2. Roles

Phases

1. Detect — alert or report; confirm user impact 2. Declare — assign SEV; open incident record and channel 3. Mitigate — restore service; may be workaround first 4. Resolve — fix deployed; monitors green 5. Review — postmortem; action items

Do not skip declare—informal war rooms lose timeline and ownership.

Roles

• Incident commander: coordinates; assigns tasks; time-boxes
• Comms lead: internal updates; drafts external status
• Tech leads: mitigation per component
• Scribe: timeline entries (can rotate)

Commander does not need to be the deepest debugger.

Incident tooling

Table of contents

1. Integration checklist 2. Timeline hygiene

Integration checklist

• [ ] Monitoring alert → paging service (dedupe, enrichment)
• [ ] Page ack → auto-create incident + Slack channel
• [ ] Ticket linked (Jira/Linear) with SEV field
• [ ] Status page integration for SEV1–2
• [ ] Postmortem doc template linked on close
• [ ] Metrics export for MTTD/MTTR dashboards

Timeline hygiene

Log with timestamps:

• SEV declared and roles assigned
• Customer comms sent
• Mitigation applied
• Resolution confirmed
• Postmortem scheduled

Avoid editing past entries; add corrections as new lines.

On-call design

Table of contents

1. Rotation patterns 2. Handoff template

Rotation patterns

Rules:

• Secondary must be able to page primary’s manager if no ack
• Limit pages from non-production environments
• Comp time or rotation fairness reviews monthly

Handoff template

## On-call handoff — [date]
### Open incidents (ID, SEV, owner, next step)
### Deploys / changes last 24h
### Flapping or noisy alerts
### Known risks this week

Postmortem process

Table of contents

1. Template 2. Metrics

Template

# Postmortem — [incident ID] — [title]
## Summary (1 paragraph)
## Impact (duration, users, revenue if known)
## Timeline (UTC)
## Root cause
## Contributing factors
## What went well
## What went poorly
## Action items
| Item | Owner | Due | Type (detect/mitigate/process) |

Blameless language; no individual blame.

Metrics

Review trends in monthly ops review.

Severity and escalation

Table of contents

1. Severity matrix 2. Escalation triggers

Severity matrix

Define mitigation vs resolution separately in runbooks.

Escalation triggers

Escalate when:

• SEV not reduced within target window
• Cross-team dependency blocks mitigation
• Customer or legal/comms involvement needed
• Unclear ownership after 15 min on SEV1

Document named approvers for customer-facing messages.