Site Reliability Engineer

Site Reliability Engineer (SRE)

When to Use

• Define SLIs, SLOs, and error budgets per service or user journey
• Configure burn-rate alerts and reliability dashboards
• Run production readiness reviews before launch or major change
• Analyze incidents for reliability gaps and SLO impact
• Plan capacity for traffic growth and failure scenarios (N+1, regional loss)
• Measure and reduce toil; prioritize automation with highest reliability ROI
• Map dependencies and failure modes; design graceful degradation
• Gate releases on SLO/error-budget policy (canary, rollback triggers)
• Conduct chaos or game days when org maturity supports it
• Partner with engineering on reliability backlog (timeouts, retries, circuit breakers)

When NOT to Use

• Build or fix Jenkins/GitHub Actions/GitLab pipelines → devops
• Design SEV levels, on-call rotations, postmortem program → incident-management-engineer
• IAM grants, VM patching, snapshot restores → cloud-system-administrator
• Stand up VPC, RDS, or new managed services → cloud-engineer
• JMeter/k6 load tests and app profiling → performance-engineer
• Blue-green cutover playbooks and change tiers → deployment-strategist
• K8s cluster upgrades and Helm platform → cluster-deployment-engineer
• Customer status page copy and comms approval → communication-lead
• Org-wide reliability posture, tiering, investment themes → vp-of-infrastructure

Related skills

Core Workflows

1. Scope and SRE principles

Service ownership, error budget policy, boundaries with DevOps and IM.

See `references/sre_scope_and_principles.md`.

2. SLI, SLO, and error budgets

Select SLIs, set targets, alert on burn.

See `references/sli_slo_error_budgets.md`.

3. Observability for reliability

Metrics, logs, traces, alert hygiene.

See `references/observability_reliability.md`.

4. Incident response (reliability lens)

Mitigation, SLO impact, follow-up actions.

See `references/incident_reliability_response.md`.

5. Capacity, toil, and automation

Scaling, toil metrics, reliability automation.

See `references/capacity_toil_automation.md`.

6. Release reliability and resilience testing

PRR, canaries, chaos, failure modes.

See `references/release_reliability_chaos.md`.

Outputs

• SLO document — SLI definition, target, window, exclusions, owners
• Error budget report — burn %, policy actions (freeze, focus week)
• PRR checklist — pass/fail with required fixes before launch
• Reliability backlog — ranked items with estimated SLO impact
• Incident reliability summary — budget consumed, contributing factors, action items
• Capacity plan — headroom, scaling triggers, regional failover notes

Principles

• User-centric SLIs — measure what customers experience
• Error budgets drive decisions — balance velocity and reliability
• Automate toil — repetitive manual work is a reliability risk
• Blameless learning — fix systems, not people
• Progressive delivery — small releases with measurable rollback criteria

Capacity, toil, and automation

Table of contents

1. Capacity planning 2. Toil definition and measurement 3. Automation priorities 4. Dependency and blast radius

Capacity planning

Inputs:

• Traffic forecast (seasonal, launch, marketing events)
• Per-request resource cost (CPU-ms, DB queries, egress)
• Headroom target (e.g., 30% at p99 peak before scale)
• Failure scenarios — AZ loss, dependency slow (2× latency)

Outputs:

• Scale triggers (HPA, queue consumers, DB read replicas)
• Quota requests before limits block (cloud-system-administrator)
• Cost vs reliability trade documented for product

Revisit quarterly or after major architecture change.

Toil definition and measurement

Toil = manual, repetitive, automatable work that scales with service growth.

Track weekly per team:

• Hours on toil vs project work
• Top toil tasks (ticket tags)
• Toil eliminated last quarter

Cap: if toil >50% of ops time, pause new features until automation lands.

Automation priorities

Score candidates:

Prefer: auto-remediation for known failures, self-service dashboards, safe runbook bots with guardrails.

Do not automate untested destructive actions without approval gates.

Dependency and blast radius

Maintain service graph:

• Sync and async callers/callees
• Shared infrastructure (DB, cache, queue, identity)
• Critical path for top user journeys

For each dependency document:

• Timeout and retry policy
• Fallback behavior
• Failure mode (fail open vs closed — explicit choice)

Use graph to prioritize incident drills and chaos experiments in release_reliability_chaos.md.

Incident response (reliability lens)

Table of contents

1. During incident 2. SLO impact assessment 3. Production readiness review 4. Post-incident reliability actions

During incident

SRE technical lead focus (process roles from incident-management-engineer):

1. Confirm customer-facing SLI movement (not only infra green) 2. Choose mitigation: rollback, scale, traffic shed, disable feature, failover 3. Estimate error budget minutes consumed in real time 4. Avoid destructive experiments unless runbook-approved 5. Capture timeline for postmortem (metrics screenshots, deploy IDs)

Escalate to devops for pipeline/artifact issues; cloud-engineer for net-new infra gaps.

SLO impact assessment

After mitigation:

• Calculate bad events or downtime minutes in SLO window
• Update budget remaining dashboard
• Classify: transient spike vs sustained breach
• Decide if policy actions (freeze, review) trigger

Production readiness review

Run PRR before launch or material architecture change.

Outcome: Go, Go with conditions, or No-go with required fixes.

Post-incident reliability actions

Separate from IM postmortem process:

• Reliability action items — ranked by SLO impact and recurrence risk
• Detection gaps — why burn alert late or absent
• Prevention — automation, circuit breaker, capacity buffer
• SLO revision — only with data and stakeholder sign-off (rare)

Blameless tone; track completion in same system as engineering backlog.

Observability for reliability

Table of contents

1. Golden signals 2. Dashboard hierarchy 3. Alert hygiene 4. Tracing and logs

Golden signals

Per service (Google SRE):

Add USE for resources and RED for request-driven services in dashboards.

Dashboard hierarchy

1. Executive / product — SLO status, budget remaining, incident count 2. Service — SLI charts, deploy markers, dependency health 3. Debugging — per-instance, per-region, per-endpoint drill-down

Mark deployments and feature flags on SLI graphs to correlate regressions.

Alert hygiene

Rules:

• Page humans only on SLO burn or imminent customer impact
• Ticket for degradation with time to fix before breach
• No pages on non-actionable thresholds (disk 70% with autoscale)
• Every page links runbook with first three mitigation steps

Review alert-to-incident ratio monthly; silence or fix noisy alerts.

Stack implementation often shared with devops — SRE owns what to alert on, not only how to wire Prometheus/Datadog.

Tracing and logs

• Correlation ID across services for incident timelines
• Structured logs; no PII/secrets in info level
• Trace sampling: higher rate during incidents or canaries
• Log-based metrics only when no better native metric exists (cost and cardinality)

For deep APM tuning of hot paths → performance-engineer.

Release reliability and chaos

Table of contents

1. Release gates 2. Canary and rollback 3. Failure mode analysis 4. Chaos and game days

Release gates

Align with deployment-strategist for ceremony; SRE owns SLO gates:

Block promotion when fast burn alert fires during canary.

Canary and rollback

• Canary traffic % and duration defined per risk tier
• Compare golden signals canary vs control (errors, latency, saturation)
• Rollback = one command or revert PR; test quarterly
• Feature flags for kill switch without full redeploy when possible

Coordinate artifact rollback with devops pipeline owners.

Failure mode analysis

Lightweight FMEA per critical service:

Update after each SEV incident.

Chaos and game days

Prerequisites: observability, runbooks, error budget culture, IM process.

Hypothesis format: If X fails, we detect in T minutes and mitigate via Y.

Never run prod chaos during budget freeze or major launch without exec approval.

Record results; convert gaps to reliability backlog items.

SLI, SLO, and error budgets

Table of contents

1. Choosing SLIs 2. Setting SLOs 3. Error budget math 4. Policy actions 5. Multi-window burn alerts

Choosing SLIs

Good SLIs are measurable, user-aligned, and actionable.

Avoid infra-only SLIs (CPU < 80%) as primary customer SLO unless no better proxy exists.

Document exclusions: maintenance windows, vendor outages, abuse traffic (policy decision).

Setting SLOs

Start from historical performance + business need:

1. Plot SLI for 30–90 days 2. Set target slightly below sustained good period (leave headroom) 3. Define measurement window (rolling 30d common for product SLOs) 4. Assign owners (service team + SRE partner)

Example: 99.9% availability → ~43.8 min downtime budget per 30 days.

Error budget math

error_budget = 1 - SLO_target   (as proportion of events or time)
budget_remaining = budget - consumed_in_window

Track budget consumed % weekly. Report alongside deploy count and incident count.

Policy actions

When budget burn exceeds thresholds (example—tune to org):

Product and eng leads agree policy in advance — not invented during outage.

Multi-window burn alerts

Google SRE pattern — alert on fast burn (page) and slow burn (ticket):

Tune multipliers to alert noise; pair every page with runbook link.

SRE scope and principles

Table of contents

1. Role boundary 2. Core principles 3. Partnership model 4. Maturity levels

Role boundary

DevOps ships change; SRE defines how much unreliability is acceptable and measures it.

Core principles

1. Availability is a feature — explicit target, not assumed 2. Error budget — shared language between eng and product for release pace 3. Toil cap — target <50% ops time on toil; automate or eliminate 4. Simplicity — fewer moving parts beats heroic on-call 5. Gradual rollouts — detect regressions before full blast radius

Partnership model

Maturity levels

Advance one level at a time; skipping levels creates alert fatigue without trust.