Sla Slo Engineer

SLA & SLO Engineer

When to Use

• Select SLIs and document measurement queries, exclusions, and data sources
• Set SLO targets, rolling windows, and per-journey or per-tier policies
• Define error-budget math, consumption tracking, and policy actions (freeze, focus)
• Design multi-window burn-rate alert policies and severity routing
• Align customer-facing SLAs with internal SLOs (credits, measurements, carve-outs)
• Tier services by criticality and map tiers to targets and review cadence
• Run SLO review meetings, executive summaries, and quarterly governance
• Estimate capacity headroom implied by latency or availability targets
• Publish SLO specs for engineering (YAML/JSON schema, dashboard contracts)

When NOT to Use

• Lead outage mitigation, paging, or on-call rotations → site-reliability-engineer, incident-management-engineer
• Negotiate contract language, credits, or legal remedies → commercial-counsel
• Build metrics/log/trace pipelines, collectors, or alertmanager config → devops, platform-engineer
• Profile application code, load-test, or tune queries only → performance-engineer
• Run production readiness reviews, chaos games, or release cutover → site-reliability-engineer, deployment-strategist
• Coordinate multi-team program milestones without SLO scope → technical-program-manager

Related skills

Core Workflows

1. Scope and principles

Service-level taxonomy, user-centric measurement, boundaries with SRE and legal.

See `references/sla_slo_scope_and_principles.md`.

2. SLI selection and measurement

Choose SLIs, define queries, exclusions, and validation.

See `references/sli_selection_and_measurement.md`.

3. SLO targeting and error budgets

Targets, windows, budget math, and policy actions.

See `references/slo_targeting_and_error_budgets.md`.

4. Alerting, burn rates, and policies

Multi-window alerts, routing, and noise control.

See `references/alerting_burn_rates_and_policies.md`.

5. Customer SLA vs internal SLO

Contract alignment, credits, carve-outs, and communication.

See `references/customer_sla_vs_internal_slo.md`.

6. Reporting, review, and governance

Cadences, dashboards, specs, and executive reporting.

See `references/reporting_review_and_governance.md`.

Outputs

• SLO specification — SLI definition, query, target, window, exclusions, owners, tier
• Error-budget policy — thresholds, actions, escalation, link to release policy
• Burn-rate alert policy — windows, multipliers, severity, runbook links
• SLA/SLO alignment matrix — customer metric ↔ internal SLI, measurement gaps, carve-outs
• Tier catalog — criticality definitions with default targets and review cadence
• SLO review pack — budget consumed, trends, top burners, proposed target changes
• Capacity note — headroom vs latency/availability target (when in scope)

Principles

• Measure user outcomes — availability and latency of journeys, not vanity infra metrics
• Internal SLO stricter than external SLA — buffer for measurement lag and goodwill
• Policy before panic — error-budget actions agreed before budget exhaustion
• Alerts prove SLO risk — every page ties to budget burn or imminent breach
• Govern with data — reviews change targets from evidence, not anecdotes
• Hand off execution — SRE and IM own incident response; this skill owns the level definitions

When to load references

• Scope and taxonomy → references/sla_slo_scope_and_principles.md
• SLI design → references/sli_selection_and_measurement.md
• Targets and budgets → references/slo_targeting_and_error_budgets.md
• Burn alerts → references/alerting_burn_rates_and_policies.md
• Customer SLA → references/customer_sla_vs_internal_slo.md
• Reviews and governance → references/reporting_review_and_governance.md

Alerting, burn rates, and policies

Table of contents

1. Alert philosophy 2. Multi-window burn rates 3. Google SRE multipliers 4. Severity routing 5. Noise control 6. Policy document template 7. Handoff to implementation

Alert philosophy

Page on budget risk, not on symptoms alone.

Symptom alerts (CPU, single 500 spike) support debugging; they do not replace burn alerts for SLO governance.

Multi-window burn rates

Burn rate = how fast error budget is consumed relative to steady consumption for the window.

steady_burn = error_budget / window_length
actual_burn = budget_consumed_in_short_window / short_window_length
burn_rate = actual_burn / steady_burn

Example: 99.9% monthly budget → 0.1% bad allowed. If 0.1% bad occurs in 1 hour, burn rate ≈ 720× for that hour (illustrative—use your recording rules).

Google SRE multipliers

For 99.9% availability over 30d (error budget 0.001):

For 99.95% (budget 0.0005), scale multipliers proportionally or recompute from budget math.

Latency SLOs: apply same structure on (1 - latency_sli) as error rate.

Document org-specific tuning when alert noise > 2 false pages per quarter.

Severity routing

Every page includes:

• Service, SLO name, window, current burn %
• Link to SLO dashboard and runbook (runbook authored by SRE)
• Recent deploys and dependency status

Noise control

False positive review: monthly with on-call + SLA/SLO engineer; adjust multipliers, not SLO target, first.

Policy document template

alert_policy:
  service: checkout-api
  slo: availability
  objective: 0.999
  windows:
    - length: 1h
      burn_multiplier: 14.4
      severity: page
      route: pagerduty-checkout
    - length: 6h
      burn_multiplier: 6
      severity: page
      route: pagerduty-checkout
    - length: 3d
      burn_multiplier: 1
      severity: ticket
      route: jira-reliability
    - length: 30d
      burn_multiplier: 1
      severity: review
      route: slo-review-queue
  maintenance:
    suppress: true
    require_ticket: true
  runbook_url: https://wiki.example/runbooks/checkout-availability

Store alongside SLO spec; version in Git.

Handoff to implementation

This skill delivers policy YAML + acceptance tests (e.g., simulate burn in staging). SRE validates paging during game day.

Customer SLA vs internal SLO

Table of contents

1. Purpose and risk 2. Alignment matrix 3. Measurement gaps 4. Buffers and carve-outs 5. Credits and remedies 6. Communication 7. Legal handoff

Purpose and risk

Risk: promising SLA metrics you cannot measure or meet with internal SLO + ops maturity.

Alignment matrix

Maintain per product or per contract tier:

Rules:

1. Every SLA metric maps to one primary SLI or explicit “non-SLO process” 2. Internal SLO stricter than SLA commitment unless executive risk acceptance 3. Document measurement endpoint (region, API version) for both sides

Measurement gaps

Common mismatches to resolve before signing:

Provide measurement appendix to legal (commercial-counsel) with query definitions—not legal advice.

Buffers and carve-outs

Recommended buffers (starting points)

Adjust for measurement error (synthetic vs real traffic, sampling).

Standard carve-outs (engineering input)

• Scheduled maintenance (notice period per tier)
• Force majeure / provider outage (define provider list)
• Customer-caused issues (misconfiguration, quota)
• Beta/preview SKUs (no SLA or separate doc)
• DDoS / abuse beyond reasonable rate limits

Each carve-out must be machine-enumerable or ticket-linked for audit.

Credits and remedies

Engineering supplies:

• Breach detection query matching contract definition
• Monthly SLA report raw data (redacted)
• Incident timeline IDs for disputed periods

Do not promise credit automation unless finance + legal approve.

Communication

Never publish internal SLO targets as customer commitments without legal review.

Legal handoff

Escalate to commercial-counsel when:

• SLA metric not measurable with current telemetry
• Credit triggers on metrics engineering disagrees with
• Unlimited liability or uncapped credits tied to availability
• SLA applies to dependencies outside your control without carve-out

This skill stops at technical feasibility and measurement spec; counsel owns contract language.

Reporting, review, and governance

Table of contents

1. Reporting layers 2. SLO review meeting 3. Executive summary 4. Governance forum 5. Target change process 6. Metrics for the program 7. Integration with TPM and SRE

Reporting layers

Automate operational and tactical from SLO spec queries; minimize manual spreadsheets.

SLO review meeting

Attendees: service owner, product owner, SRE partner, optional SLA/SLO engineer facilitator.

Agenda (45–60 min):

1. SLI snapshot — current vs objective, 30d trend 2. Budget consumed — % and policy tier 3. Incidents — budget attribution (facts from IM/SRE) 4. Deploy correlation — releases since last review 5. Top burners — routes, regions, dependencies 6. Actions — reliability backlog, target change proposals 7. Decisions — freeze lift, target change PR, capacity ask

Outputs: decision log, Jira/Linear items, updated spec PR if targets change.

Review pack template

# SLO Review — {service} — {date}

## Summary
- Tier: T0 | Budget consumed: 62% | Policy: freeze risky launches

## SLOs
| SLO | Objective | Current (30d) | Burn % | Trend |
|-----|-----------|---------------|--------|-------|
| availability | 99.95% | 99.97% | 40% | stable |
| latency | 99% <300ms | 98.2% | 80% | degrading |

## Incidents (budget impact)
- INC-1234: 12 min availability burn — root cause: DB failover

## Proposed changes
- [ ] Tighten pool timeouts (reliability backlog #456)
- [ ] Propose latency objective 98.5% → 99% after Q3 capacity

## Decisions
- Maintain release freeze until latency burn <50%

Executive summary

One page maximum:

• Red / yellow / green per T0/T1 service by budget consumed
• Customer SLA risk — any service within buffer margin of contract breach
• Investment asks — capacity, staffing, dependency upgrades
• Wins — targets met after reliability work

Avoid raw metric dumps; link to dashboards.

Governance forum

Quarterly SLO council (optional at scale):

Decisions:

• Tier promotion/demotion
• Org-wide burn multiplier defaults
• New mandatory spec fields
• Exception registry for relaxed targets

Target change process

Proposal (data) → Peer review (SRE) → Product sign-off → Legal if SLA-bound
       → PR to slo-spec repo → Announce in review → Effective date

Effective date: next rolling window start unless emergency safety issue.

Metrics for the program

Measure the SLO program, not only services:

Integration with TPM and SRE

Differentiation reminder: site-reliability-engineer runs error-budget response during incidents and PRRs; this skill owns definitions, alignment, and governance cadence for service levels.

SLA and SLO scope and principles

Table of contents

1. Definitions 2. Role boundaries 3. Service taxonomy and tiers 4. Design principles 5. Lifecycle 6. Anti-patterns

Definitions

SLA ≠ SLO. An SLA is a business/legal artifact; an SLO is an engineering control loop. Never set customer SLA equal to internal SLO without a documented buffer.

Role boundaries

Differentiate from site-reliability-engineer/references/sli_slo_error_budgets.md: that reference supports operating reliability (dashboards, PRR, incident lens). This skill owns governance—tiers, customer alignment, review cadence, and publishable SLO specs.

Service taxonomy and tiers

Define 3–5 tiers max to avoid analysis paralysis:

Document per tier:

• Default latency SLO (if any)—e.g., % requests < 300ms at p99
• Whether multi-region or single-region SLO applies
• Maintenance and force majeure handling in SLA vs SLO
• Required on-call and IM tier (coordinate with incident-management-engineer)

Design principles

1. User-journey SLIs — measure what customers experience (success + latency), not CPU. 2. Fewer, better SLOs — 2–4 SLOs per service beat twelve correlated metrics. 3. SLO drives prioritization — error budget links to release and roadmap decisions. 4. Buffer below SLA — internal SLO target ≥ SLA metric + measurement margin (often 0.05–0.1%). 5. Explicit exclusions — maintenance, client errors, abuse; document in spec and contract. 6. Versioned specs — SLO YAML in repo; changes via PR with reviewer from SRE + product.

Lifecycle

Discover journeys → Select SLIs → Baseline → Set SLO → Policy + alerts → Publish spec
       ↑                                                              ↓
       └──────────── Quarterly review ←── Report burn ←── Operate ────┘

Anti-patterns

SLI selection and measurement

Table of contents

1. Choosing SLIs 2. Availability SLIs 3. Latency SLIs 4. Freshness and throughput 5. Measurement implementation 6. Data quality and validation 7. SLO spec schema

Choosing SLIs

Good SLIs are measurable, user-aligned, and actionable.

Avoid primary SLO on infra-only signals unless no better proxy exists—and document the mapping risk.

Availability SLIs

Request-based (HTTP/gRPC)

availability_sli = successful_requests / eligible_requests

Successful usually: HTTP 2xx/3xx or gRPC OK. Eligible excludes:

• Client errors (4xx) — policy choice; document in spec
• Requests rejected by WAF for known abuse
• Synthetic probes if mixed with real traffic (prefer separate SLI)

Time-based (batch, workers)

availability_sli = uptime_minutes / total_minutes

Use when request counting is meaningless (queue workers, control loops).

Partial availability

For multi-region: define whether SLO is global or per-region minimum. Global user journey often needs weighted combination, not min of regions.

Latency SLIs

Common pattern: proportion under threshold:

latency_sli = count(latency < T) / count(eligible_requests)

Align threshold T with product expectations and capacity plans. Changing T without rebaselining invalidates history.

Exclusions: health checks, internal admin routes, prefetch endpoints—list in spec.

Freshness and throughput

Pair freshness SLO with backfill and late data policies in data-system-ops-lead when warehouse-bound.

Measurement implementation

Data sources (pick one primary)

Rule: one authoritative SLI source per SLO; others are debug only.

Label cardinality

• Slice by tier, region, method sparingly
• Do not SLO per customer ID
• Use recording rules or aggregate tables for 30d windows

Example PromQL-style recording (illustrative)

# 30d rolling availability - implementation varies by backend
sum(rate(http_requests_total{status=~"2..|3.."}[30d]))
/
sum(rate(http_requests_total{code!~"4.."}[30d]))

Document exact query in SLO spec; implementation owned by devops / observability owners.

Data quality and validation

Before committing an SLO:

1. Compare sources — LB vs app metrics; reconcile >0.1% gap 2. Missing data — treat gaps as bad or exclude? (document) 3. Deploy markers — SLI shifts after release? expected? 4. Seasonality — Black Friday baseline separate? 5. Synthetic mix — keep <5% of eligible requests unless SLO is probe-only

Validation checklist:

• [ ] 90 days history plotted
• [ ] Incidents overlay matches intuition
• [ ] Exclusions reviewed by product + legal (if customer-facing)
• [ ] Query reviewed in PR by SRE (site-reliability-engineer)

SLO spec schema

Publish machine-readable specs (YAML example):

service: checkout-api
tier: T0
owner: team-checkout
slos:
  - name: availability
    sli:
      type: request_availability
      source: prometheus
      query_ref: recording/checkout_availability_30d
      eligible: "code!~'4..'"
      success: "code=~'2..|3..'"
    objective: 0.999
    window: rolling_30d
    exclusions:
      - maintenance_windows
      - documented_vendor_outages
  - name: latency_p99_under_300ms
    sli:
      type: latency_threshold
      threshold_ms: 300
      percentile: 0.99
      query_ref: recording/checkout_latency_good_30d
    objective: 0.99
    window: rolling_30d
budget_policy_ref: policies/t0_error_budget.yaml
alert_policy_ref: policies/t0_burn_multiwindow.yaml

Version spec in Git; link from service catalog or platform-engineer portal when available.

SLO targeting and error budgets

Table of contents

1. Setting targets 2. Measurement windows 3. Error budget math 4. Budget consumption tracking 5. Policy actions 6. Multi-SLO budgets 7. Capacity implications

Setting targets

Process:

1. Plot each SLI for 30–90 days (exclude known anomalies) 2. Identify sustained good performance (p50 of monthly aggregates) 3. Set target slightly below sustained good to leave headroom (or match product minimum) 4. Compare to tier defaults and customer SLA (must be stricter than SLA metric) 5. Record rationale in review pack when deviating from tier default

Latency targets: derive from percentile history; avoid targets no month has ever met.

Target change rules:

• Increase strictness: requires capacity or reliability investment plan
• Relax target: requires product sign-off + customer comms if SLA-bound
• Never change target retroactively without versioning spec

Measurement windows

Recommendation: internal SLO on rolling 30d; SLA may use calendar month—document conversion if dashboards differ.

Error budget math

Proportion-based (availability, latency good %)

error_budget = 1 - objective          # e.g., 0.001 for 99.9%
budget_consumed = 1 - current_sli   # over same window
budget_remaining = error_budget - budget_consumed

Time-based

allowed_downtime = (1 - objective) * window_minutes
consumed_downtime = sum(incident_bad_minutes)

Budget consumed %

burn_pct = budget_consumed / error_budget * 100

Report burn_pct weekly in SLO review; tie to deploy count and incident count for context.

Budget consumption tracking

Dashboard minimum:

Attribute consumption to:

• Deploy correlation — release introduced regression?
• Dependency — upstream SLO burn?
• Traffic shape — new cohort or abuse?

Hand off deep incident analysis to site-reliability-engineer; this skill tracks accounting of budget.

Policy actions

Agree in advance with product and engineering leadership:

Recovery: define what restores budget (rolling window roll-off vs explicit reset)—usually time-based roll-off only.

Link to ci-cd-engineer / deployment-strategist when pipelines enforce gates on burn thresholds.

Multi-SLO budgets

When a service has availability and latency SLOs:

Do not multiply independent probabilities without explaining to stakeholders.

Capacity implications

Latency SLOs imply headroom:

Document minimum capacity to meet SLO in spec appendix when T0/T1:

• RPS at objective latency
• Dependency timeouts aligned with latency threshold
• Autoscaling bounds

Escalate capacity funding when sustained burn >50% without incidents (chronic tail latency).