Slo Investigate

SLO Investigator

Deep-dive investigation of breaching SLOs: dimensional breakdown, alert correlation, runbook access. For experienced operators — no hand-holding.

Core Principles

1. Use gcx commands — do not call Grafana APIs directly (no curl, no HTTP libraries) 2. Trust the user's expertise — skip obvious context, get to the root cause 3. Use -o json for agent processing, default format for user display; show graphs for time-series data 4. Errors collected at the end — do not interleave error handling in workflow steps 5. Use --from/--to for all time-range commands (never --start/--end)

Investigation Workflow

Step 1: Retrieve SLO Definition

gcx slo definitions get <UUID> -o json

Extract from the JSON response:

• .metadata.name — SLO name
• .spec.query.type — query type: ratio, freeform, or threshold
• For ratio: .spec.query.ratio.successMetric, .spec.query.ratio.totalMetric, .spec.query.ratio.groupByLabels[]
• For freeform: .spec.query.freeform.query
• .spec.objectives[0].value — objective (0–1), .spec.objectives[0].window — window
• .spec.destinationDatasource.uid — Prometheus datasource UID
• .spec.alerting.fastBurn.annotations, .spec.alerting.slowBurn.annotations — runbook/dashboard URLs
• .metadata.annotations — additional runbook/dashboard references

If no UUID is given, list SLOs and ask which to investigate:

gcx slo definitions list

Step 2: Check Status with Wide Output

gcx slo definitions status <UUID> -o wide

This shows SLI, ERROR_BUDGET, BURN_RATE, SLI_1H, SLI_1D, and STATUS.

Early exit — OK status: If STATUS is OK, report health metrics and stop:

SLO: <name> — Status: OK
SLI: <value>  |  Error budget remaining: <budget>%  |  Burn rate: <rate>x
1h SLI: <sli_1h>  |  1d SLI: <sli_1d>
No action needed.

Early exit — NODATA status: If STATUS is NODATA, branch to NODATA diagnosis:

SLO: <name> — Status: NODATA
Recording rule metrics unavailable. Likely causes:
- Destination datasource misconfigured (check .spec.destinationDatasource.uid)
- Grafana recording rules not yet evaluated (can take 1–2 minutes after creation)
- Prometheus federation/remote write issue

Check: gcx datasources list --type prometheus
Then verify the destination datasource UID matches what the SLO expects.

Lifecycle states: If status is Creating/Updating/Deleting/Error, report that the SLO is in a transient state and investigate the Grafana backend.

Step 3: Render Timeline

gcx slo definitions timeline <UUID> --from now-1h --to now

For wider trends:

gcx slo definitions timeline <UUID> --from now-24h --to now

Show the graph output (default). Use it to identify when breaching started and how severe it is.

Step 4: Dimensional Breakdown

Resolve the datasource UID. If .spec.destinationDatasource.uid is set, use it. Otherwise auto-discover:

gcx datasources list --type prometheus

For ratio queries — extract success/total metric selectors and groupByLabels, then query dimensional breakdown:

# Success rate by dimension (e.g., cluster, status_code, endpoint)
gcx metrics query <datasource-uid> \
  'sum by (<groupByLabel>) (rate(<successMetric>[5m])) / sum by (<groupByLabel>) (rate(<totalMetric>[5m]))' \
  --from now-1h --to now --step 1m

# Error rate by dimension to spot the bad actor
gcx metrics query <datasource-uid> \
  'sum by (<groupByLabel>) (rate(<totalMetric>[5m])) - sum by (<groupByLabel>) (rate(<successMetric>[5m]))' \
  --from now-1h --to now --step 1m

If groupByLabels is empty, try common dimensions: cluster, namespace, service, status_code, endpoint.

For freeform queries — use the raw PromQL expression and add by (<label>) grouping:

gcx metrics query <datasource-uid> \
  '<freeform_expression> by (cluster)' \
  --from now-1h --to now --step 1m

# Also try other likely breakdown dimensions
gcx metrics query <datasource-uid> \
  '<freeform_expression> by (namespace)' \
  --from now-1h --to now --step 1m

Use graph output to display dimensional trends visually. Use -o json to extract exact values for the report.

Step 5: Search for Related Alert Rules

gcx alert rules list -o json | jq '[.[] | .rules[]? | select(.name | test("<slo-name>"; "i"))]'

Also try searching by UUID fragment if the name-based search returns no results:

gcx alert rules list -o json | jq '[.[] | .rules[]? | select(.labels.slo_uuid == "<UUID>" or (.name | test("<slo-name>"; "i")))]'

Extract for each matching rule: name, state (firing/pending/inactive), labels, and annotations.

Step 6: Extract Runbook and Dashboard URLs

Collect URLs from:

• .spec.alerting.fastBurn.annotations.runbook_url
• .spec.alerting.fastBurn.annotations.dashboard_url
• .spec.alerting.slowBurn.annotations.runbook_url
• .spec.alerting.slowBurn.annotations.dashboard_url
• .metadata.annotations.*

If a GitHub URL is found in runbook annotations and gh is available:

# Convert GitHub web URL to API path and fetch content
gh api /repos/<owner>/<repo>/contents/<path> --jq '.content' | base64 --decode

For raw GitHub URLs (raw.githubusercontent.com), extract the content URL pattern and use gh api with the equivalent API endpoint.

Output Format

After completing the investigation, present results in this structure:

SLO: <name>
Target: <objective>% over <window>  |  Status: BREACHING
SLI: <current>%  |  Error budget remaining: <budget>%  |  Burn rate: <rate>x
1h SLI: <sli_1h>%  |  1d SLI: <sli_1d>%

[Timeline graph — show default output]

Dimensional Breakdown:
  Worst dimension: <label>=<value> at <error_rate>% error rate
  [Additional dimensions ranked by error rate]

Related Alert Rules:
  - <rule_name>: <state> [labels: <key>=<value>]

Runbook: <url>
Dashboard: <url>

[If runbook fetched]: Key runbook steps:
  <relevant excerpt>

Next actions:
1. <most specific actionable step based on findings>
2. <follow-up investigation or escalation path>
3. <if budget near zero: suggest slo-optimize for objective review>

Error Handling

• gcx slo definitions get fails with 404: SLO UUID not found. Run gcx slo definitions list and confirm the UUID.
• gcx slo definitions status returns empty: No status available — SLO may be newly created. Check if recording rules are running (STATUS may show NODATA).
• gcx datasources {kind} query fails with datasource error: Datasource UID may be wrong. Run gcx datasources list --type prometheus to find the correct UID.
• gcx datasources {kind} query returns no data: The SLO metrics may write to a separate datasource (check .spec.destinationDatasource.uid). Try both the destination datasource and the default Prometheus datasource.
• alert rules list returns empty: Alert rules may be in a different folder. Try without filters: gcx alert rules list -o json | jq length to confirm total count.
• gh api fails: If gh is not authenticated or unavailable, report the runbook URL directly and skip content fetching.
• SLO has no groupByLabels (ratio query): Try common breakdown dimensions: cluster, namespace, service, endpoint, status_code. Report which ones return data.
• Multiple SLOs with similar names: When searching alert rules by name pattern, report all matches and their states — don't silently drop duplicates.

SLO PromQL Patterns

Reference for querying Grafana-generated SLO recording rule metrics.

Metric Inventory

These metrics are written by Grafana recording rules to the destination datasource configured on the SLO. They may not be available on the default Prometheus datasource — always use .spec.destinationDatasource.uid.

Patterns by Use Case

Current SLI (Full Window)

# Current SLI value for a specific SLO
grafana_slo_sli_window{slo_uuid="<uuid>"}

# All SLOs — compare SLI to objective
grafana_slo_sli_window / on(slo_uuid) grafana_slo_objective

Short-Window SLI Snapshots

# 1-hour SLI (fast feedback on recent changes)
grafana_slo_sli_1h{slo_uuid="<uuid>"}

# 1-day SLI (catch gradual degradation)
grafana_slo_sli_1d{slo_uuid="<uuid>"}

# Both together for trend comparison
grafana_slo_sli_1h{slo_uuid="<uuid>"}
grafana_slo_sli_1d{slo_uuid="<uuid>"}
grafana_slo_sli_window{slo_uuid="<uuid>"}

Error Budget

# Error budget remaining (as fraction, 0–1)
(grafana_slo_sli_window{slo_uuid="<uuid>"} - grafana_slo_objective{slo_uuid="<uuid>"}) /
(1 - grafana_slo_objective{slo_uuid="<uuid>"})

# Error budget consumed (percentage)
(1 - (grafana_slo_sli_window{slo_uuid="<uuid>"} - grafana_slo_objective{slo_uuid="<uuid>"}) /
(1 - grafana_slo_objective{slo_uuid="<uuid>"})) * 100

Burn Rate

Burn rate measures how fast the error budget is being consumed relative to the budget allocation rate.

# Burn rate = (1 - current_SLI) / (1 - objective)
# A burn rate of 1.0 = consuming budget exactly as fast as it accrues
# A burn rate of 2.0 = consuming budget 2x faster than it accrues
(1 - grafana_slo_sli_1h{slo_uuid="<uuid>"}) /
(1 - grafana_slo_objective{slo_uuid="<uuid>"})

# Short-window burn rate (1h) — used for fast-burn alerting
(1 - grafana_slo_sli_1h{slo_uuid="<uuid>"}) /
(1 - grafana_slo_objective{slo_uuid="<uuid>"})

# Long-window burn rate (24h) — used for slow-burn alerting
(1 - grafana_slo_sli_1d{slo_uuid="<uuid>"}) /
(1 - grafana_slo_objective{slo_uuid="<uuid>"})

Burn rate interpretation:

• < 1.0 — budget accruing faster than consumed; on track
• 1.0 — exactly on track to exhaust budget at end of window
• > 1.0 — breaching; budget will exhaust before window ends
• > 14.4 — fast-burn threshold for 1h window (burns 30-day budget in 2h)

Success/Total Rate (Ratio Queries)

# Real-time success rate (5-minute window)
grafana_slo_success_rate_5m{slo_uuid="<uuid>"}
grafana_slo_total_rate_5m{slo_uuid="<uuid>"}

# Derived error rate (requests/sec failing)
grafana_slo_total_rate_5m{slo_uuid="<uuid>"} - grafana_slo_success_rate_5m{slo_uuid="<uuid>"}

# Success ratio from the 5-minute rates
grafana_slo_success_rate_5m{slo_uuid="<uuid>"} / grafana_slo_total_rate_5m{slo_uuid="<uuid>"}

Objective Value

# Configured objective for comparison in expressions
grafana_slo_objective{slo_uuid="<uuid>"}

# All SLOs sorted by how far they are from their objective (worst first)
sort_desc(grafana_slo_sli_window - on(slo_uuid) grafana_slo_objective)

Querying with gcx

# Current window SLI
gcx metrics query <datasource-uid> \
  'grafana_slo_sli_window{slo_uuid="<uuid>"}' \
  --from now-1h --to now --step 1m

# Burn rate over last hour
gcx metrics query <datasource-uid> \
  '(1 - grafana_slo_sli_1h{slo_uuid="<uuid>"}) / (1 - grafana_slo_objective{slo_uuid="<uuid>"})' \
  --from now-1h --to now --step 1m

# Error budget trend (28-day window)
gcx metrics query <datasource-uid> \
  '(grafana_slo_sli_window{slo_uuid="<uuid>"} - grafana_slo_objective{slo_uuid="<uuid>"}) / (1 - grafana_slo_objective{slo_uuid="<uuid>"})' \
  --from now-28d --to now --step 1h

Notes

• slo_uuid label matches .metadata.name in the SLO definition (UUID format)
• Metrics are only available after recording rules complete their first evaluation (typically 1–2 minutes after SLO creation)
• If metrics return NODATA, verify the destination datasource UID and that Grafana recording rules are active
• For dimensional breakdown during investigation, query the raw success/total metrics directly (not the recording rule aggregates) using selectors from .spec.query.ratio.successMetric / .spec.query.ratio.totalMetric