Investigate Alert

Grafana Alert Investigator

Investigate Grafana alerts by analyzing state, querying datasources, and identifying next steps. Be concise and direct - these are experienced operators who need actionable information, not hand-holding.

Core Principles

1. Stop early for non-actionable scenarios (recording rules, healthy inactive alerts) 2. Be concise - no fluff, no excessive formatting, no obvious advice 3. Trust the user's expertise - no timelines, no patronizing suggestions 4. Focus on actionable information

Prerequisites

User needs gcx installed with configured context and appropriate permissions. If gcx is not configured, use the setup-gcx skill first.

Investigation Workflow

Step 1: Verify Context and Locate Alert

Check context if needed (gcx config view). If multiple contexts exist and none specified, ask which to use.

Step 2: Get Alert Details and Check for Early Exit

Fetch the alert by listing all alerts and filtering by name. Replace <AlertName> with the actual alert name:

gcx alert rules list -o json | jq -r '.[] | .rules[]? | select(.name == "<AlertName>")'

Server-side filters (use instead of downloading all rules and filtering with jq):

• --state firing|pending|inactive — filter by rule state
• --group <name> — filter by group name
• --folder <uid> — filter by folder UID

Filter by name, state, cluster/environment as relevant. If multiple matches, list them and ask which to investigate. Inform the user which context you're using.

Check the type field:

• If type: recording: This is a recording rule, not an alerting rule. Report: "This is a recording rule (pre-calculates metrics), not an alerting rule. It doesn't fire alerts. Current state: [state]. Want details on what it's recording?" Stop here unless they ask for more.

Check the state field:

• If state: inactive AND the alert's query looks healthy: Report: "Alert is inactive. [Brief what it monitors]. Health: [health]. Last evaluated: [time]. Want to see historical trends?" Stop here unless they ask for more.
• If state: firing or state: pending: Continue with full investigation below.

Step 3: Full Investigation (Firing/Pending Alerts Only)

You should use the datasourceUID from the alert when you can.

If you need to query a different datasource (e.g., Loki for log correlation), resolve its UID first:

gcx datasources list --type loki

Annotation URLs often reference datasources by name — always resolve to UID before querying.

Query the datasource. Use -o json to get the data for yourself. Use with a graph visualization for showing a summary to the user:

# Prometheus
gcx metrics query <datasource-uid> '<query>' --from now-1h --to now --step 1m -o json
gcx metrics query <datasource-uid> '<query>' --from now-1h --to now --step 1m -o graph

# Loki
gcx logs query <datasource-uid> '<query>' --from now-1h --to now -o json
gcx logs query <datasource-uid> '<query>' --from now-1h --to now -o graph

Analyze the results: What's the current value? Spike or gradual? When did it start?

Step 4: Surface Resources and Provide Analysis

Extract from annotations:

• Runbook URLs (if the URL is a GitHub URL and gh is available, fetch with gh api)
• Dashboard links
• Descriptions

Provide concise analysis:

• Where: cluster/environment from labels
• What: affected system/service
• Trend: new spike vs ongoing
• Likely causes: code changes, infrastructure, resource exhaustion
• Customer impact: if relevant

Based on the error class, suggest follow-up queries to the user:

• Connection errors: Check endpoint availability (up{job="..."}) and pod restart counts
• Latency spikes: Check upstream service latency, database query duration metrics
• Error rate increase: Break down by endpoint/handler, correlate with recent deployments
• Resource exhaustion: Check container CPU/memory metrics and node capacity

Recommend incident creation if there's customer impact.

List specific next actions - queries to run, deployments to check, metrics to examine. If there are queries for logs or metrics you can run, then ask the user if they want you to run them. If infrastructure changes are a suspected cause, suggest to the user that you could investigate any infra-as-code repos, if they point you to them.

If the next suggested actions include looking at logs in any way, use gcx to do it.

Output Format

For recording rules or healthy inactive alerts (early exit):

This is a [recording rule / inactive alert]. [One sentence what it monitors]. State: [state]. Health: [health].

Want to see more details?

For firing/pending alerts (full investigation):

Alert: <name>
State: firing [in <cluster/env>]
Monitors: <brief what it checks>

[Show graph visualization]

Current value: <value>
Trend: <spike/gradual/sustained>

Likely causes:
- <cause 1>
- <cause 2>

Impact: <who/what affected>

Runbook: <link>
Dashboard: <link>

Next actions:
- <action 1>
- <action 2>

[If customer impact:] Recommend creating an incident - <why>.

Use minimal formatting. Avoid excessive bold text. No timelines like "within 24 hours". Trust the user to prioritize.

Error Handling

• If gcx fails, explain the error
• If no alerts match, show similar names and ask for clarification
• If datasource queries fail, note it and move on
• Multiple alerts with same name: list them all with UIDs and states, ask which to investigate

Tips

• Graph visualization is critical for understanding trends
• Compare current values to baselines when relevant
• Check labels and annotations for environment/context
• Follow runbooks when available
• Err toward recommending incident creation when customer impact is unclear

Reference

For alert JSON structure, query patterns by alert type, graph interpretation, and runbook fetching, see:

• `references/alert-investigation-patterns.md`

Alert Investigation Patterns

Reference for investigating Grafana alerts with gcx. Covers the alert JSON structure, common investigation query patterns, and graph interpretation.

---

Alert JSON Structure

gcx alert rules list -o json returns an array of alert groups. Each group contains an array of rules:

[
  {
    "name": "MyAlertGroup",
    "file": "grafana",
    "rules": [
      {
        "state": "firing",
        "name": "HighErrorRate",
        "query": "rate(http_requests_total{status=~\"5..\"}[5m]) / rate(http_requests_total[5m]) > 0.05",
        "duration": 300,
        "labels": {
          "severity": "critical",
          "cluster": "us-east-1"
        },
        "annotations": {
          "summary": "High error rate detected on {{ $labels.job }}",
          "description": "Error rate is {{ $value | humanizePercentage }}",
          "runbook_url": "https://github.com/myorg/runbooks/blob/main/alerts/HighErrorRate.md",
          "dashboard_url": "https://grafana.example.com/d/abc123"
        },
        "alerts": [
          {
            "labels": {
              "alertname": "HighErrorRate",
              "job": "api-server",
              "namespace": "production"
            },
            "annotations": { ... },
            "state": "firing",
            "activeAt": "2024-01-15T10:23:45Z",
            "value": "0.08"
          }
        ],
        "type": "alerting",
        "datasourceUID": "prometheus-uid-abc123"
      }
    ]
  }
]

Key Fields

Extracting the Alert Query

# Get the query for a specific alert
gcx alert rules list -o json | \
  jq -r '.[] | .rules[] | select(.name == "<AlertName>") | .query'

# Get the datasource UID for a specific alert
gcx alert rules list -o json | \
  jq -r '.[] | .rules[] | select(.name == "<AlertName>") | .datasourceUID'

# Get all currently firing instances with their label sets
gcx alert rules list -o json | \
  jq '.[] | .rules[] | select(.name == "<AlertName>") | .alerts[] | select(.state == "firing")'

---

JSON Response Envelopes

Quick reference for -o json output to avoid jq guessing:

---

Common Investigation Query Patterns

Latency Alerts

For P99/P95 latency alerts:

# Current latency percentiles
gcx metrics query <uid> \
  'histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))' \
  --from now-1h --to now --step 1m -o graph

# Latency by endpoint
gcx metrics query <uid> \
  'histogram_quantile(0.99, sum by(job, handler) (rate(http_request_duration_seconds_bucket[5m])))' \
  --from now-1h --to now --step 1m -o json

Error Rate Alerts

For alerts on HTTP 5xx or error rates:

# Overall error rate
gcx metrics query <uid> \
  'rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])' \
  --from now-1h --to now --step 1m -o graph

# Error rate by service
gcx metrics query <uid> \
  'sum by(job) (rate(http_requests_total{status=~"5.."}[5m])) / sum by(job) (rate(http_requests_total[5m]))' \
  --from now-1h --to now --step 1m -o json

Resource Exhaustion Alerts

For CPU, memory, or disk alerts:

# CPU usage by pod
gcx metrics query <uid> \
  'sum by(pod) (rate(container_cpu_usage_seconds_total[5m]))' \
  --from now-1h --to now --step 1m -o graph

# Memory usage
gcx metrics query <uid> \
  'container_memory_working_set_bytes{container!=""}' \
  --from now-30m --to now --step 1m -o json

# Disk free percentage
gcx metrics query <uid> \
  'node_filesystem_avail_bytes / node_filesystem_size_bytes' \
  --from now-6h --to now --step 5m -o graph

Certificate / TLS Alerts

For cert expiry alerts:

# Days until certificate expiry
gcx metrics query <uid> \
  '(certmanager_certificate_expiration_timestamp_seconds - time()) / 86400' \
  --from now-1h --to now --step 10m -o json

Availability / SLO Alerts

For availability or SLO breach alerts:

# Uptime over last hour
gcx metrics query <uid> \
  'avg_over_time(up[1h])' \
  --from now-6h --to now --step 5m -o graph

# Current up/down status
gcx metrics query <uid> \
  'up == 0' \
  --from now-15m --to now --step 1m -o json

---

Loki Log Investigation Patterns

After identifying an issue from metrics, correlate with logs:

# Find error logs for a service
gcx logs query <loki-uid> '{job="api-server"} |= "error"' \
  --from now-1h --to now -o json

# Find logs around the time the alert started firing (replace timestamp)
gcx logs query <loki-uid> '{namespace="production"} |= "error"' \
  --from 2024-01-15T10:00:00Z --to 2024-01-15T10:30:00Z -o json

# Rate of error log lines (for trend analysis)
gcx logs query <loki-uid> 'rate({job="api-server"} |= "error" [5m])' \
  --from now-2h --to now --step 1m -o graph

Querying at Scale

Loki metric queries (rate(), count_over_time(), etc.) produce one series per unique label combination. At scale this hits series limits (default 20K). Always aggregate:

# BAD — one series per pod/namespace/level/... combination
gcx logs query <loki-uid> 'count_over_time({job="app"} [5m])'

# GOOD — aggregate down to what you need
gcx logs query <loki-uid> 'sum(count_over_time({job="app"} [5m]))'
gcx logs query <loki-uid> 'sum by(level) (count_over_time({job="app"} | json [5m]))'
gcx logs query <loki-uid> 'topk(10, sum by(pod) (rate({job="app"} [5m])))'

Rule of thumb: if your query uses rate(), count_over_time(), or bytes_over_time(), wrap it with sum(), sum by(label), or topk().

Stream Labels vs Extracted Labels

Loki has two kinds of labels — confusing them causes silent failures:

Common mistakes:

• Filtering extracted labels in {} — fails silently: {namespace="prod", pod="app-123"} won't work if pod is extracted, not a stream label
• Using label_format to rename extracted fields before they're parsed — add the parser stage first
• Assuming a field visible in Grafana Explore is a stream label — check with gcx logs labels -d <uid> (only shows stream labels)

---

Interpreting Graph Output

-o graph renders an ASCII time-series chart in the terminal. Key patterns:

Use -o json after -o graph to extract exact values:

# Get the peak value during the alert window
gcx metrics query <uid> '<query>' --from now-2h --to now --step 1m -o json | \
  jq '[.data[].values[] | .value] | max'

---

Runbook Fetching

If the alert annotation contains a GitHub runbook URL, fetch it with:

gh api /repos/<owner>/<repo>/contents/<path> --jq '.content' | base64 -d

For non-GitHub URLs, use curl:

curl -s "<runbook_url>"

---

See Also

• Grafana Alert Rules documentation
• The setup-gcx skill for configuring gcx if not yet set up