Synth Investigate Check

Synthetic Check Investigator

Investigate Synthetic Monitoring check failures by triaging probe data, classifying failure scope, and identifying root cause. Experienced operators need actionable diagnosis, not hand-holding.

Core Principles

1. Use gcx commands — do not call Grafana APIs directly 2. Trust the user's expertise — skip background explanations 3. Use -o json for agent processing, default format for user display 4. Show timeline graphs for time-series data — they communicate trends faster than text 5. Collect errors; report them at the end, not interleaved in workflow steps

Prerequisites

gcx configured with an active context and appropriate permissions.

Investigation Workflow

Step 1: Get Check Status (with early exit)

gcx synth checks status <ID>

If the user provided a name instead of ID, list first:

gcx synth checks list -o json | jq -r '.[] | select(.job | test("<name>"; "i")) | [.id, .job, .target, .type] | @tsv'

Early exit — OK: Check success rate >= 50% across all probes. Report: "Check <job> is healthy. Success rate: <rate>%. <probe_count> probes up." Stop unless the user asks for more.

Early exit — NODATA: No Prometheus metrics available. 1. Get check config to verify enabled: true (gcx synth checks get <ID> -o json | jq .spec.enabled) 2. If disabled: report "Check is disabled — no metrics will appear until it is re-enabled." 3. If enabled: report "No metrics found. Check datasource config or whether the SM stack is healthy." Stop after reporting.

Step 2: Get Check Configuration

gcx synth checks get <ID> -o json

Extract: job name, target, check type (http/ping/dns/tcp/traceroute), probe list, frequency, timeout, alertSensitivity, enabled flag.

For HTTP checks also note: any assertion settings, TLS config, expected status codes.

Step 3: Timeline Triage

gcx synth checks timeline <ID> --from now-1h --to now

Show the graph output to the user. Then analyze the pattern:

Use a longer window if the failure started more than 1h ago:

gcx synth checks timeline <ID> --from now-6h --to now

Step 4: Classify Failure Scope and Map Probes

Get the probe list for geographic mapping:

gcx synth probes list -o json

Cross-reference probe IDs from the check config against probe regions. Map failing probes to their regions.

All probes failing: Target/service issue — likely target down, SSL error, or DNS failure.

Subset of probes failing: Regional or network issue. Note which regions are affected:

• Single region → ISP/CDN routing issue or regional outage
• Multiple contiguous regions → CDN edge or routing policy issue
• Probe-specific → private probe infra issue (if using private probes)

Intermittent failures: Flapping. Consider: rate limiting, timeout too tight, flaky connectivity.

Step 5: Per-Probe Breakdown via PromQL (when datasource is available)

Resolve datasource UID if not already known:

gcx datasources list --type prometheus

Run per-probe success rate to pinpoint failing probes:

gcx metrics query <datasource-uid> \
  'avg by (probe) (probe_success{job="<job>",instance="<target>"})' \
  --from now-1h --to now --step 1m -o json

Show as graph for the user:

gcx metrics query <datasource-uid> \
  'avg by (probe) (probe_success{job="<job>",instance="<target>"})' \
  --from now-1h --to now --step 1m -o graph

For HTTP checks, also run HTTP phase latency to locate where time is spent:

gcx metrics query <datasource-uid> \
  'avg by (phase) (probe_http_duration_seconds{job="<job>",instance="<target>"})' \
  --from now-1h --to now --step 1m -o graph

For SSL/TLS failures or near-expiry concerns:

gcx metrics query <datasource-uid> \
  '(probe_ssl_earliest_cert_expiry{job="<job>",instance="<target>"} - time()) / 86400' \
  --from now-1h --to now --step 5m -o json

See references/sm-promql-patterns.md for full PromQL pattern library.

Step 6: Classify Failure Mode

Cross-reference signals against references/failure-modes.md to select the most likely failure mode:

1. All probes failing + HTTP non-2xx or connection refused → Target down 2. Subset of probes failing → Regional/CDN 3. TLS handshake error or cert expiry < 14 days → SSL/TLS 4. DNS resolution errors across probes → DNS resolution 5. All probes timing out, phase latency high in connect or tls → Timeout 6. Probes reaching target but assertion fails (status code, body match) → Content/assertion 7. Single private probe failing, public probes healthy → Private probe infra 8. HTTP 429 responses, intermittent failures with backoff pattern → Rate limiting

Step 7: Diagnosis and Next Actions

Synthesize findings into an actionable report (see Output Format below).

Next actions depend on failure mode:

• Target down: Check service health, recent deployments, upstream dependencies
• Regional/CDN: Check CDN config, BGP routes, regional incidents for affected regions
• SSL/TLS: Renew cert if expiring; check intermediate chain; verify TLS config
• DNS resolution: Check DNS provider status, record TTLs, CNAME chains
• Timeout: Increase check timeout (must stay < frequency); investigate latency spikes
• Content/assertion: Check if response body/status code changed due to a deployment
• Private probe infra: Check private probe agent health and connectivity
• Rate limiting: Reduce check frequency or add allowlist for SM probe IPs

If deeper investigation is needed (e.g., logs, infra repos), ask the user if they want to proceed.

If check config needs changes (probe selection, frequency, assertions), route to synth-manage-checks.

Output Format

Early exit (OK):

Check: <job> (<target>)
Status: OK
Success rate: <rate>%
Probes up: <count>/<total>

Early exit (NODATA):

Check: <job> (<target>)
Status: NODATA
Enabled: <yes/no>
Next: <datasource check / re-enable instruction>

Full investigation:

Check: <job> (<target>)
Type: <http|ping|dns|tcp|traceroute>
Status: FAILING
Success rate: <rate>% (window: <from> – <to>)

[Timeline graph]

Failure classification: <Target down | Regional/CDN | SSL/TLS | DNS | Timeout | Content/assertion | Private probe infra | Rate limiting>

Affected probes: <count>/<total>
  - <probe-name> (<region>): failing since <time>
  - <probe-name> (<region>): intermittent

Onset: <time/duration or "unknown">

Diagnosis:
<2-4 sentences describing what the data shows and the most likely cause>

Next actions:
1. <action>
2. <action>
3. <action>

Use minimal formatting. Avoid excessive bold text. Trust the user to prioritize.

Error Handling

• gcx synth checks status returns no rows: check ID may be wrong — list all checks and confirm
• gcx synth probes list fails: skip geographic mapping; classify probes by name where possible
• gcx datasources {kind} query fails with datasource error: note it, skip PromQL steps, classify using timeline data only
• Multiple checks match the search name: list all with IDs and targets, ask which to investigate
• Timeline returns no data for the window: widen to --from now-6h --to now before concluding NODATA

SM Check Failure Modes

Reference for classifying Synthetic Monitoring check failures. Cross-reference signals from probe timeline and PromQL metrics against the table below.

Failure Mode Reference

Classification Decision Tree

All probes failing?
├─ YES → Check HTTP response code / connection error
│        ├─ Connection refused / timeout → Target down
│        ├─ TLS error / cert expiry <14d → SSL/TLS
│        ├─ DNS NXDOMAIN / timeout → DNS resolution
│        └─ HTTP 429 → Rate limiting
└─ NO → Subset failing?
         ├─ Geographic cluster → Regional/CDN
         ├─ Only private probes → Private probe infra
         └─ Intermittent all probes → Timeout or Flapping
              └─ Check phase latency → Timeout if connect/tls high

Probes reach target but probe_success=0?
└─ Content/assertion failure (status code or body mismatch)

Key Metrics for Classification

• probe_success — binary pass/fail per probe
• probe_http_status_code — HTTP response code (4xx/5xx indicate assertion or target error)
• probe_http_duration_seconds{phase} — time per HTTP phase (dns, connect, tls, processing, transfer)
• probe_ssl_earliest_cert_expiry — Unix timestamp of earliest cert expiry; subtract time() and divide by 86400 for days remaining
• probe_dns_lookup_time_seconds — DNS resolution latency; high values suggest DNS issues

SM PromQL Patterns

PromQL query patterns for Synthetic Monitoring metrics. Run via:

gcx metrics query <datasource-uid> '<query>' --from <start> --to <end> --step <step>

Replace <job> with the check job name and <instance> with the check target (URL or hostname).

---

probe_success Rate Over Time

Overall success rate across all probes (1 = pass, 0 = fail):

avg(probe_success{job="<job>",instance="<target>"})

Per-probe success rate (use to identify which probes are failing):

avg by (probe) (probe_success{job="<job>",instance="<target>"})

Success rate as a percentage over a 5-minute window:

100 * avg_over_time(probe_success{job="<job>",instance="<target>"}[5m])

Recommended step: 1m for short windows (< 3h), 5m for longer windows.

---

HTTP Phase Latency

Time spent in each HTTP phase (dns, connect, tls, processing, transfer):

avg by (phase) (probe_http_duration_seconds{job="<job>",instance="<target>"})

Total HTTP request duration:

avg(probe_http_duration_seconds{job="<job>",instance="<target>",phase="transfer"})
+ avg(probe_http_duration_seconds{job="<job>",instance="<target>",phase="processing"})
+ avg(probe_http_duration_seconds{job="<job>",instance="<target>",phase="tls"})
+ avg(probe_http_duration_seconds{job="<job>",instance="<target>",phase="connect"})
+ avg(probe_http_duration_seconds{job="<job>",instance="<target>",phase="dns"})

Phase latency per probe (use to identify regional latency differences):

avg by (probe, phase) (probe_http_duration_seconds{job="<job>",instance="<target>"})

DNS resolution latency (use to identify DNS issues):

avg by (probe) (probe_http_duration_seconds{job="<job>",instance="<target>",phase="dns"})

TLS handshake latency (high values suggest cert chain or cipher negotiation issues):

avg by (probe) (probe_http_duration_seconds{job="<job>",instance="<target>",phase="tls"})

---

SSL Certificate Expiry

Days until earliest certificate expiry (negative means already expired):

(probe_ssl_earliest_cert_expiry{job="<job>",instance="<target>"} - time()) / 86400

Alert threshold check — certs expiring within 14 days:

(probe_ssl_earliest_cert_expiry{job="<job>",instance="<target>"} - time()) / 86400 < 14

Per-probe cert expiry (use to detect inconsistent cert deployment across regions):

min by (probe) ((probe_ssl_earliest_cert_expiry{job="<job>",instance="<target>"} - time()) / 86400)

---

Per-Probe Error Rates

Failure rate by probe (1 = always failing, 0 = always passing):

1 - avg by (probe) (probe_success{job="<job>",instance="<target>"})

Count of failed probe executions over 10-minute window:

count by (probe) (probe_success{job="<job>",instance="<target>"} == 0)

HTTP status codes by probe (identify 4xx/5xx patterns):

avg by (probe) (probe_http_status_code{job="<job>",instance="<target>"})

---

Useful Filters

Filter by specific probe (use probe name from gcx synth probes list):

probe_success{job="<job>",instance="<target>",probe="<probe-name>"}

Filter by multiple probes (regex):

probe_success{job="<job>",instance="<target>",probe=~"<probe1>|<probe2>"}

---

Recommended Gcx Commands

Quick per-probe check (JSON for agent processing):

gcx metrics query <datasource-uid> \
  'avg by (probe) (probe_success{job="<job>",instance="<target>"})' \
  --from now-1h --to now --step 1m -o json

Graph for user display:

gcx metrics query <datasource-uid> \
  'avg by (probe) (probe_success{job="<job>",instance="<target>"})' \
  --from now-1h --to now --step 1m -o graph

Cert expiry check:

gcx metrics query <datasource-uid> \
  '(probe_ssl_earliest_cert_expiry{job="<job>",instance="<target>"} - time()) / 86400' \
  --from now-5m --to now --step 1m -o json