Synth Manage Checks

Synthetic Monitoring Check Manager

Manage SM checks using gcx. Experienced operators — no hand-holding.

Core Principles

1. Use gcx commands; never call Grafana APIs directly (no curl, no HTTP calls) 2. Trust the user's expertise — no explanations of what SM or gcx is 3. Use -o json for agent processing; default table format for user display 4. Always dry-run before pushing: --dry-run first, actual push only on success 5. Probe names are case-sensitive — always copy-paste from gcx synth probes list

Workflow 1: Create New Check

Step 1: Determine Check Type

Use the decision table in references/check-types.md:

If unsure, ask the user what they want to test (availability, DNS, port connectivity, routing).

Step 2: List and Select Probes

gcx synth probes list

Recommend at least 3 geographically distributed probes. Copy names exactly as shown — case-sensitive. Suggest probes across different continents or regions to provide meaningful coverage (e.g., one each from North America, Europe, Asia-Pacific).

Step 3: Build YAML Definition

Use the template from references/check-types.md for the chosen type. Scaffold the file locally:

apiVersion: syntheticmonitoring.ext.grafana.app/v1alpha1
kind: Check
metadata:
  name: <job-name>   # Non-numeric = create; numeric = update
spec:
  job: <job-name>
  target: <target>
  frequency: 60000    # milliseconds; 10000-120000 typical
  timeout: 10000      # milliseconds; must be < frequency
  enabled: true
  labels:
    environment: production
    team: platform
  probes:
    - Atlanta
    - Frankfurt
    - Singapore
  alertSensitivity: medium  # none, low, medium, high
  basicMetricsOnly: false   # true = fewer metrics, lower cardinality
  settings:
    http: {}   # Replace with type-specific settings

Configuration guidance:

• frequency: critical checks 10,000–60,000ms; standard checks 60,000–300,000ms
• timeout: must be strictly less than frequency; typically 5,000–30,000ms
• alertSensitivity: high = alert if >5% failing; medium = >10%; low = >25%; none = no alerts
• basicMetricsOnly: true reduces metric cardinality (fewer label dimensions); false emits full metrics

Step 4: Dry-run and Push

# Always dry-run first
gcx synth checks push <file.yaml> --dry-run

# Push only after dry-run succeeds
gcx synth checks push <file.yaml>

Push semantics:

• Non-numeric `metadata.name` (e.g., my-api-check): creates a new check; server assigns a numeric ID and updates the local file
• Numeric `metadata.name` (e.g., 12345): updates the existing check with that ID

After creation, verify with:

gcx synth checks list
gcx synth checks status <ID>

Workflow 2: Update Existing Check

Step 1: Pull Current Definition

Fetch the specific check or all checks:

# Get single check as YAML (use ID from list output)
gcx synth checks get <ID> -o yaml > check-<ID>.yaml

# Or pull all checks to a directory
gcx synth checks pull -d ./sm-checks/

Step 2: Edit and Push

Edit the pulled YAML file (the metadata.name will be the numeric ID). Modify only the fields that need changing.

# Dry-run the update
gcx synth checks push check-<ID>.yaml --dry-run

# Apply
gcx synth checks push check-<ID>.yaml

Workflow 3: GitOps Sync (Pull/Push)

Pull all checks to local directory, edit in source control, push to apply:

# Pull all checks to directory
gcx synth checks pull -d ./sm-checks/

# Edit files as needed, then push each changed file
gcx synth checks push ./sm-checks/<file>.yaml --dry-run
gcx synth checks push ./sm-checks/<file>.yaml

For bulk push from a directory, push files individually to control which checks are updated. Review dry-run output before each push.

Workflow 4: Delete Checks

# List checks to confirm IDs
gcx synth checks list

# Delete one or more checks (by numeric ID)
gcx synth checks delete <ID>

# Skip confirmation prompt
gcx synth checks delete <ID> -f

# Delete multiple checks
gcx synth checks delete <ID1> <ID2> <ID3>

Confirm the check identity (job name and target) before deleting — use gcx synth checks get <ID> to review.

Output Format

After creating or updating:

Check: <job-name>
Target: <target>
Type: <HTTP|Ping|DNS|TCP|Traceroute>
Probes: <count> selected (<list>)
Push: SUCCESS — ID: <assigned-id>

Verify status:
  gcx synth checks status <ID>

After pull:

Pulled <N> checks to <dir>/
Files: <list of filenames>

After delete:

Deleted check <ID> (<job-name> -> <target>)

Error Handling

• "probe not found": Probe names are case-sensitive. Run gcx synth probes list and copy names exactly.
• "timeout must be less than frequency": Reduce timeout value or increase frequency.
• "invalid frequency": frequency must be between 10,000ms and 120,000ms (10s–2min).
• Dry-run fails with validation error: Fix the YAML field indicated in the error before pushing.
• Push fails with "check already exists": The check job+target combination may already exist. Use gcx synth checks list to find it and update instead of create.
• No probes available: Run gcx synth probes list; if empty, verify gcx context and SM API access.
• Complex check types (MultiHTTP, Browser, Scripted): Settings map is not fully documented. Pull an existing check of that type as a template: gcx synth checks get <ID> -o yaml.

Check Type Reference

Decision Tree

What is the target?
├── URL (https:// or http://)
│   ├── Need to trace routing/hops? → Traceroute
│   └── Standard availability/response? → HTTP
├── Hostname or IP address (no port)
│   └── → Ping
├── Domain name (DNS resolution check)
│   └── → DNS
└── host:port (TCP connectivity)
    └── → TCP

HTTP Check

Tests URL availability, response codes, response time, and optional content matching.

apiVersion: syntheticmonitoring.ext.grafana.app/v1alpha1
kind: Check
metadata:
  name: my-api-http         # non-numeric = create
spec:
  job: my-api-http
  target: https://api.example.com/health
  frequency: 60000          # 60s
  timeout: 10000            # 10s, must be < frequency
  enabled: true
  labels:
    - name: environment
      value: production
  probes:
    - Atlanta
    - Frankfurt
    - Singapore
  alertSensitivity: medium
  basicMetricsOnly: false
  settings:
    http:
      validStatusCodes: [200]    # empty = accept 2xx/3xx
      validHTTPVersions: ["HTTP/1.1", "HTTP/2.0"]
      method: GET
      noFollowRedirects: false
      tlsConfig:
        insecureSkipVerify: false
      headers: {}
      body: ""
      failIfNotSSL: false
      failIfSSL: false

Key HTTP fields:

• validStatusCodes: acceptable HTTP status codes; empty = default (2xx/3xx)
• failIfNotSSL: fail if response is not HTTPS
• noFollowRedirects: true to test redirect behavior
• body: POST body; set method: POST for non-GET

Ping Check

Tests ICMP reachability and round-trip time to a hostname or IP.

apiVersion: syntheticmonitoring.ext.grafana.app/v1alpha1
kind: Check
metadata:
  name: my-server-ping
spec:
  job: my-server-ping
  target: 10.0.1.50          # hostname or IP, no port
  frequency: 60000
  timeout: 10000
  enabled: true
  labels:
    - name: environment
      value: production
  probes:
    - Atlanta
    - Frankfurt
    - Singapore
  alertSensitivity: medium
  basicMetricsOnly: false
  settings:
    ping:
      packetCount: 3
      payloadSize: 0         # 0 = default (56 bytes)
      dontFragment: false

Key Ping fields:

• packetCount: ICMP packets per check run (1–20)
• dontFragment: true to test MTU path

DNS Check

Tests DNS resolution for a domain and validates the response.

apiVersion: syntheticmonitoring.ext.grafana.app/v1alpha1
kind: Check
metadata:
  name: my-domain-dns
spec:
  job: my-domain-dns
  target: example.com        # domain to resolve
  frequency: 120000          # 2min typical for DNS
  timeout: 10000
  enabled: true
  labels:
    - name: environment
      value: production
  probes:
    - Atlanta
    - Frankfurt
    - Tokyo
  alertSensitivity: medium
  basicMetricsOnly: false
  settings:
    dns:
      recordType: A           # A, AAAA, CNAME, MX, NS, SOA, TXT
      server: ""              # empty = use probe default resolver
      port: 53
      protocol: UDP           # UDP or TCP
      validRCodes: ["NOERROR"]
      validateAnswerRRS:
        failIfMatchesRegexp: []
        failIfNotMatchesRegexp: []

Key DNS fields:

• recordType: DNS record type to query
• server: custom resolver; empty = probe default
• validRCodes: acceptable return codes; NOERROR = success

TCP Check

Tests TCP connectivity to a host:port endpoint.

apiVersion: syntheticmonitoring.ext.grafana.app/v1alpha1
kind: Check
metadata:
  name: my-db-tcp
spec:
  job: my-db-tcp
  target: db.example.com:5432   # must be host:port format
  frequency: 60000
  timeout: 10000
  enabled: true
  labels:
    - name: environment
      value: production
  probes:
    - Atlanta
    - Frankfurt
    - Singapore
  alertSensitivity: high         # database connectivity is critical
  basicMetricsOnly: false
  settings:
    tcp:
      tls: false
      tlsConfig:
        insecureSkipVerify: false
      queryResponse: []          # protocol-level send/expect pairs

Key TCP fields:

• tls: true to initiate TLS handshake after TCP connect
• queryResponse: send/expect pairs for protocol validation

Traceroute Check

Traces the network path to a URL or hostname, measuring per-hop latency.

apiVersion: syntheticmonitoring.ext.grafana.app/v1alpha1
kind: Check
metadata:
  name: my-api-traceroute
spec:
  job: my-api-traceroute
  target: https://api.example.com   # URL or hostname
  frequency: 300000                 # 5min; traceroute is slower
  timeout: 30000                    # higher timeout for traceroute
  enabled: true
  labels:
    - name: environment
      value: production
  probes:
    - Atlanta
    - Frankfurt
    - Singapore
  alertSensitivity: low
  basicMetricsOnly: false
  settings:
    traceroute:
      maxHops: 64          # maximum hops to trace
      ptrLookup: false     # true = resolve hop IPs via PTR records
      hopTimeout: 500      # ms per hop