Slo Manage

SLO Management

Create, update, sync, and delete SLO definitions using gcx.

Core Principles

1. Use gcx commands exclusively — do not call Grafana APIs directly 2. Always run --dry-run before any push operation; proceed only if dry-run succeeds 3. Trust the user's expertise — skip explanations of SLO concepts 4. Use -o json for agent processing; default table/yaml for user display 5. Auto-resolve datasource UIDs; only ask if auto-discovery fails

Query Type Decision Table

Select query type based on what the user describes:

Metric-Pattern Decision Table

Use the metric name suffix to pick the query type when the user provides a metric name:

Guardrail: Freeform is a last resort. Before choosing freeform, verify the SLI cannot be expressed as ratio or threshold.

Hard requirement: Freeform queries MUST use $__rate_interval in all rate()/increase() calls. Literal ranges like [5m] are rejected by the SLO API.

Workflow 1: Create New SLO

Step 1: Determine query type using the decision table above

Step 2: Resolve destination datasource UID

gcx datasources list --type prometheus

Use the UID from the output. If multiple Prometheus datasources exist, ask the user which to use.

Step 3: Build YAML from the appropriate template

See references/slo-templates.md for complete templates. Key structure:

apiVersion: slo.ext.grafana.app/v1alpha1
kind: SLO
metadata:
  name: ""        # leave empty for new SLO (server assigns UUID on create)
spec:
  name: "my-api-availability"
  description: "API availability over 28 days"
  query:
    type: ratio   # freeform | ratio | threshold
    ratio:        # field matches type
      successMetric:
        prometheusMetric: http_requests_total{status!~"5.."}
      totalMetric:
        prometheusMetric: http_requests_total
      groupByLabels: [cluster, service]
  objectives:
    - value: 0.999   # 0.9 to 0.9999 typical range
      window: 28d    # 7d | 14d | 28d | 30d
  destinationDatasource:
    uid: <prometheus-uid>

Step 4: Validate with dry-run, then push

gcx slo definitions push slo.yaml --dry-run
gcx slo definitions push slo.yaml

Push semantics:

• metadata.name empty → always creates (server assigns UUID)
• metadata.name set to UUID → upsert (updates if exists, creates if not)

After creation, server assigns UUID. Run gcx slo definitions list to confirm.

Workflow 2: Update Existing SLO

Step 1: Get current definition

gcx slo definitions get <UUID> -o yaml > slo.yaml

Step 2: Modify the YAML file

Edit the relevant fields (objective value, query, alerting, etc.). Do not modify metadata.name (UUID) or readOnly fields.

Step 3: Dry-run, then push

gcx slo definitions push slo.yaml --dry-run
gcx slo definitions push slo.yaml

Workflow 3: GitOps Sync (Pull/Push)

Pull all SLOs to disk

gcx slo definitions pull -d ./slos
# Writes to ./slos/SLO/<uuid>.yaml

Push directory of SLOs

gcx slo definitions push ./slos/SLO/*.yaml --dry-run
gcx slo definitions push ./slos/SLO/*.yaml

Workflow 4: Delete SLO

Step 1: Confirm SLO identity

gcx slo definitions list
gcx slo definitions get <UUID>

Confirm the UUID and name with the user before deletion.

Step 2: Delete

gcx slo definitions delete <UUID> -f

Use -f to skip confirmation prompt when running in agent mode.

Configuration Guidance

Objective values (stored as 0–1, displayed as percentage):

• Typical range: 0.9 (90%) to 0.9999 (99.99%)
• Common starting points: 0.99 (99%), 0.999 (99.9%), 0.9999 (99.99%)

Window options: 7d, 14d, 28d, 30d

• 28d is most common; matches many SLO frameworks
• Shorter windows (7d) react faster but have higher variance

Alerting best practices:

• fastBurn: Pages on-call (high burn rate, short window — catches rapid budget consumption)
• slowBurn: Creates tickets (low burn rate, long window — catches gradual degradation)

Labels: Use consistent label keys (team, service, environment, tier) for filtering and grouping.

GroupByLabels (ratio/threshold queries): Add labels like cluster, service, endpoint for dimensional breakdown in status and investigation.

Output Format

After create/update:

SLO: <name>
UUID: <uuid>
Status: Created | Updated
Objective: <value>% over <window>
Datasource: <uid>

After pull:

Pulled <N> SLO definitions to <dir>/SLO/

After delete:

Deleted: <uuid> (<name>)

Error Handling

• Push fails with 400: Check YAML structure matches template; verify destinationDatasource.uid is valid
• Push fails with 404 on update: UUID in metadata.name not found; check with gcx slo definitions list
• Pull creates empty directory: No SLOs in this context; check gcx config view for active context
• Datasource list returns empty: No Prometheus datasources configured; ask user for UID manually
• Dry-run shows unexpected diff: Show diff to user and ask for confirmation before proceeding
• Delete fails with 404: UUID already deleted or wrong UUID; verify with gcx slo definitions list

SLO YAML Templates

Use apiVersion: slo.ext.grafana.app/v1alpha1 and kind: SLO for all SLO definitions.

---

Ratio Query Template

Use when measuring a success/total ratio (e.g., HTTP success rate, request availability).

apiVersion: slo.ext.grafana.app/v1alpha1
kind: SLO
metadata:
  name: ""                          # empty = create new; UUID = upsert existing
spec:
  name: "api-availability"          # human-readable name shown in UI
  description: "HTTP API availability measured by 2xx/total request ratio"
  query:
    type: ratio
    ratio:
      successMetric:
        prometheusMetric: http_requests_total{status!~"5.."}  # requests that succeeded
      totalMetric:
        prometheusMetric: http_requests_total                 # all requests
      groupByLabels:                # optional: enables dimensional breakdown
        - cluster
        - service
  objectives:
    - value: 0.999                  # 99.9% target (range: 0.9–0.9999 typical)
      window: 28d                   # rolling window: 7d | 14d | 28d | 30d
  labels:                           # optional: for filtering and grouping
    - key: team
      value: platform
    - key: tier
      value: critical
  alerting:
    fastBurn:                       # pages on-call: high burn rate
      annotations:
        - key: runbook_url
          value: https://github.com/myorg/runbooks/blob/main/api-slo.md
        - key: summary
          value: "SLO fast burn: API availability burning error budget rapidly"
    slowBurn:                       # creates ticket: gradual degradation
      annotations:
        - key: runbook_url
          value: https://github.com/myorg/runbooks/blob/main/api-slo.md
        - key: summary
          value: "SLO slow burn: API availability degrading over time"
  destinationDatasource:
    uid: <prometheus-datasource-uid> # resolve with: gcx datasources list --type prometheus
  folder:
    uid: <folder-uid>               # optional: omit to use root folder

---

Freeform Query Template

Use when you have a raw PromQL expression that directly expresses the SLI (must return 0–1).

apiVersion: slo.ext.grafana.app/v1alpha1
kind: SLO
metadata:
  name: ""
spec:
  name: "checkout-latency-slo"
  description: "Checkout requests completing under 500ms (freeform PromQL)"
  query:
    type: freeform
    freeform:
      # PromQL expression that returns 0.0–1.0 representing the SLI
      # Must be a ratio of good events / total events
      # REQUIRED: use $__rate_interval in all rate()/increase() calls — literal ranges (e.g. [5m]) are rejected by the SLO API
      query: >
        sum(rate(http_request_duration_seconds_bucket{job="checkout",le="0.5"}[$__rate_interval]))
        /
        sum(rate(http_request_duration_seconds_count{job="checkout"}[$__rate_interval]))
  objectives:
    - value: 0.95                   # 95% of requests complete under 500ms
      window: 28d
  labels:
    - key: team
      value: checkout
  alerting:
    fastBurn:
      annotations:
        - key: runbook_url
          value: https://github.com/myorg/runbooks/blob/main/checkout-latency.md
        - key: summary
          value: "SLO fast burn: checkout latency SLO burning error budget"
    slowBurn:
      annotations:
        - key: runbook_url
          value: https://github.com/myorg/runbooks/blob/main/checkout-latency.md
        - key: summary
          value: "SLO slow burn: checkout latency degrading"
  destinationDatasource:
    uid: <prometheus-datasource-uid>

---

Threshold Query Template

Use when measuring whether a metric stays above or below a fixed threshold.

apiVersion: slo.ext.grafana.app/v1alpha1
kind: SLO
metadata:
  name: ""
spec:
  name: "database-availability"
  description: "Database instances reporting up (threshold: up >= 1)"
  query:
    type: threshold
    threshold:
      thresholdExpression: "up{job='postgres'}"  # metric to evaluate
      threshold:
        value: 1.0                  # threshold value
        operator: gte               # operator: gte | lte | gt | lt
      groupByLabels:
        - instance
        - cluster
  objectives:
    - value: 0.9999                 # 99.99% of time threshold is met
      window: 28d
  labels:
    - key: team
      value: data
    - key: tier
      value: critical
  alerting:
    fastBurn:
      annotations:
        - key: runbook_url
          value: https://github.com/myorg/runbooks/blob/main/db-availability.md
        - key: summary
          value: "SLO fast burn: database availability threshold breached"
    slowBurn:
      annotations:
        - key: runbook_url
          value: https://github.com/myorg/runbooks/blob/main/db-availability.md
        - key: summary
          value: "SLO slow burn: database availability degrading"
  destinationDatasource:
    uid: <prometheus-datasource-uid>
  folder:
    uid: <folder-uid>