Cx Alerts

Alert Management Skill

Use this skill to list, inspect, create, delete, enable, and disable Coralogix alert definitions using the cx alerts CLI commands.

CLI Commands

Output format: append -o json or -o agents to list, get, and create commands for machine-readable output.

Multi-profile: use -p <profile> (repeatable) to target multiple profiles simultaneously.

Alert Types Reference

Coralogix supports 12 alert types:

Priority Levels

Always ask the user what priority to use when creating alerts:

Create Workflow

1. Ask the user what they want to alert on (logs, metrics, traces) 2. Ask for priority (P1–P5) 3. Build the JSON payload with alertDefProperties - use the API wire format (see references/alert-schemas.md for all enum values) 4. Tip: use cx alerts get <existing-id> -o json to get a working template, modify it, and pipe into create 5. Create using: echo '<json>' | cx alerts create or cx alerts create --from-file alert.json 6. Verify with cx alerts list --name "<alert name>"

Important structural note: The type field is a string enum (e.g. "ALERT_DEF_TYPE_LOGS_THRESHOLD"), and the alert type config (e.g. "logsThreshold": {...}) is a sibling field at the same level - NOT nested inside type.

Example: Logs Threshold Alert

{
  "alertDefProperties": {
    "name": "High Error Rate",
    "description": "Alert when error logs exceed threshold",
    "priority": "ALERT_DEF_PRIORITY_P2",
    "type": "ALERT_DEF_TYPE_LOGS_THRESHOLD",
    "enabled": true,
    "logsThreshold": {
      "logsFilter": {
        "simpleFilter": {
          "luceneQuery": "severity:ERROR",
          "labelFilters": {
            "applicationName": [
              { "operation": "LOG_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED", "value": "my-app" }
            ]
          }
        }
      },
      "rules": [{
        "condition": {
          "conditionType": "LOGS_THRESHOLD_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED",
          "threshold": 100,
          "timeWindow": {
            "logsTimeWindowSpecificValue": "LOGS_TIME_WINDOW_VALUE_MINUTES_5_OR_UNSPECIFIED"
          }
        }
      }]
    }
  }
}

Example: Metric Threshold Alert

{
  "alertDefProperties": {
    "name": "CPU Usage Critical",
    "priority": "ALERT_DEF_PRIORITY_P1",
    "type": "ALERT_DEF_TYPE_METRIC_THRESHOLD",
    "enabled": true,
    "metricThreshold": {
      "metricFilter": { "promql": "avg(cpu_usage_percent)" },
      "rules": [{
        "condition": {
          "conditionType": "METRIC_THRESHOLD_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED",
          "threshold": 90,
          "ofTheLast": { "dynamicDuration": "5m" },
          "forOverPct": 100
        }
      }]
    }
  }
}

Example: Logs Immediate Alert

{
  "alertDefProperties": {
    "name": "OOM Killer Detected",
    "description": "Alert immediately when OOM killer runs",
    "priority": "ALERT_DEF_PRIORITY_P1",
    "type": "ALERT_DEF_TYPE_LOGS_IMMEDIATE_OR_UNSPECIFIED",
    "enabled": true,
    "logsImmediate": {
      "logsFilter": {
        "simpleFilter": {
          "luceneQuery": "\"Out of memory\" OR \"OOM\"",
          "labelFilters": {}
        }
      }
    }
  }
}

Investigation Workflow

Find firing alerts

# List all alerts and look for ALERTING status
cx alerts list -o json | jq '.[] | select(.status == "ALERTING")'

# Filter by name
cx alerts list --name "error"

Inspect a specific alert

cx alerts get <alert-id>
cx alerts get <alert-id> -o json

Disable a noisy alert (temporary mute)

cx alerts disable <alert-id>
# Later, re-enable:
cx alerts enable <alert-id>

Suppression Rules

Manage alert suppression rules that mute alerts during maintenance windows or known noisy periods.

# List suppression rules
cx alerts suppression-rules list -o json

# Create from template
cx alerts suppression-rules get <existing-id> -o json > suppression-rule.json
# Edit suppression-rule.json
cx alerts suppression-rules create --from-file suppression-rule.json

Key Principles

• Always ask for priority (P1–P5) when creating alerts - never assume
• Use `--name` filter for large accounts with many alerts
• Use `-o json` with `jq` for filtering and transformation
• Use `--from-file -` to pipe JSON from stdin when constructing alerts programmatically
• Verify after create - always list or get the alert after creation to confirm
• Disable, don't delete - prefer disabling alerts over deletion for auditability

---

Additional Resources

Reference Files

• [`references/alert-schemas.md`](references/alert-schemas.md) - Complete JSON schema reference for all 12 alert types: field names, enum values (condition types, time windows, filter operations), common sub-objects (logs filter, tracing filter, notification groups, activity schedules), and important gotchas
• [`references/dataprime-reference.md`](references/dataprime-reference.md) - DataPrime query language reference for log-based and span-based alert conditions (filter syntax, operators, severity values)
• [`references/logs-querying.md`](references/logs-querying.md) - Log data model, field discovery, and query patterns for building log alert conditions
• [`references/promql-guidelines.md`](references/promql-guidelines.md) - PromQL reference for metric-based alert conditions (counters, gauges, histograms, threshold patterns)
• [`references/spans-querying.md`](references/spans-querying.md) - Span data model, duration units, and query patterns for building tracing alert conditions

Related Skills

• `cx-incident-management` - incident triage workflows that involve alerts, SLO monitoring, and notification verification
• `cx-observability-setup` - setting up notification routing and webhook integrations for alerts
• `cx-telemetry-querying` - investigate the telemetry behind a firing alert

Alert Definition Schemas Reference

Complete JSON schema reference for all Coralogix alert types, using the actual REST API wire format. Use this when constructing alertDefProperties payloads for cx alerts create.

Tip: The easiest way to create a new alert is to fetch an existing one with cx alerts get <id> -o json, modify the JSON, and pipe it into cx alerts create --from-file -.

Common Structure

Every alert definition has this top-level shape. The alert type config (e.g. logsThreshold) is a sibling of type, name, priority, etc. - NOT nested inside type.

{
  "alertDefProperties": {
    "name": "My Alert (required)",
    "description": "What this alert monitors",
    "priority": "ALERT_DEF_PRIORITY_P3",
    "type": "ALERT_DEF_TYPE_LOGS_THRESHOLD",
    "enabled": true,
    "groupByKeys": [],
    "entityLabels": {},
    "phantomMode": false,
    "activeOn": { ... },
    "incidentsSettings": { ... },
    "notificationGroup": { ... },
    "logsThreshold": { ... }
  }
}

Priority values

Type values

---

Common Sub-Objects

Activity Schedule (activeOn)

{
  "dayOfWeek": ["DAY_OF_WEEK_MONDAY_OR_UNSPECIFIED", "DAY_OF_WEEK_TUESDAY"],
  "startTime": { "hours": 8, "minutes": 0 },
  "endTime": { "hours": 18, "minutes": 0 }
}

Day values: DAY_OF_WEEK_MONDAY_OR_UNSPECIFIED, DAY_OF_WEEK_TUESDAY, DAY_OF_WEEK_WEDNESDAY, DAY_OF_WEEK_THURSDAY, DAY_OF_WEEK_FRIDAY, DAY_OF_WEEK_SATURDAY, DAY_OF_WEEK_SUNDAY

Incident Settings (incidentsSettings)

{
  "minutes": 60,
  "notifyOn": "NOTIFY_ON_TRIGGERED_ONLY_UNSPECIFIED"
}

notifyOn values: NOTIFY_ON_TRIGGERED_ONLY_UNSPECIFIED, NOTIFY_ON_TRIGGERED_AND_RESOLVED

Notification Group (notificationGroup)

{
  "groupByKeys": [],
  "destinations": [
    {
      "connectorId": "uuid",
      "presetId": "uuid",
      "notifyOn": "NOTIFY_ON_TRIGGERED_AND_RESOLVED"
    }
  ],
  "webhooks": [
    {
      "integration": { "integrationId": 123 },
      "minutes": 15,
      "notifyOn": "NOTIFY_ON_TRIGGERED_ONLY_UNSPECIFIED"
    }
  ],
  "router": {
    "id": "uuid",
    "notifyOn": "NOTIFY_ON_TRIGGERED_ONLY_UNSPECIFIED"
  }
}

Logs Filter (logsFilter)

Used by all log-based alert types:

{
  "simpleFilter": {
    "luceneQuery": "severity:ERROR AND service:api",
    "labelFilters": {
      "applicationName": [
        { "operation": "LOG_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED", "value": "my-app" }
      ],
      "subsystemName": [
        { "operation": "LOG_FILTER_OPERATION_TYPE_STARTS_WITH", "value": "backend" }
      ],
      "severities": ["LOG_SEVERITY_WARNING", "LOG_SEVERITY_ERROR", "LOG_SEVERITY_CRITICAL"]
    }
  }
}

Label filter operations: LOG_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED, LOG_FILTER_OPERATION_TYPE_INCLUDES, LOG_FILTER_OPERATION_TYPE_STARTS_WITH, LOG_FILTER_OPERATION_TYPE_ENDS_WITH

Severities: LOG_SEVERITY_VERBOSE_UNSPECIFIED, LOG_SEVERITY_DEBUG, LOG_SEVERITY_INFO, LOG_SEVERITY_WARNING, LOG_SEVERITY_ERROR, LOG_SEVERITY_CRITICAL

Tracing Filter (tracingFilter)

Used by tracing-based alert types:

{
  "simpleFilter": {
    "latencyThresholdMs": "1000",
    "tracingLabelFilters": {
      "applicationName": [
        { "operation": "TRACING_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED", "values": ["my-app"] }
      ],
      "serviceName": [
        { "operation": "TRACING_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED", "values": ["api-gateway"] }
      ],
      "operationName": [],
      "subsystemName": [],
      "spanFields": [
        {
          "key": "http.status_code",
          "filterType": {
            "operation": "TRACING_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED",
            "values": ["500"]
          }
        }
      ]
    }
  }
}

Tracing filter operations: TRACING_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED, TRACING_FILTER_OPERATION_TYPE_INCLUDES, TRACING_FILTER_OPERATION_TYPE_STARTS_WITH, TRACING_FILTER_OPERATION_TYPE_ENDS_WITH, TRACING_FILTER_OPERATION_TYPE_IS_NOT

Undetected Values Management

{
  "triggerUndetectedValues": true,
  "autoRetireTimeframe": "AUTO_RETIRE_TIMEFRAME_HOUR_1"
}

Values: AUTO_RETIRE_TIMEFRAME_NEVER_OR_UNSPECIFIED, AUTO_RETIRE_TIMEFRAME_MINUTES_5, AUTO_RETIRE_TIMEFRAME_MINUTES_10, AUTO_RETIRE_TIMEFRAME_HOUR_1, AUTO_RETIRE_TIMEFRAME_HOURS_2, AUTO_RETIRE_TIMEFRAME_HOURS_6, AUTO_RETIRE_TIMEFRAME_HOURS_12, AUTO_RETIRE_TIMEFRAME_HOURS_24

---

Alert Type Schemas

Each section shows only the alert-type-specific config block. This block is placed as a sibling of type, name, priority, etc. inside alertDefProperties.

1. Logs Threshold (logsThreshold)

Trigger when log count crosses a threshold in a time window.

{
  "alertDefProperties": {
    "name": "High Error Rate",
    "description": "Alert when error logs exceed threshold",
    "priority": "ALERT_DEF_PRIORITY_P2",
    "type": "ALERT_DEF_TYPE_LOGS_THRESHOLD",
    "enabled": true,
    "logsThreshold": {
      "logsFilter": {
        "simpleFilter": {
          "luceneQuery": "severity:ERROR",
          "labelFilters": {
            "applicationName": [
              { "operation": "LOG_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED", "value": "my-app" }
            ]
          }
        }
      },
      "rules": [{
        "condition": {
          "conditionType": "LOGS_THRESHOLD_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED",
          "threshold": 100,
          "timeWindow": {
            "logsTimeWindowSpecificValue": "LOGS_TIME_WINDOW_VALUE_MINUTES_5_OR_UNSPECIFIED"
          }
        },
        "override": { "priority": "ALERT_DEF_PRIORITY_P1" }
      }],
      "notificationPayloadFilter": [],
      "undetectedValuesManagement": null,
      "evaluationDelayMs": 0
    }
  }
}

conditionType: LOGS_THRESHOLD_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED, LOGS_THRESHOLD_CONDITION_TYPE_LESS_THAN

logsTimeWindowSpecificValue: LOGS_TIME_WINDOW_VALUE_MINUTES_5_OR_UNSPECIFIED, LOGS_TIME_WINDOW_VALUE_MINUTES_10, LOGS_TIME_WINDOW_VALUE_MINUTES_15, LOGS_TIME_WINDOW_VALUE_MINUTES_20, LOGS_TIME_WINDOW_VALUE_MINUTES_30, LOGS_TIME_WINDOW_VALUE_HOUR_1, LOGS_TIME_WINDOW_VALUE_HOURS_2, LOGS_TIME_WINDOW_VALUE_HOURS_4, LOGS_TIME_WINDOW_VALUE_HOURS_6, LOGS_TIME_WINDOW_VALUE_HOURS_12, LOGS_TIME_WINDOW_VALUE_HOURS_24, LOGS_TIME_WINDOW_VALUE_HOURS_36

2. Logs Immediate (logsImmediate)

Trigger instantly on every matching log entry. No rules or time windows.

{
  "alertDefProperties": {
    "name": "OOM Killer Detected",
    "description": "Alert immediately when OOM killer runs",
    "priority": "ALERT_DEF_PRIORITY_P1",
    "type": "ALERT_DEF_TYPE_LOGS_IMMEDIATE_OR_UNSPECIFIED",
    "enabled": true,
    "logsImmediate": {
      "logsFilter": {
        "simpleFilter": {
          "luceneQuery": "\"Out of memory\" OR \"OOM\"",
          "labelFilters": {}
        }
      },
      "notificationPayloadFilter": []
    }
  }
}

3. Logs Anomaly (logsAnomaly)

ML-based anomaly detection on log volume.

{
  "alertDefProperties": {
    "name": "Unusual Log Volume",
    "priority": "ALERT_DEF_PRIORITY_P3",
    "type": "ALERT_DEF_TYPE_LOGS_ANOMALY",
    "enabled": true,
    "logsAnomaly": {
      "logsFilter": { "simpleFilter": { "luceneQuery": "*", "labelFilters": {} } },
      "rules": [{
        "condition": {
          "conditionType": "LOGS_ANOMALY_CONDITION_TYPE_MORE_THAN_USUAL_OR_UNSPECIFIED",
          "minimumThreshold": 10,
          "timeWindow": {
            "logsTimeWindowSpecificValue": "LOGS_TIME_WINDOW_VALUE_HOUR_1"
          }
        }
      }],
      "anomalyAlertSettings": { "percentageOfDeviation": 50 },
      "notificationPayloadFilter": [],
      "evaluationDelayMs": 0
    }
  }
}

conditionType: LOGS_ANOMALY_CONDITION_TYPE_MORE_THAN_USUAL_OR_UNSPECIFIED

4. Logs Ratio Threshold (logsRatioThreshold)

Alert based on ratio between two log queries.

{
  "alertDefProperties": {
    "name": "Error Rate Ratio",
    "priority": "ALERT_DEF_PRIORITY_P2",
    "type": "ALERT_DEF_TYPE_LOGS_RATIO_THRESHOLD",
    "enabled": true,
    "logsRatioThreshold": {
      "numerator": { "simpleFilter": { "luceneQuery": "severity:ERROR", "labelFilters": {} } },
      "numeratorAlias": "Errors",
      "denominator": { "simpleFilter": { "luceneQuery": "*", "labelFilters": {} } },
      "denominatorAlias": "All Logs",
      "rules": [{
        "condition": {
          "conditionType": "LOGS_RATIO_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED",
          "threshold": 0.1,
          "timeWindow": {
            "logsRatioTimeWindowSpecificValue": "LOGS_RATIO_TIME_WINDOW_VALUE_MINUTES_5_OR_UNSPECIFIED"
          }
        }
      }],
      "groupByFor": "LOGS_RATIO_GROUP_BY_FOR_BOTH_OR_UNSPECIFIED",
      "ignoreInfinity": true,
      "notificationPayloadFilter": []
    }
  }
}

conditionType: LOGS_RATIO_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED, LOGS_RATIO_CONDITION_TYPE_LESS_THAN

groupByFor: LOGS_RATIO_GROUP_BY_FOR_BOTH_OR_UNSPECIFIED, LOGS_RATIO_GROUP_BY_FOR_NUMERATOR_ONLY, LOGS_RATIO_GROUP_BY_FOR_DENUMERATOR_ONLY

logsRatioTimeWindowSpecificValue: LOGS_RATIO_TIME_WINDOW_VALUE_MINUTES_5_OR_UNSPECIFIED, ..._MINUTES_10, ..._MINUTES_15, ..._MINUTES_30, ..._HOUR_1, ..._HOURS_2, ..._HOURS_4, ..._HOURS_6, ..._HOURS_12, ..._HOURS_24, ..._HOURS_36

5. Logs Time Relative Threshold (logsTimeRelativeThreshold)

Compare current log volume to a past time period.

{
  "alertDefProperties": {
    "name": "Spike vs Yesterday",
    "priority": "ALERT_DEF_PRIORITY_P3",
    "type": "ALERT_DEF_TYPE_LOGS_TIME_RELATIVE_THRESHOLD",
    "enabled": true,
    "logsTimeRelativeThreshold": {
      "logsFilter": { "simpleFilter": { "luceneQuery": "severity:ERROR", "labelFilters": {} } },
      "rules": [{
        "condition": {
          "conditionType": "LOGS_TIME_RELATIVE_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED",
          "threshold": 1.5,
          "comparedTo": "LOGS_TIME_RELATIVE_COMPARED_TO_SAME_HOUR_YESTERDAY"
        }
      }],
      "ignoreInfinity": true,
      "notificationPayloadFilter": []
    }
  }
}

conditionType: LOGS_TIME_RELATIVE_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED, LOGS_TIME_RELATIVE_CONDITION_TYPE_LESS_THAN

comparedTo: LOGS_TIME_RELATIVE_COMPARED_TO_PREVIOUS_HOUR_OR_UNSPECIFIED, ..._SAME_HOUR_YESTERDAY, ..._SAME_HOUR_LAST_WEEK, ..._YESTERDAY, ..._SAME_DAY_LAST_WEEK, ..._SAME_DAY_LAST_MONTH

6. Logs Unique Count (logsUniqueCount)

Alert when unique value count in a field crosses a threshold.

{
  "alertDefProperties": {
    "name": "Too Many Unique IPs",
    "priority": "ALERT_DEF_PRIORITY_P3",
    "type": "ALERT_DEF_TYPE_LOGS_UNIQUE_COUNT",
    "enabled": true,
    "logsUniqueCount": {
      "logsFilter": { "simpleFilter": { "luceneQuery": "*", "labelFilters": {} } },
      "uniqueCountKeypath": "remote_addr",
      "maxUniqueCountPerGroupByKey": "1000",
      "rules": [{
        "condition": {
          "maxUniqueCount": "500",
          "timeWindow": {
            "logsUniqueValueTimeWindowSpecificValue": "LOGS_UNIQUE_VALUE_TIME_WINDOW_VALUE_MINUTES_5"
          }
        }
      }],
      "notificationPayloadFilter": []
    }
  }
}

logsUniqueValueTimeWindowSpecificValue: LOGS_UNIQUE_VALUE_TIME_WINDOW_VALUE_MINUTE_1_OR_UNSPECIFIED, ..._MINUTES_5, ..._MINUTES_10, ..._MINUTES_15, ..._MINUTES_20, ..._MINUTES_30, ..._HOURS_1, ..._HOURS_2, ..._HOURS_4, ..._HOURS_6, ..._HOURS_12, ..._HOURS_24, ..._HOURS_36

7. Logs New Value (logsNewValue)

Alert when a value not previously seen appears in a field.

{
  "alertDefProperties": {
    "name": "New IP Address",
    "priority": "ALERT_DEF_PRIORITY_P3",
    "type": "ALERT_DEF_TYPE_LOGS_NEW_VALUE",
    "enabled": true,
    "logsNewValue": {
      "logsFilter": { "simpleFilter": { "luceneQuery": "*", "labelFilters": {} } },
      "rules": [{
        "condition": {
          "keypathToTrack": "ip_address",
          "timeWindow": {
            "logsNewValueTimeWindowSpecificValue": "LOGS_NEW_VALUE_TIME_WINDOW_VALUE_HOURS_24"
          }
        }
      }],
      "notificationPayloadFilter": []
    }
  }
}

logsNewValueTimeWindowSpecificValue: LOGS_NEW_VALUE_TIME_WINDOW_VALUE_HOURS_12_OR_UNSPECIFIED, ..._HOURS_24, ..._HOURS_48, ..._HOURS_72, ..._WEEK_1, ..._MONTH_1, ..._MONTHS_2, ..._MONTHS_3

8. Metric Threshold (metricThreshold)

Trigger when a PromQL expression crosses a threshold.

{
  "alertDefProperties": {
    "name": "CPU Usage Critical",
    "priority": "ALERT_DEF_PRIORITY_P1",
    "type": "ALERT_DEF_TYPE_METRIC_THRESHOLD",
    "enabled": true,
    "metricThreshold": {
      "metricFilter": {
        "promql": "avg(cpu_usage_percent{service=\"api\"})"
      },
      "rules": [{
        "condition": {
          "conditionType": "METRIC_THRESHOLD_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED",
          "threshold": 90,
          "ofTheLast": {
            "dynamicDuration": "5m"
          },
          "forOverPct": 100
        }
      }],
      "missingValues": {
        "replaceWithZero": true,
        "minNonNullValuesPct": 0
      },
      "undetectedValuesManagement": null,
      "evaluationDelayMs": 0
    }
  }
}

conditionType: METRIC_THRESHOLD_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED, ..._MORE_THAN_OR_EQUALS, ..._LESS_THAN, ..._LESS_THAN_OR_EQUALS

dynamicDuration: any PromQL duration string (e.g. 5m, 1h, 24h) within 1-2160 minutes. This is a free-form string, not an enum.

forOverPct: percentage of data points that must breach (0-100).

9. Metric Anomaly (metricAnomaly)

ML-based anomaly detection on metrics.

{
  "alertDefProperties": {
    "name": "Unusual Request Rate",
    "priority": "ALERT_DEF_PRIORITY_P3",
    "type": "ALERT_DEF_TYPE_METRIC_ANOMALY",
    "enabled": true,
    "metricAnomaly": {
      "metricFilter": {
        "promql": "rate(http_requests_total[5m])"
      },
      "rules": [{
        "condition": {
          "conditionType": "METRIC_ANOMALY_CONDITION_TYPE_MORE_THAN_USUAL_OR_UNSPECIFIED",
          "threshold": 10,
          "ofTheLast": {
            "specificValue": "METRIC_TIME_WINDOW_VALUE_HOUR_1"
          },
          "forOverPct": 100,
          "minNonNullValuesPct": 50
        }
      }],
      "anomalyAlertSettings": { "percentageOfDeviation": 50 },
      "evaluationDelayMs": 0
    }
  }
}

conditionType: METRIC_ANOMALY_CONDITION_TYPE_MORE_THAN_USUAL_OR_UNSPECIFIED, METRIC_ANOMALY_CONDITION_TYPE_LESS_THAN_USUAL

specificValue (timeWindow): METRIC_TIME_WINDOW_VALUE_MINUTES_1_OR_UNSPECIFIED, ..._MINUTES_5, ..._MINUTES_10, ..._MINUTES_15, ..._MINUTES_20, ..._MINUTES_30, ..._HOUR_1, ..._HOURS_2, ..._HOURS_4, ..._HOURS_6, ..._HOURS_12, ..._HOURS_24, ..._HOURS_36

10. Tracing Immediate (tracingImmediate)

Trigger instantly on matching trace spans.

{
  "alertDefProperties": {
    "name": "Slow API Call",
    "priority": "ALERT_DEF_PRIORITY_P2",
    "type": "ALERT_DEF_TYPE_TRACING_IMMEDIATE",
    "enabled": true,
    "tracingImmediate": {
      "tracingFilter": {
        "simpleFilter": {
          "latencyThresholdMs": "1000",
          "tracingLabelFilters": {
            "serviceName": [
              {
                "operation": "TRACING_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED",
                "values": ["api-gateway"]
              }
            ],
            "applicationName": [],
            "operationName": [],
            "subsystemName": [],
            "spanFields": []
          }
        }
      },
      "notificationPayloadFilter": []
    }
  }
}

11. Tracing Threshold (tracingThreshold)

Trigger when span count crosses a threshold.

{
  "alertDefProperties": {
    "name": "High Span Volume",
    "priority": "ALERT_DEF_PRIORITY_P3",
    "type": "ALERT_DEF_TYPE_TRACING_THRESHOLD",
    "enabled": true,
    "tracingThreshold": {
      "tracingFilter": {
        "simpleFilter": {
          "latencyThresholdMs": "0",
          "tracingLabelFilters": {
            "serviceName": [
              {
                "operation": "TRACING_FILTER_OPERATION_TYPE_IS_OR_UNSPECIFIED",
                "values": ["api-gateway"]
              }
            ],
            "applicationName": [],
            "operationName": [],
            "subsystemName": [],
            "spanFields": []
          }
        }
      },
      "rules": [{
        "condition": {
          "conditionType": "TRACING_THRESHOLD_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED",
          "spanAmount": 100,
          "timeWindow": {
            "tracingTimeWindowSpecificValue": "TRACING_TIME_WINDOW_VALUE_MINUTES_5_OR_UNSPECIFIED"
          }
        }
      }],
      "notificationPayloadFilter": []
    }
  }
}

conditionType: TRACING_THRESHOLD_CONDITION_TYPE_MORE_THAN_OR_UNSPECIFIED

tracingTimeWindowSpecificValue: TRACING_TIME_WINDOW_VALUE_MINUTES_5_OR_UNSPECIFIED, ..._MINUTES_10, ..._MINUTES_15, ..._MINUTES_20, ..._MINUTES_30, ..._HOUR_1, ..._HOURS_2, ..._HOURS_4, ..._HOURS_6, ..._HOURS_12, ..._HOURS_24, ..._HOURS_36

12. SLO Threshold (sloThreshold)

Monitor error budget consumption or burn rate. Exactly one of errorBudget or burnRate must be set.

Error Budget variant:

{
  "alertDefProperties": {
    "name": "SLO Budget Low",
    "priority": "ALERT_DEF_PRIORITY_P2",
    "type": "ALERT_DEF_TYPE_SLO_THRESHOLD",
    "enabled": true,
    "sloThreshold": {
      "sloDefinition": { "sloId": "uuid" },
      "errorBudget": {
        "rules": [{
          "condition": { "threshold": 50 },
          "override": { "priority": "ALERT_DEF_PRIORITY_P1" }
        }]
      }
    }
  }
}

Burn Rate variant:

{
  "alertDefProperties": {
    "name": "SLO Burn Rate High",
    "priority": "ALERT_DEF_PRIORITY_P1",
    "type": "ALERT_DEF_TYPE_SLO_THRESHOLD",
    "enabled": true,
    "sloThreshold": {
      "sloDefinition": { "sloId": "uuid" },
      "burnRate": {
        "rules": [{
          "condition": { "threshold": 2.0 },
          "override": { "priority": "ALERT_DEF_PRIORITY_P1" }
        }],
        "single": {
          "timeDuration": { "duration": "1", "unit": "DURATION_UNIT_HOURS" }
        }
      }
    }
  }
}

unit values: DURATION_UNIT_UNSPECIFIED, DURATION_UNIT_HOURS

---

Important Notes

• groupByKeys for metric alerts: Leave empty to let the API infer from the PromQL by clause. If provided, keys must be in alphabetical order (the API infers alphabetically, not in query order).
• Priority: Always ask the user -- never pick a default.
• override in rules: Optional per-rule priority override using the same ALERT_DEF_PRIORITY_* enum values. Omit to use the alert-level priority.
• notificationPayloadFilter: List of log/span field paths to include in notifications (e.g. ["obj.field"]).
• Best practice: Fetch an existing alert with cx alerts get <id> -o json to see the exact response shape, then use it as a template for creating new alerts.

DataPrime Query Language Reference

Query Structure

A DataPrime query is a pipeline of commands separated by |. Each command transforms the output of the previous one:

filter $m.severity == ERROR | groupby $l.subsystemname aggregate count() as errors

Source Handling

Every query targets a source (logs, spans, etc.). The source is set by whichever cx command you use. A full query with an explicit source looks like:

source <logs|spans> | filter ... | groupby ...

When running via a source-specific command (e.g. cx logs, cx spans), the source is injected automatically - omit it from the query. When running via cx dataprime query, use the --source flag or include source in the query itself.

The examples below focus on the DataPrime query language and omit the source and CLI command prefix.

Comments

Comments are supported with # or //:

filter $m.severity == ERROR  # only errors
| limit 10                   // cap results

Data Prefixes

All fields are accessed through three namespaces:

$d is the default prefix and can sometimes be omitted, but being explicit avoids ambiguity.

Data Types

Commands

Filtering and Selection

Note on `wildfind`: It is a standalone command, not a condition within filter. You cannot combine it with other filter expressions - use it as its own pipeline stage.

Aggregation

Transformation

Operators

Type Conversions

Cast fields inline with :type:

filter $d.http_error_code:number == 500

Supported types: bool, number, string, timestamp, interval, array, object

Field Access

# Chained field names (dot notation)
filter $d.tags.user_context.email == 'test@example.com'

# Special characters require brackets
filter $d.http['status/code'] == 500

Aggregation Functions

Example - full CLI invocation:

cx dataprime query --source logs 'groupby $l.subsystemname aggregate count() as error_count, avg($d.response_time) as avg_response | orderby error_count desc'

Utility Functions

firstNonNull - Field Coalescing

Return the first non-null value from a list of fields. Useful when the same data may appear in different fields across log sources:

# Merge fields
create message from firstNonNull($d.error_message, $d.msg, $d.body)

# Use inside groupby
groupby firstNonNull($d.error_message, $d.msg) as message aggregate count() as n

Template Sampling

Find top error patterns with a sample message for each:

filter $m.severity == ERROR | groupby $m.templateid aggregate any_value($d) as sample, count() as total | orderby total desc | limit 5

Time-Based Grouping

Use roundTime() to bucket timestamps:

# Group by hour
groupby roundTime($m.timestamp, 1h) as hour aggregate count() as count

# Error rate over 15-minute intervals
filter $m.severity == ERROR | groupby roundTime($m.timestamp, 15m) as interval aggregate count() as errors

Multi-Value Matching

Use arrayContains to match against a set of values:

# Match multiple subsystems
filter ['api', 'web', 'worker'].arrayContains($l.subsystemname)

# Match multiple severity levels
filter [ERROR, CRITICAL].arrayContains($m.severity)

Text Extraction

Regex Extraction

# Extract with unnamed capture group
extract $d.email into domain using regexp(e=/@(.*)/) | distinct $d.domain._0

# Named capture groups
extract $d.email into extracted using regexp(e=/(?<username>[a-zA-Z0-9._%+-]+)@(?<domain>.*)/) | choose $d.extracted.username, $d.extracted.domain

JSON String Parsing

# Parse a JSON string field into an object for further querying
extract $d.json_payload into parsed using jsonobject() | filter $d.parsed.status == 'failed'

Deduplication

# Remove duplicates by log template
dedupeby $m.templateid

# Dedupe by a custom field
dedupeby $d.request_id

Built-In Documentation

For the full list of commands and functions with detailed syntax:

cx dataprime list                              # List all commands and functions
cx dataprime list --filter commands             # Commands only
cx dataprime list --filter functions --name time # Search functions by name
cx dataprime show filter                        # Detailed help for a specific command
cx dataprime show groupby

Validating a DataPrime query

A query that looks right can still fail on a typoed field path, an invented function, or a malformed pipeline stage. Validate before trusting the output — a short-window run through the CLI is cheap and catches almost all of these:

cx logs  '<pipeline>' --start now-15m --end now --limit 1
cx spans '<pipeline>' --start now-15m --end now --limit 1

now-15m is a good default; widen it only if 15 minutes is unlikely to exercise the pipeline. Per "Source Handling" above, omit any leading source logs / source spans — cx logs and cx spans inject the source themselves.

Check both the exit code and the output — some errors surface only in the output.

Pass = exit 0 and the output is rows or [] with no error or warning lines.

Hard fail — query is broken, fix it:

• non-zero exit
• error from profile '...': API request failed — HTTP error from the API
• Compilation errors: — parse error, unknown function, malformed expression

Soft fail (needs investigation):

• keypath does not exist — the query parsed, but no record in the window had the referenced field. This is ambiguous: the field name might be a typo, or it might be real but absent from records in this 15-minute slice. Confirm with cx search-fields "<field hint>" --dataset logs (or --dataset spans). If the field is real, the query is fine — try a wider window or accept the empty result. If it isn't, fix the field name.

On fail: re-discover fields with cx search-fields, look up command syntax with cx dataprime show <command>, fix, re-run.

Log Querying Reference

Query and analyze Coralogix logs using the cx logs command with DataPrime syntax.

DataPrime syntax: See dataprime-reference.md for the full query language reference.

Understanding Logs in Coralogix

Logs in Coralogix are largely unstructured. Every log entry has a small structured envelope - metadata and labels - but the actual application payload (userData) is free-form and varies entirely by application. There is no universal schema for $d.* fields.

This means:

• *Metadata (`$m.)** and **labels ($l.`)* are predictable - you can always filter on severity, timestamp, application name, and subsystem name without discovery.
• *User data (`$d.)** is not predictable - field names, nesting, and types depend on whatever the application chose to log. Always verify $d` fields before assuming they exist.

---

CLI Command

cx logs '<dataprime_query>'

The source logs prefix is automatically injected if the query doesn't already include a source command.

Options

---

Log Data Model

Standard Fields (Always Available)

Severity Values

Severity keywords are used without quotes in DataPrime:

DEBUG | INFO | WARNING | ERROR | CRITICAL

cx logs 'filter $m.severity == ERROR'
cx logs 'filter [ERROR, CRITICAL].arrayContains($m.severity)'

---

Essential Query Examples

# Filter by severity
cx logs 'filter $m.severity == ERROR'

# Text search in a known field
cx logs "filter \$d.message ~ 'timeout'"

# Filter by application and subsystem
cx logs "filter \$l.applicationname == 'api' && \$l.subsystemname == 'auth'"

# Aggregate errors by subsystem
cx logs 'filter $m.severity == ERROR | groupby $l.subsystemname aggregate count() as errors | orderby errors desc'

# Wider time range and archive tier
cx logs "filter \$l.subsystemname == 'payments'" --tier archive --start now-7d

Wildfind Policy

Avoid `wildfind` by default. It scans all fields and returns noisy results, especially for generic terms.

The one exception: when the user provides a specific, quoted error message or log string and you don't know which field contains it:

# User says: "Find logs with 'connection refused'"
cx logs "wildfind 'connection refused'"

In all other cases, use filter with known fields ($m.severity, $l.subsystemname, $d.<field>) or discover field names first with cx search-fields.

---

Field Discovery

Skip discovery when:

• The query only uses standard fields ($m.severity, $m.timestamp, $l.applicationname, $l.subsystemname)
• The user explicitly names the fields they want (e.g., "filter by $d.customer_id")
• You're searching for a specific error message - use wildfind directly
• The fields have already been discovered earlier in the conversation

For customer-specific $d.* fields that need discovery, use one of these approaches:

1. Infer from Source Code (Preferred)

If you have access to the application's source code, examine logger calls, structured logging configs, and log format templates to identify field names directly.

2. Semantic Search

cx search-fields "customer identifier" --dataset logs
cx search-fields "http response code" --dataset logs

Returns DataPrime paths with similarity scores:

+------------------------+-----------------------------------+-----------+
| DataPrime path         | Description                       | Similarity|
+------------------------+-----------------------------------+-----------+
| $d.customer_id         | Unique customer identifier        | 0.89      |
| $d.user.account_id     | Customer account reference        | 0.85      |
+------------------------+-----------------------------------+-----------+

3. Sample Query Inspection

cx logs "filter \$l.subsystemname == 'api'" --limit 5 -o json

Inspect the JSON output to see all available fields in the actual data.

---

Investigation Workflow

1. Understand the Request

Identify:

• What type of logs are needed (errors, info, specific events)
• Time frame of interest
• Key entities (services, users, transactions)

2. Start with Standard Fields

For basic queries, use standard fields directly:

# Recent errors - no discovery needed
cx logs 'filter $m.severity == ERROR | limit 20'

# Errors in a specific subsystem
cx logs "filter \$m.severity == ERROR && \$l.subsystemname == 'payment-service'"

3. Build and Execute Query

Start simple, add complexity:

# Step 1: Check if data exists
cx logs "filter \$l.subsystemname == 'checkout'" --limit 10

# Step 2: Add filters
cx logs "filter \$l.subsystemname == 'checkout' && \$m.severity == ERROR"

# Step 3: Add aggregation
cx logs "filter \$l.subsystemname == 'checkout' && \$m.severity == ERROR | groupby \$d.error_type aggregate count() as occurrences"

4. Troubleshooting

If a query returns no results, change one thing at a time:

1. Extend the time range: --start now-6h or --start now-24h 2. Relax filters: remove the most restrictive condition 3. Verify field names: run a sample query with -o json to inspect the actual schema 4. Try archive tier: --tier archive --start now-30d for older data

---

Common Query Patterns

Error Investigation

# All errors in last hour
cx logs 'filter $m.severity == ERROR'

# Critical errors only
cx logs 'filter $m.severity == CRITICAL'

# Errors with text search
cx logs "filter \$m.severity == ERROR && \$d.message ~ 'database connection'"

Aggregation by Service

# Error count by subsystem
cx logs 'filter $m.severity == ERROR | groupby $l.subsystemname aggregate count() as errors | orderby errors desc'

# Error count by application and subsystem
cx logs 'filter $m.severity == ERROR | groupby $l.applicationname, $l.subsystemname aggregate count() as errors'

Time-Based Analysis

# Errors per hour
cx logs 'filter $m.severity == ERROR | groupby roundTime($m.timestamp, 1h) as hour aggregate count() as count'

# Find error spikes in 5-minute windows
cx logs 'filter $m.severity == ERROR | groupby roundTime($m.timestamp, 5m) as interval aggregate count() as count | orderby count desc | limit 10'

Finding Unique Values

# List all subsystems with errors
cx logs 'filter $m.severity == ERROR | distinct $l.subsystemname'

# List unique error types
cx logs 'filter $m.severity == ERROR | distinct $d.error_type'

Fetching Sample Logs by Template

Find top error patterns with sample messages:

cx logs 'filter $m.severity == ERROR | groupby $m.templateid aggregate any_value($d) as sample, count() as total | orderby total desc | limit 5'

---

Performance Tips

• Use --limit for exploratory queries
• Use groupby with aggregations instead of fetching all raw logs
• Filter by time first when dealing with large datasets
• Use specific filters (application, subsystem) to reduce scan scope
• For large result sets, use --output agents which spills to a temp file automatically:

cx logs 'filter $m.severity == ERROR' --start now-24h --limit 1000 -o agents

PromQL Guidelines

Core Principles

1. Pick the right query type

• Instant queries (cx metrics query) evaluate an expression at a single timestamp (now, or a given --time). Use when the question requires one number or one vector as of a moment - essentially any query that does not require results over different timeframes.
• Range queries (cx metrics query-range) evaluate the expression repeatedly across [--start, --end] at a given --step. Use for time series over a period (e.g., daily active users per day).
• Note: Range queries evaluate the expression repeatedly at each step. If --step=1d, --start=now-1d, --end=now, and the query is max_over_time(metric[1d]), the query evaluates at now-1d and now - two evaluations covering two days of data.
• Prefer instant queries over range queries for most questions, except when comparing different timeframes.

2. Understand PromQL value types

• Instant vector - set of series with 1 sample each at eval time
• Range vector - series with many samples over a window [t-range, t]
• Scalar - single number
• String - rare
• Functions like *_over_time() require a range vector. Aggregations like sum/max/min/avg ... by(...) consume instant vectors.
• Important: When using *_over_time() functions with range queries, be aware that the query also evaluates at the --start time and includes the window specified in the function.
• Example: If max_over_time(metric[1d]) is used with --start=now-1d, --end=now, --step=1d, the query evaluates at now-1d and now - the result is the max over [now-2d, now]. This is a common mistake. If a user asks "What is the max of x between 2025-01-01 and 2025-01-07?" and max_over_time(x[7d]) is used with --start=2025-01-01, --end=2025-01-07, --step=1d, the evaluation at 2025-01-01 includes [2024-12-25, 2025-01-01] - which is wrong. Use an instant query with --time to avoid this.

3. Separation of concerns

• Use *_over_time() for temporal reductions across a window (e.g., max_over_time, avg_over_time, quantile_over_time).
• Use sum/max/min/avg by (...) for label-set aggregation across series at the eval point.
• Chain them as needed (temporal reduction first, then label aggregation, or vice versa).

4. Counters vs. gauges

• Counters (monotonic, suffixed _total) → use rate()/irate() or increase() over a window.
• Gauges (current value) → use avg_over_time, max_over_time, etc., or plain avg(...) depending on intent.

5. Suffix conventions

• Canonical: _total (counter), _bucket/_sum/_count (histogram), _sum/_count (summary), _created.
• Non-standard: _avg, _mean, etc. Prefer computing averages via PromQL unless the exporter dictates otherwise.

---

CLI Usage

Instant Query

cx metrics query '<expr>'
cx metrics query '<expr>' --time 2024-01-01T12:00:00Z
cx metrics query '<expr>' --output json

Example: absolute max over last 24h (single result)

cx metrics query 'max by () (max_over_time(http_requests_in_flight[24h]))'

Range Query

cx metrics query-range '<expr>' --start now-7d --end now --step 1d

Example: absolute max per day over the last 7 days

cx metrics query-range 'max by () (max_over_time(metric[1d]))' \
  --start now-7d --end now --step 1d

IMPORTANT: Align --step with any window used in temporal reduction functions. If using max_over_time(metric[1d]), set --step 1d.

---

PromQL Fundamentals

Label Matching & Aggregation

• Matchers: {label="v"}, {label!="v"}, {label=~"re.*"}, {label!~"re"}
• Aggregate by labels to keep them; use without to drop them.

sum by (job) (rate(http_requests_total[5m]))
sum without (instance) (up)

Temporal Reductions (range → instant)

max_over_time(cpu_usage[1h])
avg_over_time(node_memory_Active_bytes[30m])
quantile_over_time(0.99, queue_length[1h])

Counters: Rates, Increases, Windows

Per-instance RPS:

rate(http_requests_total[5m])

Total RPS across fleet:

sum by () (rate(http_requests_total[5m]))

Events in last day (per user, then count actives):

count( sum by (user_id) (increase(api_call_count[24h])) > 0 )

Histograms & Summaries

p95 from a histogram:

histogram_quantile(
  0.95,
  sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))
)

Average from summary parts:

sum(rate(req_duration_seconds_sum[5m]))
/
sum(rate(req_duration_seconds_count[5m]))

Max over a Period

Correct - temporal reduction, then aggregation:

max by () (max_over_time(metric[4d]))

Per-label max:

max by (label) (max_over_time(metric[4d]))

Incorrect - max() cannot take a range vector:

max(metric[4d])   ← error

Top-k / Ranking

topk(5, sum by (instance) (rate(http_requests_total[5m])))

---

Common Tasks (ready to adapt)

1. Absolute peak per instance over 7d, then pick the winner

topk(1, max by (instance) (max_over_time(my_metric[7d])))

Run as instant query (no --time needed - defaults to now).

2. Global CPU usage % (avg across cores & hosts)

avg by () (
  rate(process_cpu_seconds_total[5m])
) * 100

3. Error rate (%) per route

100 * sum by (route) (rate(http_requests_total{code=~"5.."}[5m]))
    / sum by (route) (rate(http_requests_total[5m]))

4. Daily active users over a week (time series)

Expression:

count(count by (user_id) (increase(api_call_count[1d]) > 0))

Run as range query with --step 1d --start now-6d --end now. (Starting from 6 days ago because increase looks back one full day from each evaluation point.)

---

Performance & Safety Guidelines

• Prefer short windows for rate() (e.g., 1–5m) unless data is bursty or sparse.
• Avoid unbounded fan-out (e.g., joining massive label sets).
• Keep cardinality under control; aggregate early (sum by (...)) when only totals are needed.
• Use clamp_max/clamp_min to tame outliers when needed.
• For histograms, always aggregate buckets (sum by (le, ...)) before histogram_quantile.
• Be mindful of counter resets; rate()/increase() handle resets automatically.

---

Frequent Gotchas (and fixes)

• "Why am I getting a time series when I only want one number?"

Use cx metrics query (instant) instead of cx metrics query-range.

• "`max(metric[...])` errors."

max() can't take a range vector. Use max_over_time(metric[...]), then aggregate with max by () (...).

• "`_over_time(metric[...]) by (label)` errors."

_over_time aggregations cannot include a by clause. Use max_over_time(metric[...]), then max by (label) (...).

• "Avg looks wrong for counters."

Counters need rate()/increase(), not avg_over_time.

• "p95 from a summary?"

Summaries expose quantiles directly via the quantile label. For histograms, use histogram_quantile on bucket rates.

• "Results show empty label values."

Add {label!=""} to the selector to filter out empty label values. Example: max by (deployment) (rate(cpu_usage{deployment!=""}[5m])).

---

Mini Cheat-Sheet

Span Querying Reference

Query and analyze distributed tracing data using the cx spans command with DataPrime syntax.

DataPrime syntax: See dataprime-reference.md for the full query language reference.

Understanding Spans in Coralogix

Spans are the fundamental unit of tracing data. Traces are not stored as single entities - they are logical groupings of spans that share the same traceID. To analyze a trace, you query its constituent spans.

This means:

• *Metadata (`$m.)** and **labels ($l.`)* are predictable - you can always filter on timestamp, duration, service name, and operation name without discovery.
• *User data (`$d.)** contains trace identifiers (traceID, spanID, parentSpanID) and application-specific tags/attributes that vary by service. Always verify custom $d` fields before assuming they exist.

---

CLI Command

cx spans '<dataprime_query>'

The source spans is automatically injected - do not include it in the query.

Options

---

Span Data Model

Standard Fields (Always Available)

Note on label fields: The meaning of $l.applicationName and $l.subsystemName varies by customer - they may represent environments, teams, regions, or something else entirely. Don't assume what they map to. Use cx search-fields or sample queries to verify actual values.

Duration Units

$m.duration is in microseconds:

• 500ms = 500000
• 1s = 1000000
• 1min = 60000000

When presenting duration values, always convert to human-readable units (milliseconds, seconds, or minutes) and include the unit. Never display raw microsecond values or the "µs" symbol.

# Computed field for milliseconds
create latency_ms from $m.duration / 1000

Error Detection

Spans do not have a $m.severity field like logs. Errors are typically indicated by:

• $d.tags.error == true - the most common convention (OpenTelemetry/Jaeger)
• Status codes in custom fields (e.g. $d.http.status_code, $d.grpc.status_code)
• Other application-specific error tags

The exact field depends on the instrumentation library used. If $d.tags.error returns no results, inspect sample spans with -o json to discover how errors are tagged:

cx spans "filter \$l.serviceName == 'api'" --limit 5 -o json

---

Essential Query Examples

# Get all spans for a trace
cx spans "filter \$d.traceID == '4f6a8f3c2e8a1b97'"

# Find spans for a service
cx spans "filter \$l.serviceName == 'checkout-service'"

# Find slow spans (> 1 second)
cx spans "filter \$m.duration > 1000000"

# Find error spans
cx spans "filter \$d.tags.error == true"

# Aggregate latency by operation
cx spans "groupby \$l.operationName aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc"

# Wider time range
cx spans "filter \$l.serviceName == 'api'" --start now-6h

Wildfind Policy

Avoid `wildfind` by default. It scans all fields and is expensive.

The one exception: when the user provides a specific string and you don't know which field contains it:

cx spans "wildfind 'connection refused'"

Tip: wildfind can also serve as a last-resort field discovery method - when cx search-fields doesn't find what you need, run wildfind with a known value, then inspect the matching spans to see which fields contain it.

---

Field Discovery

Skip discovery when:

• The query only uses standard fields ($m.duration, $l.serviceName, $l.operationName, $d.traceID)
• The user explicitly names the fields they want
• The fields have already been discovered earlier in the conversation

1. Infer from Source Code (Preferred)

If you have access to the application's source code, examine OpenTelemetry instrumentation, span attribute definitions, and tracing middleware to identify field names directly.

2. Semantic Search

cx search-fields "customer identifier" --dataset spans
cx search-fields "order ID" --dataset spans
cx search-fields "http response code" --dataset spans

Note: cx search-fields only has access to the most common fields. If it doesn't find what you need, fall back to sample query inspection.

3. Sample Query Inspection

cx spans "filter \$l.serviceName == 'api'" --limit 5 -o json

Inspect the JSON output to see all available fields. Especially useful for discovering fields in unstructured or deeply nested data.

---

Investigation Workflow

1. Understand the Request

Identify:

• Whether you have a trace ID, service name, or error description
• Time frame of interest
• Whether the question is about latency, errors, or request flow

2. Start with Known Information

If you have a trace ID - go straight to it:

cx spans "filter \$d.traceID == '<trace_id>'"

If you have a service name - query its spans:

cx spans "filter \$l.serviceName == '<service>'" --limit 50

If you have neither - start broad to find entry points:

# Find recent error spans
cx spans "filter \$d.tags.error == true" --limit 20

# Find the slowest spans in the last hour
cx spans "groupby \$l.serviceName, \$l.operationName aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc | limit 10"

# Then extract trace IDs from interesting spans
cx spans "filter \$l.serviceName == '<service>' && \$m.duration > 1000000 | distinct \$d.traceID"

3. Troubleshooting

If a query returns no results, change one thing at a time:

1. Extend the time range: --start now-6h or --start now-24h 2. Relax filters: remove the most restrictive condition 3. Check field availability: the field you're filtering by may only exist in a subset of spans 4. Verify field names: run a sample query with -o json to inspect the actual schema 5. Check service names: service names are case-sensitive 6. Try archive tier: --tier archive --start now-30d for older data

---

Common Query Patterns

Trace Reconstruction

# All spans for a trace
cx spans "filter \$d.traceID == '4f6a8f3c2e8a1b97'"

# Find root spans only (no parent)
cx spans "filter \$l.serviceName == 'api-gateway' | filter \$d.parentId == null"

# Find trace IDs for a service
cx spans "filter \$l.serviceName == 'payment-service' | distinct \$d.traceID"

Latency Analysis

# Spans slower than 1 second
cx spans "filter \$m.duration > 1000000"

# Top 10 slowest operations by average duration
cx spans "groupby \$l.operationName aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc | limit 10"

# Average latency by service
cx spans "groupby \$l.serviceName aggregate avg(\$m.duration) as avg_latency"

# P95 latency by operation
cx spans "groupby \$l.operationName aggregate percentile(0.95, \$m.duration) as p95_latency"

Latency Spike Detection

# Average latency per 15-minute window
cx spans "filter \$l.serviceName == 'api' | groupby roundTime(\$m.timestamp, 15m) as interval aggregate avg(\$m.duration) as avg_latency | orderby interval"

# Find the time windows with highest latency
cx spans "filter \$l.serviceName == 'api' | groupby roundTime(\$m.timestamp, 5m) as interval aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc | limit 10"

Error Investigation

# All error spans
cx spans "filter \$d.tags.error == true"

# Error spans for a specific service
cx spans "filter \$l.serviceName == 'checkout' | filter \$d.tags.error == true"

# Error rate by service
cx spans "filter \$d.tags.error == true | groupby \$l.serviceName aggregate count() as errors | orderby errors desc"

# Error rate over time
cx spans "filter \$d.tags.error == true | groupby roundTime(\$m.timestamp, 15m) as interval aggregate count() as errors"

Sampling Error Types

# Group errors by operation with a sample
cx spans "filter \$d.tags.error == true | groupby \$l.operationName aggregate any_value(\$d) as sample, count() as total | orderby total desc | limit 5"

# Group by service and operation to see where errors concentrate
cx spans "filter \$d.tags.error == true | groupby \$l.serviceName, \$l.operationName aggregate count() as errors | orderby errors desc | limit 10"

Finding Unique Values

# List all services with spans
cx spans "distinct \$l.serviceName"

# List all operations for a service
cx spans "filter \$l.serviceName == 'api' | distinct \$l.operationName"

# Find unique trace IDs for error spans
cx spans "filter \$d.tags.error == true | distinct \$d.traceID"

Correlating by Trace ID

# Find spans across services for the same trace
cx spans "filter \$d.traceID == 'abc123' | groupby \$l.serviceName aggregate count() as span_count, avg(\$m.duration) as avg_latency"

---

Performance Tips

• Use --limit for exploratory queries
• Use groupby with aggregations instead of fetching raw spans when possible
• Filter by time first when dealing with large datasets
• Use specific filters (service name, operation) to reduce scan scope
• Don't rely solely on aggregations - retrieve sample spans to find information you didn't anticipate
• For large result sets, use --output agents which spills automatically:

cx spans "filter \$l.serviceName == 'api'" --start now-24h --limit 1000 -o agents