Cx Dashboards

Create Coralogix Dashboard

Produces a Coralogix dashboard for a target service and deploys it via the cx CLI. Workflow: discover the service's telemetry, align on intent with the user, draft a plan, emit the JSON, live-verify every query through cx, then create the dashboard in a chosen folder.

Only use metric names, log fields, and span attributes you can cite from the service's code, README, configuration, or a live query that returned a result. Do not invent them.

---

Reference files

Load these files for domain-specific guidance:

For choosing the right signal (metrics / logs / traces), use cx-telemetry-querying.

---

Dashboard Management

Beyond creating dashboards, use these commands to manage existing ones:

To update an existing dashboard:

cx dashboards get <dashboard-id> -o json > dashboard.json
# Edit dashboard.json (change name, modify widgets, etc.)
cx dashboards replace --from-file dashboard.json

To duplicate a dashboard as a new copy:

cx dashboards get <dashboard-id> -o json > dashboard.json
# Remove the "id" field, then create as new:
cx dashboards create --from-file dashboard.json

---

Workflow

Track progress through this checklist:

Dashboard Progress:
- [ ] Phase 1: Discover telemetry & business meaning
- [ ] Phase 2: Gather dashboard specifications from user
- [ ] Phase 3: Draft internal dashboard plan (sections/rows/widgets)
- [ ] Phase 4: Generate the Coralogix JSON
- [ ] Phase 5: Live-verify every query through the cx CLI
- [ ] Phase 6: Self-verify structure against the checklist
- [ ] Phase 7: Deploy via `cx dashboards create`
- [ ] Phase 8: Share the dashboard link with the user

Proceed in order. Don't jump to Phase 4 before the user approves the Phase 3 plan, and don't run Phase 7 before Phases 5 and 6 both pass. Phase 8 is mandatory — the workflow is not done until the user has a clickable link.

---

Phase 1: Discover telemetry & business meaning

For the target service, gather:

1. Business purpose - read README.md and the top-level entrypoint (main.*, index.*, cmd/main.go, etc.). Summarize in 2–3 sentences what it does, its key stages, and what can go wrong. 2. Metrics - for each candidate keyword (service name, subsystem, verbs like request, error, latency, dlq) run cx metrics search --name '*<keyword>*'. When a metric looks promising, list its labels with cx metrics get-labels <metric>. Only use names cx metrics search returns - this is what prevents invented metrics from reaching Phase 5. Cross-check the service's instrumentation (prometheus_client, promauto.NewCounter/Histogram/Gauge, OTel meters, prom-client, Micrometer, metrics.py) for semantics and histogram buckets (_sum, _count, _bucket). 3. Logs - discover custom $d.* fields with cx search-fields "<description>" --dataset logs before assuming a field exists. Sample message templates and severity with cx logs "filter \$l.applicationname == '<app>'" --limit 5 -o json. Standard fields ($m.severity, $m.timestamp, $l.applicationname, $l.subsystemname) don't need discovery. 4. Spans / traces - discover span attributes with cx search-fields "<description>" --dataset spans. Sample with cx spans "filter \$l.serviceName == '<svc>'" --limit 5 -o json. Error conventions vary ($d.tags.error, $d.http.status_code); check samples before filtering. 5. Message buses & DLQs - grep for Kafka, RabbitMQ, SQS, Pub/Sub clients and any dlq/DLQ references. Note topic/queue names for DLQ panels. 6. Service configuration - check meta.yaml, Helm values.yaml, Deployment, Dockerfile, chart.yaml. Extract:

• The applicationname / subsystemname label values as they appear in Coralogix.
• Tenant/account/team identifiers used as metric or log labels.
• Deployment environments (prod, staging, dev, …).

If the signal for a question is ambiguous (e.g. "how much revenue last week"), delegate to cx-telemetry-querying first.

Produce a short internal summary before moving on. If critical telemetry is missing (e.g. no metrics), surface that to the user and ask whether they want a log-only or trace-only dashboard.

---

Phase 2: Gather dashboard specifications

Ask the user a focused set (≤6). Prefer AskQuestion:

1. Audience & use - on-call triage, product/business tracking, capacity planning, customer success? 2. Default time range - typical viewing window (e.g. 24h, 7d). Queries still use ${__range} so users can zoom. 3. Slicing dimensions - top-level filters (tenant_id, account_id, subsystem_name, region, env, …). 4. Environment scope - which environments to include/exclude (common default: exclude dev, staging, test). 5. SLO-ish signals - success-rate, latency, or throughput targets to highlight? 6. Priorities - what to see first (drives row ordering and which section is collapsed: true).

Don't block on answers you can reasonably infer - state the inference and continue.

---

Phase 3: Draft the internal plan

Write a markdown plan the user can approve before JSON generation:

## Dashboard: <Service> - <Purpose>

### Section 1: <Overview> (collapsed: false)
- Row 1: [widget type] <title> - <what it shows> - source: metrics|logs|spans
- Row 2: ...

### Section 2: <Deep dive> (collapsed: false)
...

### Section N: <Logs & errors> (collapsed: true)
...

### Top-level filters
- <label> (<source>)

### Assumptions / gaps
- ...

Section design:

• First section: at-a-glance health (gauges + key rates), always expanded.
• Pair related time-series in the same row (rate + latency).
• Final section (raw logs, rare breakdowns): collapsed: true.
• Aim for 3–5 sections, 6–20 widgets total.

Widget-type selection:

Don't use other widget types unless the user asks.

Wait for the user to approve or adjust the plan before emitting JSON.

---

Phase 4: Generate the Coralogix JSON

Produce a single JSON document following `references/widget-templates.md`. Key rules:

1. Top-level shape:

   {
     "id": "<21-char-nanoid>",
     "name": "<Dashboard Name>",
     "layout": { "sections": [ ... ] },
     "variables": [],
     "variablesV2": [],
     "filters": [ ... ],
     "relativeTimeFrame": "<seconds>s",
     "annotations": [],
     "off": {},
     "actions": []
   }

2. IDs - fresh UUIDs for every section, row, widget, and query id. 3. Row height - "appearance": { "height": 19 } unless there's a reason to change. 4. Section options - include options.custom.name, collapsed, and color.predefined: "SECTION_PREDEFINED_COLOR_UNSPECIFIED". 5. Filters - one entry per slicing dimension from Phase 2. Default operator equals with empty values so users can fill in. Use notEquals for environment exclusions (see `references/widget-templates.md`). 6. relativeTimeFrame - default "172800s" (48h) unless the user specified otherwise.

For query syntax follow `references/query-syntax.md`; for the full query languages load `references/dataprime-reference.md` and `references/promql-guidelines.md`.

---

Phase 5: Live-verify every query through the cx CLI

Every PromQL and DataPrime query in the draft has to successfully run through cx before Phase 7. This catches invented metric names, typoed field paths, and malformed pipelines.

Frequent vs Archive (what / when / where in JSON)

What:

• Frequent (TIER_FREQUENT_SEARCH): hot tier for fast search on recent logs/spans.
• Archive (TIER_ARCHIVE): cold tier for older logs/spans (long-term).

When to choose:

• Choose Frequent for on-call and recent investigations (hours/days).
• Choose Archive for long lookbacks (weeks/months) or when the time range is beyond hot retention.

The two languages are verified against different windows:

• PromQL: map relativeTimeFrame to a $RANGE token (e.g. 48h for 172800s), substitute ${__range} with [$RANGE] for the CLI call, then restore ${__range} in the JSON before Phase 6. Range vectors are window-sensitive, so the check has to match what the dashboard will evaluate.
• DataPrime: verify against a fixed short window (now-15m → now, --limit 1). The goal is syntax / field / pipeline validation, not data-presence on the dashboard's window — a short window is faster and a cleaner fail signal.

Full procedure (CLI invocations, $RANGE mapping table, retry budget, failure modes): `references/verification.md`.

If a query can't be made to pass within the retry budget, surface it to the user with the CLI error verbatim - don't ship a broken widget.

---

Phase 6: Self-verify structure

Run this checklist against the final JSON. Fix and re-check if any item fails before Phase 7.

Query syntax (dashboard-specific)

• [ ] Every PromQL range vector in a metrics widget uses [${__range}] - never [$__range], never [5m] (unless the panel is intentionally a sliding window).
• [ ] promqlQueryType is PROM_QL_QUERY_TYPE_INSTANT for single-value widgets (gauge, pieChart, dataTable). Omitted for lineChart.
• [ ] DataPrime log queries use $d.message / $l.applicationname / unquoted severity enums (full rules: `references/dataprime-reference.md`).
• [ ] Every DataPrime widget query starts with source logs or source spans (dashboard widgets require the source prefix; Phase 5 verification strips it before handing the pipeline to cx logs / cx spans).
• [ ] Success-rate denominators wrapped in clamp_min(..., 1).
• [ ] Histogram queries use the correct suffix (_sum, _count, _bucket).
• [ ] Widget queries are valid without the dashboard-level filters - Coralogix injects them at render time.

Structure

• [ ] Each section has id.value, rows, and options.custom.
• [ ] Each row has id.value, appearance.height, and widgets.
• [ ] Each widget has a unique id.value and a definition with exactly one of gauge / pieChart / lineChart / dataTable.
• [ ] Every gauge has numeric min and max, and min < max.
• [ ] Success-rate gauges use thresholdType: "THRESHOLD_TYPE_ABSOLUTE" with green at high values; error/DLQ gauges use red at high values.
• [ ] "Total" / "stat" widgets are encoded as gauge, not as a stat type.
• [ ] Top-level filters includes each slicing dimension from Phase 2.
• [ ] All IDs are freshly generated UUIDs, unique within the document.

Content

• [ ] Dashboard name is descriptive ("<Service> - <Purpose>").
• [ ] Widget titles are short, human-readable, and match what the query computes.
• [ ] The logs/errors section is collapsed: true unless the user said otherwise.

---

Phase 7: Deploy via cx dashboards create

Don't tell the user to paste JSON into the Coralogix UI - deploy it directly.

1. List folders: cx dashboards folders list -o json. 2. Suggest the best folder match (team, product area, or a folder named after the service). Default to root (omit --folder) if nothing fits. 3. Write the verified JSON to a temp file and run cx dashboards create --from-file /tmp/cx-dashboard-<slug>.json --folder <id>. The CLI generates the requestId envelope and prints the created dashboard ID.

Full procedure (folder-picking UX, command templates, idempotency note): `references/deploy.md`.

On failure: show the CLI error verbatim and return to Phase 5. The most common cause is a query that parses locally but the live API rejects.

---

Phase 8: Share the dashboard link

The workflow is not done until the user has a clickable link to the dashboard. Printing the ID alone forces the user to navigate the Coralogix UI by hand, which defeats the point of automating deployment.

After Phase 7 succeeds, capture the dashboard id returned by cx dashboards create, build the URL using the region → webapp host mapping in `references/deploy.md` § "Share the link", and emit the output template below. Render the dashboard name as the link text — that's what the user clicks.

---

Output format for the user

When the webapp host is resolvable (any built-in region, or a Custom endpoint that follows the api.<host> convention):

````

Plan

<the approved Phase 3 plan>

Verification

• PromQL queries verified: <N>/<N>
• DataPrime queries verified: <N>/<N>

Deployed

• Dashboard: [<Name>](https://<region>.app.coralogix.com/#/dashboards/<id>)
• ID: <id>
• Folder: <folder name or "root">
• Profile: <cx profile>

Open it: <Name>

Adjust filter values (e.g. account_id) after opening it. ````

When the webapp host cannot be derived (custom endpoint that doesn't match api.<host>), omit the link entirely — do not invent a URL. Use this template instead:

````

Plan

<the approved Phase 3 plan>

Verification

• PromQL queries verified: <N>/<N>
• DataPrime queries verified: <N>/<N>

Deployed

• Dashboard: <Name> (open via the Coralogix UI; ID <id>)
• ID: <id>
• Folder: <folder name or "root">
• Profile: <cx profile>

Adjust filter values (e.g. account_id) after opening it. ````

---

References

• Dashboard query gotchas & cross-references: `references/query-syntax.md`
• Widget JSON templates: `references/widget-templates.md`
• Live-verification procedure: `references/verification.md`
• Deploy procedure: `references/deploy.md`
• DataPrime language reference: `references/dataprime-reference.md`
• PromQL reference: `references/promql-guidelines.md`
• Log querying patterns: `references/logs-querying.md`
• Span querying patterns: `references/spans-querying.md`
• Inline DataPrime help: cx dataprime list, cx dataprime show <command>
• Coralogix Custom Dashboards docs: <https://www.coralogix.com/docs/user-guides/custom-dashboards/introduction/>

Related Skills

• `cx-observability-setup` - full monitoring setup workflow (views, webhooks, notifications, integrations)
• `cx-incident-management` - SLO and alert-connected dashboards, incident triage
• `cx-telemetry-querying` - discover the right telemetry signal before building dashboards

DataPrime Query Language Reference

Query Structure

A DataPrime query is a pipeline of commands separated by |. Each command transforms the output of the previous one:

filter $m.severity == ERROR | groupby $l.subsystemname aggregate count() as errors

Source Handling

Every query targets a source (logs, spans, etc.). The source is set by whichever cx command you use. A full query with an explicit source looks like:

source <logs|spans> | filter ... | groupby ...

When running via a source-specific command (e.g. cx logs, cx spans), the source is injected automatically - omit it from the query. When running via cx dataprime query, use the --source flag or include source in the query itself.

The examples below focus on the DataPrime query language and omit the source and CLI command prefix.

Comments

Comments are supported with # or //:

filter $m.severity == ERROR  # only errors
| limit 10                   // cap results

Data Prefixes

All fields are accessed through three namespaces:

$d is the default prefix and can sometimes be omitted, but being explicit avoids ambiguity.

Data Types

Commands

Filtering and Selection

Note on `wildfind`: It is a standalone command, not a condition within filter. You cannot combine it with other filter expressions - use it as its own pipeline stage.

Aggregation

Transformation

Operators

Type Conversions

Cast fields inline with :type:

filter $d.http_error_code:number == 500

Supported types: bool, number, string, timestamp, interval, array, object

Field Access

# Chained field names (dot notation)
filter $d.tags.user_context.email == 'test@example.com'

# Special characters require brackets
filter $d.http['status/code'] == 500

Aggregation Functions

Example - full CLI invocation:

cx dataprime query --source logs 'groupby $l.subsystemname aggregate count() as error_count, avg($d.response_time) as avg_response | orderby error_count desc'

Utility Functions

firstNonNull - Field Coalescing

Return the first non-null value from a list of fields. Useful when the same data may appear in different fields across log sources:

# Merge fields
create message from firstNonNull($d.error_message, $d.msg, $d.body)

# Use inside groupby
groupby firstNonNull($d.error_message, $d.msg) as message aggregate count() as n

Template Sampling

Find top error patterns with a sample message for each:

filter $m.severity == ERROR | groupby $m.templateid aggregate any_value($d) as sample, count() as total | orderby total desc | limit 5

Time-Based Grouping

Use roundTime() to bucket timestamps:

# Group by hour
groupby roundTime($m.timestamp, 1h) as hour aggregate count() as count

# Error rate over 15-minute intervals
filter $m.severity == ERROR | groupby roundTime($m.timestamp, 15m) as interval aggregate count() as errors

Multi-Value Matching

Use arrayContains to match against a set of values:

# Match multiple subsystems
filter ['api', 'web', 'worker'].arrayContains($l.subsystemname)

# Match multiple severity levels
filter [ERROR, CRITICAL].arrayContains($m.severity)

Text Extraction

Regex Extraction

# Extract with unnamed capture group
extract $d.email into domain using regexp(e=/@(.*)/) | distinct $d.domain._0

# Named capture groups
extract $d.email into extracted using regexp(e=/(?<username>[a-zA-Z0-9._%+-]+)@(?<domain>.*)/) | choose $d.extracted.username, $d.extracted.domain

JSON String Parsing

# Parse a JSON string field into an object for further querying
extract $d.json_payload into parsed using jsonobject() | filter $d.parsed.status == 'failed'

Deduplication

# Remove duplicates by log template
dedupeby $m.templateid

# Dedupe by a custom field
dedupeby $d.request_id

Built-In Documentation

For the full list of commands and functions with detailed syntax:

cx dataprime list                              # List all commands and functions
cx dataprime list --filter commands             # Commands only
cx dataprime list --filter functions --name time # Search functions by name
cx dataprime show filter                        # Detailed help for a specific command
cx dataprime show groupby

Validating a DataPrime query

A query that looks right can still fail on a typoed field path, an invented function, or a malformed pipeline stage. Validate before trusting the output — a short-window run through the CLI is cheap and catches almost all of these:

cx logs  '<pipeline>' --start now-15m --end now --limit 1
cx spans '<pipeline>' --start now-15m --end now --limit 1

now-15m is a good default; widen it only if 15 minutes is unlikely to exercise the pipeline. Per "Source Handling" above, omit any leading source logs / source spans — cx logs and cx spans inject the source themselves.

Check both the exit code and the output — some errors surface only in the output.

Pass = exit 0 and the output is rows or [] with no error or warning lines.

Hard fail — query is broken, fix it:

• non-zero exit
• error from profile '...': API request failed — HTTP error from the API
• Compilation errors: — parse error, unknown function, malformed expression

Soft fail (needs investigation):

• keypath does not exist — the query parsed, but no record in the window had the referenced field. This is ambiguous: the field name might be a typo, or it might be real but absent from records in this 15-minute slice. Confirm with cx search-fields "<field hint>" --dataset logs (or --dataset spans). If the field is real, the query is fine — try a wider window or accept the empty result. If it isn't, fix the field name.

On fail: re-discover fields with cx search-fields, look up command syntax with cx dataprime show <command>, fix, re-run.

Phase 7: Deploy via cx dashboards create

Don't tell the user to paste JSON into the Coralogix UI - deploy the dashboard directly.

---

1. Pick a folder

List folders and suggest the best match:

cx dashboards folders list -o json

Rank the existing folders by relevance (service name, team, product area) and present the top matches with AskQuestion:

• "Folder X (id: <id>) - best match by name"
• "Folder Y (id: <id>)"
• "Root (no folder)"
• "None of these - I'll create a folder in the Coralogix UI first"

Default to "Root" if nothing fits.

---

2. If the user wants a new folder

Ask them for a folder name (and an optional parent folder id - omit for a top-level folder), then create it directly:

cx dashboards folders create --name "<Folder Name>"
# or, as a sub-folder of an existing one:
cx dashboards folders create --name "<Sub-folder>" --parent-id <parent-folder-id>

The command prints the new folder id. Use that id as --folder in step 3.

If folder creation fails (most common cause: API key missing the team-dashboards:Update permission), fall back to the Coralogix UI - Dashboards → Folders → + New folder - then rerun cx dashboards folders list -o json and proceed with the chosen id.

---

3. Save and deploy

1. Write the verified JSON to /tmp/cx-dashboard-<slug>.json (use the file-write tool; don't prescribe a specific shell idiom). 2. Deploy into the chosen folder (omit --folder for root):

   cx dashboards create --from-file /tmp/cx-dashboard-<slug>.json --folder <folder-id>

The CLI generates the requestId envelope automatically and prints the created dashboard ID and name on success. Pipe into -o json or -o agents for structured output.

On failure: show the CLI error verbatim, return to Phase 5 (most common cause: a query that parses locally but the live API rejects), fix, and redeploy.

On success: continue to step 4 below — the workflow is not finished until the user has the link.

---

4. Share the link (final step — mandatory)

Don't stop at "dashboard created". The very last action is to give the user a clickable link to the dashboard.

Build the URL from the active profile's region and the dashboard id returned by cx dashboards create:

https://<region>.app.coralogix.com/#/dashboards/<id>

Region → webapp host mapping:

If the custom endpoint doesn't follow the api. prefix convention, omit the link entirely — do not invent a URL. Use the second ("webapp host cannot be derived") template in SKILL.md § "Output format for the user", which drops the markdown link from the Deployed line and drops the standalone Open it: line so the user is never shown a broken URL.

Render the link as a markdown link using the dashboard name as the link text, e.g.:

Dashboard: **[Order Service - Health](https://eu2.app.coralogix.com/#/dashboards/abc123def456)**

Then emit the summary defined in the main SKILL.md § "Output format for the user".

---

5. Replace an existing dashboard

To update a dashboard that already exists (instead of creating a new one), use the replace workflow:

1. Get the current definition:

   cx dashboards get <dashboard-id> -o json > dashboard.json

2. Edit dashboard.json (change widgets, queries, filters, etc.). The id field must remain intact.

3. Deploy the updated version:

   cx dashboards replace --from-file dashboard.json --yes

This is a full replacement - the entire dashboard definition is overwritten. The id field in the JSON determines which dashboard is updated.

Use replace when:

• The user asks to update, modify, or iterate on an existing dashboard.
• You're refining a dashboard after Phase 5 verification found issues.
• The user exported a dashboard and wants to push changes back.

Use create (not replace) when:

• Building a new dashboard from scratch.
• Duplicating an existing dashboard (remove the id field first).

---

6. Idempotency note

Each create run generates a fresh top-level id (21-char nanoid), so re-running creates a new dashboard rather than overwriting an existing one. Use replace to update in place.

Log Querying Reference

Query and analyze Coralogix logs using the cx logs command with DataPrime syntax.

DataPrime syntax: See dataprime-reference.md for the full query language reference.

Understanding Logs in Coralogix

Logs in Coralogix are largely unstructured. Every log entry has a small structured envelope - metadata and labels - but the actual application payload (userData) is free-form and varies entirely by application. There is no universal schema for $d.* fields.

This means:

• *Metadata (`$m.)** and **labels ($l.`)* are predictable - you can always filter on severity, timestamp, application name, and subsystem name without discovery.
• *User data (`$d.)** is not predictable - field names, nesting, and types depend on whatever the application chose to log. Always verify $d` fields before assuming they exist.

---

CLI Command

cx logs '<dataprime_query>'

The source logs prefix is automatically injected if the query doesn't already include a source command.

Options

---

Log Data Model

Standard Fields (Always Available)

Severity Values

Severity keywords are used without quotes in DataPrime:

DEBUG | INFO | WARNING | ERROR | CRITICAL

cx logs 'filter $m.severity == ERROR'
cx logs 'filter [ERROR, CRITICAL].arrayContains($m.severity)'

---

Essential Query Examples

# Filter by severity
cx logs 'filter $m.severity == ERROR'

# Text search in a known field
cx logs "filter \$d.message ~ 'timeout'"

# Filter by application and subsystem
cx logs "filter \$l.applicationname == 'api' && \$l.subsystemname == 'auth'"

# Aggregate errors by subsystem
cx logs 'filter $m.severity == ERROR | groupby $l.subsystemname aggregate count() as errors | orderby errors desc'

# Wider time range and archive tier
cx logs "filter \$l.subsystemname == 'payments'" --tier archive --start now-7d

Wildfind Policy

Avoid `wildfind` by default. It scans all fields and returns noisy results, especially for generic terms.

The one exception: when the user provides a specific, quoted error message or log string and you don't know which field contains it:

# User says: "Find logs with 'connection refused'"
cx logs "wildfind 'connection refused'"

In all other cases, use filter with known fields ($m.severity, $l.subsystemname, $d.<field>) or discover field names first with cx search-fields.

---

Field Discovery

Skip discovery when:

• The query only uses standard fields ($m.severity, $m.timestamp, $l.applicationname, $l.subsystemname)
• The user explicitly names the fields they want (e.g., "filter by $d.customer_id")
• You're searching for a specific error message - use wildfind directly
• The fields have already been discovered earlier in the conversation

For customer-specific $d.* fields that need discovery, use one of these approaches:

1. Infer from Source Code (Preferred)

If you have access to the application's source code, examine logger calls, structured logging configs, and log format templates to identify field names directly.

2. Semantic Search

cx search-fields "customer identifier" --dataset logs
cx search-fields "http response code" --dataset logs

Returns DataPrime paths with similarity scores:

+------------------------+-----------------------------------+-----------+
| DataPrime path         | Description                       | Similarity|
+------------------------+-----------------------------------+-----------+
| $d.customer_id         | Unique customer identifier        | 0.89      |
| $d.user.account_id     | Customer account reference        | 0.85      |
+------------------------+-----------------------------------+-----------+

3. Sample Query Inspection

cx logs "filter \$l.subsystemname == 'api'" --limit 5 -o json

Inspect the JSON output to see all available fields in the actual data.

---

Investigation Workflow

1. Understand the Request

Identify:

• What type of logs are needed (errors, info, specific events)
• Time frame of interest
• Key entities (services, users, transactions)

2. Start with Standard Fields

For basic queries, use standard fields directly:

# Recent errors - no discovery needed
cx logs 'filter $m.severity == ERROR | limit 20'

# Errors in a specific subsystem
cx logs "filter \$m.severity == ERROR && \$l.subsystemname == 'payment-service'"

3. Build and Execute Query

Start simple, add complexity:

# Step 1: Check if data exists
cx logs "filter \$l.subsystemname == 'checkout'" --limit 10

# Step 2: Add filters
cx logs "filter \$l.subsystemname == 'checkout' && \$m.severity == ERROR"

# Step 3: Add aggregation
cx logs "filter \$l.subsystemname == 'checkout' && \$m.severity == ERROR | groupby \$d.error_type aggregate count() as occurrences"

4. Troubleshooting

If a query returns no results, change one thing at a time:

1. Extend the time range: --start now-6h or --start now-24h 2. Relax filters: remove the most restrictive condition 3. Verify field names: run a sample query with -o json to inspect the actual schema 4. Try archive tier: --tier archive --start now-30d for older data

---

Common Query Patterns

Error Investigation

# All errors in last hour
cx logs 'filter $m.severity == ERROR'

# Critical errors only
cx logs 'filter $m.severity == CRITICAL'

# Errors with text search
cx logs "filter \$m.severity == ERROR && \$d.message ~ 'database connection'"

Aggregation by Service

# Error count by subsystem
cx logs 'filter $m.severity == ERROR | groupby $l.subsystemname aggregate count() as errors | orderby errors desc'

# Error count by application and subsystem
cx logs 'filter $m.severity == ERROR | groupby $l.applicationname, $l.subsystemname aggregate count() as errors'

Time-Based Analysis

# Errors per hour
cx logs 'filter $m.severity == ERROR | groupby roundTime($m.timestamp, 1h) as hour aggregate count() as count'

# Find error spikes in 5-minute windows
cx logs 'filter $m.severity == ERROR | groupby roundTime($m.timestamp, 5m) as interval aggregate count() as count | orderby count desc | limit 10'

Finding Unique Values

# List all subsystems with errors
cx logs 'filter $m.severity == ERROR | distinct $l.subsystemname'

# List unique error types
cx logs 'filter $m.severity == ERROR | distinct $d.error_type'

Fetching Sample Logs by Template

Find top error patterns with sample messages:

cx logs 'filter $m.severity == ERROR | groupby $m.templateid aggregate any_value($d) as sample, count() as total | orderby total desc | limit 5'

---

Performance Tips

• Use --limit for exploratory queries
• Use groupby with aggregations instead of fetching all raw logs
• Filter by time first when dealing with large datasets
• Use specific filters (application, subsystem) to reduce scan scope
• For large result sets, use --output agents which spills to a temp file automatically:

cx logs 'filter $m.severity == ERROR' --start now-24h --limit 1000 -o agents

PromQL Guidelines

Core Principles

1. Pick the right query type

• Instant queries (cx metrics query) evaluate an expression at a single timestamp (now, or a given --time). Use when the question requires one number or one vector as of a moment - essentially any query that does not require results over different timeframes.
• Range queries (cx metrics query-range) evaluate the expression repeatedly across [--start, --end] at a given --step. Use for time series over a period (e.g., daily active users per day).
• Note: Range queries evaluate the expression repeatedly at each step. If --step=1d, --start=now-1d, --end=now, and the query is max_over_time(metric[1d]), the query evaluates at now-1d and now - two evaluations covering two days of data.
• Prefer instant queries over range queries for most questions, except when comparing different timeframes.

2. Understand PromQL value types

• Instant vector - set of series with 1 sample each at eval time
• Range vector - series with many samples over a window [t-range, t]
• Scalar - single number
• String - rare
• Functions like *_over_time() require a range vector. Aggregations like sum/max/min/avg ... by(...) consume instant vectors.
• Important: When using *_over_time() functions with range queries, be aware that the query also evaluates at the --start time and includes the window specified in the function.
• Example: If max_over_time(metric[1d]) is used with --start=now-1d, --end=now, --step=1d, the query evaluates at now-1d and now - the result is the max over [now-2d, now]. This is a common mistake. If a user asks "What is the max of x between 2025-01-01 and 2025-01-07?" and max_over_time(x[7d]) is used with --start=2025-01-01, --end=2025-01-07, --step=1d, the evaluation at 2025-01-01 includes [2024-12-25, 2025-01-01] - which is wrong. Use an instant query with --time to avoid this.

3. Separation of concerns

• Use *_over_time() for temporal reductions across a window (e.g., max_over_time, avg_over_time, quantile_over_time).
• Use sum/max/min/avg by (...) for label-set aggregation across series at the eval point.
• Chain them as needed (temporal reduction first, then label aggregation, or vice versa).

4. Counters vs. gauges

• Counters (monotonic, suffixed _total) → use rate()/irate() or increase() over a window.
• Gauges (current value) → use avg_over_time, max_over_time, etc., or plain avg(...) depending on intent.

5. Suffix conventions

• Canonical: _total (counter), _bucket/_sum/_count (histogram), _sum/_count (summary), _created.
• Non-standard: _avg, _mean, etc. Prefer computing averages via PromQL unless the exporter dictates otherwise.

---

CLI Usage

Instant Query

cx metrics query '<expr>'
cx metrics query '<expr>' --time 2024-01-01T12:00:00Z
cx metrics query '<expr>' --output json

Example: absolute max over last 24h (single result)

cx metrics query 'max by () (max_over_time(http_requests_in_flight[24h]))'

Range Query

cx metrics query-range '<expr>' --start now-7d --end now --step 1d

Example: absolute max per day over the last 7 days

cx metrics query-range 'max by () (max_over_time(metric[1d]))' \
  --start now-7d --end now --step 1d

IMPORTANT: Align --step with any window used in temporal reduction functions. If using max_over_time(metric[1d]), set --step 1d.

---

PromQL Fundamentals

Label Matching & Aggregation

• Matchers: {label="v"}, {label!="v"}, {label=~"re.*"}, {label!~"re"}
• Aggregate by labels to keep them; use without to drop them.

sum by (job) (rate(http_requests_total[5m]))
sum without (instance) (up)

Temporal Reductions (range → instant)

max_over_time(cpu_usage[1h])
avg_over_time(node_memory_Active_bytes[30m])
quantile_over_time(0.99, queue_length[1h])

Counters: Rates, Increases, Windows

Per-instance RPS:

rate(http_requests_total[5m])

Total RPS across fleet:

sum by () (rate(http_requests_total[5m]))

Events in last day (per user, then count actives):

count( sum by (user_id) (increase(api_call_count[24h])) > 0 )

Histograms & Summaries

p95 from a histogram:

histogram_quantile(
  0.95,
  sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))
)

Average from summary parts:

sum(rate(req_duration_seconds_sum[5m]))
/
sum(rate(req_duration_seconds_count[5m]))

Max over a Period

Correct - temporal reduction, then aggregation:

max by () (max_over_time(metric[4d]))

Per-label max:

max by (label) (max_over_time(metric[4d]))

Incorrect - max() cannot take a range vector:

max(metric[4d])   ← error

Top-k / Ranking

topk(5, sum by (instance) (rate(http_requests_total[5m])))

---

Common Tasks (ready to adapt)

1. Absolute peak per instance over 7d, then pick the winner

topk(1, max by (instance) (max_over_time(my_metric[7d])))

Run as instant query (no --time needed - defaults to now).

2. Global CPU usage % (avg across cores & hosts)

avg by () (
  rate(process_cpu_seconds_total[5m])
) * 100

3. Error rate (%) per route

100 * sum by (route) (rate(http_requests_total{code=~"5.."}[5m]))
    / sum by (route) (rate(http_requests_total[5m]))

4. Daily active users over a week (time series)

Expression:

count(count by (user_id) (increase(api_call_count[1d]) > 0))

Run as range query with --step 1d --start now-6d --end now. (Starting from 6 days ago because increase looks back one full day from each evaluation point.)

---

Performance & Safety Guidelines

• Prefer short windows for rate() (e.g., 1–5m) unless data is bursty or sparse.
• Avoid unbounded fan-out (e.g., joining massive label sets).
• Keep cardinality under control; aggregate early (sum by (...)) when only totals are needed.
• Use clamp_max/clamp_min to tame outliers when needed.
• For histograms, always aggregate buckets (sum by (le, ...)) before histogram_quantile.
• Be mindful of counter resets; rate()/increase() handle resets automatically.

---

Frequent Gotchas (and fixes)

• "Why am I getting a time series when I only want one number?"

Use cx metrics query (instant) instead of cx metrics query-range.

• "`max(metric[...])` errors."

max() can't take a range vector. Use max_over_time(metric[...]), then aggregate with max by () (...).

• "`_over_time(metric[...]) by (label)` errors."

_over_time aggregations cannot include a by clause. Use max_over_time(metric[...]), then max by (label) (...).

• "Avg looks wrong for counters."

Counters need rate()/increase(), not avg_over_time.

• "p95 from a summary?"

Summaries expose quantiles directly via the quantile label. For histograms, use histogram_quantile on bucket rates.

• "Results show empty label values."

Add {label!=""} to the selector to filter out empty label values. Example: max by (deployment) (rate(cpu_usage{deployment!=""}[5m])).

---

Mini Cheat-Sheet

Coralogix Dashboard Query Syntax - Dashboard-Specific Gotchas

This file documents only the non-obvious rules specific to authoring a Coralogix dashboard JSON - the things that silently break an imported dashboard even when the query itself is valid.

For the underlying query languages, use the sibling skills:

• DataPrime (filters, aggregations, operators, type conversions, extract, roundTime): cx-dataprime skill → skills/cx-dataprime/references/dataprime-reference.md.
• PromQL (counters vs gauges, histograms, label matching, temporal reductions, full function list): cx-metrics-query skill → skills/cx-metrics-query/references/promql-guidelines.md.
• Inline help: cx dataprime list and cx dataprime show <command>.

---

1. The ${__range} dashboard variable

Coralogix injects the dashboard-level time range as ${__range} - this is a dashboard-only variable, not a PromQL feature.

• Correct: increase(foo_total[${__range}])
• Wrong: increase(foo_total[$__range]) - missing braces; Coralogix drops it
• Wrong: increase(foo_total[5m]) - hard-codes a 5-minute window; the dashboard time picker has no effect

Use fixed ranges ([5m], [1h]) only when the panel intentionally shows a rolling window independent of the dashboard time picker (rare; document it in the panel title).

Verifying ${__range} queries with the CLI

cx metrics query doesn't expand ${__range} (it's a dashboard-side substitution). During Phase 5 verification, swap ${__range} for the concrete token that matches the dashboard's relativeTimeFrame (e.g. [48h] for the default "172800s"). Restore ${__range} in the JSON before deploy.

---

2. promqlQueryType - instant vs time-series widgets

Every metrics widget has a promqlQueryType:

• PROM_QL_QUERY_TYPE_INSTANT - single point in time. Required for gauge, pieChart, and dataTable.
• Omit (or default) - time series. Use for lineChart.

Leaving an instant widget in time-series mode makes the panel render a single average across the window instead of the intended single-point value, which breaks success-rate gauges and top-N tables.

---

3. DataPrime inside widget JSON

The DataPrime language itself is documented in the cx-dataprime skill. A few rules matter specifically inside dashboard widgets:

• DataPrime widget queries must start with source logs or source spans - Coralogix doesn't infer the source for DataPrime widgets the way it does for metrics. Example: source logs | filter $m.severity == ERROR | agg count().
• The JSON key is dataprimeQuery.text (a string), not dataprimeQuery.value.
• filter / groupby / aggregate forms use the same syntax as the CLI - the cx-dataprime skill is authoritative.
• Severity enums are unquoted in DataPrime: $m.severity == ERROR, not "ERROR".

Widget-side gotchas that commonly appear in reviews

• $d.message, not $m.text.
• contains is a method on the field: $d.message.contains('timeout'), not $d.message contains 'timeout'.
• Application filter uses $l.applicationname (lowercase), not $m.applicationName.

If you're unsure, consult the cx-dataprime skill reference rather than guessing.

---

4. PromQL idioms that recur in dashboards

The full PromQL reference is in the cx-metrics-query skill. The patterns below show up in almost every dashboard and are the ones to copy-paste.

Histogram average over the dashboard range:

sum by (label) (rate(foo_latency_sum[${__range}]))
/
sum by (label) (rate(foo_latency_count[${__range}]))

Histogram P95:

histogram_quantile(0.95,
  sum by (le, label) (rate(foo_latency_bucket[${__range}]))
)

Counter increments over the range:

sum by (account_id) (increase(foo_total[${__range}]))

Success rate (%) with safe denominator (wrap denominators in clamp_min(..., 1)):

100 *
  sum(increase(foo_success_total[${__range}]))
  /
  clamp_min(
    sum(increase(foo_success_total[${__range}]))
    + sum(increase(foo_failure_total[${__range}])),
    1
  )

Propagating a label across metrics (metric A has account_id; metric B has both account_id and account_slug):

sum by (account_id) (increase(foo_total[${__range}]))
* on (account_id) group_left(account_slug)
max by (account_id, account_slug) (
  (max_over_time(bar_total{account_slug!=""}[365d] @ end()) * 0) + 1
)

---

5. Lucene (legacy logs query)

Only use if the user explicitly requests Lucene - prefer DataPrime.

• Severity: coralogix.metadata.severity:ERROR
• Application: coralogix.metadata.applicationName:"my-service"
• Field match: message:"is stuck"

---

6. Cross-references: how Olly and Coralogix consume this JSON

Queries authored by this skill are eventually consumed by the Coralogix dashboard-rendering layer and by the Olly agent that analyses dashboards. The rules below reflect how the JSON is actually treated - verified against the cx-olly, olly, and olly-knowledge-base repositories.

Field precedence Olly enforces

Source: olly/apps/api/src/api/agent/handlers/utils/dashboard_enrichment.py (_get_important_fields) surfaces exactly two dashboard fields to the LLM as "important":

• `relativeTimeFrame` - Olly uses this as the default time range when the user doesn't pass from/to. Our Phase 5 CLI verification therefore has to run with the same relativeTimeFrame - don't verify a 48h dashboard with a [5m] window.
• `filters` - Olly injects the dashboard-level filters array into every DataPrime and PromQL query before sending it. DASHBOARD_FOCUSED_PROMPT in olly/apps/api/src/api/agent/agents/orchestrator_agent/dashboard_prompts.py instructs: "EDIT ALL queries to INCLUDE THE FILTERS FROM THE DASHBOARD DATA". Consequence: widget queries must be valid before the dashboard filters are injected - do not pre-bake the slicing dimensions into the widget's query text.

JSON keys Olly consumes

Source: WHITELIST_DASHBOARD_KEYS in olly/apps/api/src/api/agent/handlers/utils/dashboard_enrichment.py:

name, description, title, isVisible, columns, query, unit, scaleType,
variables, variablesV2, filters, relativeTimeFrame, annotations,
twoMinutes, slugName, actions, updatedAt, createdAt, updaterAuthorId,
updaterName, authorId, authorName, updatedOriginType, createdOriginType

Prefer these exact field names over synonyms. Keys outside this list are dropped from Olly's view of the dashboard.

Allowed query languages in a widget

Source: olly-knowledge-base/common/src/common/models/dashboard_context_models.py (QueryEmbedding.query_type): only dataprime, promql, or lucene. Nothing else will be indexed or embedded downstream.

Sibling Olly skill

The dashboard-analysis counterpart to this skill lives at cx-olly/apps/api/src/api/agent/agents/skills_agent/skills/dashboards.md. It covers the consume-side rules (accept dashboard links, honor var-* parameters, always add filters + relativeTimeFrame to derived queries). Our authoring rules above mirror its expectations.

Coralogix docs

• Overview: <https://www.coralogix.com/docs/user-guides/custom-dashboards/introduction/>
• Widgets index: <https://www.coralogix.com/docs/user-guides/custom-dashboards/widgets/>
• Query builder: <https://www.coralogix.com/docs/user-guides/custom-dashboards/tutorials/query-builder/>
• Dashboard settings (slugs, folders): <https://www.coralogix.com/docs/user-guides/custom-dashboards/tutorials/manage-dashboard-settings/>

---

7. Dashboard-syntax checklist (apply before Phase 5)

• [ ] PromQL range vectors inside widgets use [${__range}].
• [ ] promqlQueryType is PROM_QL_QUERY_TYPE_INSTANT for gauge / pieChart / dataTable; omitted for lineChart.
• [ ] DataPrime widget queries start with source logs or source spans (required for dashboard widgets; stripped during Phase 5 CLI verification).
• [ ] DataPrime contains is written as .contains(...).
• [ ] DataPrime severity enums are unquoted (ERROR, CRITICAL, …).
• [ ] Success-rate denominators wrapped in clamp_min(..., 1).
• [ ] Histogram queries use the correct suffix (_sum, _count, _bucket).
• [ ] No invented metric names - every PromQL metric appeared in cx metrics search during Phase 1.
• [ ] Widget queries remain valid without the dashboard-level filters (Coralogix/Olly inject them at render time).
• [ ] Widget query language is one of dataprime, promql, or lucene - nothing else.
• [ ] relativeTimeFrame in the final JSON matches the window used for Phase 5 verification.

Span Querying Reference

Query and analyze distributed tracing data using the cx spans command with DataPrime syntax.

DataPrime syntax: See dataprime-reference.md for the full query language reference.

Understanding Spans in Coralogix

Spans are the fundamental unit of tracing data. Traces are not stored as single entities - they are logical groupings of spans that share the same traceID. To analyze a trace, you query its constituent spans.

This means:

• *Metadata (`$m.)** and **labels ($l.`)* are predictable - you can always filter on timestamp, duration, service name, and operation name without discovery.
• *User data (`$d.)** contains trace identifiers (traceID, spanID, parentSpanID) and application-specific tags/attributes that vary by service. Always verify custom $d` fields before assuming they exist.

---

CLI Command

cx spans '<dataprime_query>'

The source spans is automatically injected - do not include it in the query.

Options

---

Span Data Model

Standard Fields (Always Available)

Note on label fields: The meaning of $l.applicationName and $l.subsystemName varies by customer - they may represent environments, teams, regions, or something else entirely. Don't assume what they map to. Use cx search-fields or sample queries to verify actual values.

Duration Units

$m.duration is in microseconds:

• 500ms = 500000
• 1s = 1000000
• 1min = 60000000

When presenting duration values, always convert to human-readable units (milliseconds, seconds, or minutes) and include the unit. Never display raw microsecond values or the "µs" symbol.

# Computed field for milliseconds
create latency_ms from $m.duration / 1000

Error Detection

Spans do not have a $m.severity field like logs. Errors are typically indicated by:

• $d.tags.error == true - the most common convention (OpenTelemetry/Jaeger)
• Status codes in custom fields (e.g. $d.http.status_code, $d.grpc.status_code)
• Other application-specific error tags

The exact field depends on the instrumentation library used. If $d.tags.error returns no results, inspect sample spans with -o json to discover how errors are tagged:

cx spans "filter \$l.serviceName == 'api'" --limit 5 -o json

---

Essential Query Examples

# Get all spans for a trace
cx spans "filter \$d.traceID == '4f6a8f3c2e8a1b97'"

# Find spans for a service
cx spans "filter \$l.serviceName == 'checkout-service'"

# Find slow spans (> 1 second)
cx spans "filter \$m.duration > 1000000"

# Find error spans
cx spans "filter \$d.tags.error == true"

# Aggregate latency by operation
cx spans "groupby \$l.operationName aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc"

# Wider time range
cx spans "filter \$l.serviceName == 'api'" --start now-6h

Wildfind Policy

Avoid `wildfind` by default. It scans all fields and is expensive.

The one exception: when the user provides a specific string and you don't know which field contains it:

cx spans "wildfind 'connection refused'"

Tip: wildfind can also serve as a last-resort field discovery method - when cx search-fields doesn't find what you need, run wildfind with a known value, then inspect the matching spans to see which fields contain it.

---

Field Discovery

Skip discovery when:

• The query only uses standard fields ($m.duration, $l.serviceName, $l.operationName, $d.traceID)
• The user explicitly names the fields they want
• The fields have already been discovered earlier in the conversation

1. Infer from Source Code (Preferred)

If you have access to the application's source code, examine OpenTelemetry instrumentation, span attribute definitions, and tracing middleware to identify field names directly.

2. Semantic Search

cx search-fields "customer identifier" --dataset spans
cx search-fields "order ID" --dataset spans
cx search-fields "http response code" --dataset spans

Note: cx search-fields only has access to the most common fields. If it doesn't find what you need, fall back to sample query inspection.

3. Sample Query Inspection

cx spans "filter \$l.serviceName == 'api'" --limit 5 -o json

Inspect the JSON output to see all available fields. Especially useful for discovering fields in unstructured or deeply nested data.

---

Investigation Workflow

1. Understand the Request

Identify:

• Whether you have a trace ID, service name, or error description
• Time frame of interest
• Whether the question is about latency, errors, or request flow

2. Start with Known Information

If you have a trace ID - go straight to it:

cx spans "filter \$d.traceID == '<trace_id>'"

If you have a service name - query its spans:

cx spans "filter \$l.serviceName == '<service>'" --limit 50

If you have neither - start broad to find entry points:

# Find recent error spans
cx spans "filter \$d.tags.error == true" --limit 20

# Find the slowest spans in the last hour
cx spans "groupby \$l.serviceName, \$l.operationName aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc | limit 10"

# Then extract trace IDs from interesting spans
cx spans "filter \$l.serviceName == '<service>' && \$m.duration > 1000000 | distinct \$d.traceID"

3. Troubleshooting

If a query returns no results, change one thing at a time:

1. Extend the time range: --start now-6h or --start now-24h 2. Relax filters: remove the most restrictive condition 3. Check field availability: the field you're filtering by may only exist in a subset of spans 4. Verify field names: run a sample query with -o json to inspect the actual schema 5. Check service names: service names are case-sensitive 6. Try archive tier: --tier archive --start now-30d for older data

---

Common Query Patterns

Trace Reconstruction

# All spans for a trace
cx spans "filter \$d.traceID == '4f6a8f3c2e8a1b97'"

# Find root spans only (no parent)
cx spans "filter \$l.serviceName == 'api-gateway' | filter \$d.parentId == null"

# Find trace IDs for a service
cx spans "filter \$l.serviceName == 'payment-service' | distinct \$d.traceID"

Latency Analysis

# Spans slower than 1 second
cx spans "filter \$m.duration > 1000000"

# Top 10 slowest operations by average duration
cx spans "groupby \$l.operationName aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc | limit 10"

# Average latency by service
cx spans "groupby \$l.serviceName aggregate avg(\$m.duration) as avg_latency"

# P95 latency by operation
cx spans "groupby \$l.operationName aggregate percentile(0.95, \$m.duration) as p95_latency"

Latency Spike Detection

# Average latency per 15-minute window
cx spans "filter \$l.serviceName == 'api' | groupby roundTime(\$m.timestamp, 15m) as interval aggregate avg(\$m.duration) as avg_latency | orderby interval"

# Find the time windows with highest latency
cx spans "filter \$l.serviceName == 'api' | groupby roundTime(\$m.timestamp, 5m) as interval aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc | limit 10"

Error Investigation

# All error spans
cx spans "filter \$d.tags.error == true"

# Error spans for a specific service
cx spans "filter \$l.serviceName == 'checkout' | filter \$d.tags.error == true"

# Error rate by service
cx spans "filter \$d.tags.error == true | groupby \$l.serviceName aggregate count() as errors | orderby errors desc"

# Error rate over time
cx spans "filter \$d.tags.error == true | groupby roundTime(\$m.timestamp, 15m) as interval aggregate count() as errors"

Sampling Error Types

# Group errors by operation with a sample
cx spans "filter \$d.tags.error == true | groupby \$l.operationName aggregate any_value(\$d) as sample, count() as total | orderby total desc | limit 5"

# Group by service and operation to see where errors concentrate
cx spans "filter \$d.tags.error == true | groupby \$l.serviceName, \$l.operationName aggregate count() as errors | orderby errors desc | limit 10"

Finding Unique Values

# List all services with spans
cx spans "distinct \$l.serviceName"

# List all operations for a service
cx spans "filter \$l.serviceName == 'api' | distinct \$l.operationName"

# Find unique trace IDs for error spans
cx spans "filter \$d.tags.error == true | distinct \$d.traceID"

Correlating by Trace ID

# Find spans across services for the same trace
cx spans "filter \$d.traceID == 'abc123' | groupby \$l.serviceName aggregate count() as span_count, avg(\$m.duration) as avg_latency"

---

Performance Tips

• Use --limit for exploratory queries
• Use groupby with aggregations instead of fetching raw spans when possible
• Filter by time first when dealing with large datasets
• Use specific filters (service name, operation) to reduce scan scope
• Don't rely solely on aggregations - retrieve sample spans to find information you didn't anticipate
• For large result sets, use --output agents which spills automatically:

cx spans "filter \$l.serviceName == 'api'" --start now-24h --limit 1000 -o agents

Phase 5: Live-verify every query through the cx CLI

Every PromQL and DataPrime query in the draft dashboard must successfully run through cx before Phase 7 ships it. This is where invented metric names, typoed field paths, and malformed DataPrime pipelines get caught.

---

1. Resolve the dashboard time range (PromQL only)

Parse relativeTimeFrame from the draft (default "172800s" = 48h) into a human token and call it $RANGE:

$RANGE is used only for PromQL verification (§2): range vectors are window-sensitive, so the CLI check has to match the window the dashboard will evaluate. DataPrime verification (§3) uses a fixed short window instead — see that section.

---

2. Verify each PromQL query

For every widget whose definition contains a promqlQuery, substitute ${__range} in the expression with [$RANGE] (e.g. [48h]). Leave any other fixed window ([5m], [1h]) untouched - those were placed intentionally for sliding-rate panels.

Instant-style widgets (gauge / pieChart / dataTable with promqlQueryType: PROM_QL_QUERY_TYPE_INSTANT):

cx metrics query '<expression-with-[$RANGE]-substituted>' -o agents

Time-series widgets (lineChart):

cx metrics query-range '<expression>' --start now-$RANGE --end now --step <auto> -o agents

Pick <step> proportional to $RANGE: 1m for 1–6h, 5m for 24h, 1h for 7d+. Match any window used by a *_over_time / rate / increase inside the expression if it's narrower than $RANGE.

A query passes when the CLI returns a 200 response and either has data or an empty-but-well-formed result. Fails include unknown metric names, parse errors, non-200 responses, or cx error output.

On failure: consult the cx-metrics-query skill for PromQL help, re-search for the real metric name with cx metrics search, re-list labels with cx metrics get-labels, and fix the query in the draft JSON. Budget ≤5 retry attempts per query.

---

3. Verify each DataPrime query

For every widget whose definition contains a dataprimeQuery, pick the CLI command from the widget's source prefix and strip the leading `source logs` / `source spans` before handing the pipeline to cx:

The dashboard runtime requires the source … prefix inside the widget JSON (see query-syntax.md §3). cx logs and cx spans inject the source themselves; if you leave a leading source … in the pipeline they silently run against the command's own source, which masks pillar mismatches. Strip it for verification; restore nothing - the widget JSON keeps the prefix.

Verify against a fixed short window (now-15m → now), not the dashboard's $RANGE. The goal here is syntax / field / pipeline validation — proving the query parses and references real fields. The dashboard runs against ${__range} itself at render time; we don't need to re-prove data presence on the dashboard's window during the build. A short window is faster, cheaper, and a clean fail signal (a query that fails on now-15m is broken regardless of range).

Choose the tier to verify:

• --tier frequent (default): hot storage, fast, recent data.
• --tier archive: cold/long-term storage, older data.

Use Frequent Search unless you have a reason to validate against Archive. Switch to Archive when:

• The dashboard is intended for long lookbacks (weekly/monthly trends, retrospectives).
• Frequent Search returns empty for known-good queries because the time range is beyond hot retention.
• The user explicitly says “this dashboard should work on archived data.”

Log-backed widgets:

cx logs '<pipeline-without-leading-source-logs>' --start now-15m --end now --limit 1 --tier <frequent|archive>

Span-backed widgets:

cx spans '<pipeline-without-leading-source-spans>' --start now-15m --end now --limit 1 --tier <frequent|archive>

Check both the exit code and the output — some errors surface only in the output. A query passes when cx exits 0 and the output is rows or [] with no error or warning lines (an empty result on a low-volume signal is fine). It hard-fails on a non-zero exit, an error from profile '...': API request failed line, or a Compilation errors: block — the query is broken. A keypath does not exist warning is a soft fail: the query parsed but no record in the window had the referenced field. Verify with cx search-fields "<hint>" --dataset logs|spans; if the field is real the query is fine (widen the window or accept the empty result), if it isn't, fix the field name. On failure: consult the cx-dataprime skill (cx dataprime show <command> for inline help), re-discover fields with cx search-fields, fix, retry. Budget ≤5 retry attempts per query.

---

4. Restore ${__range} before Phase 6

Once every PromQL and DataPrime query passes, restore ${__range} (and any other variables) in the emitted JSON. PromQL verification swapped ${__range} for the concrete [$RANGE]; the final JSON keeps the injected variable intact. (DataPrime queries don't carry ${__range}, so nothing to restore there beyond keeping the source logs / source spans prefix in the widget JSON.)

If any query can't be made to pass within the retry budget, surface it to the user with the CLI error verbatim - don't silently ship a broken widget.

Coralogix Dashboard JSON - Widget Templates

Copy these templates when generating a dashboard. Always replace every UUID (id.value, query id) with a freshly generated UUID, and adapt queries to the target service.

For query-language rules:

• Dashboard-specific gotchas (${__range}, promqlQueryType, widget filters): `query-syntax.md`.
• Full DataPrime syntax: cx-dataprime skill → skills/cx-dataprime/references/dataprime-reference.md.
• Full PromQL reference: cx-metrics-query skill → skills/cx-metrics-query/references/promql-guidelines.md.

A "stat" widget in Coralogix docs is actually emitted as gauge in the JSON. There is no separate stat type.

Every query below must pass live verification via cx metrics query / cx logs / cx spans before deploy - see SKILL.md Phase 5.

---

Top-level skeleton

{
  "id": "<21-char-nanoid>",
  "name": "<Service> - <Purpose>",
  "layout": {
    "sections": [
      { "id": {"value": "<uuid>"}, "rows": [ ... ], "options": { "custom": { "name": "Section name", "collapsed": false, "color": {"predefined": "SECTION_PREDEFINED_COLOR_UNSPECIFIED"} } } }
    ]
  },
  "variables": [],
  "variablesV2": [],
  "filters": [ /* see "Top-level filters" below */ ],
  "relativeTimeFrame": "172800s",
  "annotations": [],
  "off": {},
  "actions": []
}

A row always looks like:

{
  "id": {"value": "<uuid>"},
  "appearance": {"height": 19},
  "widgets": [ /* 1–2 widgets per row */ ]
}

---

Widget: gauge (also used for "stat"/total)

Use for headline numbers: counts, percentages, success rates.

Gauge min and max must be numeric and min < max. This API constraint applies even when thresholdType is THRESHOLD_TYPE_ABSOLUTE; use a real range such as 0..100 for percentages, a known capacity/limit when available, or a max above the highest threshold.

{
  "id": {"value": "<uuid>"},
  "title": "Success Rate",
  "definition": {
    "gauge": {
      "query": {
        "metrics": {
          "promqlQuery": {
            "value": "100 * sum(increase(foo_success_total[${__range}])) / clamp_min(sum(increase(foo_success_total[${__range}])) + sum(increase(foo_failure_total[${__range}])), 1)"
          },
          "aggregation": "AGGREGATION_UNSPECIFIED",
          "filters": [],
          "editorMode": "METRICS_QUERY_EDITOR_MODE_TEXT",
          "promqlQueryType": "PROM_QL_QUERY_TYPE_INSTANT"
        }
      },
      "min": 0,
      "max": 100,
      "showInnerArc": true,
      "showOuterArc": true,
      "unit": "UNIT_PERCENT",
      "thresholds": [
        {"from": 0,  "color": "var(--c-visualization-red-05)"},
        {"from": 80, "color": "var(--c-visualization-yellow-05)"},
        {"from": 95, "color": "var(--c-visualization-green-05)"}
      ],
      "dataModeType": "DATA_MODE_TYPE_HIGH_UNSPECIFIED",
      "thresholdBy": "THRESHOLD_BY_UNSPECIFIED",
      "decimal": 2,
      "thresholdType": "THRESHOLD_TYPE_ABSOLUTE",
      "legend": {"isVisible": true, "columns": [], "groupByQuery": true, "placement": "LEGEND_PLACEMENT_AUTO"},
      "legendBy": "LEGEND_BY_GROUPS",
      "displaySeriesName": false,
      "decimalPrecision": false
    }
  }
}

For "count of bad things" (errors, DLQ): use unit: "UNIT_NUMBER", green at low values, red at high:

"thresholds": [
  {"from": 0,  "color": "var(--c-visualization-green-05)"},
  {"from": 1,  "color": "var(--c-visualization-yellow-05)"},
  {"from": 10, "color": "var(--c-visualization-red-05)"}
],
"thresholdType": "THRESHOLD_TYPE_ABSOLUTE"

For DataPrime-driven count (e.g. error log count): swap metrics.promqlQuery for dataprime.dataprimeQuery:

"query": {
  "dataprime": {
    "dataprimeQuery": {"text": "source logs | filter $m.severity == ERROR || $m.severity == CRITICAL | agg count()"},
    "filters": []
  }
}

---

Widget: pieChart

Use for small-cardinality breakdowns (≤8 slices).

{
  "id": {"value": "<uuid>"},
  "title": "Messages Per Env",
  "definition": {
    "pieChart": {
      "query": {
        "metrics": {
          "promqlQuery": {"value": "sum by (subsystem_name) (increase(foo_total[${__range}]))"},
          "filters": [],
          "groupNames": ["subsystem_name"],
          "editorMode": "METRICS_QUERY_EDITOR_MODE_TEXT",
          "promqlQueryType": "PROM_QL_QUERY_TYPE_INSTANT",
          "aggregation": "AGGREGATION_UNSPECIFIED"
        }
      },
      "maxSlicesPerChart": 8,
      "minSlicePercentage": 1,
      "stackDefinition": {"maxSlicesPerStack": 4},
      "labelDefinition": {"labelSource": "LABEL_SOURCE_INNER", "isVisible": true, "showName": true, "showValue": true, "showPercentage": true},
      "showLegend": true,
      "unit": "UNIT_UNSPECIFIED",
      "colorScheme": "classic",
      "dataModeType": "DATA_MODE_TYPE_HIGH_UNSPECIFIED",
      "decimal": 2,
      "legend": {"isVisible": true, "columns": [], "groupByQuery": true, "placement": "LEGEND_PLACEMENT_AUTO"},
      "hashColors": false,
      "decimalPrecision": false,
      "showTotal": false
    }
  }
}

---

Widget: lineChart

Use for anything over time (rates, latencies, counts per bucket).

{
  "id": {"value": "<uuid>"},
  "title": "Latency P95 by Stage",
  "definition": {
    "lineChart": {
      "legend": {"isVisible": true, "columns": [], "groupByQuery": true, "placement": "LEGEND_PLACEMENT_AUTO"},
      "tooltip": {"showLabels": false, "type": "TOOLTIP_TYPE_ALL"},
      "queryDefinitions": [
        {
          "id": "<uuid>",
          "query": {
            "metrics": {
              "promqlQuery": {"value": "histogram_quantile(0.95, sum by (le, stage) (rate(foo_latency_bucket[${__range}])))"},
              "filters": [],
              "editorMode": "METRICS_QUERY_EDITOR_MODE_TEXT",
              "seriesLimitType": "METRICS_SERIES_LIMIT_TYPE_BY_SERIES_COUNT"
            }
          },
          "seriesCountLimit": "20",
          "unit": "UNIT_SECONDS",
          "scaleType": "SCALE_TYPE_LINEAR",
          "isVisible": true,
          "colorScheme": "classic",
          "resolution": {"bucketsPresented": 96},
          "dataModeType": "DATA_MODE_TYPE_HIGH_UNSPECIFIED",
          "decimal": 2,
          "hashColors": false,
          "decimalPrecision": false,
          "intervalResolution": {"auto": {"minimumInterval": "15s", "maximumDataPoints": 96}}
        }
      ],
      "stackedLine": "STACKED_LINE_UNSPECIFIED",
      "connectNulls": false
    }
  }
}

• For counts (not latency) use "unit": "UNIT_UNSPECIFIED".
• Multiple lines in the same panel: add more objects to queryDefinitions (each with its own id).

---

Widget: dataTable

Use for top-N tables and raw log listings.

Metrics-backed table (e.g. top accounts by count):

{
  "id": {"value": "<uuid>"},
  "title": "Top Accounts by Message Count",
  "definition": {
    "dataTable": {
      "query": {
        "metrics": {
          "promqlQuery": {"value": "sum by (account_id) (increase(foo_total[${__range}]))"},
          "filters": [],
          "editorMode": "METRICS_QUERY_EDITOR_MODE_TEXT",
          "promqlQueryType": "PROM_QL_QUERY_TYPE_INSTANT"
        }
      },
      "resultsPerPage": 10,
      "rowStyle": "ROW_STYLE_UNSPECIFIED",
      "columns": [
        {"field": "account_id"},
        {"field": "#value"}
      ],
      "dataModeType": "DATA_MODE_TYPE_HIGH_UNSPECIFIED"
    }
  }
}

DataPrime-backed table (e.g. last errors):

{
  "id": {"value": "<uuid>"},
  "title": "Last errors",
  "definition": {
    "dataTable": {
      "query": {
        "dataprime": {
          "dataprimeQuery": {"text": "source logs | filter $m.severity == ERROR || $m.severity == CRITICAL | orderby $m.timestamp desc"},
          "filters": []
        }
      },
      "resultsPerPage": 100,
      "rowStyle": "ROW_STYLE_ONE_LINE",
      "columns": [
        {"field": "$m.severity",  "width": 121},
        {"field": "$m.timestamp", "width": 190},
        {"field": "$d",           "width": 600}
      ],
      "dataModeType": "DATA_MODE_TYPE_HIGH_UNSPECIFIED"
    }
  }
}

---

Top-level filters

Add one entry per slicing dimension the user approved. Users fill in values at view time.

Exclude non-prod environments (replace the example values with whatever the target deployment actually uses, e.g. ["dev", "staging", "test"]):

{
  "source": {
    "metrics": {
      "label": "subsystem_name",
      "operator": {"notEquals": {"selection": {"list": {"values": ["<non-prod-env-1>", "<non-prod-env-2>"]}}}}
    }
  },
  "enabled": true,
  "collapsed": false,
  "id": {"value": "<uuid>"}
}

User-fillable slicing filter:

{
  "source": {
    "metrics": {
      "label": "account_id",
      "operator": {"equals": {"selection": {"list": {"values": []}}}}
    }
  },
  "enabled": true,
  "collapsed": false,
  "id": {"value": "<uuid>"}
}

---

Section template

{
  "id": {"value": "<uuid>"},
  "rows": [ /* rows go here */ ],
  "options": {
    "custom": {
      "name": "Errors",
      "collapsed": true,
      "color": {"predefined": "SECTION_PREDEFINED_COLOR_UNSPECIFIED"}
    }
  }
}

Set collapsed: true for logs/debug sections and any section that isn't the dashboard's primary purpose.

---

Tier (dataModeType) quick-reference

dataModeType lives per widget under the widget's definition (e.g. definition.gauge.dataModeType, definition.dataTable.dataModeType, definition.lineChart.queryDefinitions[].dataModeType). It controls which storage tier the widget reads from at render time.

The templates above ship with DATA_MODE_TYPE_HIGH_UNSPECIFIED. When the user wants archive, replace DATA_MODE_TYPE_HIGH_UNSPECIFIED → DATA_MODE_TYPE_ARCHIVE on every widget before deploy (not just the dashboard root).

---

Unit enum quick-reference

---

Threshold type

• THRESHOLD_TYPE_ABSOLUTE - thresholds compared against the raw value. Use for success rates and fixed-meaning counts.
• THRESHOLD_TYPE_RELATIVE - thresholds as % of min/max. Use when the scale is arbitrary.

Default to ABSOLUTE for rates and DLQ counts; RELATIVE only when the absolute scale is unknown.