Cx Telemetry Querying

Telemetry Querying Skill

Use this skill as the entry point for any investigation, debugging, or data question that may be answered from telemetry data. It helps you decide where the relevant signal lives (metrics, logs, traces, RUM) and tells you which reference files to load before querying.

Loading References

Before querying, load the reference files for the chosen pillar:

---

Safety

All query commands (cx logs, cx spans, cx metrics, cx dataprime, cx search-fields) are read-only and work in --read-only mode. They never modify data and can be run freely without --yes.

---

Quick Routing Guide

Use this table for obvious cases where one pillar is the clear first choice:

For ambiguous questions (e.g., "How much money did users spend last week?"), the signal could live in any pillar. Follow the Discovery Workflow below.

---

Discovery Workflow

When the answer could reside in multiple pillars, run discovery in parallel to find the best source.

Step 1: Search Metrics

Check if a relevant metric exists:

cx metrics search --name '*transaction*'
cx metrics search --name '*payment*'
cx metrics search --name '*revenue*'
cx metrics search --description "total purchase amount"

If a matching metric is found, load references/promql-guidelines.md + references/metrics-querying.md and continue.

Step 2: Search Log and Span Fields

Use semantic field search to find relevant DataPrime paths:

cx search-fields "transaction amount" --dataset logs
cx search-fields "payment total" --dataset spans
cx search-fields "purchase value" --dataset logs --limit 10

If you know a concrete value that should appear in the data but don't know which field holds it, use value search instead. It returns the matching field keys alongside sample values, which also lets you infer the field's type (string, numeric, enum, etc.):

cx search-fields "payment_failed" -s value --dataset logs
cx search-fields "grpc.status.UNAVAILABLE" -s value --dataset spans
cx search-fields "eu-west-1" -s value --dataset all

Requirements: cx search-fields needs a Coralogix API key or OAuth on the active profile. If credentials are missing, prompt the user to run cx profiles add <name>.

If matching fields are found:

• For logs: load references/dataprime-reference.md + references/logs-querying.md
• For spans: load references/dataprime-reference.md + references/spans-querying.md

Step 3: Search the Codebase

When discovery results are ambiguous or you need to validate what a metric/field actually represents, search the codebase:

• Look for metric registration code (e.g., prometheus.NewCounter, metrics.record)
• Look for log statements that emit the field (e.g., logger.info("transaction", ...))
• Look for span attributes (e.g., span.setAttribute("purchase.amount", ...))

This confirms the semantic meaning and helps you choose the right pillar.

Step 4: Choose and Query

Based on discovery results, pick the pillar with the clearest signal, load its reference files (see Loading References), then query.

---

Fallback and Pivoting

If your initial route yields no results, pivot to another pillar.

Example pivot paths:

• Metrics empty → try traces (per-request data) or logs (event records)
• Logs empty → try traces (structured span attributes) or metrics (aggregated counters)
• Traces empty → try logs (text-based debug output)

Do not stop after one failed attempt. Try at least two pillars before concluding the data does not exist.

---

CLI Commands Reference

---

Examples

Example 1: Business Question (Ambiguous Source)

Question: "How much money did people spend on the platform last week?"

Approach: 1. Search metrics: cx metrics search --name '*revenue*' and cx metrics search --name '*transaction*' 2. Search log fields: cx search-fields "transaction amount" --dataset logs 3. Search span fields: cx search-fields "payment total" --dataset spans 4. If a metric like payment_total_usd exists, load metrics references and run a range query 5. If only logs have the data, load logs references and use DataPrime aggregation 6. If traces have purchase.amount attribute, load spans references

Example 2: Latency Question (Clear First Choice)

Question: "What's the average latency of the checkout route?"

Approach: 1. First try metrics: cx metrics search --name '*checkout*latency*' or cx metrics search --name '*http*duration*' 2. If a histogram metric exists, load metrics references and use histogram_quantile 3. If no metric, fall back to traces: load spans references and aggregate span durations

Example 3: Frontend Performance (RUM)

Question: "Why is the dashboard page loading slowly for users?"

Approach: 1. This is clearly a RUM question - load references/rum-querying.md + references/rum-fields.md + references/logs-querying.md + references/dataprime-reference.md 2. Query web vitals and page load times 3. If RUM shows backend calls are slow, pivot to spans references for the API calls

Example 4: Error Investigation (Logs + Traces)

Question: "Why are users getting 500 errors on the payment endpoint?"

Approach: 1. Check error rate metrics → load metrics references 2. Search for error logs → load logs references 3. Get traces for failed requests → load spans references 4. Cross-reference: find trace IDs in logs, then fetch full traces for root cause

---

Beyond Investigation

Not every question is answered by querying data. If the user's intent is operational rather than investigative, route to the appropriate workflow skill:

---

Key Principles

• Load references before querying: check the Loading References table first
• Discover before querying: always run search/discovery to find the right source
• Parallel discovery: for ambiguous questions, search metrics, logs, and spans concurrently
• Validate with code: when unsure what a metric or field represents, check the codebase
• Pivot on failure: if one pillar is empty, try another before giving up

DataPrime Query Language Reference

Query Structure

A DataPrime query is a pipeline of commands separated by |. Each command transforms the output of the previous one:

filter $m.severity == ERROR | groupby $l.subsystemname aggregate count() as errors

Source Handling

Every query targets a source (logs, spans, etc.). The source is set by whichever cx command you use. A full query with an explicit source looks like:

source <logs|spans> | filter ... | groupby ...

When running via a source-specific command (e.g. cx logs, cx spans), the source is injected automatically - omit it from the query. When running via cx dataprime query, use the --source flag or include source in the query itself.

The examples below focus on the DataPrime query language and omit the source and CLI command prefix.

Comments

Comments are supported with # or //:

filter $m.severity == ERROR  # only errors
| limit 10                   // cap results

Data Prefixes

All fields are accessed through three namespaces:

$d is the default prefix and can sometimes be omitted, but being explicit avoids ambiguity.

Data Types

Commands

Filtering and Selection

Note on `wildfind`: It is a standalone command, not a condition within filter. You cannot combine it with other filter expressions - use it as its own pipeline stage.

Aggregation

Transformation

Operators

Type Conversions

Cast fields inline with :type:

filter $d.http_error_code:number == 500

Supported types: bool, number, string, timestamp, interval, array, object

Field Access

# Chained field names (dot notation)
filter $d.tags.user_context.email == 'test@example.com'

# Special characters require brackets
filter $d.http['status/code'] == 500

Aggregation Functions

Example - full CLI invocation:

cx dataprime query --source logs 'groupby $l.subsystemname aggregate count() as error_count, avg($d.response_time) as avg_response | orderby error_count desc'

Utility Functions

firstNonNull - Field Coalescing

Return the first non-null value from a list of fields. Useful when the same data may appear in different fields across log sources:

# Merge fields
create message from firstNonNull($d.error_message, $d.msg, $d.body)

# Use inside groupby
groupby firstNonNull($d.error_message, $d.msg) as message aggregate count() as n

Template Sampling

Find top error patterns with a sample message for each:

filter $m.severity == ERROR | groupby $m.templateid aggregate any_value($d) as sample, count() as total | orderby total desc | limit 5

Time-Based Grouping

Use roundTime() to bucket timestamps:

# Group by hour
groupby roundTime($m.timestamp, 1h) as hour aggregate count() as count

# Error rate over 15-minute intervals
filter $m.severity == ERROR | groupby roundTime($m.timestamp, 15m) as interval aggregate count() as errors

Multi-Value Matching

Use arrayContains to match against a set of values:

# Match multiple subsystems
filter ['api', 'web', 'worker'].arrayContains($l.subsystemname)

# Match multiple severity levels
filter [ERROR, CRITICAL].arrayContains($m.severity)

Text Extraction

Regex Extraction

# Extract with unnamed capture group
extract $d.email into domain using regexp(e=/@(.*)/) | distinct $d.domain._0

# Named capture groups
extract $d.email into extracted using regexp(e=/(?<username>[a-zA-Z0-9._%+-]+)@(?<domain>.*)/) | choose $d.extracted.username, $d.extracted.domain

JSON String Parsing

# Parse a JSON string field into an object for further querying
extract $d.json_payload into parsed using jsonobject() | filter $d.parsed.status == 'failed'

Deduplication

# Remove duplicates by log template
dedupeby $m.templateid

# Dedupe by a custom field
dedupeby $d.request_id

Built-In Documentation

For the full list of commands and functions with detailed syntax:

cx dataprime list                              # List all commands and functions
cx dataprime list --filter commands             # Commands only
cx dataprime list --filter functions --name time # Search functions by name
cx dataprime show filter                        # Detailed help for a specific command
cx dataprime show groupby

Validating a DataPrime query

A query that looks right can still fail on a typoed field path, an invented function, or a malformed pipeline stage. Validate before trusting the output — a short-window run through the CLI is cheap and catches almost all of these:

cx logs  '<pipeline>' --start now-15m --end now --limit 1
cx spans '<pipeline>' --start now-15m --end now --limit 1

now-15m is a good default; widen it only if 15 minutes is unlikely to exercise the pipeline. Per "Source Handling" above, omit any leading source logs / source spans — cx logs and cx spans inject the source themselves.

Check both the exit code and the output — some errors surface only in the output.

Pass = exit 0 and the output is rows or [] with no error or warning lines.

Hard fail — query is broken, fix it:

• non-zero exit
• error from profile '...': API request failed — HTTP error from the API
• Compilation errors: — parse error, unknown function, malformed expression

Soft fail (needs investigation):

• keypath does not exist — the query parsed, but no record in the window had the referenced field. This is ambiguous: the field name might be a typo, or it might be real but absent from records in this 15-minute slice. Confirm with cx search-fields "<field hint>" --dataset logs (or --dataset spans). If the field is real, the query is fine — try a wider window or accept the empty result. If it isn't, fix the field name.

On fail: re-discover fields with cx search-fields, look up command syntax with cx dataprime show <command>, fix, re-run.

Log Querying Reference

Query and analyze Coralogix logs using the cx logs command with DataPrime syntax.

DataPrime syntax: See dataprime-reference.md for the full query language reference.

Understanding Logs in Coralogix

Logs in Coralogix are largely unstructured. Every log entry has a small structured envelope - metadata and labels - but the actual application payload (userData) is free-form and varies entirely by application. There is no universal schema for $d.* fields.

This means:

• *Metadata (`$m.)** and **labels ($l.`)* are predictable - you can always filter on severity, timestamp, application name, and subsystem name without discovery.
• *User data (`$d.)** is not predictable - field names, nesting, and types depend on whatever the application chose to log. Always verify $d` fields before assuming they exist.

---

CLI Command

cx logs '<dataprime_query>'

The source logs prefix is automatically injected if the query doesn't already include a source command.

Options

---

Log Data Model

Standard Fields (Always Available)

Severity Values

Severity keywords are used without quotes in DataPrime:

DEBUG | INFO | WARNING | ERROR | CRITICAL

cx logs 'filter $m.severity == ERROR'
cx logs 'filter [ERROR, CRITICAL].arrayContains($m.severity)'

---

Essential Query Examples

# Filter by severity
cx logs 'filter $m.severity == ERROR'

# Text search in a known field
cx logs "filter \$d.message ~ 'timeout'"

# Filter by application and subsystem
cx logs "filter \$l.applicationname == 'api' && \$l.subsystemname == 'auth'"

# Aggregate errors by subsystem
cx logs 'filter $m.severity == ERROR | groupby $l.subsystemname aggregate count() as errors | orderby errors desc'

# Wider time range and archive tier
cx logs "filter \$l.subsystemname == 'payments'" --tier archive --start now-7d

Wildfind Policy

Avoid `wildfind` by default. It scans all fields and returns noisy results, especially for generic terms.

The one exception: when the user provides a specific, quoted error message or log string and you don't know which field contains it:

# User says: "Find logs with 'connection refused'"
cx logs "wildfind 'connection refused'"

In all other cases, use filter with known fields ($m.severity, $l.subsystemname, $d.<field>) or discover field names first with cx search-fields.

---

Field Discovery

Skip discovery when:

• The query only uses standard fields ($m.severity, $m.timestamp, $l.applicationname, $l.subsystemname)
• The user explicitly names the fields they want (e.g., "filter by $d.customer_id")
• You're searching for a specific error message - use wildfind directly
• The fields have already been discovered earlier in the conversation

For customer-specific $d.* fields that need discovery, use one of these approaches:

1. Infer from Source Code (Preferred)

If you have access to the application's source code, examine logger calls, structured logging configs, and log format templates to identify field names directly.

2. Semantic Search

cx search-fields "customer identifier" --dataset logs
cx search-fields "http response code" --dataset logs

Returns DataPrime paths with similarity scores:

+------------------------+-----------------------------------+-----------+
| DataPrime path         | Description                       | Similarity|
+------------------------+-----------------------------------+-----------+
| $d.customer_id         | Unique customer identifier        | 0.89      |
| $d.user.account_id     | Customer account reference        | 0.85      |
+------------------------+-----------------------------------+-----------+

3. Sample Query Inspection

cx logs "filter \$l.subsystemname == 'api'" --limit 5 -o json

Inspect the JSON output to see all available fields in the actual data.

---

Investigation Workflow

1. Understand the Request

Identify:

• What type of logs are needed (errors, info, specific events)
• Time frame of interest
• Key entities (services, users, transactions)

2. Start with Standard Fields

For basic queries, use standard fields directly:

# Recent errors - no discovery needed
cx logs 'filter $m.severity == ERROR | limit 20'

# Errors in a specific subsystem
cx logs "filter \$m.severity == ERROR && \$l.subsystemname == 'payment-service'"

3. Build and Execute Query

Start simple, add complexity:

# Step 1: Check if data exists
cx logs "filter \$l.subsystemname == 'checkout'" --limit 10

# Step 2: Add filters
cx logs "filter \$l.subsystemname == 'checkout' && \$m.severity == ERROR"

# Step 3: Add aggregation
cx logs "filter \$l.subsystemname == 'checkout' && \$m.severity == ERROR | groupby \$d.error_type aggregate count() as occurrences"

4. Troubleshooting

If a query returns no results, change one thing at a time:

1. Extend the time range: --start now-6h or --start now-24h 2. Relax filters: remove the most restrictive condition 3. Verify field names: run a sample query with -o json to inspect the actual schema 4. Try archive tier: --tier archive --start now-30d for older data

---

Common Query Patterns

Error Investigation

# All errors in last hour
cx logs 'filter $m.severity == ERROR'

# Critical errors only
cx logs 'filter $m.severity == CRITICAL'

# Errors with text search
cx logs "filter \$m.severity == ERROR && \$d.message ~ 'database connection'"

Aggregation by Service

# Error count by subsystem
cx logs 'filter $m.severity == ERROR | groupby $l.subsystemname aggregate count() as errors | orderby errors desc'

# Error count by application and subsystem
cx logs 'filter $m.severity == ERROR | groupby $l.applicationname, $l.subsystemname aggregate count() as errors'

Time-Based Analysis

# Errors per hour
cx logs 'filter $m.severity == ERROR | groupby roundTime($m.timestamp, 1h) as hour aggregate count() as count'

# Find error spikes in 5-minute windows
cx logs 'filter $m.severity == ERROR | groupby roundTime($m.timestamp, 5m) as interval aggregate count() as count | orderby count desc | limit 10'

Finding Unique Values

# List all subsystems with errors
cx logs 'filter $m.severity == ERROR | distinct $l.subsystemname'

# List unique error types
cx logs 'filter $m.severity == ERROR | distinct $d.error_type'

Fetching Sample Logs by Template

Find top error patterns with sample messages:

cx logs 'filter $m.severity == ERROR | groupby $m.templateid aggregate any_value($d) as sample, count() as total | orderby total desc | limit 5'

---

Performance Tips

• Use --limit for exploratory queries
• Use groupby with aggregations instead of fetching all raw logs
• Filter by time first when dealing with large datasets
• Use specific filters (application, subsystem) to reduce scan scope
• For large result sets, use --output agents which spills to a temp file automatically:

cx logs 'filter $m.severity == ERROR' --start now-24h --limit 1000 -o agents

Metrics Querying Reference

Query and analyze Coralogix metrics using the cx metrics CLI commands with PromQL.

PromQL syntax: See promql-guidelines.md for the full query language reference.

CLI Commands

All metrics operations use cx metrics with four subcommands:

Output format: append -o json or -o agents to any command for machine-readable output.

Search Examples

# Exact substring match
cx metrics search --name http_requests

# Wildcard: find all CPU metrics
cx metrics search --name '*cpu*'

# List all metrics
cx metrics search --name '*'

Instant Query Examples

# Current state
cx metrics query 'up'

# At a specific time
cx metrics query 'rate(http_requests_total[5m])' --time 2024-01-01T12:00:00Z

# With output for further processing
cx metrics query 'sum by (service) (rate(http_errors_total[5m]))' -o agents

Range Query Examples

# Last hour, default step (1m)
cx metrics query-range 'rate(http_requests_total[5m])'

# Custom window and step
cx metrics query-range 'sum by (service) (rate(http_requests_total[5m]))' \
  --start now-6h --end now --step 5m

# Daily aggregation over the last week
cx metrics query-range 'max by () (max_over_time(cpu_usage[1d]))' \
  --start now-7d --end now --step 1d

Label Discovery Example

cx metrics get-labels http_requests_total
# Returns: job, instance, method, route, status_code, ...

Time Syntax

All time arguments accept:

• Relative: now, now-1h, now-30m, now-2d, now-1w
• Absolute: RFC3339/ISO 8601 - 2024-01-01T00:00:00Z

---

Investigation Workflow

1. Initial Assessment

When given a vague problem, ask 1–2 focused clarifying questions before proceeding:

• What exactly is failing or behaving unexpectedly?
• When did it start? What is the affected time window?

Prefer to start investigating immediately if the question is specific enough.

2. Metric Discovery

Always start by searching for relevant metrics before querying:

# Try domain-specific patterns first
cx metrics search --name '*http*'
cx metrics search --name '*error*'
cx metrics search --name '*latency*'
cx metrics search --name '*cpu*'
cx metrics search --name '*memory*'

# If nothing found, broaden the search
cx metrics search --name '*request*'
cx metrics search --name '*'   # full list as last resort

When two similar metrics are found and one is suffixed with _count, prefer the one without the suffix - _count typically tracks the number of observations, not the measured value itself.

3. Label Discovery

Once a relevant metric is identified, discover its labels before filtering:

cx metrics get-labels <metric_name>

Use the returned label names to build precise PromQL filters. Note: label values are not directly queryable via the CLI - infer them from query results or domain knowledge.

4. Query Construction & Execution

Choose the right query type:

• Instant query (cx metrics query) - use for current state, single values, or absolute aggregations over a window. Use --time to query historical data at a specific moment.
• Range query (cx metrics query-range) - use when comparing across time periods (e.g., per-day DAU, hourly error rate trend). Set --step to match any [window] used in temporal functions.

Start simple, add complexity as needed:

# Step 1: Check if metric exists and has data
cx metrics query 'http_requests_total'

# Step 2: Add label filters and aggregation
cx metrics query 'sum by (status) (rate(http_requests_total[5m]))'

# Step 3: Build the final diagnostic query
cx metrics query 'sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))'

5. Retry Logic

If a query returns no results or an error: 1. Check metric name - run cx metrics search --name '*<keyword>*' with a broader term 2. Check label names - run cx metrics get-labels <metric> to verify filter keys 3. Widen the time range or shorten the rate window 4. If filtering on a label that may be empty, exclude empty values: {label!=""} 5. Try an alternative metric name or structure

Maximum 5 retry attempts per query, each with a concrete improvement.

6. Pattern Recognition & Root Cause Analysis

After collecting results:

• Correlate across metrics (e.g., error spike matches CPU spike?)
• Look for temporal patterns - recurring peaks, sudden step changes
• Cross-layer analysis: app → services → infrastructure → dependencies
• Provide actionable next steps, not just data

7. Summarize Frequently

PromQL results can be large. After every few queries, summarize:

• Key findings so far
• Queries already run
• Next planned queries
• Ask to continue if more investigation is needed

---

Common Investigation Patterns

HTTP Errors

1. Check error rate: sum by (service) (rate(http_requests_total{status=~"5.."}[5m])) 2. Compare to total RPS: sum by (service) (rate(http_requests_total[5m])) 3. Check pod/deployment health metrics 4. Check dependency latency

Performance / Latency

1. Check p95/p99 latency via histograms: histogram_quantile(0.95, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))) 2. Check resource saturation: CPU, memory, disk 3. Check autoscaling metrics 4. Check dependency response times

Availability

1. Check up metric across services: cx metrics query 'up' 2. Check pod restart counts 3. Check node health 4. Check service discovery metrics

---

Key Principles

• Discover before querying: always search for metric names first
• Instant over range: prefer instant queries unless the question requires a time series
• Align step with window: when using max_over_time(metric[1d]), set --step 1d
• Filter empty labels: if results have blank label values, add {label!=""} to the filter
• Aggregate early: use sum by (...) to reduce cardinality before further operations

PromQL Guidelines

Core Principles

1. Pick the right query type

• Instant queries (cx metrics query) evaluate an expression at a single timestamp (now, or a given --time). Use when the question requires one number or one vector as of a moment - essentially any query that does not require results over different timeframes.
• Range queries (cx metrics query-range) evaluate the expression repeatedly across [--start, --end] at a given --step. Use for time series over a period (e.g., daily active users per day).
• Note: Range queries evaluate the expression repeatedly at each step. If --step=1d, --start=now-1d, --end=now, and the query is max_over_time(metric[1d]), the query evaluates at now-1d and now - two evaluations covering two days of data.
• Prefer instant queries over range queries for most questions, except when comparing different timeframes.

2. Understand PromQL value types

• Instant vector - set of series with 1 sample each at eval time
• Range vector - series with many samples over a window [t-range, t]
• Scalar - single number
• String - rare
• Functions like *_over_time() require a range vector. Aggregations like sum/max/min/avg ... by(...) consume instant vectors.
• Important: When using *_over_time() functions with range queries, be aware that the query also evaluates at the --start time and includes the window specified in the function.
• Example: If max_over_time(metric[1d]) is used with --start=now-1d, --end=now, --step=1d, the query evaluates at now-1d and now - the result is the max over [now-2d, now]. This is a common mistake. If a user asks "What is the max of x between 2025-01-01 and 2025-01-07?" and max_over_time(x[7d]) is used with --start=2025-01-01, --end=2025-01-07, --step=1d, the evaluation at 2025-01-01 includes [2024-12-25, 2025-01-01] - which is wrong. Use an instant query with --time to avoid this.

3. Separation of concerns

• Use *_over_time() for temporal reductions across a window (e.g., max_over_time, avg_over_time, quantile_over_time).
• Use sum/max/min/avg by (...) for label-set aggregation across series at the eval point.
• Chain them as needed (temporal reduction first, then label aggregation, or vice versa).

4. Counters vs. gauges

• Counters (monotonic, suffixed _total) → use rate()/irate() or increase() over a window.
• Gauges (current value) → use avg_over_time, max_over_time, etc., or plain avg(...) depending on intent.

5. Suffix conventions

• Canonical: _total (counter), _bucket/_sum/_count (histogram), _sum/_count (summary), _created.
• Non-standard: _avg, _mean, etc. Prefer computing averages via PromQL unless the exporter dictates otherwise.

---

CLI Usage

Instant Query

cx metrics query '<expr>'
cx metrics query '<expr>' --time 2024-01-01T12:00:00Z
cx metrics query '<expr>' --output json

Example: absolute max over last 24h (single result)

cx metrics query 'max by () (max_over_time(http_requests_in_flight[24h]))'

Range Query

cx metrics query-range '<expr>' --start now-7d --end now --step 1d

Example: absolute max per day over the last 7 days

cx metrics query-range 'max by () (max_over_time(metric[1d]))' \
  --start now-7d --end now --step 1d

IMPORTANT: Align --step with any window used in temporal reduction functions. If using max_over_time(metric[1d]), set --step 1d.

---

PromQL Fundamentals

Label Matching & Aggregation

• Matchers: {label="v"}, {label!="v"}, {label=~"re.*"}, {label!~"re"}
• Aggregate by labels to keep them; use without to drop them.

sum by (job) (rate(http_requests_total[5m]))
sum without (instance) (up)

Temporal Reductions (range → instant)

max_over_time(cpu_usage[1h])
avg_over_time(node_memory_Active_bytes[30m])
quantile_over_time(0.99, queue_length[1h])

Counters: Rates, Increases, Windows

Per-instance RPS:

rate(http_requests_total[5m])

Total RPS across fleet:

sum by () (rate(http_requests_total[5m]))

Events in last day (per user, then count actives):

count( sum by (user_id) (increase(api_call_count[24h])) > 0 )

Histograms & Summaries

p95 from a histogram:

histogram_quantile(
  0.95,
  sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))
)

Average from summary parts:

sum(rate(req_duration_seconds_sum[5m]))
/
sum(rate(req_duration_seconds_count[5m]))

Max over a Period

Correct - temporal reduction, then aggregation:

max by () (max_over_time(metric[4d]))

Per-label max:

max by (label) (max_over_time(metric[4d]))

Incorrect - max() cannot take a range vector:

max(metric[4d])   ← error

Top-k / Ranking

topk(5, sum by (instance) (rate(http_requests_total[5m])))

---

Common Tasks (ready to adapt)

1. Absolute peak per instance over 7d, then pick the winner

topk(1, max by (instance) (max_over_time(my_metric[7d])))

Run as instant query (no --time needed - defaults to now).

2. Global CPU usage % (avg across cores & hosts)

avg by () (
  rate(process_cpu_seconds_total[5m])
) * 100

3. Error rate (%) per route

100 * sum by (route) (rate(http_requests_total{code=~"5.."}[5m]))
    / sum by (route) (rate(http_requests_total[5m]))

4. Daily active users over a week (time series)

Expression:

count(count by (user_id) (increase(api_call_count[1d]) > 0))

Run as range query with --step 1d --start now-6d --end now. (Starting from 6 days ago because increase looks back one full day from each evaluation point.)

---

Performance & Safety Guidelines

• Prefer short windows for rate() (e.g., 1–5m) unless data is bursty or sparse.
• Avoid unbounded fan-out (e.g., joining massive label sets).
• Keep cardinality under control; aggregate early (sum by (...)) when only totals are needed.
• Use clamp_max/clamp_min to tame outliers when needed.
• For histograms, always aggregate buckets (sum by (le, ...)) before histogram_quantile.
• Be mindful of counter resets; rate()/increase() handle resets automatically.

---

Frequent Gotchas (and fixes)

• "Why am I getting a time series when I only want one number?"

Use cx metrics query (instant) instead of cx metrics query-range.

• "`max(metric[...])` errors."

max() can't take a range vector. Use max_over_time(metric[...]), then aggregate with max by () (...).

• "`_over_time(metric[...]) by (label)` errors."

_over_time aggregations cannot include a by clause. Use max_over_time(metric[...]), then max by (label) (...).

• "Avg looks wrong for counters."

Counters need rate()/increase(), not avg_over_time.

• "p95 from a summary?"

Summaries expose quantiles directly via the quantile label. For histograms, use histogram_quantile on bucket rates.

• "Results show empty label values."

Add {label!=""} to the selector to filter out empty label values. Example: max by (deployment) (rate(cpu_usage{deployment!=""}[5m])).

---

Mini Cheat-Sheet

RUM Field Reference

All fields are under $d.cx_rum.*.

Table of Contents

• Event Context
• Error Context & Grouping
• Session Context
• Version & Environment
• Page Context
• Network Request Context
• Web Vitals Context
• Interaction Context
• Resource Context
• Mobile Contexts
• Other Fields

---

Event Context

event_context.*

Error Context & Grouping

error_context.*

Top-level error grouping fields:

Session Context

session_context.*

Version & Environment

Page Context

page_context.*

Network Request Context

network_request_context.*

Web Vitals Context

web_vitals_context.*

Interaction Context

interaction_context.*

Resource Context

resource_context.*

Mobile Contexts

Device Context (device_context.*): device, device_name, os, osVersion, emulator

Mobile SDK (mobile_sdk.*): framework, sdk_version

View Context (view_context.*): view, view_activity, view_fragment

Mobile Vitals (mobile_vitals_context.*):

• CPU: cpu.cpu_usage, cpu.total_cpu_time, cpu.main_thread_cpu_time
• Memory: memory.memory_utilization, memory.heap_max, memory.heap_used
• Performance: fps, cold, warm, slow_frozen.slow_frames, slow_frozen.frozen_frames, anr

Other Fields

RUM Querying Reference

Query and analyze Coralogix Real User Monitoring data using the cx logs command with DataPrime syntax.

DataPrime syntax: See dataprime-reference.md for the full query language reference.

Log querying basics: See logs-querying.md for field discovery, wildfind policy, and general log query patterns.

Complete RUM field catalog: See rum-fields.md.

Understanding RUM in Coralogix

RUM captures real user interactions from browsers and mobile apps - errors, performance metrics, network requests, web vitals, and user interactions. RUM data is stored as regular logs in the cx_rum subsystem, queried with the same cx logs command and DataPrime syntax used for any other logs.

This means:

• *Metadata (`$m.)** and **labels ($l.`)* work the same as regular logs - you can filter on timestamp, severity, etc.
• *User data (`$d.cx_rum.`) contains all RUM-specific fields - event types, errors, sessions, web vitals, interactions, and more. See rum-fields.md** for the complete field catalog.
• Session replay and session flows are not available - only individual RUM log events can be queried.

---

CLI Command

cx logs '<dataprime_query>'

The source logs prefix is automatically injected if the query doesn't already include a source command.

Options

Note: Use --start now-7d (or wider) for web vitals and page performance queries. Short time ranges produce unreliable percentiles - low-traffic pages have too few data points.

---

RUM Data Model

Identifying RUM Logs

Every RUM query must include $l.subsystemname == 'cx_rum'.

Application filtering in RUM uses dedicated fields - $l.applicationname does not map to the RUM application name:

# RUM application name
cx logs "filter \$l.subsystemname == 'cx_rum' && \$d.cx_rum.version_metadata.app_name == 'my-app'"

# Micro-frontend app label
cx logs "filter \$l.subsystemname == 'cx_rum' && \$d.cx_rum.labels.mfeApp == 'my-app'"

# WRONG - $l.applicationname is not the RUM application name
cx logs "filter \$l.subsystemname == 'cx_rum' && \$l.applicationname == 'my-app'"

Event Types

Filter by $d.cx_rum.event_context.type:

Key Fields

All RUM fields live under $d.cx_rum.*. The most commonly used:

Error Detection

RUM errors can come from multiple event types (error, network-request, custom-log). The universal error marker is event_context.severity == 5, which applies regardless of event type.

The rum_template_id field groups similar error events into distinct issues - always group by it when analyzing errors, and filter out nulls:

cx logs "filter \$l.subsystemname == 'cx_rum' && \$d.cx_rum.event_context.severity:num == 5 && \$d.cx_rum.rum_template_id != null | groupby \$d.cx_rum.rum_template_id aggregate count() as error_count, any_value(\$d.cx_rum.version_metadata.app_name) as app_name, any_value(\$d.cx_rum.event_context.type) as event_type, any_value(\$d.cx_rum.error_context.error_message) as error_message, any_value(\$d.cx_rum.network_request_context.method) as method, any_value(\$d.cx_rum.network_request_context.fragments) as url_fragments, any_value(\$d.cx_rum.network_request_context.status_code) as status_code, any_value(\$d.cx_rum.custom_log_context.message) as custom_log_message, distinct_count(\$d.cx_rum.session_context.user_id) as affected_users | orderby error_count desc" --start now-7d

Include any_value() for descriptive fields from all error types - irrelevant fields will be null. When composing error descriptions from grouped results, the relevant fields depend on the event type:

• error → error_message
• network-request → "<method> <url_fragments> (status <status_code>)"
• custom-log → custom_log_context.message

---

Essential Query Examples

# All RUM errors in the last 7 days
cx logs "filter \$l.subsystemname == 'cx_rum' && \$d.cx_rum.event_context.severity:num == 5" --start now-7d

# Network request errors
cx logs "filter \$l.subsystemname == 'cx_rum' && \$d.cx_rum.event_context.severity:num == 5 && \$d.cx_rum.event_context.type == 'network-request' | groupby \$d.cx_rum.rum_template_id aggregate count() as error_count, any_value(\$d.cx_rum.network_request_context.method) as method, any_value(\$d.cx_rum.network_request_context.fragments) as fragments, any_value(\$d.cx_rum.network_request_context.status_code) as status_code | orderby error_count desc" --start now-7d

# Slow loading pages (LT p75)
cx logs "filter \$l.subsystemname == 'cx_rum' && \$d.cx_rum.event_context.type == 'web-vitals' && \$d.cx_rum.web_vitals_context.name == 'LT' | groupby \$d.cx_rum.page_context.page_fragments aggregate distinct_count(\$d.cx_rum.session_context.user_id:string) as users, percentile(0.75, \$d.cx_rum.web_vitals_context.value) as LT_p75_ms | orderby users desc" --start now-7d

# User interactions on a page
cx logs "filter \$l.subsystemname == 'cx_rum' && \$d.cx_rum.event_context.type == 'user-interaction' && \$d.cx_rum.page_context.page_fragments ~ '/some/page' && \$d.cx_rum.interaction_context.target_element_inner_text != null && \$d.cx_rum.interaction_context.target_element_inner_text != '' | groupby \$d.cx_rum.interaction_context.target_element_inner_text aggregate count() as click_count, distinct_count(\$d.cx_rum.session_context.user_id) as unique_users | orderby click_count desc" --start now-7d

# Affected users per error
cx logs "filter \$l.subsystemname == 'cx_rum' && \$d.cx_rum.event_context.severity:num == 5 && \$d.cx_rum.rum_template_id != null | groupby \$d.cx_rum.rum_template_id aggregate distinct_count(\$d.cx_rum.session_context.user_id) as affected_users, count() as error_count, any_value(\$d.cx_rum.error_context.error_message) as error_message | orderby affected_users desc" --start now-7d

# LCP by page
cx logs "filter \$l.subsystemname == 'cx_rum' && \$d.cx_rum.event_context.type == 'web-vitals' && \$d.cx_rum.web_vitals_context.name == 'LCP' | groupby \$d.cx_rum.page_context.page_fragments aggregate percentile(0.75, \$d.cx_rum.web_vitals_context.value) as LCP_p75_ms, count() as samples | orderby LCP_p75_ms desc" --start now-7d

---

Querying Patterns

Web Vitals

Web vitals use percentile(0.75, ...) for p75 values - avg is skewed by outliers. Use $d.cx_rum.web_vitals_context.value without :num cast.

Only query the specific vitals the user asks about. For "loading times" query LT, for "LCP" query LCP. Include all vitals only when the user explicitly asks for a full overview.

For multiple vitals in one query, use conditional if() inside percentile:

cx logs "filter \$l.subsystemname == 'cx_rum' && \$d.cx_rum.event_context.type == 'web-vitals' | groupby \$d.cx_rum.page_context.page_fragments aggregate percentile(0.75, if(\$d.cx_rum.web_vitals_context.name == 'LT', \$d.cx_rum.web_vitals_context.value)) as LT_p75, percentile(0.75, if(\$d.cx_rum.web_vitals_context.name == 'LCP', \$d.cx_rum.web_vitals_context.value)) as LCP_p75" --start now-7d

User Interactions

Always aggregate results - raw interaction events are noisy. Group by interaction_context.target_element_inner_text (the button/link text the user sees), and filter out null/empty values.

Do not group by target_element (HTML tag like DIV, SPAN) or target_selector - these are not meaningful to users. The correct field prefix is interaction_context, not user_interaction_context.

Network Requests

Filter network requests by event type $d.cx_rum.event_context.type == 'network-request'. For failed requests, combine with event_context.severity:num == 5. Compose descriptions as "<method> <fragments> (status <status_code>)".

Page Performance

Use the LT (Load Time) web vital for page loading time questions. Group by $d.cx_rum.page_context.page_fragments (not page_url), and include user count for context with distinct_count($d.cx_rum.session_context.user_id:string) as users.

---

Troubleshooting

If a query returns no results, change one thing at a time:

1. Extend the time range: --start now-7d or --start now-30d 2. Relax filters: remove the most restrictive condition 3. Verify field names: run a sample query with -o json to inspect actual fields 4. Try archive tier: --tier archive --start now-30d for older data

Note: Filtering by cx_rum fields will show only RUM/frontend logs and hide backend logs. This is expected when analyzing RUM data.

Span Querying Reference

Query and analyze distributed tracing data using the cx spans command with DataPrime syntax.

DataPrime syntax: See dataprime-reference.md for the full query language reference.

Understanding Spans in Coralogix

Spans are the fundamental unit of tracing data. Traces are not stored as single entities - they are logical groupings of spans that share the same traceID. To analyze a trace, you query its constituent spans.

This means:

• *Metadata (`$m.)** and **labels ($l.`)* are predictable - you can always filter on timestamp, duration, service name, and operation name without discovery.
• *User data (`$d.)** contains trace identifiers (traceID, spanID, parentSpanID) and application-specific tags/attributes that vary by service. Always verify custom $d` fields before assuming they exist.

---

CLI Command

cx spans '<dataprime_query>'

The source spans is automatically injected - do not include it in the query.

Options

---

Span Data Model

Standard Fields (Always Available)

Note on label fields: The meaning of $l.applicationName and $l.subsystemName varies by customer - they may represent environments, teams, regions, or something else entirely. Don't assume what they map to. Use cx search-fields or sample queries to verify actual values.

Duration Units

$m.duration is in microseconds:

• 500ms = 500000
• 1s = 1000000
• 1min = 60000000

When presenting duration values, always convert to human-readable units (milliseconds, seconds, or minutes) and include the unit. Never display raw microsecond values or the "µs" symbol.

# Computed field for milliseconds
create latency_ms from $m.duration / 1000

Error Detection

Spans do not have a $m.severity field like logs. Errors are typically indicated by:

• $d.tags.error == true - the most common convention (OpenTelemetry/Jaeger)
• Status codes in custom fields (e.g. $d.http.status_code, $d.grpc.status_code)
• Other application-specific error tags

The exact field depends on the instrumentation library used. If $d.tags.error returns no results, inspect sample spans with -o json to discover how errors are tagged:

cx spans "filter \$l.serviceName == 'api'" --limit 5 -o json

---

Essential Query Examples

# Get all spans for a trace
cx spans "filter \$d.traceID == '4f6a8f3c2e8a1b97'"

# Find spans for a service
cx spans "filter \$l.serviceName == 'checkout-service'"

# Find slow spans (> 1 second)
cx spans "filter \$m.duration > 1000000"

# Find error spans
cx spans "filter \$d.tags.error == true"

# Aggregate latency by operation
cx spans "groupby \$l.operationName aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc"

# Wider time range
cx spans "filter \$l.serviceName == 'api'" --start now-6h

Wildfind Policy

Avoid `wildfind` by default. It scans all fields and is expensive.

The one exception: when the user provides a specific string and you don't know which field contains it:

cx spans "wildfind 'connection refused'"

Tip: wildfind can also serve as a last-resort field discovery method - when cx search-fields doesn't find what you need, run wildfind with a known value, then inspect the matching spans to see which fields contain it.

---

Field Discovery

Skip discovery when:

• The query only uses standard fields ($m.duration, $l.serviceName, $l.operationName, $d.traceID)
• The user explicitly names the fields they want
• The fields have already been discovered earlier in the conversation

1. Infer from Source Code (Preferred)

If you have access to the application's source code, examine OpenTelemetry instrumentation, span attribute definitions, and tracing middleware to identify field names directly.

2. Semantic Search

cx search-fields "customer identifier" --dataset spans
cx search-fields "order ID" --dataset spans
cx search-fields "http response code" --dataset spans

Note: cx search-fields only has access to the most common fields. If it doesn't find what you need, fall back to sample query inspection.

3. Sample Query Inspection

cx spans "filter \$l.serviceName == 'api'" --limit 5 -o json

Inspect the JSON output to see all available fields. Especially useful for discovering fields in unstructured or deeply nested data.

---

Investigation Workflow

1. Understand the Request

Identify:

• Whether you have a trace ID, service name, or error description
• Time frame of interest
• Whether the question is about latency, errors, or request flow

2. Start with Known Information

If you have a trace ID - go straight to it:

cx spans "filter \$d.traceID == '<trace_id>'"

If you have a service name - query its spans:

cx spans "filter \$l.serviceName == '<service>'" --limit 50

If you have neither - start broad to find entry points:

# Find recent error spans
cx spans "filter \$d.tags.error == true" --limit 20

# Find the slowest spans in the last hour
cx spans "groupby \$l.serviceName, \$l.operationName aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc | limit 10"

# Then extract trace IDs from interesting spans
cx spans "filter \$l.serviceName == '<service>' && \$m.duration > 1000000 | distinct \$d.traceID"

3. Troubleshooting

If a query returns no results, change one thing at a time:

1. Extend the time range: --start now-6h or --start now-24h 2. Relax filters: remove the most restrictive condition 3. Check field availability: the field you're filtering by may only exist in a subset of spans 4. Verify field names: run a sample query with -o json to inspect the actual schema 5. Check service names: service names are case-sensitive 6. Try archive tier: --tier archive --start now-30d for older data

---

Common Query Patterns

Trace Reconstruction

# All spans for a trace
cx spans "filter \$d.traceID == '4f6a8f3c2e8a1b97'"

# Find root spans only (no parent)
cx spans "filter \$l.serviceName == 'api-gateway' | filter \$d.parentId == null"

# Find trace IDs for a service
cx spans "filter \$l.serviceName == 'payment-service' | distinct \$d.traceID"

Latency Analysis

# Spans slower than 1 second
cx spans "filter \$m.duration > 1000000"

# Top 10 slowest operations by average duration
cx spans "groupby \$l.operationName aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc | limit 10"

# Average latency by service
cx spans "groupby \$l.serviceName aggregate avg(\$m.duration) as avg_latency"

# P95 latency by operation
cx spans "groupby \$l.operationName aggregate percentile(0.95, \$m.duration) as p95_latency"

Latency Spike Detection

# Average latency per 15-minute window
cx spans "filter \$l.serviceName == 'api' | groupby roundTime(\$m.timestamp, 15m) as interval aggregate avg(\$m.duration) as avg_latency | orderby interval"

# Find the time windows with highest latency
cx spans "filter \$l.serviceName == 'api' | groupby roundTime(\$m.timestamp, 5m) as interval aggregate avg(\$m.duration) as avg_latency | orderby avg_latency desc | limit 10"

Error Investigation

# All error spans
cx spans "filter \$d.tags.error == true"

# Error spans for a specific service
cx spans "filter \$l.serviceName == 'checkout' | filter \$d.tags.error == true"

# Error rate by service
cx spans "filter \$d.tags.error == true | groupby \$l.serviceName aggregate count() as errors | orderby errors desc"

# Error rate over time
cx spans "filter \$d.tags.error == true | groupby roundTime(\$m.timestamp, 15m) as interval aggregate count() as errors"

Sampling Error Types

# Group errors by operation with a sample
cx spans "filter \$d.tags.error == true | groupby \$l.operationName aggregate any_value(\$d) as sample, count() as total | orderby total desc | limit 5"

# Group by service and operation to see where errors concentrate
cx spans "filter \$d.tags.error == true | groupby \$l.serviceName, \$l.operationName aggregate count() as errors | orderby errors desc | limit 10"

Finding Unique Values

# List all services with spans
cx spans "distinct \$l.serviceName"

# List all operations for a service
cx spans "filter \$l.serviceName == 'api' | distinct \$l.operationName"

# Find unique trace IDs for error spans
cx spans "filter \$d.tags.error == true | distinct \$d.traceID"

Correlating by Trace ID

# Find spans across services for the same trace
cx spans "filter \$d.traceID == 'abc123' | groupby \$l.serviceName aggregate count() as span_count, avg(\$m.duration) as avg_latency"

---

Performance Tips

• Use --limit for exploratory queries
• Use groupby with aggregations instead of fetching raw spans when possible
• Filter by time first when dealing with large datasets
• Use specific filters (service name, operation) to reduce scan scope
• Don't rely solely on aggregations - retrieve sample spans to find information you didn't anticipate
• For large result sets, use --output agents which spills automatically:

cx spans "filter \$l.serviceName == 'api'" --start now-24h --limit 1000 -o agents