Axiom Apl

Axiom Processing Language (APL)

APL is Axiom's query language for analyzing observability data. This skill provides comprehensive guidance for writing, debugging, and optimizing APL queries.

Quick Reference

Documentation: https://axiom.co/docs/apl/introduction

CLI usage: See references/cli.md

Core Workflow

1. List Available Datasets

axiom dataset list -f json

2. Discover Schema (CRITICAL - Always Do First)

['<dataset>'] | getschema

Never guess field names. The schema shows all fields with their types.

3. Sample Data

['<dataset>'] | limit 10

4. Write Query

See references for operators, functions, and patterns.

APL Syntax Essentials

Dataset Reference

['dataset-name']           // Bracket notation (required for names with dots/dashes)
dataset_name               // Plain identifier (only for simple names)

Field Reference

field_name                 // Plain field
['field.with.dots']        // Bracket notation for dotted fields
['service.name']           // OTel data (see references/otel.md for field mappings)

Basic Query Structure

['dataset']
| where <condition>
| extend <new_field> = <expression>
| summarize <aggregation> by <grouping>
| project <fields>
| sort by <field> desc
| limit 100

Time Handling

Always filter by time first - it's the most selective filter.

// Relative time
| where _time >= ago(1h)
| where _time >= ago(24h) and _time < ago(1h)

// Absolute time
| where _time >= datetime(2024-01-15T10:00:00Z)
| where _time between (datetime(2024-01-15) .. datetime(2024-01-16))

Time functions:

• ago(timespan) - Relative past time
• now() - Current time
• datetime(string) - Parse datetime
• bin(_time, 5m) - Time bucketing
• bin_auto(_time) - Automatic bucketing

When NOT to Use

• Simple field lookup: Use getschema directly instead of invoking the full skill
• Known query patterns: If you already have a working query, don't re-invoke for syntax help
• Real-time alerting: Use Axiom Monitors for continuous alerting, not ad-hoc queries

References

• [CLI Usage](references/cli.md) - Command flags and execution
• [Operators](references/operators.md) - Tabular and scalar operators
• [Functions](references/functions.md) - String, datetime, aggregation functions
• [Patterns](references/patterns.md) - Query patterns by use case
• [Common Gotchas](references/gotchas.md) - Mistakes and fixes
• [OpenTelemetry](references/otel.md) - OTel field mappings and trace patterns

Axiom CLI Reference

Documentation: https://axiom.co/docs/reference/cli

Configuration & Deployments

The CLI reads configuration from ~/.axiom.toml. Environment variables (AXIOM_TOKEN, AXIOM_URL, AXIOM_ORG_ID) override config values.

Introspect Configuration

# Active deployment name
axiom config get active_deployment

# Deployment URL and org ID (replace <name> with deployment alias)
axiom config get "deployments.<name>.url"
axiom config get "deployments.<name>.org_id"

Do NOT use `config get` to read tokens. If you need to export credentials (e.g., to .env or .envrc files), use eval $(axiom config export --force) instead.

Switch Deployments

# Per-command override (does not persist)
axiom -D staging query "['logs'] | limit 10" --start-time -1h

# Persistent switch
axiom auth select <alias>

# Switch organization within current deployment
axiom auth switch-org <org-id>

Verify Connectivity

# Check all deployments
axiom auth status

# Check specific deployment
axiom auth status <alias>

Self-Discovery

axiom help credentials    # Token types and authentication guidance
axiom help environment    # All supported environment variables
axiom --help              # Top-level command overview
axiom <command> --help    # Subcommand details

Query Execution

axiom query "<APL>" [flags]

Flags

Time Range Formats

# Relative (recommended)
--start-time -20m             # 20 minutes ago
--start-time -1h              # 1 hour ago
--start-time -24h             # 24 hours ago
--start-time -7d              # 7 days ago
--start-time -2w              # 2 weeks ago

# Mixed relative
--start-time -1d12h           # 1 day and 12 hours ago

# Absolute
--start-time 2024-01-15       # Date
--start-time 2024-01-15T10:00:00Z  # ISO timestamp

Supported relative units: w (week), d (day), h (hour), m (minute), s (second), ms, us, ns.

Output Formats

Examples

# Simple query
axiom query "['logs'] | limit 10" --start-time -1h

# JSON output for parsing
axiom query "['logs'] | summarize count() by status" -f json --start-time -1h

# Specific time range
axiom query "['logs'] | where error == true" --start-time -24h --end-time -1h

# Query against a different deployment
axiom -D prod query "['logs'] | count" --start-time -1h

Dataset Operations

# List all datasets
axiom dataset list
axiom dataset list -f json

# Aliases
axiom dataset ls

Livestream

# Stream live data (no filtering; dataset argument required)
axiom stream <dataset>
axiom stream logs

Query Best Practices

1. Always specify time range with --start-time to limit data scanned 2. Use JSON for parsing with -f json for programmatic access 3. Prefer aggregations to reduce data volume returned 4. Limit results with | limit N during exploration 5. Discover schema first with | getschema before writing queries

APL Functions Reference

Documentation: https://axiom.co/docs/apl/scalar-functions

String Functions

String Predicates

Performance tip: has_cs is 5-10x faster than contains. Prefer case-sensitive variants.

Extract Functions

// Extract with regex
| extend user_id = extract("user=([^&]+)", 1, url)

// Extract all matches
| extend numbers = extract_all(@"\d+", message)

DateTime Functions

DateTime Parts

Time Arithmetic

| extend tomorrow = _time + 1d
| extend last_hour = _time - 1h
| extend duration_hours = (end_time - start_time) / 1h

Type Conversion

Null Handling

Safe Field Access

// Ensure field exists with default type
| extend error = coalesce(ensure_field("error", typeof(bool)), false)

JSON Functions

// Access nested JSON
| extend props = parse_json(properties)
| where props.level == "error"

// Access dotted attribute
| extend dataset = ['attributes.custom'].dataset

Mathematical Functions

Conditional Functions

// Simple if
| extend level = iff(status >= 500, "error", "ok")

// Case/switch
| extend severity = case(
    status >= 500, "critical",
    status >= 400, "error",
    status >= 300, "warning",
    "info"
  )

Array Functions

// Check membership
| where url in ("login", "logout", "home")
| where status !in (200, 201, 204)

Common APL Gotchas

Mistakes that cause errors or unexpected results, and how to fix them.

Bracket Notation

Problem: Dotted field names without brackets

// WRONG - syntax error or wrong field
| where service.name == "api"
| project status.code, service.name

// CORRECT - bracket notation required
| where ['service.name'] == "api"
| project ['status.code'], ['service.name']

Rule: Any field with dots, dashes, or spaces needs ['field.name'] notation.

When to use brackets

Type Mismatches

Problem: Comparing wrong types

// WRONG - comparing string to integer
| where ['status.code'] == 2

// CORRECT - status.code is a string
| where ['status.code'] == "ERROR"

Problem: Arithmetic on strings

// WRONG - duration might be stored as string
| extend duration_ms = duration / 1000

// CORRECT - convert first
| extend duration_ms = tolong(duration) / 1000

Rule: Use getschema to check field types before operating on them.

Duration Units

Problem: Assuming wrong time units

OTel traces store duration in nanoseconds, not milliseconds.

// WRONG - looking for 1ms threshold
| where duration > 1

// CORRECT - 1ms = 1,000,000 nanoseconds
| where duration > 1000000

// BETTER - explicit conversion
| extend duration_ms = duration / 1000000.0
| where duration_ms > 1

Conversion table:

Time Filtering

Problem: Missing time filter

// WRONG - scans entire dataset (expensive!)
['logs']
| where status >= 500

// CORRECT - always filter by time first
['logs']
| where _time >= ago(1h)
| where status >= 500

Rule: Always include time filter. Put it first for best performance.

Problem: Wrong time syntax

// WRONG - missing ago()
| where _time >= 1h

// CORRECT - use ago() for relative time
| where _time >= ago(1h)

// ALSO CORRECT - explicit datetime
| where _time >= datetime(2024-01-15)

Null Handling

Problem: Null comparisons fail silently

// WRONG - rows with null error field are excluded
| where error == true

// CORRECT - handle nulls explicitly
| where coalesce(error, false) == true

// OR use ensure_field
| extend error = coalesce(ensure_field("error", typeof(bool)), false)
| where error == true

Aggregation Issues

Problem: Counting vs distinct counting

// Counts all rows
| summarize count() by user_id

// Counts unique values
| summarize dcount(user_id)

Problem: Missing group-by context

// WRONG - avg over entire result
| extend avg_duration = avg(duration)

// CORRECT - use summarize for aggregations
| summarize avg_duration = avg(duration) by service

Search Performance

Problem: Using expensive search

// SLOW - scans all fields
search "error"

// FASTER - target specific field
| where message contains "error"

// FASTEST - case-sensitive word match
| where message has_cs "error"

Performance ranking: 1. has_cs (fastest) - case-sensitive word boundary 2. has - word boundary match 3. contains_cs - case-sensitive substring 4. contains - substring match 5. search (slowest) - full-text all fields

Result Limits

Problem: Unbounded queries

// WRONG - could return millions of rows
['logs']
| where _time >= ago(24h)

// CORRECT - always limit or aggregate
['logs']
| where _time >= ago(24h)
| limit 1000

// OR aggregate
['logs']
| where _time >= ago(24h)
| summarize count() by status

Rule: Either limit results or summarize to aggregate.

Query Structure

Problem: Wrong operator order

// WRONG - project before where loses fields
['logs']
| project _time, message
| where status >= 500  // status no longer exists!

// CORRECT - filter before projecting
['logs']
| where status >= 500
| project _time, message

Recommended order: 1. where _time - Time filter first 2. where - Other filters 3. extend - Add computed fields 4. summarize - Aggregate 5. project - Select final fields 6. sort - Order results 7. limit - Restrict count

CLI Issues

Problem: Quoting in shell

# WRONG - shell interprets special chars
axiom query ['logs'] | where status >= 500

# CORRECT - quote the entire query
axiom query "['logs'] | where status >= 500"

# For complex queries, use heredoc
axiom query "$(cat <<'EOF'
['logs']
| where _time >= ago(1h)
| where message contains "error"
| limit 100
EOF
)"

Problem: Missing time range

# WRONG - uses default (might be too broad)
axiom query "['logs'] | limit 10"

# CORRECT - explicit time range
axiom query "['logs'] | limit 10" --start-time -1h

OTel Field Mismatches

Problem: Using standard OTel field paths

Axiom promotes some resource attributes to top level.

// WRONG - Axiom promotes these fields
| where ['resource.service.name'] == "api"
| where ['resource.telemetry.sdk.version'] == "1.0"

// CORRECT - No resource. prefix for promoted fields
| where ['service.name'] == "api"
| where ['telemetry.sdk.version'] == "1.0"

Rule: Check getschema output. See otel.md for complete field mappings.

Problem: Custom attribute access

Non-semconv attributes are stored in a map field.

// WRONG - field doesn't exist at this path
| where ['attributes.my_field'] == "value"

// CORRECT - access via custom map
| where ['attributes.custom']['my_field'] == "value"

// WRONG - aggregation without cast fails
| summarize count() by ['attributes.custom']['field']

// CORRECT - explicit cast required for aggregations
| summarize count() by tostring(['attributes.custom']['field'])

APL Operators Reference

Documentation: https://axiom.co/docs/apl/tabular-operators

Tabular Operators

Filtering

where

Filter rows based on condition.

| where status >= 500
| where ['service.name'] == "api-gateway"
| where message contains "error"
| where _time >= ago(1h)

search

Full-text search across all fields. Use sparingly - expensive.

search "error" or "exception"
search in (['logs']) "timeout"

Projection

project

Select and rename specific fields.

| project _time, status, message
| project timestamp=_time, code=status

project-away

Remove specific fields.

| project-away internal_id, debug_info

project-rename

Rename fields without changing selection.

| project-rename responseTime=['duration'], path=['url']

extend

Add calculated fields.

| extend duration_ms = duration / 1000000
| extend is_error = status >= 400
| extend full_name = strcat(first_name, " ", last_name)

Aggregation

summarize

Aggregate data with grouping.

// Count by field
| summarize count() by status

// Multiple aggregations
| summarize
    total = count(),
    errors = countif(status >= 500),
    avg_duration = avg(duration)
  by ['service.name']

// Time bucketing
| summarize count() by bin(_time, 5m)
| summarize count() by bin_auto(_time)

// Percentiles
| summarize
    p50 = percentile(duration, 50),
    p95 = percentile(duration, 95),
    p99 = percentile(duration, 99)
  by endpoint

Sorting & Limiting

sort / order

Sort results.

| sort by _time desc
| sort by count_ desc, name asc
| order by duration desc  // alias for sort

top

Get top N by field.

| top 10 by count_
| top 5 by duration desc

limit / take

Limit result count.

| limit 100
| take 50  // alias for limit

Joining

join

Combine datasets.

| join kind=inner (
    ['other-dataset'] | where _time >= ago(1h)
  ) on user_id

// Join kinds: inner, leftouter, rightouter, fullouter, leftanti, rightanti

union

Combine multiple datasets.

union ['logs-app-1'], ['logs-app-2']
union ['logs-*']  // wildcard

Data Shaping

mv-expand

Expand arrays into rows.

| mv-expand tag = tags
| mv-expand item = parse_json(items)

parse

Extract fields from strings.

| parse message with * "user=" user " action=" action
| parse-kv message as (duration:long, error:string) with (pair_delimiter=",")

getschema

Show dataset schema.

['dataset'] | getschema

Aggregation Functions

Set Functions

Special Aggregations

arg_min / arg_max

Get field value at min/max of another field.

// Get operation name at earliest timestamp
| summarize first_op = arg_min(_time, name) by trace_id

// Get slowest operation name
| summarize slowest = arg_max(duration, name) by trace_id

histogram

Create histogram buckets.

| summarize histogram(duration, 100) by endpoint

topk

Top K values with counts.

| summarize topk(status, 5)

OpenTelemetry Field Mappings

Axiom transforms OpenTelemetry data during ingestion. Field names differ from standard OTel conventions.

Promoted Resource Fields

These resource attributes are stored at top level (without resource. prefix):

Service: service.name, service.version, service.namespace, service.instance.id

Telemetry SDK: telemetry.sdk.name, telemetry.sdk.version, telemetry.sdk.language, telemetry.distro.name, telemetry.distro.version

All other resource attributes: resource.<key> or resource.custom map.

Standard Span Fields

Always present in trace datasets:

Attribute Storage

Semantic Convention Attributes

Standard OTel semconv attributes stored flat under attributes.:

attributes.http.request.method
attributes.http.response.status_code
attributes.db.system
attributes.db.statement
attributes.rpc.service

AI Observability Attributes

GenAI and Eval attributes are stored flat (not in custom map):

attributes.gen_ai.system
attributes.gen_ai.request.model
attributes.gen_ai.response.model
attributes.gen_ai.usage.input_tokens
attributes.gen_ai.usage.output_tokens
attributes.eval.name
attributes.eval.score.value

Custom Attributes

Non-standard attributes go to a map field:

attributes.custom = {"my_field": "value", "user_id": "123"}

Access pattern:

// Filter/project - works directly
| where ['attributes.custom']['user_id'] == "123"
| project ['attributes.custom']['user_id']

// Aggregation - MUST cast to string
| summarize count() by tostring(['attributes.custom']['user_id'])

Without tostring(), aggregations fail: "grouping by field of type unknown is not supported".

Common Mistakes

Duration Conversion

OTel durations are in nanoseconds:

Convert for display:

| extend duration_ms = duration / 1000000.0

APL Query Patterns

Common query patterns organized by use case.

Log Analysis

Error Rate by Service

['logs']
| where _time >= ago(1h)
| summarize
    total = count(),
    errors = countif(status >= 500)
  by ['service.name']
| extend error_rate = round(errors * 100.0 / total, 2)
| where errors > 0
| sort by error_rate desc

Error Timeline

['logs']
| where _time >= ago(6h)
| where status >= 500
| summarize errors = count() by bin(_time, 15m)
| sort by _time asc

Top Error Messages

['logs']
| where _time >= ago(1h)
| where level == "error" or status >= 500
| summarize count() by message
| top 20 by count_

Latency Percentiles by Endpoint

['logs']
| where _time >= ago(1h)
| summarize
    p50 = percentile(duration, 50),
    p90 = percentile(duration, 90),
    p95 = percentile(duration, 95),
    p99 = percentile(duration, 99)
  by endpoint
| sort by p99 desc

Request Volume Over Time

['logs']
| where _time >= ago(24h)
| summarize requests = count() by bin(_time, 1h)
| sort by _time asc

Trace Analysis

Get All Spans for a Trace

['traces']
| where trace_id == "<TRACE_ID>"
| sort by _time asc
| limit 100

Find Error Spans in Trace

['traces']
| where trace_id == "<TRACE_ID>"
| where error == true
| project _time, ['service.name'], name, duration, ['status.message']

Find Traces by Criteria

['traces']
| where _time >= ago(1h)
| where ['service.name'] == "<SERVICE>"
| where error == true
| extend error = coalesce(ensure_field("error", typeof(bool)), false)
| summarize
    start_time = min(_time),
    total_duration = max(duration),
    span_count = count(),
    error_count = countif(error),
    services = make_set(['service.name']),
    root_operation = arg_min(_time, name)
  by trace_id
| sort by start_time desc
| limit 20

Slow Traces (> 1 second)

['traces']
| where _time >= ago(1h)
| where duration >= 1000000000  // 1s in nanoseconds
| summarize
    start_time = min(_time),
    total_duration = max(duration),
    span_count = count(),
    services = make_set(['service.name'])
  by trace_id
| sort by total_duration desc
| limit 20

Service Dependencies

['traces']
| where _time >= ago(1h)
| where kind == "CLIENT"
| summarize calls = count() by caller=['service.name'], callee=name
| sort by calls desc

Metrics & Performance

Service Performance Summary

['logs']
| where _time >= ago(1h)
| summarize
    requests = count(),
    errors = countif(status >= 500),
    p50_latency = percentile(duration, 50),
    p95_latency = percentile(duration, 95),
    p99_latency = percentile(duration, 99)
  by ['service.name']
| extend error_rate = round(errors * 100.0 / requests, 2)
| sort by requests desc

High Latency Services

['logs']
| where _time >= ago(1h)
| summarize
    p99 = percentile(duration, 99)
  by ['service.name']
| where p99 > 1000000000  // > 1s
| extend p99_ms = p99 / 1000000.0
| sort by p99 desc

Throughput Over Time

['logs']
| where _time >= ago(6h)
| summarize
    requests = count(),
    errors = countif(status >= 500)
  by bin(_time, 5m)
| extend error_rate = round(errors * 100.0 / requests, 2)
| sort by _time asc

Anomaly Detection

Volume Anomaly (Z-Score)

['logs']
| where _time >= ago(24h)
| summarize count() by bin(_time, 1h)
| extend
    avg_count = avg(count_),
    stdev_count = stdev(count_),
    z_score = (count_ - avg(count_)) / stdev(count_)
| where abs(z_score) > 2

New Values Detection

// Find new error codes in last hour not seen in previous 24h
['logs']
| where _time >= ago(1h)
| summarize by error_code
| join kind=leftanti (
    ['logs']
    | where _time between (ago(25h) .. ago(1h))
    | summarize by error_code
  ) on error_code

Statistical Outliers

['logs']
| where _time >= ago(1h)
| summarize
    avg_duration = avg(duration),
    stdev_duration = stdev(duration)
| extend
    lower_bound = avg_duration - 3 * stdev_duration,
    upper_bound = avg_duration + 3 * stdev_duration

Data Exploration

Schema Discovery

['dataset'] | getschema

Sample Data

['dataset']
| where _time >= ago(1h)
| limit 10

Field Cardinality

['dataset']
| where _time >= ago(1h)
| summarize
    total = count(),
    unique_values = dcount(field_name)
  by field_name

Value Distribution

['dataset']
| where _time >= ago(1h)
| summarize count() by field_name
| top 20 by count_

Time Distribution

['dataset']
| where _time >= ago(24h)
| summarize count() by bin(_time, 1h)
| sort by _time asc

Cross-Dataset Correlation

Event Correlation by Time

union
  (['dataset1'] | summarize d1_count = count() by bin(_time, 5m)),
  (['dataset2'] | summarize d2_count = count() by bin(_time, 5m))
| summarize
    dataset1 = sum(d1_count),
    dataset2 = sum(d2_count)
  by _time
| sort by _time asc

Sequential Pattern Mining

['logs']
| where _time >= ago(1h)
| sort by _time asc
| extend
    next_event = next(event_type, 1),
    time_to_next = next(_time, 1) - _time
| where time_to_next < 5m
| summarize count() by pattern = strcat(event_type, " -> ", next_event)
| top 20 by count_

OpenTelemetry Specific

OTel Field Reference

Duration Conversion

OTel durations are in nanoseconds:

| extend duration_ms = duration / 1000000.0
| where duration >= 1000000000  // >= 1 second