Exploring Llm Traces

Exploring LLM traces with MCP tools

PostHog captures LLM/AI agent activity as traces. Each trace is a tree of events representing a single AI interaction — from the top-level agent invocation down to individual LLM API calls.

Available tools

Event hierarchy

See the event reference for the full schema.

$ai_trace (top-level container)
  └── $ai_span (logical groupings, e.g. "RAG retrieval", "tool execution")
        ├── $ai_generation (individual LLM API call)
        └── $ai_embedding (embedding creation)

Events are linked via $ai_parent_id → parent's $ai_span_id or $ai_trace_id.

Workflow: debug a trace or session from a URL

Step 1 — Classify the URL

First inspect the path. Do not treat every UUID-looking value as a trace ID.

• /ai-observability/traces/<trace_id> or legacy /llm-analytics/traces/<trace_id> / /llm-observability/traces/<trace_id> is a single trace. Fetch it with posthog:query-llm-trace.
• /ai-observability/sessions/<session_id> or legacy /llm-analytics/sessions/<session_id> is an AI session, not a trace. Fetch traces with posthog:query-llm-traces-list filtered by event property $ai_session_id.

Preserve date_from / date_to query parameters from the URL when present. If none are present but the URL has a timestamp query parameter, use that timestamp as the anchor and query an absolute window around it, for example timestamp - 36h to timestamp + 36h. This handles exact session links whose UI timestamp may be offset from the stored event timestamps while keeping the query bounded. If the URL has neither explicit dates nor timestamp, use a safe default like {"date_from": "-7d"}.

For exact trace and session URLs, skip schema discovery for the standard $ai_* fields used below. These are AI observability built-ins, not project-specific custom properties.

Step 2 — Fetch trace data

For a trace URL, call posthog:query-llm-trace with:

{
  "traceId": "<trace_id>",
  "dateRange": { "date_from": "-7d" }
}

For a session URL, call posthog:query-llm-traces-list with:

{
  "dateRange": { "date_from": "<timestamp_minus_36h>", "date_to": "<timestamp_plus_36h>" },
  "filterTestAccounts": false,
  "limit": 20,
  "properties": [{ "type": "event", "key": "$ai_session_id", "value": ["<session_id>"], "operator": "exact" }]
}

Use the URL's date_from / date_to values in the session query if present. If the URL only has timestamp, calculate the absolute date range from that timestamp instead of using a relative range like -1h. Set filterTestAccounts: false for an exact URL so the requested trace is not hidden by account filters.

The result contains the event tree with all properties. The response may be large — when it exceeds the inline limit, Claude Code auto-persists it to a file.

From the result you get:

• Every event with its type ($ai_span, $ai_generation, etc.)
• Span names ($ai_span_name) — these are the tool/step names
• Latency, error flags, models used
• Parent-child relationships via $ai_parent_id
• _posthogUrl — always include this in your response so the user can click through to the UI

Step 3 — Parse large results with scripts

When the result is persisted to a file (large traces with full $ai_input/$ai_output_choices), use the parsing scripts to explore it.

Start with the summary to get the full picture, then drill into specifics:

# 1. Overview: metadata, tool calls, final output, errors
python3 scripts/print_summary.py /path/to/persisted-file.json

# 2. Timeline: chronological event list with truncated I/O
python3 scripts/print_timeline.py /path/to/persisted-file.json

# 3. Drill into a specific span's full input/output
SPAN="tool_name" python3 scripts/extract_span.py /path/to/persisted-file.json

# 4. Full conversation with thinking blocks and tool calls
python3 scripts/extract_conversation.py /path/to/persisted-file.json

# 5. Search for a keyword across all properties
SEARCH="keyword" python3 scripts/search_traces.py /path/to/persisted-file.json

All scripts support MAX_LEN=N env var to control truncation (0 = unlimited).

Investigation patterns

"Did the agent use the tool correctly?"

1. Find the $ai_span for the tool call (look at $ai_span_name) 2. Check $ai_input_state — what arguments were passed to the tool? 3. Check $ai_output_state — what did the tool return? 4. Check $ai_is_error — did the tool call fail?

"Was the context correct?" / "Were the right files surfaced?"

1. Find the $ai_generation event where the LLM made the decision 2. Check $ai_input — this is the full message history the LLM saw 3. Look at preceding $ai_span events for retrieval/search steps 4. Check their $ai_output_state — what content was retrieved and fed to the LLM?

"Did the subagent work?"

1. In the structural overview, find spans that are children of other spans (via $ai_parent_id) 2. The parent span is the orchestrator; child spans are subagent steps 3. Check each child's $ai_output_state and $ai_is_error 4. If a child span contains $ai_generation events, those are the subagent's LLM calls

"Why did the LLM say X?"

1. Use search_traces.py to find where the text appears: SEARCH="the text" python3 scripts/search_traces.py FILE 2. This shows which event and property path contains it 3. Check the $ai_input of that generation to see what the LLM was told before it said X

Constructing UI links

The trace tools return _posthogUrl — always surface this to the user.

You can also construct links manually:

• Trace detail: https://app.posthog.com/ai-observability/traces/<trace_id>?timestamp=<url_encoded_timestamp>&event=<optional_event_id>
• Traces list with filters: returned in _posthogUrl from query-llm-traces-list

The timestamp query param is required — use the createdAt of the earliest event in the trace, URL-encoded (e.g. timestamp=2026-04-01T19%3A39%3A20Z).

When presenting findings, always include the relevant PostHog URL so the user can verify.

Finding traces

Use posthog:query-llm-traces-list to search and filter traces.

CRITICAL: Never assume event names, property names, or property values from training data. Every project instruments different custom properties. For open-ended searches and custom filters, call posthog:read-data-schema first to discover what properties and values actually exist in the project's data before constructing filters.

The exception is exact AI observability trace/session URLs: use the built-in $ai_trace_id / $ai_session_id fields directly and skip schema discovery.

Discovering the schema first

Before filtering traces, discover what's available:

1. Confirm AI events exist — call posthog:read-data-schema with kind: "events" and look for $ai_* events 2. Find filterable properties — call posthog:read-data-schema with kind: "event_properties" and event_name: "$ai_generation" (or another AI event) to see what properties are captured 3. Get actual values — call posthog:read-data-schema with kind: "event_property_values", event_name: "$ai_generation", and property_name: "$ai_model" to see real model names in use

Only then construct the query-llm-traces-list call with property filters.

This is especially important for custom properties like project_id, conversation_id, user_tier, etc. — these vary per project and cannot be guessed.

Do not confirm $ai_* properties, but confirm any other like email of a person.

By filters

posthog:query-llm-traces-list
{
  "dateRange": {"date_from": "-1h"},
  "filterTestAccounts": true,
  "limit": 20,
  "properties": [
    {"type": "event", "key": "$ai_model", "value": "gpt-4o", "operator": "exact"}
  ]
}

Multiple filters are AND-ed together:

posthog:query-llm-traces-list
{
  "dateRange": {"date_from": "-1h"},
  "filterTestAccounts": true,
  "properties": [
    {"type": "event", "key": "$ai_provider", "value": "anthropic", "operator": "exact"},
    {"type": "event", "key": "$ai_is_error", "value": ["true"], "operator": "exact"}
  ]
}

You can also filter by person properties (discover them via read-data-schema with kind: "entity_properties" and entity: "person"):

posthog:query-llm-traces-list
{
  "dateRange": {"date_from": "-1h"},
  "filterTestAccounts": true,
  "properties": [
    {"type": "person", "key": "email", "value": "@company.com", "operator": "icontains"}
  ]
}

By external identifiers

Customers often store their own IDs as event or person properties. Use posthog:read-data-schema to discover what custom properties exist, then filter:

1. Call posthog:read-data-schema with kind: "event_properties" and event_name: "$ai_trace" to find custom properties 2. Review the returned properties and their sample values 3. Construct the filter using the discovered property key and a known value

posthog:query-llm-traces-list
{
  "dateRange": {"date_from": "-7d"},
  "properties": [
    {"type": "event", "key": "project_id", "value": "proj_abc123", "operator": "exact"}
  ]
}

For more complex SQL patterns, read these references:

• Single trace retrieval — fetches a single trace by ID with all events and properties (renders the TraceQuery HogQL)
• Traces list with aggregated metrics — two-phase query: find trace IDs first, then fetch aggregated latency, tokens, costs, and error counts

Parsing large trace results

Trace tool results are JSON. When too large to read inline, Claude Code persists them to a file.

Persisted file format

[{ "type": "text", "text": "{\"results\": [...], \"_posthogUrl\": \"...\"}" }]

Trace JSON structure

results (array for list, object for single trace)
  ├── id, traceName, createdAt, totalLatency, totalCost
  ├── inputState, outputState (trace-level state)
  └── events[]
        ├── event ($ai_span | $ai_generation | $ai_embedding | $ai_metric | $ai_feedback)
        ├── id, createdAt
        └── properties
              ├── $ai_span_name, $ai_latency, $ai_is_error
              ├── $ai_input_state, $ai_output_state (span tool I/O)
              ├── $ai_input, $ai_output_choices (generation messages)
              ├── $ai_model, $ai_provider
              └── $ai_input_tokens, $ai_output_tokens, $ai_total_cost_usd

Available scripts

Tips

• Always set dateRange — queries without a time range are slow. Use narrow windows (-30m, -1h) for broad listing queries; wider windows (-7d, -30d) are fine for narrow queries filtered by trace ID or specific property values
• Always include the _posthogUrl in your response so the user can click through
• $ai_input_state / $ai_output_state on spans contain tool call inputs and outputs
• $ai_input / $ai_output_choices on generations contain the full LLM conversation — can be megabytes; when the result is persisted to a file, use the parsing scripts
• In raw SQL, heavy content ($ai_input / $ai_output / $ai_output_choices / $ai_input_state / $ai_output_state / $ai_tools) lives only on the posthog.ai_events table, not events.properties — see the event reference for the column mapping and trace-id-anchored query patterns
• Use filterTestAccounts: true to exclude internal/test traffic when searching
• $ai_trace events are NOT in the events array — their data is surfaced via trace-level inputState, outputState, and traceName

AI observability event and property reference

Event types

$ai_trace

Top-level container for a trace. Emitted last, after all child events.

$ai_span

Logical grouping within a trace (e.g. "RAG retrieval", "tool execution", "routing").

$ai_generation

Individual LLM API call (e.g. a chat completion request).

$ai_embedding

Embedding creation event (text to vector).

Where heavy content lives: events vs ai_events

The heavy LLM properties are not stored on `events` — they live as native columns on a dedicated ClickHouse table, referenced in HogQL as `posthog.ai_events`. The events table keeps only the lightweight metadata (token counts, costs, model, provider, $ai_trace_id, latency, error flags).

posthog.ai_events is ORDER BY (team_id, trace_id, timestamp), so `trace_id` is the access path, not `timestamp`. Rows are dropped after the retention period (30 days by default), so traces older than that have no content. Nothing restricts which heavy columns an event can carry, but the typical shape is: $ai_generation carries input / output_choices / tools (embeddings carry input); $ai_span and $ai_trace carry input_state / output_state.

For trace inspection, prefer the query-llm-trace / query-llm-traces-list tools — they read posthog.ai_events for you. Drop to the SQL below only for custom analysis (aggregations, joins, batch extraction) or when you're already at the SQL layer.

Single trace — when you already have a trace_id (e.g. from a trace URL or query-llm-traces-list): read it directly.

SELECT timestamp, span_id, event, model, input, output_choices
FROM posthog.ai_events
WHERE trace_id = '<trace_id>'
ORDER BY timestamp

Batch / analytics (a time window across many traces): filter on the timestamp-indexed events table first to get the trace IDs, then fetch the heavy content from posthog.ai_events anchored on trace_id.

WITH matching_traces AS (
    SELECT DISTINCT properties.$ai_trace_id AS trace_id
    FROM events
    WHERE event = '$ai_generation'
        AND timestamp >= now() - INTERVAL 7 DAY
        AND properties.$ai_model = 'gpt-4o'  -- token/cost/model/ids stay on events
)
SELECT a.trace_id, a.span_id, a.model, a.input, a.output_choices
FROM posthog.ai_events AS a
WHERE a.trace_id IN (SELECT trace_id FROM matching_traces)
ORDER BY a.trace_id, a.timestamp

Common patterns

Linking events in a trace

All events share $ai_trace_id. The hierarchy is built via $ai_parent_id:

$ai_trace (id: "trace-1", $ai_trace_id: "trace-1")
  └── $ai_span (id: "span-1", $ai_trace_id: "trace-1", $ai_parent_id: "trace-1")
        └── $ai_generation (id: "gen-1", $ai_trace_id: "trace-1", $ai_parent_id: "span-1")

Cost aggregation

Costs are only on $ai_generation and $ai_embedding events. Sum $ai_total_cost_usd across these for the same $ai_trace_id to get total trace cost.

Large properties warning

These properties can contain megabytes of data:

• $ai_input — full conversation history, system prompts
• $ai_input_state / $ai_output_state — application state snapshots

Use contentDetail: "preview" or "none" when querying via MCP tools. When using contentDetail: "full", dump results to a file.

In raw SQL these live only on posthog.ai_events, not events.properties — see Where heavy content lives for the column mapping and query patterns.

LLM Trace query

This query might return a very large blob of JSON data. You should either only include data you need in case it's minimal or dump the results to a file and use bash commands to explore it. This query must always have time ranges set. You can calculate the time range as -30 to +30 minutes from the source event. The typical order of event capture for a trace is: $ai_span -> $ai_generation/$ai_embedding -> $ai_trace. Explore $ai\_\*-prefixed properties to find data related to traces, generations, embeddings, spans, feedback, and metric. Key properties of the $ai_generation event: $ai_input and $ai_output_choices.

IMPORTANT: The $ai_input, $ai_input_state, and $ai_output_state properties can be extremely large (containing full conversation histories, system prompts, or application state). When your query selects these properties, you MUST dump the results to a file and use bash commands to explore the output. Never output them directly into the conversation.

This content lives only on posthog.ai_events (read it directly by trace_id), not on events.properties — see where heavy content lives.

SELECT
    trace_id AS id,
    any(session_id) AS ai_session_id,
    min(timestamp) AS first_timestamp,
    max(timestamp) AS last_timestamp,
    ifNull(nullIf(argMinIf(distinct_id, timestamp, equals(event, '$ai_trace')), ''), argMin(distinct_id, timestamp)) AS first_distinct_id,
    round(if(and(equals(countIf(and(greater(latency, 0), notEquals(event, '$ai_generation'))), 0), greater(countIf(and(greater(latency, 0), equals(event, '$ai_generation'))), 0)), sumIf(latency, and(equals(event, '$ai_generation'), greater(latency, 0))), sumIf(latency, or(equals(parent_id, NULL), equals(parent_id, trace_id)))), 2) AS total_latency,
    nullIf(sumIf(input_tokens, in(event, tuple('$ai_generation', '$ai_embedding'))), 0) AS input_tokens,
    nullIf(sumIf(output_tokens, in(event, tuple('$ai_generation', '$ai_embedding'))), 0) AS output_tokens,
    nullIf(round(sumIf(input_cost_usd, in(event, tuple('$ai_generation', '$ai_embedding'))), 10), 0) AS input_cost,
    nullIf(round(sumIf(output_cost_usd, in(event, tuple('$ai_generation', '$ai_embedding'))), 10), 0) AS output_cost,
    nullIf(round(sumIf(total_cost_usd, in(event, tuple('$ai_generation', '$ai_embedding'))), 10), 0) AS total_cost,
    arrayDistinct(arraySort(x -> x.3, groupArrayIf(tuple(uuid, event, timestamp, properties, input, output, output_choices, input_state, output_state, tools), notEquals(event, '$ai_trace')))) AS events,
    argMinIf(input_state, timestamp, equals(event, '$ai_trace')) AS input_state,
    argMinIf(output_state, timestamp, equals(event, '$ai_trace')) AS output_state,
    ifNull(argMinIf(ifNull(nullIf(span_name, ''), nullIf(trace_name, '')), timestamp, equals(event, '$ai_trace')), argMin(ifNull(nullIf(span_name, ''), nullIf(trace_name, '')), timestamp)) AS trace_name
FROM
    ai_events
WHERE
    and(in(event, tuple('$ai_span', '$ai_generation', '$ai_embedding', '$ai_metric', '$ai_feedback', '$ai_trace')), and(greaterOrEquals(ai_events.timestamp, assumeNotNull(toDateTime('2025-12-09 23:35:41'))), lessOrEquals(ai_events.timestamp, assumeNotNull(toDateTime('2025-12-10 00:25:41'))), equals(trace_id, '79955c94-7453-488f-a84a-eabb6f084e4c')))
GROUP BY
    trace_id
LIMIT 1

LLM Traces list query

List multiple LLM traces with aggregated latency, token usage, costs, and error counts. This is a two-phase query for performance: first find matching trace IDs, then fetch full trace data. Time ranges are always required. Results can be large — dump to a file if needed.

This query intentionally omits large content fields ($ai_input, $ai_output, $ai_output_choices, $ai_input_state, $ai_output_state, $ai_tools). These live only on the dedicated posthog.ai_events table (not events), retained 30 days by default. Use the single trace query (or the query-llm-trace wrapper) to retrieve them for a specific trace, or read posthog.ai_events directly anchored on trace_id — see where heavy content lives for the column mapping.

Phase 1 — Find trace IDs

Use this subquery to find trace IDs matching your criteria. Add property filters here for efficiency.

SELECT
    properties.$ai_trace_id AS trace_id,
    min(timestamp) AS first_ts,
    max(timestamp) AS last_ts
FROM events
WHERE
    event IN ('$ai_span', '$ai_generation', '$ai_embedding', '$ai_metric', '$ai_feedback', '$ai_trace')
    AND isNotNull(properties.$ai_trace_id)
    AND properties.$ai_trace_id != ''
    AND timestamp >= now() - INTERVAL 1 HOUR
    AND timestamp <= now()
    -- Add property filters here, e.g.:
    -- AND properties.$ai_model = 'gpt-4o'
    -- AND properties.$ai_is_error = 'true'
GROUP BY trace_id
ORDER BY min(timestamp) DESC
LIMIT 20

Phase 2 — Fetch trace data

Use the trace IDs from phase 1 to fetch aggregated metrics. Replace the IN (...) clause with the IDs found above.

SELECT
    properties.$ai_trace_id AS id,
    any(properties.$ai_session_id) AS ai_session_id,
    min(timestamp) AS first_timestamp,
    ifNull(
        nullIf(argMinIf(distinct_id, timestamp, event = '$ai_trace'), ''),
        argMin(distinct_id, timestamp)
    ) AS first_distinct_id,
    round(
        CASE
            WHEN countIf(toFloat(properties.$ai_latency) > 0 AND event != '$ai_generation') = 0
                 AND countIf(toFloat(properties.$ai_latency) > 0 AND event = '$ai_generation') > 0
            THEN sumIf(toFloat(properties.$ai_latency),
                       event = '$ai_generation' AND toFloat(properties.$ai_latency) > 0)
            ELSE sumIf(toFloat(properties.$ai_latency),
                       properties.$ai_parent_id IS NULL
                       OR toString(properties.$ai_parent_id) = toString(properties.$ai_trace_id))
        END, 2
    ) AS total_latency,
    sumIf(toFloat(properties.$ai_input_tokens),
          event IN ('$ai_generation', '$ai_embedding')) AS input_tokens,
    sumIf(toFloat(properties.$ai_output_tokens),
          event IN ('$ai_generation', '$ai_embedding')) AS output_tokens,
    round(sumIf(toFloat(properties.$ai_input_cost_usd),
          event IN ('$ai_generation', '$ai_embedding')), 10) AS input_cost,
    round(sumIf(toFloat(properties.$ai_output_cost_usd),
          event IN ('$ai_generation', '$ai_embedding')), 10) AS output_cost,
    round(sumIf(toFloat(properties.$ai_total_cost_usd),
          event IN ('$ai_generation', '$ai_embedding')), 10) AS total_cost,
    ifNull(
        argMinIf(
            ifNull(properties.$ai_span_name, properties.$ai_trace_name),
            timestamp, event = '$ai_trace'
        ),
        argMin(
            ifNull(properties.$ai_span_name, properties.$ai_trace_name),
            timestamp
        )
    ) AS trace_name,
    countIf(
        isNotNull(properties.$ai_error) OR properties.$ai_is_error = 'true'
    ) AS error_count
FROM events
WHERE
    event IN ('$ai_span', '$ai_generation', '$ai_embedding', '$ai_metric', '$ai_feedback', '$ai_trace')
    AND timestamp >= now() - INTERVAL 1 HOUR
    AND timestamp <= now()
    AND properties.$ai_trace_id IN ('trace-id-1', 'trace-id-2')
GROUP BY properties.$ai_trace_id
ORDER BY first_timestamp DESC