Pandaprobe

PandaProbe

PandaProbe is an agent engineering platform for tracing, evaluating, and monitoring AI agents and LLM apps. This skill supports four goals — one per numbered section below:

1. Onboard a project onto PandaProbe — a guided, interactive setup that builds and runs a first traced agent example from scratch, ideal for an empty or new project (section 1). 2. Instrument an application with the PandaProbe SDK so its runs are traced (section 2); see references/instrumentation.md. 3. Read and evaluate data — traces, sessions, spans, scores, eval runs — with the pandaprobe CLI (section 3); see references/cli.md. 4. Access the documentation — fetch the latest PandaProbe docs, the source of truth that instrumentation relies on (section 4).

Core Principles

Follow these for ALL PandaProbe work:

1. Documentation first. NEVER instrument from memory. The SDK is under active development — always fetch the latest docs before writing instrumentation code (section 4). 2. CLI for data access. Use the pandaprobe CLI to read or evaluate data — never hand-build API calls. It is agent-first: JSON on stdout, errors on stderr, meaningful exit codes, no interactive prompts (section 3). 3. Best practices by use case. Read the relevant reference file before implementing — instrumentation → references/instrumentation.md; CLI → references/cli.md. 4. Use the latest versions. Unless the user specifies otherwise, use the latest PandaProbe SDK and CLI. 5. Never echo the API key. Don't print, log, or paste keys into chat; let the CLI and SDK manage credentials (pandaprobe config show masks the key).

1. Set up PandaProbe (guided onboarding)

When the user says "set up PandaProbe", "get started with PandaProbe", or similar, and is starting from an empty or new project, run the guided onboarding — an interactive flow that builds and runs a small traced agent example so they see their first traces. Follow the step-by-step playbook in references/setup.md.

If they instead want to add tracing to an app they already have, that's section 2 (instrument), not onboarding.

2. Instrument an agent application (SDK)

Add PandaProbe tracing to the user's app. The SDK changes often and exact APIs differ per framework, so work doc-first — do not instrument from memory. Keep your reasoning high-level here and pull specifics from the docs.

Procedure:

1. Identify the stack — which agent framework (LangGraph, LangChain, OpenAI Agents, CrewAI, Google ADK, Claude Agent SDK, DeepAgents) or LLM provider (OpenAI, Anthropic, Gemini, Mistral, Bedrock) the app uses. 2. Choose the layer (priority order): agent-framework integration → provider wrapper → manual decorators. Prefer the highest that fits; layers compose. 3. Fetch the latest docs for that exact framework/provider (section 4) and confirm the package extra, import path, class name, and wiring pattern before writing code. 4. Install, set credentials, wire the chosen layer, then verify the trace landed with the CLI (section 3).

The high-level playbook — layers, install extras, the integration wiring patterns, credentials, and verification — lives in references/instrumentation.md. Always reconcile exact names and signatures with the current docs.

3. Read & evaluate data (CLI)

Use the pandaprobe CLI to inspect traces, sessions, spans, scores, and evaluation runs. Reads (list/get/spans/metrics) are safe to run freely; the three eval write commands — evals runs create, evals runs batch, evals scores submit — create data, so run them only when the user explicitly asks.

Install, authentication (Cloud and self-hosted), the full command tree, output contract, exit codes, and jq recipes are in references/cli.md — read it before running commands. For anything beyond it, run pandaprobe <command> --help or see https://docs.pandaprobe.com/tools/cli.

4. Access PandaProbe documentation

The docs are the source of truth that instrumentation relies on. All product docs live at docs.pandaprobe.com. Prefer your application's native fetch tools (e.g. WebFetch) over curl when available — the URLs below work with any method.

a. Documentation index (llms.txt) — every page with title, URL, and one-line summary; use it to find the right page for a topic:

curl -s https://docs.pandaprobe.com/llms.txt

b. Fetch a page as markdown — append .md to its path (or send Accept: text/markdown):

curl -s "https://docs.pandaprobe.com/tracing/integrations/langgraph.md"

Key instrumentation pages: tracing/overview, tracing/concepts, tracing/integrations/overview (+ the per-framework page), tracing/wrappers/overview, tracing/manual/decorators, tracing/configuration/environment-variables. CLI: tools/cli.

Workflow: start at llms.txt to orient → fetch the specific .md page → verify exact names and signatures against it before writing code, and adopt the latest SDK version.

PandaProbe CLI Reference

Operate the pandaprobe CLI to read traces, sessions, spans, scores, and evaluation runs. The CLI is agent-first: JSON on stdout, errors on stderr, no interactive prompts, meaningful exit codes. Parse output with jq and branch on exit codes.

Official docs: https://docs.pandaprobe.com/tools/cli

Install

# macOS / Linux
curl -fsSL https://cli.pandaprobe.com/install.sh | sh

# Windows (PowerShell)
irm https://cli.pandaprobe.com/install.ps1 | iex

# From source (requires Go)
go install github.com/chirpz-ai/pandaprobe-cli@latest

The binary is pandaprobe. Verify:

pandaprobe version

Discover

pandaprobe --help              # top-level command groups
pandaprobe traces --help       # actions in a group
pandaprobe traces list --help  # flags for one action

Authenticate

Two paths:

Cloud (recommended) — auth login opens the browser, mints a 90-day API key, and writes api_key + project_name to ~/.pandaprobe/config.yaml:

pandaprobe auth login              # browser-based login (Cloud only)
pandaprobe auth login --no-browser # headless/SSH: prints the URL instead
pandaprobe auth status             # confirm login (key masked)
pandaprobe auth logout             # remove stored credentials

Local / self-hosted (auth disabled or manual key) — set values yourself. Never print the key.

pandaprobe config set endpoint http://localhost:8000
pandaprobe config set project_name my-project
pandaprobe config set api_key <key>   # only if the deployment requires one

Configuration & precedence

Values resolve highest → lowest: *flags > `PANDAPROBE_ env > ~/.pandaprobe/config.yaml`

defaults.**

Inspect the effective config (API key masked):

pandaprobe config show
pandaprobe config get endpoint
pandaprobe config path

Output contract

JSON by default → data to stdout, errors to stderr, so output pipes cleanly into jq. Use --format table only for human display.

List responses wrap items with pagination:

{ "items": [ /* ... */ ], "pagination": { "total": 150, "limit": 20, "offset": 0 } }

Errors are JSON on stderr:

{ "error": { "code": "validation_error", "message": "...", "status": 422, "request_id": "...", "details": {} } }

Global flags: --verbose, --debug (logs HTTP to stderr, key masked), --no-color, --config <path>.

Exit codes (branch on these)

Pagination everywhere: --limit (1–200) and --offset. Filtering is server-side.

Command reference

Traces (read-only)

pandaprobe traces list --status ERROR --sort-by started_at --sort-order desc --limit 20
pandaprobe traces get <trace_id>
pandaprobe traces spans <trace_id> --kind LLM --status ERROR

• traces list — filters: --status (PENDING|RUNNING|COMPLETED|ERROR), --session-id,

--user-id, --name, --tags, --started-after, --started-before, --sort-by (started_at|ended_at|name|latency|status), --sort-order (asc|desc), --limit (1–200), --offset.

• traces get <trace_id> — full trace with inline spans. --spans-only returns just the

spans array; --kind / --status filter spans client-side.

• traces spans <trace_id> — --kind (AGENT|TOOL|LLM|RETRIEVER|CHAIN|EMBEDDING|OTHER),

--status (OK|ERROR|UNSET).

{ "items": [ { "trace_id": "tr_123", "name": "support-agent", "status": "ERROR", "started_at": "..." } ],
  "pagination": { "total": 42, "limit": 20, "offset": 0 } }

Sessions (read-only)

pandaprobe sessions list --has-error --sort-by recent --limit 20
pandaprobe sessions get <session_id> --include-traces

• sessions list — --user-id, --has-error, --started-after, --started-before,

--tags, --query, --sort-by (recent|trace_count|latency|cost), --sort-order, --limit, --offset.

• sessions get <session_id> — --include-traces (default true), --limit, --offset.

Evaluations

Target traces or sessions with --target trace|session (default trace). There are multiple read commands along with three write commands that are flagged below.

Read:

pandaprobe evals metrics --target trace
pandaprobe evals runs list --target trace --status COMPLETED --limit 20
pandaprobe evals runs get <run_id> --target trace
pandaprobe evals runs scores <run_id> --target trace
pandaprobe evals scores list --target trace --name coherence --source AUTOMATED
pandaprobe evals scores get <trace_id> --target trace

• evals metrics — --target trace|session (default trace).
• evals runs list — --target, --status (PENDING|RUNNING|COMPLETED|FAILED),

--limit, --offset.

• evals runs get <run_id> / evals runs scores <run_id> — --target.
• evals scores list — --target; trace filters --trace-id, --name, --source

(AUTOMATED|ANNOTATION|PROGRAMMATIC), --status (SUCCESS|FAILED|PENDING), --data-type (NUMERIC|BOOLEAN|CATEGORICAL), --eval-run-id, --environment, --date-from, --date-to; session uses --session-id; --limit, --offset.

• evals scores get <trace_id|session_id> — --target.

Write — run only when the user explicitly asks or you need to evaluate an agent:

# Run metrics over traces matching filters
pandaprobe evals runs create --target trace --metrics coherence,tool_correctness --status COMPLETED

# Run metrics over an explicit set of traces/sessions
pandaprobe evals runs batch --target trace --trace-ids <id1>,<id2> --metrics coherence

# Submit a manual score for a trace (trace target only)
pandaprobe evals scores submit --trace-id <trace_id> --name accuracy --value 0.92

• evals runs create — --target, --metrics m1,m2 (required), --sampling-rate,

--model, --name. Trace filters: --date-from, --date-to, --status, --session-id, --user-id, --tags, --filter-name. Session filters: --user-id, --has-error, --tags, --min-trace-count, --signal-weights. (write op)

• evals runs batch — --target, --trace-ids/--session-ids (required), --metrics

(required), --name, --model. (write op)

• evals scores submit — --trace-id, --name, --value (required); optional

--data-type, --source, --reason, --metadata. Trace-only; --target session errors. (write op)

Utility

pandaprobe version
pandaprobe completion zsh   # also bash | fish | powershell
pandaprobe config show      # set | get | show | path

Recipes

list/get are read-only and safe to run freely. The recipes below only read data.

# 1. Overview: count traces by status
pandaprobe traces list --limit 200 \
  | jq '[.items[].status] | group_by(.) | map({status: .[0], count: length})'

# 2. Find a failed trace, then read the failing span's error
ID=$(pandaprobe traces list --status ERROR --limit 1 | jq -r '.items[0].trace_id')
pandaprobe traces get "$ID" | jq '.spans[] | select(.status == "ERROR") | {name, kind, error}'

# 3. Eval-run drill-down: list -> get -> scores
RUN=$(pandaprobe evals runs list --status COMPLETED --limit 1 | jq -r '.items[0].id')
pandaprobe evals runs get "$RUN"
pandaprobe evals runs scores "$RUN" | jq '.items[] | {name, value, data_type}'

# 4. Per-trace scores
pandaprobe evals scores get "$ID" | jq '.[] | {name, value, data_type, source}'

PandaProbe SDK Instrumentation

Add PandaProbe tracing to a developer's AI agent or LLM application. PandaProbe captures execution as traces (one logical run) composed of spans (steps within a run).

Instrumentation has three layers, in priority order:

1. Agent framework integrations — the primary path. If the app runs on a supported framework, this gives end-to-end traces (LLM + tools + sub-agents) with almost no code. 2. LLM provider wrappers — when there's no agent framework, or you only need LLM-call visibility. 3. Manual instrumentation — decorators/context managers for custom runtimes or extra spans around your own logic.

Prefer the highest layer that fits; they compose (a wrapper call inside an integration or @pandaprobe.trace nests automatically as a child span).

Fetch current docs first

The SDK evolves — never instrument from memory. Before writing code, fetch the latest docs and adopt the latest SDK version. Treat every snippet below as a starting point and verify it against the current docs — especially exact import paths and class names, which are framework-specific.

• Index (discover all pages): https://docs.pandaprobe.com/llms.txt
• Overview & concepts: https://docs.pandaprobe.com/tracing/overview · https://docs.pandaprobe.com/tracing/concepts
• Integrations (agent frameworks): https://docs.pandaprobe.com/tracing/integrations/overview
• Wrappers (LLM providers): https://docs.pandaprobe.com/tracing/wrappers/overview
• Manual decorators / context managers: https://docs.pandaprobe.com/tracing/manual/decorators
• Env vars: https://docs.pandaprobe.com/tracing/configuration/environment-variables

Append .md to any docs URL for clean markdown (e.g. …/tracing/integrations/langgraph.md). Always open the specific framework's integration page before wiring it up.

Install

pip install pandaprobe        # Python 3.10+ (or: uv add pandaprobe)

Install only the extras for the providers/frameworks in use (combine with commas, e.g. pip install "pandaprobe[langgraph,openai]"):

Credentials

Set via environment variables (read at client init). Never hardcode or echo the key.

export PANDAPROBE_API_KEY="your-api-key"
export PANDAPROBE_PROJECT_NAME="my-project"
# Optional:
export PANDAPROBE_ENDPOINT="https://api.pandaprobe.com"   # change for self-hosted
export PANDAPROBE_ENVIRONMENT="production"                # tag traces
export PANDAPROBE_ENABLED="true"                          # set false to no-op the SDK

The SDK auto-initializes from the environment. You can also configure programmatically via pandaprobe.init() — see https://docs.pandaprobe.com/tracing/configuration/project-configuration. If keys are missing, ask the user to set them in their shell or .env; don't ask them to paste a key into chat.

Layer 1 — Agent framework integrations (recommended)

Hook into a framework to automatically trace the full agent lifecycle — LLM calls, tool invocations, sub-agent handoffs, guardrails — as a properly nested span tree. You don't create traces or spans yourself.

All integrations share the same constructor params: session_id, user_id, tags, metadata (all optional). They differ only in how you wire them in, via one of two patterns:

• Callback handler (LangChain-family: LangGraph, LangChain, DeepAgents): construct a

handler and pass it per invocation via config={"callbacks": [handler]}.

• Adapter `.instrument()` (OpenAI Agents, CrewAI, Google ADK, Claude Agent SDK):

construct an adapter and call adapter.instrument() once at startup, before creating agents/runners — then use the framework normally.

Callback-handler pattern (LangGraph):

# verify against current docs
from pandaprobe.integrations.langgraph import LangGraphCallbackHandler

handler = LangGraphCallbackHandler(session_id="conversation-123", user_id="user-abc")
result = graph.invoke(
    {"messages": [{"role": "user", "content": "Hello!"}]},
    config={"callbacks": [handler]},   # pass on every invocation
)

Adapter pattern (OpenAI Agents — same shape for CrewAI, Google ADK, Claude Agent SDK):

# verify against current docs
from pandaprobe.integrations.openai_agents import OpenAIAgentsAdapter

OpenAIAgentsAdapter(session_id="conversation-123", user_id="user-abc").instrument()
# ...then build and run agents as usual; runs are traced automatically.

Each framework's page documents exactly what gets traced and any framework-specific caveats — read it before wiring: https://docs.pandaprobe.com/tracing/integrations/overview

Layer 2 — LLM provider wrappers

When there's no agent framework, wrap the provider client once. The wrapper returns the same client type, so existing call sites are unchanged, and every call is traced (input, output, model, token usage, params, TTFT for streaming).

# verify against current docs
from pandaprobe.wrappers import wrap_openai
from openai import OpenAI

client = wrap_openai(OpenAI())   # use exactly as before — all calls now traced

Available wrappers: wrap_openai, wrap_anthropic, wrap_gemini, wrap_mistral, wrap_bedrock (beta). Details: https://docs.pandaprobe.com/tracing/wrappers/overview

Layer 3 — Manual instrumentation

For custom runtimes or to add spans around your own logic. Use @pandaprobe.trace on a top-level entry point and @pandaprobe.span on inner steps; both auto-detect sync/async and capture inputs/outputs. pandaprobe.start_trace() / t.span() context managers cover cases decorators don't fit.

# verify against current docs
import pandaprobe

@pandaprobe.trace(name="support-agent")               # entry point -> a trace
def handle_request(query: str) -> str:
    docs = retrieve_docs(query)
    return generate_answer(query, docs)

@pandaprobe.span(name="retrieve", kind="RETRIEVER")   # inner step -> a span
def retrieve_docs(query: str) -> list[str]:
    ...

@pandaprobe.span requires an active trace context. Span kinds: LLM, TOOL, AGENT, CHAIN, RETRIEVER, EMBEDDING, OTHER. Details: https://docs.pandaprobe.com/tracing/manual/decorators

Verify instrumentation worked

1. Confirm the SDK imports and check the version:

   python -c "import pandaprobe; print(pandaprobe.__version__)"

2. Run the app with debug logging to see traces being sent (key stays masked):

   PANDAPROBE_DEBUG=true python your_app.py

3. Confirm the trace landed using the CLI (see cli.md):

   pandaprobe traces list --limit 5
   pandaprobe traces get <trace_id>     # inspect spans, token usage, status

If no trace appears, check PANDAPROBE_API_KEY / PANDAPROBE_PROJECT_NAME are set, PANDAPROBE_ENABLED is not false, and the endpoint is reachable. See https://docs.pandaprobe.com/tracing/configuration/troubleshooting.

Set up PandaProbe (guided onboarding)

Goal: take a new user starting from an empty or new project from zero to their first traces by building and running a small, working agent example for them, then pointing them to next steps. This path is strictly for onboarding from scratch — it favors a runnable example over theory.

If the user already has an application and wants to wire tracing into it, this is the wrong path — do not use this file. Switch to instrumentation.md (SKILL.md section 2), which covers instrumenting an existing app.

This is an orchestrator: it makes the choices interactive, implements the example doc-first (from instrumentation.md + the live docs), and verifies with the CLI (cli.md).

Ground rules

• Onboarding from scratch only. Build a fresh example in a new/empty project. The moment

the task becomes "instrument my existing code," hand off to instrumentation.md.

• Be interactive. Ask the user the questions in Step 1 and wait for answers — don't

assume. Keep it friendly and concrete.

• Confirm before side effects. Get explicit confirmation before installing packages,

creating files, or authenticating.

• Doc-first — never from memory. Fetch the latest docs for the chosen provider/framework

and implement the example from them (exact imports, integration wiring). The SDK changes often.

Step 0 — Confirm the starting point

Confirm this is a from-scratch onboarding, not an existing app:

• Empty/new project, or the user just wants a guided first example → continue here.
• They already have an app to trace → stop and switch to instrumentation.md.

Confirm Python 3.10+ is available (PandaProbe's SDK is Python; there is no TS SDK yet), and pick/confirm a working directory (create one and a virtualenv if helpful).

Step 1 — Ask what to build (interactive)

Ask the user two questions and wait for answers:

1. Which model provider do they want to use — OpenAI, Anthropic, Gemini, Mistral, or Bedrock? (They'll need that provider's API key.) 2. Which agent framework would they like to build with — LangGraph, LangChain, OpenAI Agents, CrewAI, Google ADK, or Claude Agent SDK? Recommend one that pairs well with their chosen provider and confirm.

You'll build a small agent with one tool, traced via that framework's integration, so they see a multi-span trace (agent + LLM + tool). Summarize the plan (provider + framework + packages you'll install) and confirm before proceeding.

Step 2 — Install & authenticate the CLI, then set credentials

Confirm before installing. Credentials go in a .env file (not shell export).

a. Install and authenticate the CLI. This signs the user in and writes their api_key and project_name to ~/.pandaprobe/config.yaml:

curl -fsSL https://cli.pandaprobe.com/install.sh | sh   # Windows: irm https://cli.pandaprobe.com/install.ps1 | iex
pandaprobe version
pandaprobe auth login        # opens the browser; add --no-browser on headless/SSH

b. Reuse those credentials for the SDK — copy the RAW values. After login, the CLI stores the raw api_key and project_name in ~/.pandaprobe/config.yaml (find it with pandaprobe config path). Open that file, read the raw values, and write them into a .env file in the project root in exactly this format:

PANDAPROBE_API_KEY=<raw value>
PANDAPROBE_PROJECT_NAME=<raw value>

Use the raw, unmasked key exactly as stored in config.yaml. Do not use the masked form (e.g. sk_pp_****43dc) printed by pandaprobe config show or pandaprobe config get — a masked value will not authenticate. Read it from the config file, not from config get.

c. Add the provider key. The only credential the user must supply is their LLM provider key. Ask which way they'd prefer to add it:

• a) Paste in chat — you add it. The user pastes their provider API key in the chat and

you write it into .env yourself, using the correct variable name for the provider.

• b) The user adds it manually. prompt them to put the credential in .env (the right

variable name for their provider) and wait for them to confirm it's set.

Either way .env ends up with the provider key, e.g.:

OPENAI_API_KEY=<provider key>   # variable name depends on the provider
# PANDAPROBE_DEBUG=true   # optional: log traces being sent

d. Ignore secrets in git. Create or update .gitignore in the project root and add .env so the credentials are never committed:

.env

e. Install the SDK with the extra for the chosen framework, plus python-dotenv to load .env (and any provider/framework library the example imports; see the extras table in instrumentation.md):

pip install "pandaprobe[<framework-extra>]" python-dotenv   # e.g. "pandaprobe[langgraph]"

Step 3 — Fetch docs & implement the example (doc-first)

Fetch the latest relevant docs (SKILL.md section 4), then write a minimal, runnable agent example from them — not from memory. Confirm the file with the user before creating it.

• Fetch tracing/integrations/overview + the chosen framework's page (and

get-started/quickstart), then wire the integration (callback handler or adapter .instrument()) around a small agent with one tool.

• Load the .env before the PandaProbe client initializes (the SDK reads env vars at

init), e.g. call load_dotenv() at the very top of the file.

Use instrumentation.md for the layer overview, but follow the docs for exact names and wiring.

Step 4 — Run the example

Run it so it emits a trace (it loads credentials from .env):

python example.py

Set PANDAPROBE_DEBUG=true in .env to log traces being sent. If nothing is sent: check PANDAPROBE_API_KEY / PANDAPROBE_PROJECT_NAME (and the provider key) are present in .env and loaded, PANDAPROBE_ENABLED is not false, and the endpoint is reachable (https://docs.pandaprobe.com/tracing/configuration/troubleshooting).

Step 5 — Verify the trace

The CLI is already installed and authenticated (Step 2), so just confirm the trace landed (details: cli.md):

pandaprobe traces list --limit 5
pandaprobe traces get <trace_id> # inspect spans, token usage, status

Then point the user to the dashboard at https://app.pandaprobe.com to see the trace visually.

Step 6 — Next steps

Offer where to go from here:

• Build another agent example with a different framework or provider.
• Instrument their real app — switch to instrumentation.md.
• Read data & run evaluations over captured traces — see cli.md.