Deepeval Tracing

DeepEval Tracing

Use this skill to instrument an AI application — an LLM app, agent, RAG pipeline, or chatbot — with DeepEval's native tracing so its execution is visible span by span in Confident AI's Observatory. The work is: pick a supported integration when one exists, fall back to manual @observe otherwise, give each span a meaningful type, and add tags and metadata.

This skill stops at producing well-formed traces. Attaching evaluation metrics and running evals is the deepeval skill's job.

Scope: AI Applications Only

Instrument only the AI parts of the system — agent loops and planning, LLM calls, retrieval / vector search, and tool calls. The span types (llm, retriever, tool, agent) describe AI components. Do not trace non-AI software (web servers, CRUD backends, infrastructure). If the target has no LLM, agent, retrieval, or tool-calling component, this skill does not apply.

When to Use vs the deepeval and deepeval-otel Skills

• This skill (`deepeval-tracing`) — instrument an app with the DeepEval SDK

(@observe, framework integrations) so traces reach Confident AI.

• `deepeval` skill — build pytest eval suites: datasets, metrics, traced

evals, deepeval test run, iteration. It runs evals against an app this skill instrumented.

• `deepeval-otel` skill — instrument with the vendor-neutral OpenTelemetry

SDK instead of the DeepEval SDK (raw OTLP, including non-Python apps).

The three are complementary. If unsure between this skill and deepeval-otel: use this one when the app is Python and you want the DeepEval SDK; use deepeval-otel when you want raw OpenTelemetry or the app is not Python.

Prerequisites

• An AI application in Python with pip install deepeval.
• For traces to reach Confident AI: deepeval login, or an exported

CONFIDENT_API_KEY (preferred for CI and non-interactive runs).

Workflow

1. Confirm the target is an AI application (it has LLM calls, an agent loop, retrieval, or tool calls). If it has none of these, stop — this skill does not apply. 2. Detect the framework, model provider, agent SDK, and vector database in use. 3. Read references/integrations.md and the exact integration doc for what was detected. Prefer a native integration over manual instrumentation. 4. If no native integration fits, instrument manually with @observe. Read references/tracing.md. 5. Give each span a meaningful type (llm, retriever, tool, agent) and capture inputs/outputs. 6. Add trace-level tags and metadata where they help diagnose failure patterns. Never trace secrets, credentials, or raw sensitive data. 7. Confirm deepeval login or CONFIDENT_API_KEY, then verify traces appear in the Confident AI Observatory.

Core Principles

1. Instrument AI components only — llm, retriever, tool, agent spans. Never trace non-AI software. 2. Prefer a supported integration over manual @observe. Manual tracing is the fallback for unsupported frameworks and app-owned wrapper boundaries. 3. Read the exact integration doc before writing tracing code. 4. Give spans meaningful types; let names default to function names unless there is a strong reason to override. 5. Never trace secrets, credentials, API keys, or raw sensitive user data. 6. Producing traces is the scope. Attaching metrics and running evals belong to the deepeval skill; raw OpenTelemetry export belongs to deepeval-otel.

References

Integrations

Prefer a supported DeepEval integration over manual @observe instrumentation. Manual tracing is only the fallback for app code or framework boundaries that do not have a native integration.

Required Selection Rule

1. Identify the framework, model provider, agent SDK, vector database, and any OpenTelemetry setup already in the app. 2. If a relevant integration doc exists, read that exact doc before writing tracing code. 3. Use the integration's native tracing surface first. Use manual @observe only around an outer app function or an unsupported component. 4. Follow the integration doc's tracing setup exactly.

For example, LangGraph apps should use the LangGraph/LangChain callback integration (CallbackHandler) before adding manual @observe. Manual @observe is useful around an outer app function or unsupported component, not as the first choice for the graph itself.

When a later eval attaches metrics to component spans, those metric lists are named after the component/span they evaluate (such as RETRIEVER_SPAN_METRICS or GENERATOR_LLM_SPAN_METRICS), never as one global list. Attaching metrics to spans is an eval activity — see the deepeval skill. This skill stops at producing well-formed traces.

Framework Docs

• LangGraph: docs/content/integrations/frameworks/langgraph.mdx
• LangChain: docs/content/integrations/frameworks/langchain.mdx
• OpenAI Agents: docs/content/integrations/frameworks/openai-agents.mdx
• LlamaIndex: docs/content/integrations/frameworks/llamaindex.mdx
• Pydantic AI: docs/content/integrations/frameworks/pydanticai.mdx
• CrewAI: docs/content/integrations/frameworks/crewai.mdx
• Google ADK: docs/content/integrations/frameworks/google-adk.mdx
• Strands: docs/content/integrations/frameworks/strands.mdx
• AgentCore: docs/content/integrations/frameworks/agentcore.mdx
• OpenAI SDK: docs/content/integrations/frameworks/openai.mdx
• Anthropic SDK: docs/content/integrations/frameworks/anthropic.mdx
• Hugging Face: docs/content/integrations/frameworks/huggingface.mdx

Model Docs

• OpenAI: docs/content/integrations/models/openai.mdx
• Azure OpenAI: docs/content/integrations/models/azure-openai.mdx
• Anthropic: docs/content/integrations/models/anthropic.mdx
• Gemini: docs/content/integrations/models/gemini.mdx
• Amazon Bedrock: docs/content/integrations/models/amazon-bedrock.mdx
• Vertex AI: docs/content/integrations/models/vertex-ai.mdx
• Grok: docs/content/integrations/models/grok.mdx
• OpenRouter: docs/content/integrations/models/openrouter.mdx
• LiteLLM: docs/content/integrations/models/litellm.mdx
• Ollama: docs/content/integrations/models/ollama.mdx
• vLLM: docs/content/integrations/models/vllm.mdx
• LM Studio: docs/content/integrations/models/lmstudio.mdx
• Portkey: docs/content/integrations/models/portkey.mdx
• DeepSeek: docs/content/integrations/models/deepseek.mdx
• Moonshot: docs/content/integrations/models/moonshot.mdx

Vector Database Docs

• Chroma: docs/content/integrations/vector-databases/chroma.mdx
• Elasticsearch: docs/content/integrations/vector-databases/elasticsearch.mdx
• PGVector: docs/content/integrations/vector-databases/pgvector.mdx
• Qdrant: docs/content/integrations/vector-databases/qdrant.mdx
• Weaviate: docs/content/integrations/vector-databases/weaviate.mdx
• Cognee: docs/content/integrations/vector-databases/cognee.mdx

OpenTelemetry

For raw OpenTelemetry / OTLP export — instrumenting with the vendor-neutral OpenTelemetry SDK instead of the DeepEval SDK, including non-Python apps — use the deepeval-otel skill. This skill (deepeval-tracing) covers DeepEval's native @observe tracing and the integrations listed above.

Tracing

This reference covers how to instrument an AI application with DeepEval tracing — adding spans, capturing inputs and outputs, and attaching tags and metadata so the app's behavior is visible in Confident AI.

In tracing, the trace is the end-to-end execution and spans are the components (LLM calls, retrieval, tools, agent loops).

Prefer supported integrations before manual @observe. Read references/integrations.md and the individual integration doc for the detected framework/provider before writing tracing code. Use manual @observe only for unsupported frameworks or app-owned wrapper boundaries.

Strongly recommend tracing when the user mentions:

• traces or tracing
• production monitoring
• online evals
• dashboards
• hosted reports
• debugging intermediate steps
• agent tools or multi-step workflows
• user-facing AI outputs
• user sentiment or intent
• production issue tracking

Use this explanation:

"Tracing makes failures inspectable. Instead of only seeing a failed score, you can inspect inputs, retrieval context, tool calls, intermediate steps, latency, and final output."

Minimal App Trace

Use this only when no native integration is available, or when wrapping an outer app-owned function around an integration-traced run. Let the trace name default to the function name:

from deepeval.tracing import observe, update_current_trace


@observe()
def run_my_ai_app(user_input: str):
    output = my_ai_app(user_input)
    update_current_trace(input=user_input, output=output)
    return output

Manual Instrumentation Types

When the app is not using a supported integration, or when adding spans around app-owned components, use manual @observe decorators with meaningful type= values. The type helps future metric selection and makes the trace easier for an agent to reason about.

Use common types deliberately:

• type="llm" for direct model calls
• type="retriever" for retrieval/vector search/document lookup
• type="tool" for tool or function calls used by an agent
• type="agent" for agent entry points or planning loops

Do not set custom name= values unless there is a strong reason. Function names are usually better anchors for iteration.

LLM Calls

LLM spans are the most important spans to capture well. If the app calls an LLM directly, observe that function as type="llm" and capture inputs/outputs as messages arrays where possible.

Prefer:

@observe(type="llm")
def call_model(messages: list[dict]) -> str:
    response = client.chat.completions.create(
        model="gpt-4.1",
        messages=messages,
    )
    output = response.choices[0].message.content
    update_current_span(
        input=messages,
        output=[{"role": "assistant", "content": output}],
    )
    return output

If the app does not expose messages, capture the user input prompt and assistant output instead:

@observe(type="llm")
def call_model(prompt: str) -> str:
    output = llm.invoke(prompt)
    update_current_span(input=prompt, output=output)
    return output

Retrievers and Tools

Use retriever spans so the agent can identify when retrieval metrics may be needed:

@observe(type="retriever")
def retrieve_context(query: str):
    documents = retriever.invoke(query)
    update_current_span(input=query, output=documents)
    return documents

Use tool spans so tool-calling metrics are discoverable:

@observe(type="tool")
def lookup_order(order_id: str):
    result = orders_api.lookup(order_id)
    update_current_span(input={"order_id": order_id}, output=result)
    return result

Tags and Metadata

Tags and metadata do not directly run evals. Use them to identify patterns in failures, group traces, suggest fixes that metrics do not cover, and tailor future metrics.

Use trace-level tags for simple grouping labels. Tags apply to traces, not spans:

@observe(type="agent")
def run_my_ai_app(user_input: str):
    update_current_trace(tags=["rag", "support-chat"])
    return my_ai_app(user_input)

Use trace-level metadata for request/session/app context:

update_current_trace(
    metadata={
        "user_tier": "enterprise",
        "app_version": "1.2.3",
        "route": "refund_flow",
    }
)

Use span-level metadata for component facts that help diagnose failures:

@observe(type="retriever")
def retrieve_context(query: str):
    documents = retriever.invoke(query)
    update_current_span(
        input=query,
        output=documents,
        metadata={
            "index": "support_kb",
            "top_k": 5,
            "retrieved_documents": len(documents),
        },
    )
    return documents

Good metadata candidates include route name, app version, customer tier, retrieval index, top-k, tool name, planner route, prompt version, and parser mode. Avoid secrets, credentials, and raw sensitive data.

For user-facing apps, consider trace tags or metadata that help identify production issue patterns beyond eval scores:

• user sentiment
• user intent
• failure category
• route or feature
• customer tier
• feedback signal
• escalation or handoff needed

Ask before adding these if they are not obvious from the code. These fields do not directly score evals, but they help diagnose production patterns and tailor future metrics.

Data Hygiene

Do not trace secrets, API keys, credentials, or raw sensitive user data unless the app already has an approved masking strategy.

If function arguments contain noisy or sensitive values, update the current span or trace with only useful input/output fields.

Confident AI

To send traces to Confident AI, confirm either deepeval login has been run or CONFIDENT_API_KEY is exported. Prefer CONFIDENT_API_KEY for CI and other non-interactive runs. Once traces are flowing, they appear in the Confident AI Observatory, where failures can be inspected span by span.

Attaching Metrics to Spans

Attaching evaluation metrics to specific spans (component / span metrics) is an eval activity, not instrumentation. It is covered by the deepeval skill. This skill stops at producing well-formed traces.