Deepeval Otel

DeepEval OpenTelemetry Export

Use this skill to instrument an AI application — an LLM app, agent, RAG pipeline, or chatbot — with raw OpenTelemetry so its traces land in Confident AI's Observatory. No deepeval package is needed — it works with any OTLP-capable OpenTelemetry SDK. The job is exactly two things: export to the correct Confident AI OTLP endpoint, and set the confident.* attributes Confident AI reads off each span.

Scope: AI Applications Only

This skill instruments AI applications only. The confident.* attributes and span types — agent, llm, retriever, tool — describe AI components, and Confident AI's Observatory is built to evaluate and monitor AI behavior.

Instrument only the AI parts of the system: agent loops and planning, LLM calls, retrieval / vector search, and tool calls. Do not apply confident.* attributes to non-AI software (web servers, CRUD backends, database layers, infrastructure) or to non-AI spans inside an otherwise-AI app — that data does not belong in Confident AI and will not render meaningfully. If the target has no LLM, agent, retrieval, or tool-calling component, this skill does not apply.

When to Use vs the deepeval Skill

Use this skill for vendor-neutral OTLP export to Confident AI — pointing an OpenTelemetry exporter at Confident AI and setting confident.* attributes.

Use the `deepeval` skill when the user wants to build a Python pytest eval suite, generate datasets or goldens, write metrics, run deepeval test run, or instrument with the deepeval SDK's @observe decorator. The two skills are complementary, not alternatives.

Prerequisites

• A Confident AI account and a CONFIDENT_API_KEY.
• An OpenTelemetry SDK for the application's language. For Python:

opentelemetry-sdk and opentelemetry-exporter-otlp-proto-http.

• The Confident AI OTLP endpoint accepts HTTP only — never gRPC.

How It Works

Confident AI exposes an OTLP/HTTP traces endpoint. Point any OpenTelemetry span exporter at it with the x-confident-api-key header. Confident AI's exporter then reads confident.* attributes off each span to build the trace and span structure. Parent/child nesting comes from native OpenTelemetry span context, not from any attribute.

Workflow

1. Confirm the target is an AI application (it has LLM calls, an agent loop, retrieval, or tool calls). If it has none of these, stop — this skill does not apply. Then inspect for an existing OpenTelemetry setup (a TracerProvider, span exporters, or an OpenTelemetry Collector) and prefer repointing what exists over adding a parallel pipeline. 2. Choose the endpoint from the API key's region prefix. Read references/endpoint-and-exporter.md. 3. Wire (or repoint) an OTLP/HTTP span exporter with the x-confident-api-key header. For Python, start from templates/confident_otel_setup.py. 4. If the process runs other OpenTelemetry instrumentation or an APM agent (auto-instrumentation for HTTP/DB, Datadog, etc.), isolate the Confident AI export so only AI spans reach it — a dedicated pipeline or a span filter. Read "Export Only AI Spans" in references/endpoint-and-exporter.md. 5. Set confident.span.* attributes on spans; set confident.trace.* for trace-wide fields. Read references/span-attributes.md and references/trace-attributes.md. 6. Honor the OTLP data-type rules: JSON-encode dicts/metadata, use native arrays for string lists. See the Data-Type Rules in span-attributes.md. 7. If the app already emits OpenTelemetry GenAI semantic conventions, read references/gen-ai-fallbacks.md before adding redundant attributes. 8. Verify traces appear in the Confident AI Observatory.

Core Principles

1. Instrument AI components only — agent, LLM, retriever, and tool spans. Never apply confident.* attributes to non-AI software or non-AI spans. 2. Export only AI spans. If the process has other OpenTelemetry instrumentation or an APM agent, isolate the Confident AI pipeline (a dedicated provider or a span filter) so non-AI spans — HTTP requests, DB queries, infra — are never exported to Confident AI. 3. Prefer repointing an existing OTLP exporter over adding a parallel one. 4. The confident.* attribute keys are the entire contract — they are the same in every language, so language choice is irrelevant. 5. Always use OTLP/HTTP. Confident AI's endpoint does not accept gRPC. 6. Honor OTLP data-type rules: attribute values must be primitives or homogeneous primitive lists; JSON-encode dicts and metadata. 7. Set confident.span.type explicitly when it is known; rely on gen_ai.* inference only as a fallback. 8. Never put secrets, credentials, or raw sensitive data into span attributes.

References

Templates

Endpoint and Exporter

Where to export OpenTelemetry traces and how to authenticate so they land in Confident AI's Observatory.

Endpoints

Confident AI exposes one OTLP/HTTP traces endpoint per region. There are exactly two.

When configuring an OTLP/HTTP span exporter directly, the endpoint value must include the /v1/traces suffix. When configuring through the standard OTEL_EXPORTER_OTLP_ENDPOINT environment variable, supply only the base endpoint — the SDK appends /v1/traces itself.

Choosing the Endpoint by API Key Region

Confident AI API keys are region-prefixed. Pick the endpoint from the prefix of the project's CONFIDENT_API_KEY:

Only confident_eu_… keys use the EU endpoint. When in doubt, ask the user for project region or use the default endpoint.

Authentication

Every request must carry the Confident AI API key in an HTTP header:

x-confident-api-key: <CONFIDENT_API_KEY>

Set this as a header on the OTLP exporter. Read the key from the CONFIDENT_API_KEY environment variable — never hardcode it into source.

Transport: HTTP Only

Confident AI's OTLP endpoint accepts OTLP/HTTP only — never gRPC.

• Python: use the HTTP exporter OTLPSpanExporter from

opentelemetry.exporter.otlp.proto.http.trace_exporter (package opentelemetry-exporter-otlp-proto-http). Do not use the opentelemetry.exporter.otlp.proto.grpc variant.

• OpenTelemetry Collector: use the otlphttp exporter, not otlp (gRPC).
• Other SDKs: choose the OTLP/HTTP exporter (proto-http, HttpProtobuf, or

the language's equivalent).

Standard OTel Environment Variables

The exporter honors standard OpenTelemetry environment variables, so the endpoint and headers can be configured without code changes:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://otel.confident-ai.com"
export OTEL_EXPORTER_OTLP_HEADERS="x-confident-api-key=<CONFIDENT_API_KEY>"

With env-var configuration the base endpoint is given; the SDK appends /v1/traces automatically.

Exporter Wiring (Python)

The minimal path:

1. Create a TracerProvider. 2. Attach a BatchSpanProcessor wrapping an OTLP/HTTP OTLPSpanExporter pointed at <endpoint>/v1/traces with the x-confident-api-key header. 3. Register the provider as the global tracer provider. 4. Get a tracer, start spans, and set confident.* attributes on them.

import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

api_key = os.environ["CONFIDENT_API_KEY"]
endpoint = (
    "https://eu.otel.confident-ai.com"
    if api_key.startswith("confident_eu_")
    else "https://otel.confident-ai.com"
)

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint=f"{endpoint}/v1/traces",
            headers={"x-confident-api-key": api_key},
        )
    )
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("my-llm-app") as span:
    span.set_attribute("confident.span.type", "agent")
    span.set_attribute("confident.trace.name", "my-llm-app")
    # ... see trace-attributes.md and span-attributes.md

See templates/confident_otel_setup.py for the full runnable version.

Other Languages

The wiring shape is identical in any OpenTelemetry SDK — only the class and package names differ. In every language:

1. Construct an OTLP/HTTP span exporter. 2. Set its endpoint/url to <region endpoint>/v1/traces. 3. Add the x-confident-api-key header. 4. Register it with a batch span processor on the tracer provider.

Then emit spans as usual and set the confident.* attributes documented in references/trace-attributes.md and references/span-attributes.md. The attribute keys are the entire contract — they are the same regardless of language.

Export Only AI Spans (Isolate From Other Instrumentation)

Most real applications run more OpenTelemetry instrumentation than just the AI code. Auto-instrumentation libraries and APM agents (Datadog, New Relic, Grafana, OpenTelemetry auto-instrumentation, etc.) emit spans for HTTP requests, database queries, cache calls, outbound network calls, and framework internals. This happens in any runtime — Node.js, Python, Java, Go, and others — not just one.

If the Confident AI exporter shares a tracer provider or processor pipeline with that instrumentation, all of those unrelated spans get shipped to Confident AI's Observatory, where they bury the AI trace in non-AI noise. Confident AI's Observatory is for AI behavior; only AI spans belong there.

Rule: the Confident AI export pipeline must carry AI spans only. Use one of two approaches.

Approach 1 — Dedicated pipeline (preferred when feasible)

Register the Confident AI exporter on a tracer provider / processor used only by the AI instrumentation, separate from the global provider that auto-instrumentation and APM agents feed. Create AI spans with that dedicated provider's tracer. Non-AI spans never reach the Confident AI exporter because they were never on its pipeline.

Approach 2 — Filter the pipeline

When AI spans and other spans unavoidably share a provider (common when an AI framework emits its spans onto the global provider), wrap the Confident AI-bound processor or exporter in a filter that forwards only AI spans and drops everything else.

Identify an AI span by any of:

• it has a confident.span.type attribute set; or
• it carries gen_ai.* semantic-convention attributes; or
• its span name matches a known AI-framework prefix (for example, the Vercel

AI SDK emits spans named ai.*).

Implement the filter as either:

• a span processor that no-ops onStart/onEnd for non-AI spans, so only

AI spans are handed to the underlying Confident AI processor; or

• an exporter wrapper that removes non-AI spans from each batch before

calling the real OTLP exporter.

A working reference implementation is the deepeval TypeScript SDK's DeepEvalBatchFilterProcessor (a name-prefix span-processor filter) and DeepEvalExporterWrapper (an exporter wrapper) in deepeval.ts/src/integrations/ai-sdk/index.ts. Mirror that shape in whatever language and SDK the app uses.

Caveat — preserve span nesting when filtering. Dropping an intermediate non-AI span can orphan its AI child spans (their parentSpanId now points at a span that was never exported). When filtering, re-parent orphaned AI spans onto the nearest exported ancestor, or strip the dangling parent reference so the child becomes a clean root. The DeepEvalExporterWrapper above does exactly this for root spans.

GenAI Semantic-Convention Fallbacks

When a confident.* attribute is absent, Confident AI's exporter falls back to the standard OpenTelemetry GenAI semantic-convention attributes (gen_ai.*). This matters when the app is already instrumented by a GenAI-aware library that emits gen_ai.* spans — those spans carry useful data into Confident AI without any extra confident.* attributes.

Read this file only when the app already produces gen_ai.* spans. For new instrumentation, set confident.* attributes directly (see span-attributes.md).

Span-Type Inference

If confident.span.type is not set, the span type is inferred:

Attribute Fallback Table

When the confident.* key is absent, the exporter reads the gen_ai.* key:

Guidance

• confident.* attributes always win when both are present.
• For new instrumentation, prefer explicit confident.* attributes — they map

directly and unambiguously.

• Rely on fallbacks only to avoid duplicating attributes an existing GenAI

integration already emits. Do not add confident.* copies of gen_ai.* attributes the app already sets unless you need to override them.

Span-Level Attributes

Span-level attributes describe one span — a single component of the trace. Set them as confident.span.* (and the per-type confident.llm.*, confident.agent.*, confident.retriever.*, confident.tool.*) attributes on that span.

confident.span.type selects which per-type keys are meaningful. Set it first.

The span types — llm, tool, agent, retriever, and the generic base — describe AI-application components. Apply confident.* attributes only to the AI parts of the system (LLM calls, agent loops, retrieval, tool calls), not to non-AI spans such as HTTP handlers, database queries, or infrastructure.

Common Span Attributes

Valid on every span type:

The allowed values for confident.span.type are exactly llm, tool, agent, retriever, and base. base is a generic span with no extra typed fields.

Span Errors

Span errors are not a confident.* attribute. Use the native OpenTelemetry span Status:

from opentelemetry.trace import Status, StatusCode

try:
    ...
except Exception as e:
    span.set_status(Status(StatusCode.ERROR), str(e))
    span.record_exception(e)

A span with StatusCode.ERROR is rendered as errored in the Observatory; if it is the root span, the whole trace is marked errored.

LLM Spans

Set confident.span.type to llm.

If the span references a prompt managed in Confident AI, set the discrete prompt fields:

Set only the prompt fields that apply.

Agent Spans

Set confident.span.type to agent.

Retriever Spans

Set confident.span.type to retriever.

Put the retrieved chunks on confident.span.retrieval_context.

Tool Spans

Set confident.span.type to tool.

Put the tool's arguments on confident.span.input and its result on confident.span.output.

Data-Type Rules

OpenTelemetry attribute values must be primitives (string, bool, int, float) or homogeneous lists of primitives. There is no map/object attribute type. Encode accordingly:

• Objects / dicts (confident.span.metadata, confident.trace.metadata)

must be JSON-encoded strings — json.dumps(...).

• String lists (tags, context, retrieval_context,

available_tools, agent_handoffs) can be native OTLP string arrays (a Python list/tuple of str). A JSON array string is also accepted.

• `ToolCall` lists (tools_called, expected_tools) must be a **native

OTLP list where each element is a JSON-serialized ToolCall string** — i.e. a list of JSON strings, not one JSON string of a list.

• `input` / `output` are passthrough. If the value is not already a

string, JSON-encode it before setting.

• Numbers (top_k, chunk_size, token counts, costs) are set as native

int/float, not strings.

Span Nesting

Parent/child relationships come from native OpenTelemetry span context — start a child span inside the parent span's context. There is no confident.* attribute for parenthood. With tracer.start_as_current_span(...), spans opened inside the with block are automatically nested under it.

Trace-Level Attributes

Trace-level attributes describe the whole trace, not a single span. Set them as confident.trace.* attributes on any span in the trace — most naturally on the root span. Confident AI aggregates them up to the trace.

A trace is one end-to-end execution; its spans are the components. Span-level fields are documented separately in span-attributes.md.

Attribute Table

All attributes are optional. Set only what is meaningful for the app.

Environment Resolution

confident.trace.environment accepts a deployment environment string (commonly "production", "staging", "development", or "testing"). The default is "production".

It can be set in two places, and the Resource attribute wins over the span attribute:

• As a span attribute: confident.trace.environment on a span.
• As an OpenTelemetry Resource attribute confident.trace.environment on

the TracerProvider's Resource. This is the recommended way to stamp the environment once for an entire process.

Data-Type Rules

The encoding rules for objects (JSON strings) versus lists (native OTLP arrays) versus ToolCall lists apply identically to trace-level and span-level attributes. They are detailed once, in span-attributes.md under Data-Type Rules — read that section before encoding tags, metadata, context, retrieval_context, tools_called, or expected_tools.