Otel Tracing

OTel Tracing

Create spans for logical operations, propagate context across every boundary, use semantic conventions for attribute names. Every tracing decision — span granularity, attribute selection, sampling strategy — trades off cost against visibility.

References

Spans

A span represents a single unit of work in a trace — an HTTP request handler, a database query, a message publish. Not every function call.

Span Naming

Name spans for the class of operation, not the instance. Low-cardinality names enable aggregation; high-cardinality names destroy it.

• Use the most general string that identifies a statistically interesting class
• Never embed high-cardinality values (IDs, emails, paths with IDs)
• HTTP span names: {METHOD} {route} — server: GET /users/:id; client: GET
• Database span names: {operation} {table} — SELECT users, INSERT orders

SpanKind

SpanKind tells backends how to assemble the trace tree. Set it correctly.

• A single span MUST NOT serve more than one purpose
• Create a new span before injecting context for outgoing calls
• CLIENT -> SERVER for synchronous calls; PRODUCER -> CONSUMER for async
• Using INTERNAL for HTTP handlers or DB calls is wrong — use SERVER or CLIENT

Span Status

• Leave status Unset on success — it already means "no error"
• Ok is a "final call" — once set, subsequent Error attempts are ignored
• Instrumentation libraries SHOULD NOT set Ok — leave to application code

Span Lifetime

• Every created span MUST be ended — leaked spans cause memory issues and

incomplete traces

• After end(), the span becomes non-recording; further mutations are ignored
• Use language-idiomatic patterns: defer (Go), try/finally (Java/JS),

context manager (Python)

Span Data: Attributes, Events, Links

Attributes

Key-value pairs annotating a span. Value types: string, boolean, integer, float, or arrays of these.

• Use semantic conventions for attribute names — http.request.method, db.system,

not custom names. Consistent naming enables cross-service analysis.

• Add sampling-relevant attributes at span creation — samplers can only see attributes

present at creation time.

• Keep cardinality low — attribute values should be from a bounded set.
• Check `IsRecording` before expensive attribute computation — sampled-out spans

discard all data.

• Never store PII in attributes — trace data flows to shared backends. Use opaque

identifiers.

• Respect attribute limits — SDK enforces AttributeCountLimit (default 128). Excess

attributes are silently dropped.

Events

Timestamped annotations — structured log entries attached to a span.

• Use events when the timestamp matters; use attributes for metadata with no meaningful

timestamp

• Add events for key domain occurrences ("page became interactive", "retry attempted")
• Use events for verbose data instead of additional spans

Recording Exceptions

• Always pair `RecordException` with `setStatus(ERROR)` — RecordException alone does

not change span status

• Never swallow application exceptions in instrumentation — catch instrumentation errors,

log them, but always rethrow application exceptions

Links

Associate a span with spans from other traces without parent-child hierarchy.

• Batch processing: link consumer span to each producer span
• Async follow-up: job span links back to triggering span
• Add links at span creation when possible — samplers can consider them

See ${CLAUDE_SKILL_DIR}/references/span-data.md for events format, links format, and SDK limits table.

Context Propagation

Context propagation correlates spans across service boundaries. Without it, each service produces isolated spans — no distributed trace.

Inject / Extract Pattern

1. Sender creates a span, makes it current, injects context into outgoing carrier (HTTP headers, message metadata) 2. Receiver extracts context from incoming carrier and uses it as parent for new spans

• Call inject AFTER creating the outgoing CLIENT or PRODUCER span
• Call extract BEFORE creating the server/consumer span
• Use W3C TraceContext as the default propagation format

In-Process Propagation

• Make spans active/current — enables automatic log correlation, nested

auto-instrumentation picking up correct parent, context propagation to child ops

• Pass context explicitly in async code — automatic propagation may break across

thread boundaries, goroutines, or async callbacks

• Never propagate request context to background work — background tasks should start

new traces and link back to the triggering span

Baggage

Propagates arbitrary key-value pairs across service boundaries alongside trace context.

• Use for: tenant ID, request priority, feature flags, sampling hints
• NEVER put PII, credentials, or API keys in baggage — visible to all downstream

services

See ${CLAUDE_SKILL_DIR}/references/context-propagation.md for W3C header format, propagator selection table, baggage details, and security considerations.

Instrumentation

Approaches

Getting a Tracer

• Name the tracer after your library, package, or module — not the application
• Include a version string matching your library version
• Libraries: accept TracerProvider via dependency injection or use the global one
• Applications: configure the SDK and set the global TracerProvider

What to Instrument

Good candidates: public API methods with I/O or significant computation, request/message handlers, outbound calls (HTTP, DB, RPC), background jobs.

Poor candidates: every function call (noise), thin wrapper libraries (already instrumented underneath), pure computation with no I/O and sub-ms duration.

Decision: Is it a network call or significant I/O? -> Create span (CLIENT/SERVER). Is it a meaningful business operation? -> Create span (INTERNAL). Would a span event on parent suffice? -> Add event. Otherwise -> don't instrument.

Library Instrumentation Rules

• Depend on OpenTelemetry API only — never SDK. The API is a no-op without SDK, so

zero overhead for users who don't use OTel.

• Follow semantic conventions for your domain
• Set the schema_url to record which semantic convention version you use
• Prefer events over spans for verbose internal details
• Support optional TracerProvider injection for testability

See ${CLAUDE_SKILL_DIR}/references/instrumentation.md for server-side, client-side, and async producer/consumer patterns, plus testing guidance.

Sampling

Sampling controls which traces are recorded and exported — the primary mechanism for managing tracing costs.

Head Sampling

Decision at trace creation time, before any spans complete.

ParentBased is the most common production configuration — respects parent decisions, applies custom root sampling. Default SDK sampler: ParentBased(root=AlwaysOn).

Tail Sampling

Decision after all spans in a trace complete. Requires collector infrastructure.

• All services export at 100%
• OTel Collector with tail sampling processor buffers spans by trace ID
• All spans with the same trace ID MUST reach the same collector — use trace-ID-aware

load balancing

Sampling Principles

• Start with no sampling — add only when cost or volume requires it
• Always sample errors — configure tail sampling to keep error traces
• Use ParentBased — children follow parent decisions for complete traces
• Provide attributes at span creation — samplers cannot see late-added attributes
• Filter health checks — high volume, low value; sample aggressively or filter entirely

See ${CLAUDE_SKILL_DIR}/references/sampling.md for combined head+tail strategies, decision guide, and per-scenario recommendations.

Semantic Conventions

Use semantic conventions for span names and attributes. Consistent naming enables cross-service analysis without learning custom attribute names.

• HTTP server spans: kind SERVER, name {METHOD} {http.route}, never full URI path
• HTTP client spans: kind CLIENT, name {METHOD}
• Database spans: kind CLIENT, name {operation} {target}, always sanitize queries
• Messaging producer: kind PRODUCER, name {destination} publish
• Messaging consumer: kind CONSUMER, name {destination} process

See ${CLAUDE_SKILL_DIR}/references/semantic-conventions.md for required/recommended attributes per domain, status mapping rules, and general conventions.

SDK Components

TracerProvider

• Initialize once, early in application startup
• Always call shutdown() on exit — flushes remaining spans
• Configure Resource with service.name — identifies your service in backends
• Use BatchSpanProcessor in production — SimpleSpanProcessor is for dev/testing only

See ${CLAUDE_SKILL_DIR}/references/sdk-components.md for Resource attributes, BatchSpanProcessor tuning, exporter types, environment variable configuration, and Collector deployment patterns.

Application

When writing tracing code:

• Apply all conventions silently — don't narrate each rule being followed.
• Use semantic conventions for attribute names. Check the reference for your domain

(HTTP, DB, messaging) before inventing names.

• If an existing codebase contradicts a convention, follow the codebase and flag the

divergence once.

When reviewing tracing code:

• Cite the specific violation and show the fix inline.
• Focus on: span naming cardinality, missing context propagation, missing error

recording, wrong SpanKind, missing span.end().

Integration

The coding skill governs workflow; this skill governs tracing implementation choices. Language-specific skills handle SDK API differences.

Context Propagation

Context propagation is the mechanism that correlates spans across service boundaries. Without it, each service produces isolated spans — no distributed trace.

How It Works

1. Sender creates a span, makes it current, and injects context into the outgoing carrier (HTTP headers, message metadata) 2. Carrier transports the serialized context across the network 3. Receiver extracts context from the incoming carrier and uses it as the parent for new spans

Service A                    Network               Service B
─────────                    ───────               ─────────
Create span
  │
  ├─ inject(context, headers)
  │        ────────────────────►
  │         traceparent: 00-{traceId}-{spanId}-01
  │                                          extract(context, headers)
  │                                                 │
  │                                          Create child span
  │                                          (parent = extracted context)

W3C Trace Context

The default propagation format. Uses two HTTP headers:

traceparent

{version}-{trace-id}-{parent-id}-{trace-flags}
00-a0892f3577b34da6a3ce929d0e0e4736-f03067aa0ba902b7-01

tracestate

Vendor-specific key-value pairs. Carried alongside traceparent for multi-vendor interoperability. OpenTelemetry uses ot= prefix for its own entries (e.g., sampling threshold).

Propagator API

Inject

Serialize context into a carrier for outgoing requests.

// Pseudocode
propagator.inject(context, carrier, setter)

• Call inject AFTER creating the outgoing CLIENT or PRODUCER span
• The span must be current/active so the propagator can read its SpanContext
• The setter writes key-value pairs into the carrier (e.g., HTTP headers)

Extract

Deserialize context from an incoming carrier.

// Pseudocode
extractedContext = propagator.extract(context, carrier, getter)

• Call extract BEFORE creating the server/consumer span
• Use the extracted context as parent for the new span
• The getter reads key-value pairs from the carrier

Composite Propagator

Combine multiple propagators (e.g., W3C Trace Context + Baggage):

propagator = CompositeTextMapPropagator([
    TraceContextPropagator(),
    BaggagePropagator()
])

In-Process Propagation

Within a single process, context flows through the "current" or "active" span:

1. Make spans active/current — this enables:

• Automatic log correlation (trace ID in log entries)
• Nested auto-instrumentation picking up the correct parent
• Context propagation to child operations

2. Capture context early on public APIs — active context may change during callbacks or async operations. Capture it at the API boundary.

3. Pass context explicitly in async code — automatic context propagation may break across thread boundaries, goroutines, or async callbacks. Pass the context object explicitly when starting background work.

Baggage

Baggage propagates arbitrary key-value pairs across service boundaries alongside trace context. Unlike span attributes, baggage crosses process boundaries.

Use cases:

• Propagate tenant ID, request priority, feature flags
• Share sampling decisions or routing hints

Security rules:

• NEVER put PII, credentials, or API keys in baggage
• Baggage is visible to all downstream services
• Baggage is often logged and may be sent to untrusted services

Security Considerations

Incoming Context from External Sources

• Malicious actors can send forged trace headers
• Consider ignoring or sanitizing context from untrusted sources
• Validate that trace IDs and span IDs are well-formed

Outgoing Context to External Services

• Internal trace IDs may reveal architectural information
• Configure propagators to suppress context to external/public endpoints
• Review what baggage values are being sent downstream

Propagator Selection

Instrumentation

Instrumentation is how you add tracing to your code. OpenTelemetry supports three approaches: automatic, manual (code-based), and hybrid.

Instrumentation Approaches

Automatic Instrumentation

Library hooks or monkey-patching that intercept calls to known libraries (HTTP clients, database drivers, messaging systems) and create spans automatically.

Advantages:

• Zero code changes for supported libraries
• Consistent span naming and attribute population
• Automatic context propagation

Limitations:

• Only covers supported libraries
• Cannot add business-specific attributes
• May not work in all environments (e.g., some serverless runtimes)

Manual (Code-Based) Instrumentation

Explicit span creation using the OpenTelemetry SDK.

When to use:

• Business-critical operations with custom attributes
• Libraries not covered by auto-instrumentation
• Fine-grained control over span boundaries and attributes

Hybrid (Recommended for Production)

Combine automatic instrumentation for infrastructure spans with manual instrumentation for business logic. This provides:

• Automatic context propagation and correlation
• Infrastructure visibility without code changes
• Business-specific spans where they matter

Getting a Tracer

// Pseudocode — pattern is the same across languages
tracer = tracerProvider.getTracer(
    "com.example.my-service",  // instrumentation scope name
    "1.0.0"                     // version
)

Rules: 1. Name the tracer after your library, package, or module — not the application 2. Include a version string matching your library version 3. The tracer name appears in telemetry and helps debug instrumentation issues 4. Libraries: accept TracerProvider via dependency injection or use the global one 5. Applications: configure the SDK and set the global TracerProvider

What to Instrument

Good Candidates for Spans

1. Public API methods that do I/O or significant computation 2. Request/message handlers — the entry point for external work 3. Outbound calls — HTTP requests, database queries, RPC calls 4. Background jobs — scheduled tasks, queue consumers

Poor Candidates for Spans

1. Every function call — creates excessive noise and overhead 2. Thin wrapper libraries — if the underlying call is already instrumented 3. Pure computation with no I/O and sub-millisecond duration 4. Getters/setters — trivial accessors add no observability value

Decision Guide

Is this a network call or significant I/O?
  └─ Yes → Create a span (CLIENT or SERVER kind)
  └─ No
      Is this a meaningful business operation?
        └─ Yes → Create a span (INTERNAL kind)
        └─ No
            Would a span event on the parent suffice?
              └─ Yes → Add an event instead
              └─ No → Don't instrument

Instrumentation Patterns

Server-Side (Incoming Request)

// 1. Extract context from incoming request
extractedContext = propagator.extract(context, request, getter)

// 2. Create server span with extracted context as parent
span = tracer.startSpan("GET /users/{id}",
    kind: SERVER,
    parent: extractedContext,
    attributes: { "http.request.method": "GET", "url.path": path }
)

// 3. Make span active for nested instrumentation
try (scope = span.makeCurrent()) {
    result = handleRequest(request)
    span.setAttribute("http.response.status_code", result.status)
} catch (error) {
    span.recordException(error)
    span.setStatus(ERROR, error.message)
    throw error
} finally {
    span.end()
}

Client-Side (Outgoing Request)

// 1. Create client span
span = tracer.startSpan("GET",
    kind: CLIENT,
    attributes: { "http.request.method": "GET", "server.address": host }
)

// 2. Inject context into outgoing request
try (scope = span.makeCurrent()) {
    propagator.inject(context, request, setter)
    response = httpClient.send(request)
    span.setAttribute("http.response.status_code", response.status)
} catch (error) {
    span.recordException(error)
    span.setStatus(ERROR, error.message)
    throw error
} finally {
    span.end()
}

Async Producer/Consumer

// Producer: create span and inject context into message
span = tracer.startSpan("send",
    kind: PRODUCER,
    attributes: { "messaging.destination.name": queue }
)
try (scope = span.makeCurrent()) {
    propagator.inject(context, message.headers, setter)
    queue.send(message)
} finally {
    span.end()
}

// Consumer: extract context and link or parent
extractedContext = propagator.extract(context, message.headers, getter)
span = tracer.startSpan("process",
    kind: CONSUMER,
    links: [Link(extractedContext)],  // or parent: extractedContext
    attributes: { "messaging.destination.name": queue }
)

Batch Processing with Links

When a consumer processes multiple messages at once, link to each producer:

links = messages.map(msg =>
    Link(propagator.extract(context, msg.headers, getter))
)
span = tracer.startSpan("process_batch",
    kind: CONSUMER,
    links: links
)

Library Instrumentation Rules

When adding tracing to a library (not an application):

1. Depend on OpenTelemetry API only — never the SDK. The API is a no-op without SDK, so your library has zero overhead for users who don't use OTel. 2. Use the earliest stable API version (1.0.) to minimize dependency conflicts 3. Follow semantic conventions for your domain (HTTP, DB, messaging) 4. Set the `schema_url` to record which semantic convention version you use 5. Don't create spans for thin wrappers — instrument at the logical level, not the network level 6. Prefer events over spans for verbose internal details 7. Support optional TracerProvider injection* for testability

Testing Instrumentation

1. Use a mock or in-memory SpanExporter to capture spans in tests 2. Verify: span names, kinds, attributes, status, parent-child relationships 3. Test with auto-instrumentation enabled to check for span duplication 4. Verify context propagation across service boundaries in integration tests

Sampling

Sampling controls which traces are recorded and exported. It is the primary mechanism for managing tracing costs without losing visibility.

Why Sample

• Cost control — at high throughput, 100% collection is expensive
• Focus on interesting traces — errors, high latency, specific users
• Reduce noise — health checks, synthetic traffic, routine operations
• Representativeness — a small sample can accurately represent the whole

When Not to Sample

• Low-volume services (tens of traces per second)
• Regulatory requirements that prohibit dropping data
• When pre-aggregation is sufficient (metrics, not traces)
• When the cost of implementing sampling exceeds storage savings

Head Sampling

Decision made at trace creation time, before any spans complete.

How It Works

The sampler evaluates the trace ID, span name, initial attributes, and parent context to decide: record, record-and-sample, or drop.

Built-in Head Samplers

ParentBased Sampler

The most common production configuration. Respects parent sampling decisions while allowing custom root sampling:

ParentBased(
    root = TraceIdRatioBased(0.1),           // 10% of root spans
    remoteParentSampled = AlwaysOn,          // respect sampled parent
    remoteParentNotSampled = AlwaysOff,      // respect unsampled parent
    localParentSampled = AlwaysOn,           // respect local sampled
    localParentNotSampled = AlwaysOff        // respect local unsampled
)

Default SDK sampler: ParentBased(root=AlwaysOn) — samples everything but respects parent decisions.

Head Sampling Trade-offs

Tail Sampling

Decision made after all (or most) spans in a trace have completed. Requires collecting all spans before deciding.

How It Works

1. All services export spans at 100% (or use AlwaysOn sampler) 2. An OpenTelemetry Collector with tail sampling processor collects spans 3. The processor buffers spans by trace ID, waits for completion 4. Sampling policies evaluate complete traces and decide keep/drop

Common Tail Sampling Policies

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    policies:
      - name: errors-policy
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-traces
        type: latency
        latency: { threshold_ms: 5000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

Tail Sampling Architecture

┌─────────┐     ┌─────────┐     ┌─────────────────────┐     ┌─────────┐
│Service A │────►│Load Bal.│────►│ Collector             │────►│ Backend │
│Service B │────►│Exporter │────►│ (tail sampling proc.) │────►│         │
│Service C │────►│         │────►│                       │────►│         │
└─────────┘     └─────────┘     └─────────────────────┘     └─────────┘

Critical: All spans with the same trace ID MUST reach the same collector instance. Use a load-balancing exporter that routes by trace ID.

Tail Sampling Trade-offs

Combined Head + Tail Sampling

For very high-volume systems:

1. Head sampling at the SDK level reduces volume (e.g., 10% baseline) 2. Tail sampling at the Collector further refines (keep all errors from the 10%)

This protects the pipeline from overload while still capturing interesting traces.

Sampling Decision Guidance

Key Principles

1. Start with no sampling — add it only when cost or volume requires it 2. Always sample errors — configure tail sampling to keep error traces 3. Use ParentBased — children should follow parent decisions for complete traces 4. Provide attributes at span creation — samplers cannot see late-added attributes 5. Monitor your samplers — dropped traces mean lost visibility; track drop rates 6. Filter health checks — they're high volume, low value; sample aggressively or filter entirely

SDK Components

The OpenTelemetry SDK implements the tracing API and provides the pipeline for processing and exporting spans. Understanding the SDK architecture is essential for configuring tracing in applications.

Architecture Overview

Application Code
       │
       ▼
   TracerProvider ──► Tracer ──► Span
       │
       ▼
   SpanProcessor(s)
       │
       ▼
   SpanExporter(s) ──► Backend (Jaeger, OTLP Collector, etc.)

TracerProvider

The entry point for tracing. Holds all configuration and creates Tracers.

Responsibilities:

• Factory for Tracer instances
• Owns SpanProcessors, Sampler, IdGenerator, and SpanLimits
• Manages lifecycle (initialization, shutdown, flush)

Configuration:

// Pseudocode
provider = TracerProvider(
    sampler: ParentBased(root: TraceIdRatioBased(0.1)),
    spanProcessors: [BatchSpanProcessor(OTLPExporter())],
    resource: Resource(service.name: "my-service", service.version: "1.0"),
    spanLimits: SpanLimits(attributeCountLimit: 128)
)

// Set as global
setGlobalTracerProvider(provider)

Lifecycle: 1. Create and configure at application startup 2. Register as global provider 3. Call shutdown() at application exit — flushes remaining spans 4. After shutdown, returns no-op Tracers

Rules: 1. Initialize once, early in application startup 2. One provider per application (unless testing) 3. Always call shutdown() on exit — losing final spans is common 4. Configure Resource with service.name — this identifies your service in backends

Tracer

Created by TracerProvider. Responsible for creating spans.

tracer = provider.getTracer("com.example.my-library", "1.0.0")
span = tracer.startSpan("operation-name")

Rules: 1. Name the tracer after the instrumentation scope (library/package name) 2. Include the version 3. Tracers are lightweight — don't cache aggressively, but don't create per-request 4. Configuration changes on the provider apply to all existing Tracers

SpanProcessor

Hooks into span lifecycle for processing. Receives spans at creation (OnStart) and completion (OnEnd).

Simple Span Processor

Exports each span synchronously when it ends. Development/testing only.

SimpleSpanProcessor(exporter)

• Blocks the application thread during export
• No batching — one export call per span
• Useful for debugging: spans appear immediately

Batch Span Processor

Buffers spans and exports in batches. Use in production.

BatchSpanProcessor(
    exporter: OTLPExporter(),
    maxQueueSize: 2048,          // buffer capacity
    scheduledDelayMillis: 5000,  // max delay between exports
    maxExportBatchSize: 512,     // spans per batch
    exportTimeoutMillis: 30000   // export timeout
)

Behavior:

• Exports when batch reaches maxExportBatchSize OR scheduledDelayMillis elapses
• Drops spans when queue is full (maxQueueSize)
• Exports remaining spans on shutdown() and forceFlush()

Tuning guidance:

Multiple Processors

Register multiple processors for different purposes:

provider = TracerProvider(
    spanProcessors: [
        BatchSpanProcessor(OTLPExporter()),        // export to backend
        SimpleSpanProcessor(ConsoleExporter())      // debug output
    ]
)

Processors are invoked in registration order.

SpanExporter

Serializes and transmits spans to a backend. The exporter is the end of the processing pipeline.

Common Exporters

OTLP is the recommended exporter for production. Send to an OpenTelemetry Collector which then routes to your backend(s).

Exporter Interface

interface SpanExporter {
    export(batch: Span[]) → Success | Failure
    shutdown()
    forceFlush()
}

• export() must not block indefinitely — has a timeout
• Retry logic is the exporter's responsibility
• After shutdown(), export() returns Failure

Resource

A Resource describes the entity producing telemetry. Attached to all spans from the provider.

Essential resource attributes:

Rules: 1. Always set service.name — backends use it to group traces 2. Set service.version for version-aware debugging 3. Resource is immutable after provider creation

Configuration Patterns

Minimal Production Setup

resource = Resource(
    service.name: "my-service",
    service.version: "1.0.0",
    deployment.environment.name: "production"
)

exporter = OTLPExporter(endpoint: "http://collector:4317")
processor = BatchSpanProcessor(exporter)
sampler = ParentBased(root: TraceIdRatioBased(0.1))

provider = TracerProvider(
    resource: resource,
    sampler: sampler,
    spanProcessors: [processor]
)
setGlobalTracerProvider(provider)

Development Setup

provider = TracerProvider(
    resource: Resource(service.name: "my-service"),
    sampler: AlwaysOn,
    spanProcessors: [SimpleSpanProcessor(ConsoleExporter())]
)

Environment Variable Configuration

Many SDKs support configuration via environment variables:

OpenTelemetry Collector

A standalone process that receives, processes, and exports telemetry. Decouples applications from backends.

Benefits:

• Central configuration for sampling, filtering, and routing
• Backend changes don't require application redeployment
• Supports tail sampling, attribute processing, and enrichment
• Can fan-out to multiple backends simultaneously

Typical deployment:

┌─────────┐     ┌───────────┐     ┌─────────┐
│ App      │────►│ Collector │────►│ Backend │
│ (SDK)    │OTLP │           │OTLP │ (Jaeger)│
└─────────┘     │ - sampling│     └─────────┘
                │ - filtering│────►│ Metrics │
                │ - routing  │     └─────────┘
                └───────────┘

Collector deployment patterns:

• Sidecar — one collector per application instance (agent mode)
• Gateway — shared collector(s) for multiple services
• Agent + Gateway — local agents forward to central gateway

Semantic Conventions

Semantic conventions standardize span names, attributes, and status across languages and libraries. Using conventions enables cross-service analysis without learning each library's custom attribute names.

Why Conventions Matter

• Consistency — all HTTP spans use the same attribute names regardless of language
• Tooling — backends build dashboards and alerts on well-known attribute names
• Correlation — standardized names allow joining spans from different services
• Vendor neutrality — no proprietary attribute schemas

HTTP Spans

HTTP Server Spans

Span kind: SERVER

Span name: {METHOD} {http.route} (e.g., GET /users/:id)

• If no route available: {METHOD} (e.g., GET)
• Never use the full URI path as span name

Required attributes:

Conditionally required:

Recommended:

Status rules:

• 1xx, 2xx, 3xx → leave Unset (unless other error like network failure)
• 4xx → leave Unset for server spans (client's problem)
• 5xx → set Error
• Don't set status description if http.response.status_code explains it

HTTP Client Spans

Span kind: CLIENT

Span name: {METHOD} (e.g., GET)

• Add {url.template} if available: GET /users/{id}

Required attributes:

Status rules:

• 4xx → set Error for client spans (server rejected our request)
• 5xx → set Error

HTTP Span Example

Client span:

name: "GET"
kind: CLIENT
attributes:
  http.request.method: "GET"
  url.full: "https://example.com:8080/webshop/articles/4?s=1"
  server.address: "example.com"
  server.port: 8080
  http.response.status_code: 200
  network.protocol.version: "1.1"

Corresponding server span:

name: "GET /webshop/articles/:article_id"
kind: SERVER
attributes:
  http.request.method: "GET"
  url.path: "/webshop/articles/4"
  url.query: "s=1"
  url.scheme: "https"
  http.route: "/webshop/articles/:article_id"
  server.address: "example.com"
  server.port: 8080
  http.response.status_code: 200
  client.address: "192.0.2.4"

Database Spans

Span kind: CLIENT

Span name: {operation} {target} (e.g., SELECT users)

Key attributes:

Rules:

• Always sanitize queries — replace parameter values with ? or $N
• db.query.text is opt-in due to potential sensitivity
• Use db.operation.name even when the full query is not recorded

Messaging Spans

Producer Spans

Span kind: PRODUCER

Span name: {destination} publish (e.g., orders publish)

Consumer Spans

Span kind: CONSUMER

Span name: {destination} process (e.g., orders process)

For batch consumers: Create one span per batch with links to each producer span.

General Conventions

Error Reporting

Set error.type when the operation ends with an error. The value should be predictable and low-cardinality.

Network Attributes

Server/Client Attributes

Sampling-Critical Attributes

These attributes SHOULD be provided at span creation time because samplers can only see attributes present at creation:

HTTP server: http.request.method, url.path, url.scheme, server.address, server.port, client.address, user_agent.original

HTTP client: http.request.method, url.full, server.address, server.port

Span Data: Attributes, Events, and Links

Spans carry three kinds of additional data: attributes for metadata, events for timestamped annotations, and links for cross-trace references.

Attributes

Attributes are key-value pairs that annotate a span with metadata about the operation.

Value types: string, boolean, integer, float, or arrays of these types. Keys must be non-null strings.

When to Add Attributes

1. At span creation — preferred. Samplers can only see attributes present at creation time. Add sampling-relevant attributes here. 2. After creation — for attributes only available after the operation starts (e.g., response status code).

Attribute Best Practices

1. Use semantic conventions. http.request.method, db.system, not custom names. Consistent naming enables cross-service analysis. 2. Keep cardinality low. Attribute values should be from a bounded set. http.request.method = "GET" is good; user.email = "john@example.com" creates unbounded cardinality. 3. Check `IsRecording` before expensive computation. If the span is not recording (sampled out), attribute computation is wasted work:

   if span.IsRecording() {
       span.SetAttribute("db.statement", sanitize(query))
   }

4. Never store PII in attributes. Trace data often flows to shared backends. Use opaque identifiers instead of names, emails, or addresses. 5. Respect attribute limits. SDKs enforce AttributeCountLimit (default 128). Excess attributes are silently dropped.

Common Attribute Patterns

Events

Span events are timestamped annotations — structured log entries attached to a span. They represent meaningful, singular points in time during a span's duration.

When to Use Events vs Attributes

Event Best Practices

1. Events have a name, timestamp, and optional attributes 2. Attach events to the span your instrumentation created, not the active span 3. Events preserve insertion order 4. Use events for verbose data instead of additional spans 5. The exception event is a special semantic convention (see RecordException below)

Recording Exceptions

RecordException is a specialized event for recording exceptions:

span.recordException(error)
span.setStatus(StatusCode.ERROR, error.message)

This creates an event with semantic convention attributes:

• exception.type — the exception class name
• exception.message — the error message
• exception.stacktrace — the stack trace string

Always pair `RecordException` with `setStatus(ERROR)`. RecordException alone does not change the span status.

Links

Links associate a span with one or more spans from the same or different traces, implying a causal relationship without a parent-child hierarchy.

When to Use Links

1. Batch processing — A consumer span processes multiple messages; link to each producer span 2. Async follow-up — An operation triggers a deferred job; the job span links back to the triggering span 3. Fan-in — A span aggregates results from multiple upstream spans

Link Best Practices

1. Add links at span creation when possible — samplers can consider them 2. Links added after creation may not influence sampling decisions 3. Links carry a SpanContext and optional attributes 4. SDK enforces LinkCountLimit (default 128)

Links vs Parent-Child

SDK Limits

The SDK enforces limits to prevent unbounded memory growth:

Excess items are silently dropped. The SDK logs a warning once per span when limits are hit.

Spans

A span represents a single unit of work in a trace. Spans are the building blocks of distributed traces — they carry timing, identity, and causal relationships.

Span Anatomy

Every span contains:

Span Naming

The span name identifies the class of operation, not the instance. It must be low-cardinality to enable aggregation in backends.

Rules: 1. Use the most general string that identifies a statistically interesting class 2. Prioritize generality over human-readability 3. Never embed high-cardinality values (IDs, emails, paths with IDs)

HTTP span names follow the pattern {METHOD} {route}:

• Server: GET /users/:id (use http.route)
• Client: GET (if no low-cardinality target available)

Database span names follow: {operation} {table}:

• SELECT users, INSERT orders

SpanKind

SpanKind describes the relationship between spans in a trace. It tells backends how to assemble the trace tree.

Rules: 1. A single span MUST NOT serve more than one purpose (e.g., don't use a SERVER span to also describe an outgoing call) 2. Create a new span before injecting context for outgoing calls 3. CLIENT → SERVER is the typical parent-child pair for synchronous calls 4. PRODUCER → CONSUMER is the typical pair for async operations 5. Default is INTERNAL if not specified

Common mistakes:

• Using INTERNAL for HTTP handlers → use SERVER
• Using INTERNAL for database calls → use CLIENT
• Using SERVER for outgoing HTTP requests → use CLIENT

Span Status

Rules: 1. Leave status Unset on success — it already means "no error" 2. Set Error with a description when operations fail 3. Ok is a "final call" — once set, subsequent Error attempts are ignored 4. Status has total order: Ok > Error > Unset 5. Instrumentation libraries SHOULD NOT set Ok — leave that to application code 6. Description is ONLY used with Error status

Span Lifetime

1. Start time is recorded at span creation 2. End time is recorded when span.end() is called 3. Every created span MUST be ended — leaked spans cause memory issues 4. After end(), the span becomes non-recording; further mutations are ignored 5. Ending a parent does NOT end children — children may outlive parents

Idiomatic patterns for ensuring spans end:

// Go: defer
span := tracer.Start(ctx, "operation")
defer span.End()

// Java: try-with-resources or try/finally
Span span = tracer.spanBuilder("operation").startSpan();
try (Scope scope = span.makeCurrent()) {
    // work
} finally {
    span.end();
}

// Python: context manager
with tracer.start_as_current_span("operation") as span:
    # work

// JavaScript: try/finally
const span = tracer.startSpan("operation");
try {
    // work
} finally {
    span.end();
}

Root Spans

A root span has no parent — it starts a new trace. Root spans:

• Get a new, randomly generated trace ID
• Have an empty parent span ID
• Are typically the entry point of a service (HTTP handler, message consumer)

Child spans inherit:

• The parent's trace ID
• All trace state values