Opentelemetry

OpenTelemetry

Quick Start (signal design)

• Export OTLP via an OpenTelemetry Collector (vendor-neutral endpoint).
• Standardize resource attributes: service.name, service.version, deployment.environment.
• Start with auto-instrumentation, then add manual spans and log correlation.

Load Next (References)

• references/concepts.md — traces/metrics/logs, context propagation, sampling, semantic conventions
• references/collector-and-otlp.md — Collector pipelines, processors, deployment patterns, tail sampling
• references/instrumentation-and-troubleshooting.md — manual spans, propagation pitfalls, cardinality, debugging

Collector and OTLP

Why Use the Collector

The Collector provides:

• A stable OTLP endpoint for all services
• Vendor-neutral export (swap backends without app changes)
• Centralized processing (batching, filtering, sampling, redaction)

Minimal Collector Config (Example)

receivers:
  otlp:
    protocols:
      grpc: {}
      http: {}

processors:
  batch: {}

exporters:
  otlphttp:
    endpoint: https://telemetry-backend.example/v1/otlp

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

Deployment Patterns

• Agent/DaemonSet: per-node collection (common in Kubernetes).
• Gateway: centralized Collector (good for tail sampling and heavy processing).
• Sidecar: per-pod Collector (less common; increases resource usage).

Tail Sampling (High-Value Traces)

Tail sampling keeps important traces while dropping noise:

• Sample errors at 100%
• Sample slow traces above a latency threshold
• Sample a small percentage of normal traffic

Use a gateway Collector for tail sampling to ensure full trace visibility.

Security and Reliability

Checklist:

• TLS for OTLP endpoints
• Auth at the Collector ingress (mTLS, token, or network isolation)
• Memory limits and backpressure (batch + memory limiter)
• Redact sensitive attributes at the Collector

Concepts: Traces, Metrics, Logs

Signals

• Traces: end-to-end request flow across services, represented as spans.
• Metrics: numeric time series (counters, gauges, histograms).
• Logs: discrete events; correlate with traces via trace_id and span_id.

Tracing Terms

• Trace: tree of spans for a single request/operation.
• Span: timed operation with attributes and events.
• Span kind: server/client/producer/consumer/internal (helps topology and analysis).
• Attributes: key/value metadata for filtering and grouping.
• Events: timestamped annotations on a span.

Prefer:

• Low-cardinality attributes (status codes, route names)
• High-detail information as span events or structured logs

Context Propagation

Distributed tracing relies on propagation of context across process boundaries:

• HTTP headers (W3C Trace Context)
• Messaging metadata (producer → consumer)

Propagation breaks when:

• Requests are queued without copying context
• Background tasks start without parent context
• Async boundaries drop context

Resources vs Span Attributes

Resource attributes describe the emitter:

• service.name
• service.version
• deployment.environment
• cloud.region (if applicable)

Span attributes describe the operation:

• HTTP: route, method, status code
• DB: system, statement name (avoid raw queries), duration

Sampling

Sampling controls cost and overhead:

• Head sampling: decide at trace start (simple, may miss rare errors)
• Tail sampling: decide after seeing span data (requires Collector, better for “keep errors”)

Common patterns:

• Parent-based sampling (respect upstream decision)
• Always sample error traces (tail sampling policy)

Cardinality (Common Pitfall)

Avoid high-cardinality labels/attributes in metrics (user IDs, request IDs). For traces, high-cardinality attributes are less harmful but still increase storage/search costs.

Instrumentation Patterns and Troubleshooting

Start Strategy

1) Enable auto-instrumentation for HTTP server/client and common libraries. 2) Add manual spans around business operations (checkout, sync, provision). 3) Add metrics for SLOs (latency histograms, error rate counters). 4) Correlate logs with trace context.

Manual Span Pattern

Prefer naming spans by operation, not by URL or dynamic IDs:

• Good: db.query.users_by_email, payment.charge, cache.get
• Avoid: GET /users/123 (high cardinality)

Attach attributes:

• route templates, not raw paths
• stable identifiers (tenant tier, feature flag name), not user IDs

Log Correlation

Emit structured fields in logs:

• trace_id
• span_id
• service.name
• deployment.environment

Avoid copying full baggage into logs.

Troubleshooting Checklist

No telemetry arrives

• Verify exporter endpoint and protocol (OTLP gRPC vs HTTP)
• Confirm the Collector receiver is enabled and reachable
• Validate TLS settings and auth requirements

Spans exist but traces look broken

• Propagation missing (headers stripped, message metadata lost)
• Background work detached from parent context
• Async context lost across task boundaries

High CPU/memory or backend cost

• High-cardinality metric labels
• Too many span attributes/events
• Sampling too permissive

Fixes:

• Move detail from attributes to logs/events
• Apply sampling and filtering at the Collector
• Add batch processing

Incorrect service grouping

If services appear merged or fragmented:

• Standardize service.name and service.version
• Avoid environment suffixes in service.name (use deployment.environment instead)