Logging Observability

Logging & Observability

Structured logging, distributed tracing, and metrics for production systems. Covers the full observability stack from log formatting to alert routing.

When to Use

Activate on: "structured logging", "distributed tracing", "OpenTelemetry", "OTel", "correlation ID", "log levels", "Grafana dashboard", "alerting thresholds", "SLI SLO", "Prometheus metrics", "PagerDuty integration", "observability stack", "Winston setup", "Pino logger", "log aggregation", "Datadog", "Honeycomb"

NOT for: Performance profiling (CPU/memory flamegraphs) | Load testing | Database query optimization | Security auditing

Decision Tree: What to Log at Each Level

flowchart TD
    E[Event Occurs] --> Q1{Does it represent\na system failure?}
    Q1 -->|Yes| Q2{Is it recoverable\nwithout human?}
    Q2 -->|No| FATAL[FATAL: Service cannot\ncontinue — trigger pager]
    Q2 -->|Yes| ERROR[ERROR: Operation failed,\nwill retry or degrade]
    Q1 -->|No| Q3{Is it unexpected\nbut not failing?}
    Q3 -->|Yes| WARN[WARN: Unusual condition,\ncircuit breaker open,\ndeprecation used]
    Q3 -->|No| Q4{Is it a meaningful\nbusiness event?}
    Q4 -->|Yes| INFO[INFO: User action,\npayment processed,\nservice started]
    Q4 -->|No| Q5{Needed to debug\na specific issue?}
    Q5 -->|Yes| DEBUG[DEBUG: DB queries,\ncache hits/misses,\nfunction inputs]
    Q5 -->|No| TRACE[TRACE: Fine-grained\nloop iterations,\nOTel spans]

Rule of thumb: Production runs INFO and above. DEBUG only enabled per-service via dynamic config, never always-on in prod.

Core Patterns

Structured Log Format (JSON)

Every log line must be parseable. String concatenation is not a log.

Node.js with Pino:

import pino from 'pino';

const logger = pino({
  level: process.env.LOG_LEVEL ?? 'info',
  base: {
    service: 'payment-service',
    version: process.env.SERVICE_VERSION,
    env: process.env.NODE_ENV,
  },
  redact: {
    paths: ['req.headers.authorization', 'body.password', 'body.cardNumber', '*.ssn'],
    censor: '[REDACTED]',
  },
});

// Good: structured fields
logger.info({ orderId, userId, amountCents }, 'Payment processed');

// Bad: string interpolation
logger.info(`Payment processed for user ${userId} order ${orderId}`);

Python with structlog:

import structlog

log = structlog.get_logger()

structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,
        structlog.stdlib.add_log_level,
        structlog.stdlib.add_logger_name,
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.JSONRenderer(),
    ]
)

# Good: key-value pairs
log.info("payment_processed", order_id=order_id, user_id=user_id, amount_cents=amount)

Correlation IDs

Every request needs a trace ID that flows through all downstream calls. This is the minimum viable distributed tracing without OTel.

// Express middleware
import { randomUUID } from 'crypto';
import { AsyncLocalStorage } from 'async_hooks';

const requestContext = new AsyncLocalStorage<{ traceId: string; spanId: string }>();

export function correlationMiddleware(req, res, next) {
  const traceId = req.headers['x-trace-id'] ?? randomUUID();
  const spanId = randomUUID().slice(0, 8);

  requestContext.run({ traceId, spanId }, () => {
    res.setHeader('x-trace-id', traceId);
    next();
  });
}

// Logger that auto-includes context
export function getLogger(name: string) {
  return {
    info: (msg: string, fields?: object) => {
      const ctx = requestContext.getStore();
      logger.info({ ...ctx, ...fields, logger: name }, msg);
    },
    // ... error, warn, debug
  };
}

OpenTelemetry Setup

See references/opentelemetry-setup.md for complete OTel collector config, SDK initialization per language, and span attribute conventions.

Minimal Node.js bootstrap:

// Must be first import in entrypoint
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';

const sdk = new NodeSDK({
  serviceName: 'payment-service',
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

Distributed Trace Propagation

sequenceDiagram
    participant C as Client
    participant GW as API Gateway
    participant SVC as Payment Service
    participant DB as Database

    C->>GW: POST /checkout<br/>(no trace header)
    Note over GW: Generate trace-id: abc123<br/>span-id: 0001
    GW->>SVC: POST /payment<br/>traceparent: 00-abc123-0001-01
    Note over SVC: Inherit trace-id: abc123<br/>New span-id: 0002
    SVC->>DB: INSERT payment<br/>traceparent: 00-abc123-0002-01
    Note over DB: Inherit trace-id: abc123<br/>New span-id: 0003
    DB-->>SVC: OK (span 0003 ends)
    SVC-->>GW: 200 OK (span 0002 ends)
    GW-->>C: 200 OK (span 0001 ends)<br/>x-trace-id: abc123

The W3C traceparent header format: 00-{traceId}-{spanId}-{flags}. Always propagate this header on every downstream HTTP call.

Reference Files

Anti-Patterns (Shibboleths)

Anti-Pattern 1: Logging PII or Secrets in Production

Novice thinking: "I'll just log the full request body to debug this auth issue."

Why wrong: GDPR/CCPA violations carry fines up to 4% of global revenue. Secrets in logs propagate to log aggregators, S3 exports, audit trails — all places with different access controls. A single console.log(req.body) can expose thousands of user passwords in your Datadog dashboard.

Detection signature: Search your logs for password, ssn, cardNumber, authorization as field values (not keys). If any appear, you have a PII leak.

Fix — Allowlist approach:

// Never log what you don't explicitly approve
const SAFE_BODY_FIELDS = ['orderId', 'productId', 'quantity', 'currency'];

logger.info({
  body: pick(req.body, SAFE_BODY_FIELDS), // only known-safe fields
  path: req.path,
  method: req.method,
}, 'Request received');

Fix — Redaction in logger config:

// Pino's redact runs before any transport
const logger = pino({
  redact: {
    paths: [
      'req.headers.authorization',
      'req.headers.cookie',
      'body.password',
      'body.*.password',   // nested objects too
      'body.cardNumber',
      'body.ssn',
      '*.token',
      '*.secret',
    ],
    censor: '[REDACTED]',
  },
});

Shibboleth: An expert sets up redaction at logger initialization, not as a reminder comment. Redaction must be structural, not ad-hoc.

---

Anti-Pattern 2: Unstructured String Logs Instead of Structured JSON

Novice thinking: logger.info('User ' + userId + ' purchased ' + productId + ' for $' + amount)

Why wrong: You cannot filter, aggregate, or alert on string-interpolated data in any log aggregator. A Grafana query for amount > 1000 requires amount to be a numeric field, not embedded in a sentence. String logs are write-only — you can read them but not query them at scale.

Impact: Your 10 million daily log lines become unsearchable. MTTR (mean time to recovery) during incidents doubles because engineers grep through strings instead of filtering structured fields.

Fix:

// Bad: string log — amount is buried in text
logger.info(`User ${userId} purchased ${productId} for $${amount}`);

// Good: structured — every field is queryable
logger.info({ userId, productId, amountDollars: amount / 100 }, 'purchase_completed');

Consistent event naming: Use snake_case verb-noun event names (payment_processed, user_signed_up, order_failed) as the message string. This creates a stable vocabulary for dashboards and alerts.

Shibboleth: The log message string is for humans scanning log tails. All queryable data lives in structured fields. A logger that produces {} as its output shape is better than one that produces readable strings.

---

Anti-Pattern 3: Log-and-Throw (Duplicate Log Entries)

Novice thinking: Log the error, then re-throw so the caller also knows about it.

// BAD: log-and-throw
async function processPayment(orderId: string) {
  try {
    return await chargeCard(orderId);
  } catch (err) {
    logger.error({ err, orderId }, 'Payment failed'); // Logged here
    throw err; // And the caller logs it again
  }
}

async function handleCheckout(req, res) {
  try {
    await processPayment(req.body.orderId);
  } catch (err) {
    logger.error({ err }, 'Checkout failed'); // Same error logged twice
    res.status(500).json({ error: 'Checkout failed' });
  }
}

Why wrong: Every error appears 2-5 times in your logs depending on call depth. Alerting on error count becomes unreliable. Incident review is confusing — engineers think there were multiple failures. Log volume costs money (Datadog charges per ingested GB).

Fix — Log only at the boundary where you handle the error:

// Good: log only where you decide what to do with the error
async function processPayment(orderId: string) {
  // No try-catch: let errors propagate naturally
  return await chargeCard(orderId);
}

async function handleCheckout(req, res) {
  try {
    await processPayment(req.body.orderId);
    res.json({ success: true });
  } catch (err) {
    // One log, at the boundary where we're deciding to return 500
    logger.error({ err, orderId: req.body.orderId }, 'checkout_failed');
    res.status(500).json({ error: 'Checkout failed' });
  }
}

Rule: Log where you handle. Don't log where you propagate. The call stack in the error object already tells you where it originated.

Shibboleth: If you see the same traceId appear in more than two error log lines for a single request, you have a log-and-throw chain somewhere.

Quality Checklist

[ ] All log lines are JSON (no string concatenation)
[ ] Log level strategy documented: what goes at each level
[ ] PII/secrets redacted at logger config level, not call site
[ ] Correlation IDs propagated on all outbound HTTP calls
[ ] OTel SDK initialized before any other imports
[ ] Error logs include the error object (not just message)
[ ] No log-and-throw patterns in error handling
[ ] DEBUG logs use conditional guards or sampling
[ ] SLI/SLO defined for each critical user journey
[ ] Alert routing: notify vs page threshold documented
[ ] Runbook linked from every paging alert
[ ] Log retention policy set (cost vs compliance)

Output Artifacts

1. Logger configuration — Pino/Winston/structlog setup with redaction rules 2. OTel bootstrap file — SDK init with auto-instrumentation 3. Correlation middleware — AsyncLocalStorage request context 4. Prometheus metrics module — Counter/histogram/gauge definitions 5. Grafana dashboard JSON — Four golden signals panels 6. Alertmanager rules YAML — SLO-based alert definitions

Alerting Patterns Reference

SLI/SLO design, alert routing, fatigue prevention, and PagerDuty integration.

SLI/SLO Design

Vocabulary

• SLI (Service Level Indicator): A measurement. "Our 99th percentile latency."
• SLO (Service Level Objective): A target. "99th percentile latency < 500ms over 30 days."
• SLA (Service Level Agreement): A contract with consequences if the SLO is missed.
• Error Budget: How much failure your SLO allows. SLO of 99.9% = 43.8 minutes/month of allowed downtime.

The Four Golden Signals (Google SRE Book)

Every service needs SLIs for these:

SLO Templates by Service Type

HTTP API:

slos:
  - name: availability
    sli: http_requests_total{status!~"5.."}  /  http_requests_total
    target: 0.999  # 99.9% — 43.8 min/month budget
    window: 30d

  - name: latency_p99
    sli: histogram_quantile(0.99, http_request_duration_seconds_bucket)
    target: 0.5  # < 500ms at p99
    window: 30d

  - name: latency_p50
    sli: histogram_quantile(0.50, http_request_duration_seconds_bucket)
    target: 0.1  # < 100ms at p50
    window: 30d

Async Worker / Queue Consumer:

slos:
  - name: job_success_rate
    sli: jobs_completed_total / (jobs_completed_total + jobs_failed_total)
    target: 0.995  # 99.5% — more lenient for async

  - name: queue_lag
    sli: queue_oldest_unprocessed_message_age_seconds
    target: 60  # < 60 seconds lag
    window: 1h  # shorter window for queue health

Error Budget Burn Rate Alerting

Burn rate alerts are more actionable than threshold alerts. They tell you how fast you're consuming your budget.

Formula: If your SLO is 99.9% over 30 days, your error budget is 43.8 minutes.

• Burn rate 1x = consuming budget at exactly the pace it's refreshed (sustainable)
• Burn rate 14.4x = consuming 30-day budget in 2 days (page immediately)
• Burn rate 6x = consuming budget in ~5 days (page within 1 hour)

Multi-window, multi-burn-rate alerts (the recommended pattern from Google SRE Workbook):

# Prometheus alerting rules
groups:
  - name: slo.payment-service
    rules:
      # Fast burn: 2% of budget in 1 hour (14.4x rate) — page now
      - alert: PaymentServiceFastBurn
        expr: |
          (
            rate(http_requests_total{service="payment",status=~"5.."}[1h]) /
            rate(http_requests_total{service="payment"}[1h])
          ) > 0.144
        for: 2m
        labels:
          severity: critical
          team: payments
        annotations:
          summary: "Fast error budget burn on payment-service"
          description: "Burning error budget at {{ $value | humanizePercentage }} error rate (14.4x)"
          runbook: "https://runbooks.internal/payment-service/fast-burn"

      # Slow burn: 5% of budget in 6 hours (6x rate) — ticket or Slack
      - alert: PaymentServiceSlowBurn
        expr: |
          (
            rate(http_requests_total{service="payment",status=~"5.."}[6h]) /
            rate(http_requests_total{service="payment"}[6h])
          ) > 0.06
        for: 15m
        labels:
          severity: warning
          team: payments
        annotations:
          summary: "Slow error budget burn on payment-service"
          runbook: "https://runbooks.internal/payment-service/slow-burn"

Alert Routing and Severity

Severity Taxonomy

Do not use ad-hoc severity labels. Define a taxonomy and stick to it.

PagerDuty Integration

# Alertmanager config
global:
  resolve_timeout: 5m

receivers:
  - name: pagerduty-critical
    pagerduty_configs:
      - routing_key: ${PAGERDUTY_INTEGRATION_KEY}
        severity: critical
        description: '{{ .GroupLabels.alertname }}: {{ .Annotations.summary }}'
        details:
          runbook: '{{ .Annotations.runbook }}'
          service: '{{ .Labels.service }}'
          environment: '{{ .Labels.env }}'

  - name: slack-warning
    slack_configs:
      - api_url: ${SLACK_WEBHOOK_URL}
        channel: '#alerts-warning'
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ .Annotations.description }}'
        actions:
          - type: button
            text: 'Runbook'
            url: '{{ .Annotations.runbook }}'

  - name: slack-info
    slack_configs:
      - api_url: ${SLACK_WEBHOOK_URL}
        channel: '#alerts-info'

route:
  group_by: ['alertname', 'service', 'env']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: slack-info  # default
  routes:
    - match:
        severity: critical
      receiver: pagerduty-critical
      repeat_interval: 1h  # re-page every hour if not resolved
    - match:
        severity: warning
      receiver: slack-warning
      repeat_interval: 8h

Alert Fatigue Prevention

The Alert Fatigue Death Spiral

1. Team adds alert for every possible condition 2. Alerts fire constantly, mostly for non-urgent things 3. Team starts ignoring alerts 4. Real incident fires, nobody notices 5. Outage

Rules for Alert-Worthiness

An alert should only fire if: 1. It requires human action — can the system fix it automatically? If yes, it should. 2. It cannot wait until morning — if the on-call can sleep through it, it's not a page. 3. It's actionable — is there a runbook? If not, write one or don't alert. 4. It's not already covered — does a higher-level alert catch this?

Audit question: For each alert in the last 30 days, was there a runbook entry written? If an alert fires and nobody writes anything down, it's noise.

Inhibition Rules

Suppress child alerts when parent is already firing:

# alertmanager inhibition rules
inhibit_rules:
  # If the whole service is down, don't also alert on latency
  - source_match:
      alertname: ServiceDown
    target_match_re:
      alertname: (HighLatency|ErrorRateHigh|QueueLag)
    equal: [service, env]

  # If database is down, suppress application errors (they're caused by DB)
  - source_match:
      alertname: DatabaseDown
    target_match_re:
      alertname: (PaymentFailed|OrderCreateFailed)
    equal: [env]

Alert Deduplication

Group alerts so the on-call gets one notification for a correlated event, not ten:

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s      # collect related alerts for 30s before firing
  group_interval: 5m   # how often to send updates on ongoing group
  repeat_interval: 4h  # re-notify if still firing after 4h

Runbook Template

Every paging alert must link to a runbook. Without this, the alert is not production-ready.

# Runbook: PaymentServiceFastBurn

## Summary
Payment service is consuming its error budget at 14.4x the sustainable rate.

## Impact
Users are experiencing payment failures. Checkout is degraded.

## Immediate Steps (first 5 minutes)
1. Check Grafana dashboard: [link]
2. Look at recent deploys: `git log --oneline -10`
3. Check downstream dependencies: Stripe, database, fraud service

## Diagnosis

### Is it a deploy?
- Compare error rate before/after deploy
- If yes: rollback with `kubectl rollout undo deploy/payment-service`

### Is it a downstream dependency?
- Check Stripe status: https://status.stripe.com
- Check database: `kubectl exec -it postgres-0 -- psql -c "SELECT 1"`

### Is it traffic-related?
- Check traffic volume: unusual spike in requests?
- Rate limiting might need adjustment

## Escalation
- Exhaust the above steps within 15 minutes
- If unresolved: escalate to payment-team-lead
- If infrastructure: escalate to platform-team

## Resolution
Once resolved, write a postmortem in Notion: [link]

Grafana Dashboard Design

Four Golden Signals Dashboard Layout

Row 1: Service Health Overview
  [Availability SLO gauge] [Error Budget remaining gauge] [Current error rate]

Row 2: Traffic & Errors
  [Request rate time series]  [HTTP 5xx rate time series]

Row 3: Latency
  [p50/p95/p99 latency heatmap]  [Latency distribution histogram]

Row 4: Saturation
  [CPU/Memory usage]  [Queue depth]  [Connection pool usage]

Row 5: Downstream Dependencies
  [Database query latency]  [External API success rate]

Key Grafana Panel Configs

// Error rate panel — use a threshold annotation for SLO
{
  "type": "timeseries",
  "title": "HTTP Error Rate",
  "targets": [{
    "expr": "rate(http_requests_total{status=~'5..'}[5m]) / rate(http_requests_total[5m])",
    "legendFormat": "Error Rate"
  }],
  "thresholds": {
    "mode": "absolute",
    "steps": [
      { "color": "green", "value": 0 },
      { "color": "yellow", "value": 0.001 },  // 0.1% warning
      { "color": "red", "value": 0.01 }        // 1% critical
    ]
  }
}

SLO Gauge Panel

{
  "type": "gauge",
  "title": "30-Day Availability",
  "targets": [{
    "expr": "1 - (increase(http_requests_total{status=~'5..'}[30d]) / increase(http_requests_total[30d]))",
    "instant": true
  }],
  "options": {
    "minVizValue": 0.99,
    "maxVizValue": 1,
    "thresholds": [
      { "color": "red", "value": 0 },
      { "color": "yellow", "value": 0.999 },
      { "color": "green", "value": 0.9995 }
    ]
  }
}

OpenTelemetry Setup Reference

Complete configuration for OTel SDK initialization, collector deployment, and trace propagation.

SDK Initialization by Language

Node.js

Install:

npm install @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http \
  @opentelemetry/exporter-metrics-otlp-http \
  @opentelemetry/resources \
  @opentelemetry/semantic-conventions

src/telemetry.ts — must be the first import in your entrypoint:

import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';

const resource = new Resource({
  [SemanticResourceAttributes.SERVICE_NAME]: process.env.SERVICE_NAME ?? 'unknown-service',
  [SemanticResourceAttributes.SERVICE_VERSION]: process.env.SERVICE_VERSION ?? '0.0.0',
  [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: process.env.NODE_ENV ?? 'development',
});

const sdk = new NodeSDK({
  resource,
  traceExporter: new OTLPTraceExporter({
    url: `${process.env.OTEL_EXPORTER_OTLP_ENDPOINT}/v1/traces`,
    headers: {
      'Authorization': `Bearer ${process.env.OTEL_EXPORTER_TOKEN}`,
    },
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: `${process.env.OTEL_EXPORTER_OTLP_ENDPOINT}/v1/metrics`,
    }),
    exportIntervalMillis: 15000, // match Prometheus scrape interval
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': { enabled: false }, // too noisy
      '@opentelemetry/instrumentation-http': {
        ignoreIncomingRequestHook: (req) => req.url?.includes('/health'),
      },
    }),
  ],
});

sdk.start();

process.on('SIGTERM', async () => {
  await sdk.shutdown();
});

src/index.ts — always import telemetry first:

import './telemetry';  // Must be first
import express from 'express';
// ... rest of app

Python

Install:

pip install opentelemetry-sdk \
  opentelemetry-exporter-otlp \
  opentelemetry-instrumentation-fastapi \
  opentelemetry-instrumentation-requests \
  opentelemetry-instrumentation-sqlalchemy

app/telemetry.py:

from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME, SERVICE_VERSION
import os

def init_telemetry():
    resource = Resource.create({
        SERVICE_NAME: os.getenv("SERVICE_NAME", "unknown-service"),
        SERVICE_VERSION: os.getenv("SERVICE_VERSION", "0.0.0"),
        "deployment.environment": os.getenv("ENVIRONMENT", "development"),
    })

    otlp_endpoint = os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4318")

    # Traces
    tracer_provider = TracerProvider(resource=resource)
    tracer_provider.add_span_processor(
        BatchSpanProcessor(
            OTLPSpanExporter(endpoint=f"{otlp_endpoint}/v1/traces")
        )
    )
    trace.set_tracer_provider(tracer_provider)

    # Metrics
    metric_reader = PeriodicExportingMetricReader(
        OTLPMetricExporter(endpoint=f"{otlp_endpoint}/v1/metrics"),
        export_interval_millis=15000,
    )
    meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
    metrics.set_meter_provider(meter_provider)

# FastAPI app setup
from fastapi import FastAPI
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor

app = FastAPI()
init_telemetry()
FastAPIInstrumentor.instrument_app(app)
RequestsInstrumentor().instrument()

Go

Install:

go get go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp \
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

// telemetry/setup.go
package telemetry

import (
    "context"
    "os"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/sdk/resource"
    "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
)

func InitTracer(ctx context.Context) (func(context.Context) error, error) {
    exporter, err := otlptracehttp.New(ctx,
        otlptracehttp.WithEndpoint(os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")),
    )
    if err != nil {
        return nil, err
    }

    res := resource.NewWithAttributes(
        semconv.SchemaURL,
        semconv.ServiceName(os.Getenv("SERVICE_NAME")),
        semconv.ServiceVersion(os.Getenv("SERVICE_VERSION")),
    )

    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(res),
        trace.WithSampler(trace.ParentBased(trace.TraceIDRatioBased(0.1))), // 10% sampling
    )
    otel.SetTracerProvider(tp)

    return tp.Shutdown, nil
}

OTel Collector Config

Deploy the OTel Collector as a sidecar or daemonset. It receives from your services and fans out to Jaeger, Prometheus, and your cloud provider.

otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 5s
    send_batch_size: 512

  memory_limiter:
    limit_mib: 512
    spike_limit_mib: 128
    check_interval: 5s

  # Add environment and cluster metadata
  resource:
    attributes:
      - key: k8s.cluster.name
        value: ${CLUSTER_NAME}
        action: upsert

  # Filter health check spans
  filter:
    traces:
      exclude:
        match_type: regexp
        span_names: [".*health.*", ".*readiness.*", ".*liveness.*"]

exporters:
  # Jaeger (for trace visualization)
  otlp/jaeger:
    endpoint: jaeger-collector:4317
    tls:
      insecure: true

  # Prometheus (for metrics scraping)
  prometheus:
    endpoint: 0.0.0.0:8889
    namespace: otel

  # Grafana Tempo (alternative to Jaeger)
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

  # Cloud providers (pick one)
  otlp/datadog:
    endpoint: https://trace.agent.datadoghq.com
    headers:
      DD-API-KEY: ${DD_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, filter, resource, batch]
      exporters: [otlp/jaeger]

    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [prometheus]

Span Attributes — Semantic Conventions

Use the OpenTelemetry semantic conventions for span attributes. Do not invent your own attribute names.

HTTP spans (auto-instrumented, but good to know):

http.method        = "POST"
http.url           = "https://api.example.com/payment"
http.status_code   = 200
http.route         = "/payment/:id"

Database spans (auto-instrumented with sqlalchemy/pg):

db.system          = "postgresql"
db.name            = "payments"
db.operation       = "INSERT"
db.sql.table       = "orders"

Custom business spans — add to your code:

const tracer = trace.getTracer('payment-service');

async function processPayment(orderId: string, amountCents: number) {
  return tracer.startActiveSpan('payment.process', async (span) => {
    span.setAttributes({
      'payment.order_id': orderId,
      'payment.amount_cents': amountCents,
      'payment.currency': 'USD',
      // Never set: cardNumber, cvv, or any PII
    });

    try {
      const result = await chargeCard(orderId, amountCents);
      span.setStatus({ code: SpanStatusCode.OK });
      span.setAttribute('payment.transaction_id', result.transactionId);
      return result;
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      throw err;
    } finally {
      span.end();
    }
  });
}

Context Propagation

The W3C traceparent header propagates trace context across service boundaries.

Format: 00-{traceId-32hex}-{spanId-16hex}-{flags-2hex} Example: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Propagation in fetch/axios:

import { context, propagation } from '@opentelemetry/api';

async function callDownstreamService(url: string, body: object) {
  const headers: Record<string, string> = { 'Content-Type': 'application/json' };

  // OTel injects traceparent/tracestate automatically if using auto-instrumentation
  // Manual injection if needed:
  propagation.inject(context.active(), headers);

  return fetch(url, { method: 'POST', headers, body: JSON.stringify(body) });
}

With getNodeAutoInstrumentations(), the http and undici instrumentations handle this automatically for all outbound calls.

Sampling Strategy

Full trace sampling at 100% is expensive and often unnecessary. Use head-based sampling:

// Always sample errors, sample 10% of successes
class ErrorAlwaysSampler implements Sampler {
  shouldSample(context, traceId, spanName, spanKind, attributes) {
    if (attributes['http.status_code'] >= 400) {
      return { decision: SamplingDecision.RECORD_AND_SAMPLED };
    }
    // 10% of everything else
    if (Math.random() < 0.1) {
      return { decision: SamplingDecision.RECORD_AND_SAMPLED };
    }
    return { decision: SamplingDecision.NOT_RECORD };
  }
}