Logging Best Practices

Logging Best Practices

Expert guidance for production-grade logging based on Boris Tane's loggingsucks.com philosophy.

Core Philosophy

Stop logging "what your code is doing." Start logging "what happened to this request."

Traditional logging is optimized for writing, not querying. Developers emit logs for immediate debugging convenience without considering how they'll be searched later. This creates massive signal-to-noise ratios at scale.

The Wide Events Architecture

Instead of scattered log statements throughout your code, build one comprehensive event per request per service:

// ❌ Traditional scattered logging
logger.info("Request started");
logger.info(`User ${userId} found`);
logger.info("Fetching cart");
logger.debug(`Cart has ${items.length} items`);
logger.info("Processing payment");
logger.error(`Payment failed: ${error.message}`);

// ✅ Wide event - build throughout request, emit once
const event = {
  request_id: req.id,
  timestamp: Date.now(),
  service: "checkout",
  version: "2.3.1",

  user: { id: userId, tier: "premium", account_age_days: 847 },
  cart: { id: cartId, item_count: 3, total_cents: 15999 },
  payment: { method: "card", provider: "stripe", latency_ms: 234 },

  outcome: "failure",
  error: { type: "PaymentDeclined", code: "card_declined", retriable: true }
};

logger.info(event);

Key Concepts

When to Apply This Skill

• Implementing logging in new services
• Reviewing code with log statements
• Debugging production issues
• Designing observability strategy
• Migrating from printf-style to structured logging
• Reducing log volume while improving queryability

What This Skill Provides

1. Wide event patterns for different frameworks and languages 2. Field design guidance for high-cardinality debugging 3. Sampling strategies that preserve signal 4. Anti-pattern detection in existing logging code 5. Query-first thinking for log architecture

Logging Best Practices Rules

Sections

Architecture

Core architectural patterns for production logging.

• architecture-wide-events.md - The wide event pattern
• architecture-event-lifecycle.md - Building events through request lifecycle

Field Design

What to include in your events.

• fields-high-cardinality.md - Embrace high-cardinality fields
• fields-required-context.md - Essential fields for every event
• fields-business-context.md - Capturing business-relevant data

Sampling

Smart sampling strategies.

• sampling-tail-sampling.md - Sample after completion, not before
• sampling-always-keep.md - What to never sample away

Anti-Patterns

Common mistakes to avoid.

• antipattern-scattered-logs.md - Stop scattering log statements
• antipattern-string-search.md - Design for queries, not grep
• antipattern-otel-silver-bullet.md - OTel is delivery, not strategy

Anti-Pattern: OpenTelemetry as Silver Bullet

Impact: MEDIUM-HIGH

OpenTelemetry is a delivery protocol, not a solution architecture. It doesn't determine what to log, add business context, or fix flawed mental models.

The Misconception

"We use OpenTelemetry, so observability is solved."

Reality: Most OTel implementations capture only span names, duration, and status—insufficient for effective debugging.

What OTel Is vs. Isn't

Minimal OTel Implementation (Common)

// ❌ What most teams end up with
// Auto-instrumented, no custom context
const tracer = trace.getTracer('checkout-service');

async function checkout(req, res) {
  const span = tracer.startSpan('checkout');

  try {
    await processPayment();
    span.setStatus({ code: SpanStatusCode.OK });
  } catch (error) {
    span.setStatus({ code: SpanStatusCode.ERROR });
    span.recordException(error);
    throw error;
  } finally {
    span.end();
  }
}

// Result: You know checkout took 500ms and failed
// You DON'T know: which user, what cart value, what payment method,
// which feature flags, what error code, is this a VIP customer...

OTel + Wide Events (Correct)

// ✅ OTel for tracing, wide events for context
const tracer = trace.getTracer('checkout-service');

async function checkout(req, res) {
  const span = tracer.startSpan('checkout');

  // Build wide event alongside span
  const event = {
    trace_id: span.spanContext().traceId,
    span_id: span.spanContext().spanId,
    request_id: req.id,
    timestamp: Date.now(),
    service: 'checkout'
  };

  try {
    const user = await getUser(req.userId);
    event.user = { id: user.id, tier: user.tier, ltv_cents: user.ltv };
    span.setAttribute('user.tier', user.tier);

    const cart = await getCart(req.cartId);
    event.cart = { id: cart.id, total_cents: cart.total, item_count: cart.items.length };
    span.setAttribute('cart.total_cents', cart.total);

    const payment = await processPayment(cart, user);
    event.payment = { provider: payment.provider, method: payment.method };

    event.outcome = 'success';
    span.setStatus({ code: SpanStatusCode.OK });

  } catch (error) {
    event.outcome = 'failure';
    event.error = { type: error.name, code: error.code, message: error.message };
    span.setStatus({ code: SpanStatusCode.ERROR });
    span.recordException(error);
    throw error;

  } finally {
    event.duration_ms = Date.now() - event.timestamp;
    span.end();

    // Wide event provides queryable context
    logger.info(event);
  }
}

Complementary, Not Competitive

OTel Attributes Are Not Wide Events

// ❌ Stuffing everything into span attributes
span.setAttribute('user.id', userId);
span.setAttribute('user.tier', tier);
span.setAttribute('cart.total', total);
span.setAttribute('cart.items', itemCount);
// ... 40 more attributes

// Problems:
// - Most trace backends have attribute cardinality limits
// - Query capabilities vary wildly between vendors
// - Not designed for analytics/aggregation

Checklist

• [ ] OTel traces for cross-service flow visualization
• [ ] Wide events for rich, queryable context within services
• [ ] Link wide events to traces via trace_id/span_id
• [ ] Business context in wide events, not just span attributes
• [ ] Don't rely solely on auto-instrumentation
• [ ] Understand your trace backend's query limitations

Anti-Pattern: Scattered Log Statements

Impact: CRITICAL

Scattered console.log or logger.info statements throughout code create debugging nightmares at scale. Replace with the wide events pattern.

The Problem

// ❌ SCATTERED LOGGING ANTI-PATTERN
async function processCheckout(req, res) {
  console.log('Checkout started');                    // No context

  const user = await getUser(req.userId);
  console.log(`User found: ${user.id}`);              // Different format

  const cart = await getCart(req.cartId);
  logger.info('Cart retrieved', { cartId: cart.id }); // Yet another format

  console.log(`Cart total: ${cart.total}`);

  try {
    const payment = await processPayment(cart, user);
    console.log('Payment successful');
    logger.info('Order created', { orderId: payment.orderId });
  } catch (error) {
    console.error('Payment failed:', error.message);  // Lost context
    throw error;
  }

  console.log('Checkout completed');
}

Why This Fails at Scale

The Solution: One Wide Event

// ✅ WIDE EVENT PATTERN
async function processCheckout(req, res) {
  const event = {
    request_id: req.id,
    timestamp: Date.now(),
    service: 'checkout',
    path: '/checkout',
    method: 'POST'
  };

  try {
    const user = await getUser(req.userId);
    event.user = {
      id: user.id,
      tier: user.tier,
      account_age_days: user.accountAgeDays
    };

    const cart = await getCart(req.cartId);
    event.cart = {
      id: cart.id,
      item_count: cart.items.length,
      total_cents: cart.total
    };

    const paymentStart = Date.now();
    const payment = await processPayment(cart, user);
    event.payment = {
      provider: payment.provider,
      latency_ms: Date.now() - paymentStart,
      order_id: payment.orderId
    };

    event.outcome = 'success';
    event.duration_ms = Date.now() - event.timestamp;

  } catch (error) {
    event.outcome = 'failure';
    event.error = {
      type: error.name,
      code: error.code,
      message: error.message
    };
    event.duration_ms = Date.now() - event.timestamp;
    throw error;

  } finally {
    // ONE log per request, with FULL context
    logger.info(event);
  }
}

Comparison

Detection Checklist

When reviewing code, flag these patterns:

• [ ] Multiple console.log/logger.x in same function
• [ ] Log statements without request/correlation ID
• [ ] String interpolation in logs: logger.info(\User ${id}\)
• [ ] console.log('starting...') / console.log('done')
• [ ] Error logs that don't include request context
• [ ] Mixed console and structured logger usage

Refactoring Steps

1. Identify all log statements in the function 2. Create single event object at function start 3. Replace each log with event enrichment 4. Move emit to finally block (or completion middleware) 5. Include error context in catch blocks 6. Delete all intermediate log statements

Anti-Pattern: String Search Logging

Impact: HIGH

Designing logs for grep/string search treats them as character bags with no structural understanding. Design for queries, not grep.

The Problem

// ❌ STRING-BASED LOGGING
logger.info(`User user-123 logged in from 192.168.1.1`);
logger.info(`user_id=user-123 action=login ip=192.168.1.1`);
logger.info(`[LOGIN] User: user-123, IP: 192.168.1.1`);
logger.info(`{"userId": "user-123", "action": "login"}`); // JSON string, not object!

// Same entity, 4 different formats. Grep hell:
// grep "user-123" | grep "user_id=user-123" | grep "User: user-123" | grep '"userId": "user-123"'

Why String Search Fails

The Solution: Structured Fields

// ✅ STRUCTURED LOGGING
logger.info({
  event_type: 'user_login',
  user: {
    id: 'user-123',
    email_domain: 'company.com',
    tier: 'enterprise'
  },
  client: {
    ip: '192.168.1.1',
    user_agent: 'Mozilla/5.0...',
    geo_country: 'US'
  },
  auth: {
    method: 'password',
    mfa_used: true
  },
  request_id: 'req-abc123',
  timestamp: Date.now()
});

Query Power

-- Structured: Simple, fast, exact
SELECT * FROM logs
WHERE user.id = 'user-123'
  AND event_type = 'user_login';

-- Aggregation enabled
SELECT
  client.geo_country,
  COUNT(*) as logins,
  COUNT(DISTINCT user.id) as unique_users
FROM logs
WHERE event_type = 'user_login'
  AND timestamp > NOW() - INTERVAL 24 HOUR
GROUP BY client.geo_country;

-- String search: Fragile, slow, incomplete
grep "user-123" logs.txt | grep -E "(login|logged in|LOGIN)"

Common String-Based Anti-Patterns

// ❌ String interpolation
logger.info(`Processing order ${orderId} for user ${userId}`);

// ❌ Key-value string format
logger.info(`order_id=${orderId} user_id=${userId} status=processing`);

// ❌ JSON as string (not parsed)
logger.info(JSON.stringify({ orderId, userId, status: 'processing' }));

// ❌ Inconsistent separators
logger.info(`Order: ${orderId}, User: ${userId}, Status: processing`);
logger.info(`[Order ${orderId}] [User ${userId}] processing`);

Correct Patterns

// ✅ Structured object
logger.info({
  event_type: 'order_processing',
  order: { id: orderId, status: 'processing' },
  user: { id: userId },
  request_id: requestId
});

// ✅ With proper logger configuration
const logger = pino({
  formatters: {
    level: (label) => ({ level: label })
  }
});

Logger Configuration

Ensure your logger outputs proper JSON:

// pino (recommended)
import pino from 'pino';
const logger = pino({ level: 'info' });

// winston
import winston from 'winston';
const logger = winston.createLogger({
  format: winston.format.json(),
  transports: [new winston.transports.Console()]
});

// bunyan
import bunyan from 'bunyan';
const logger = bunyan.createLogger({ name: 'app' });

Checklist

• [ ] No string interpolation in log messages
• [ ] All context passed as structured object
• [ ] Logger configured to output JSON
• [ ] Consistent field names across codebase
• [ ] Can query any field with WHERE field.subfield = 'value'
• [ ] No regex required to extract values

Event Lifecycle Management

Impact: HIGH

Build your wide event progressively through the request lifecycle, enriching with context at each stage, then emit once at completion.

Pattern: Request Context Builder

// middleware/requestContext.ts
import { AsyncLocalStorage } from 'async_hooks';

interface RequestEvent {
  request_id: string;
  timestamp: number;
  service: string;
  version: string;
  method: string;
  path: string;
  user?: UserContext;
  business?: Record<string, unknown>;
  metrics?: Record<string, number>;
  error?: ErrorContext;
  outcome?: 'success' | 'failure';
  duration_ms?: number;
}

const eventStorage = new AsyncLocalStorage<RequestEvent>();

// Initialize at request start
export function initRequestEvent(req: Request): void {
  const event: RequestEvent = {
    request_id: req.id ?? crypto.randomUUID(),
    timestamp: Date.now(),
    service: process.env.SERVICE_NAME!,
    version: process.env.SERVICE_VERSION!,
    method: req.method,
    path: req.path,
  };
  eventStorage.enterWith(event);
}

// Enrich throughout request
export function enrichEvent(data: Partial<RequestEvent>): void {
  const event = eventStorage.getStore();
  if (event) {
    Object.assign(event, data);
  }
}

// Add nested context
export function addUserContext(user: UserContext): void {
  enrichEvent({ user });
}

export function addBusinessContext(key: string, value: unknown): void {
  const event = eventStorage.getStore();
  if (event) {
    event.business = { ...event.business, [key]: value };
  }
}

export function recordMetric(key: string, value: number): void {
  const event = eventStorage.getStore();
  if (event) {
    event.metrics = { ...event.metrics, [key]: value };
  }
}

// Emit at request end
export function finalizeEvent(outcome: 'success' | 'failure'): RequestEvent {
  const event = eventStorage.getStore()!;
  event.outcome = outcome;
  event.duration_ms = Date.now() - event.timestamp;
  return event;
}

Good Example: Express Middleware

// Middleware chain that builds event progressively

app.use((req, res, next) => {
  initRequestEvent(req);
  next();
});

app.use(authMiddleware); // Adds user context

app.post('/checkout', async (req, res) => {
  const startDb = Date.now();
  const cart = await getCart(req.body.cartId);
  recordMetric('db_latency_ms', Date.now() - startDb);

  addBusinessContext('cart', {
    id: cart.id,
    item_count: cart.items.length,
    total_cents: cart.total
  });

  const startPayment = Date.now();
  const result = await processPayment(cart);
  recordMetric('payment_latency_ms', Date.now() - startPayment);

  addBusinessContext('payment', {
    method: result.method,
    provider: result.provider
  });

  res.json({ orderId: result.orderId });
});

// Final middleware emits event
app.use((req, res, next) => {
  res.on('finish', () => {
    const outcome = res.statusCode < 400 ? 'success' : 'failure';
    const event = finalizeEvent(outcome);
    event.status_code = res.statusCode;
    logger.info(event);
  });
  next();
});

Bad Example: Context Lost Across Boundaries

// ❌ Context doesn't flow through async boundaries
app.post('/checkout', async (req, res) => {
  console.log('checkout started'); // No context

  await someAsyncOperation(); // Context lost

  console.log('payment done'); // Still no context
});

Framework Patterns

Checklist

• [ ] Initialize event at request entry point
• [ ] Use framework-appropriate context propagation
• [ ] Enrich at each significant stage (auth, business logic, external calls)
• [ ] Record timing metrics for each operation
• [ ] Capture errors with full context before emitting
• [ ] Emit exactly once at request completion

Wide Events Architecture

Impact: CRITICAL

Emit one comprehensive event per request per service instead of scattered log statements. This is the canonical log line pattern—consolidating all debugging information into a single queryable record.

Why This Matters

Traditional scattered logging creates:

• Signal-to-noise problems at scale
• Inconsistent field naming across statements
• Impossible cross-request correlation
• String-search archaeology instead of analytics

Wide events transform logging from archaeology → analytics.

Good Example

// Build event throughout request lifecycle
const event = {
  // Request metadata
  request_id: req.id,
  trace_id: req.headers['x-trace-id'],
  timestamp: Date.now(),

  // Service context
  service: "checkout",
  version: "2.3.1",
  region: "us-east-1",
  instance_id: process.env.INSTANCE_ID,

  // User context
  user: {
    id: userId,
    tier: "premium",
    account_age_days: 847,
    lifetime_value_cents: 2340000
  },

  // Business context
  cart: {
    id: cartId,
    item_count: 3,
    total_cents: 15999,
    coupon_applied: "SAVE20"
  },

  // Operation metrics
  payment: {
    method: "card",
    provider: "stripe",
    latency_ms: 234,
    attempt: 1
  },

  // Outcome
  outcome: "success",
  duration_ms: 487,

  // Feature flags active
  feature_flags: ["new_checkout_v2", "express_payment"]
};

// Emit ONCE at request completion
logger.info(event);

Bad Example

// ❌ Scattered logging - debugging nightmare at scale
logger.info("Request started");
logger.debug(`Processing user ${userId}`);
logger.info("User authenticated successfully");
logger.debug(`Found cart with ${items.length} items`);
logger.info("Initiating payment");
logger.debug(`Using provider: stripe`);
logger.info("Payment processed");
logger.info("Order created");
logger.info("Request completed");

Queryable Power

With wide events, answer complex questions in one query:

-- "Why are premium users failing checkout more than usual?"
SELECT
  error.code,
  COUNT(*) as failures,
  AVG(payment.latency_ms) as avg_latency
FROM logs
WHERE
  service = 'checkout'
  AND outcome = 'failure'
  AND user.tier = 'premium'
  AND timestamp > NOW() - INTERVAL 1 HOUR
GROUP BY error.code
ORDER BY failures DESC

Checklist

• [ ] One event per request per service (not multiple log statements)
• [ ] Build event progressively through request lifecycle
• [ ] Include request, user, business, and technical context
• [ ] Emit at request completion (success or failure)
• [ ] 40+ fields for rich dimensionality
• [ ] Consistent field naming across services

Capture Business Context

Impact: HIGH

Technical context alone is insufficient. Capture business-relevant data that enables domain-specific queries and debugging.

Why Business Context Matters

"Show me failed checkouts" is technical.

"Show me failed checkouts for enterprise customers buying > $1000 with new checkout enabled" is actionable.

Good Example: E-Commerce Context

const event = {
  // ... required fields ...

  // === USER BUSINESS CONTEXT ===
  user: {
    id: "user_12345",
    tier: "enterprise",           // Subscription level
    account_age_days: 847,        // Tenure
    lifetime_value_cents: 2340000,// LTV
    orders_count: 47,             // Purchase history
    support_tier: "priority",     // Support level
    is_beta_user: true,           // Beta program
    acquisition_channel: "referral"
  },

  // === TRANSACTION CONTEXT ===
  cart: {
    id: "cart_789",
    item_count: 5,
    unique_products: 3,
    total_cents: 45999,
    currency: "USD",
    has_subscription_items: true,
    coupon_code: "SAVE20",
    discount_cents: 9200
  },

  // === BUSINESS OPERATION ===
  checkout: {
    flow_type: "express",         // Which checkout variant
    payment_method: "saved_card",
    shipping_method: "express",
    is_gift: false,
    requires_signature: true
  },

  // === FEATURE CONTEXT ===
  experiment: {
    checkout_variant: "new_v2",   // A/B test bucket
    recommendation_model: "ml_v3",
    pricing_tier: "dynamic"
  }
};

Good Example: SaaS/API Context

const event = {
  // ... required fields ...

  // === CUSTOMER CONTEXT ===
  customer: {
    id: "cust_abc123",
    plan: "enterprise",
    mrr_cents: 299900,            // Monthly revenue
    seats_used: 45,
    seats_limit: 50,
    contract_end_date: "2025-12-31",
    is_churning_risk: false,
    customer_success_tier: "high_touch"
  },

  // === API USAGE ===
  api: {
    endpoint: "/v2/documents/analyze",
    api_version: "2024-01",
    quota_remaining: 8547,
    quota_limit: 10000,
    is_rate_limited: false
  },

  // === RESOURCE CONTEXT ===
  resource: {
    type: "document",
    id: "doc_xyz789",
    size_bytes: 1024000,
    page_count: 47,
    owner_id: "user_456"
  },

  // === BILLING IMPACT ===
  billing: {
    metered_units: 47,            // Usage-based billing
    estimated_cost_cents: 235,
    is_overage: false
  }
};

Domain-Specific Queries Enabled

-- "Which enterprise customers are hitting rate limits?"
SELECT customer.id, customer.plan, COUNT(*) as limit_hits
FROM logs
WHERE api.is_rate_limited = true
  AND customer.plan = 'enterprise'
  AND timestamp > NOW() - INTERVAL 24 HOUR
GROUP BY customer.id, customer.plan;

-- "Revenue impact of checkout failures by experiment variant"
SELECT
  experiment.checkout_variant,
  COUNT(*) as failures,
  SUM(cart.total_cents) / 100 as lost_revenue_dollars
FROM logs
WHERE outcome = 'failure'
  AND path = '/checkout'
GROUP BY experiment.checkout_variant;

-- "API errors for customers nearing quota"
SELECT customer.id, error.code, api.quota_remaining
FROM logs
WHERE outcome = 'failure'
  AND api.quota_remaining < 100
  AND customer.mrr_cents > 100000;

Bad Example: No Business Context

// ❌ Pure technical—can't answer business questions
const event = {
  request_id: "abc123",
  method: "POST",
  path: "/checkout",
  status: 500,
  duration_ms: 234
};

// "Are enterprise customers affected?" — Can't answer
// "What's the revenue impact?" — Can't answer
// "Which A/B variant is failing?" — Can't answer

Checklist

• [ ] User tier/subscription level included
• [ ] Business entity details (cart total, order value, etc.)
• [ ] Feature flag / experiment bucketing captured
• [ ] Domain-specific metadata present
• [ ] Revenue-relevant fields included
• [ ] Can answer "which customers are affected?" from logs alone

Embrace High-Cardinality Fields

Impact: CRITICAL

High-cardinality fields (user IDs, order IDs, session IDs) are where real debugging power lies. Traditional observability treated these as expensive or problematic—modern columnar databases handle them efficiently.

Understanding Cardinality

The Misconception

"High-cardinality data is expensive and should be avoided"

Reality: Modern columnar databases (ClickHouse, BigQuery, Snowflake) are designed for high-cardinality queries. The cost of NOT having this data is hours of debugging.

Good Example: Rich High-Cardinality Context

const event = {
  // High-cardinality identifiers (THE GOOD STUFF)
  request_id: "req_7f3b2a1c",
  trace_id: "trace_abc123def456",
  user_id: "user_12345",
  account_id: "acc_98765",
  session_id: "sess_xyz789",
  order_id: "ord_555666",
  cart_id: "cart_111222",

  // Also high-cardinality but often missed
  correlation_id: "corr_parent_request",
  idempotency_key: "idem_client_generated",

  // User attributes (enable cohort analysis)
  user: {
    id: "user_12345",
    email_domain: "company.com", // Not full email—privacy
    signup_date: "2023-06-15",
    subscription_tier: "enterprise",
    lifetime_orders: 47,
    lifetime_value_cents: 234500
  },

  // Transaction details
  transaction: {
    id: "txn_789",
    amount_cents: 15999,
    currency: "USD",
    payment_method_id: "pm_card_visa_1234"
  }
};

Bad Example: Only Low-Cardinality Data

// ❌ Useless for debugging specific issues
const event = {
  method: "POST",
  path: "/checkout",
  status: 500,
  service: "checkout",
  region: "us-east-1"
};

// "500 errors are up in checkout" - but WHICH users? WHICH orders?

Query Power Unlocked

-- With high-cardinality: "What happened to this specific user?"
SELECT * FROM logs
WHERE user_id = 'user_12345'
ORDER BY timestamp DESC
LIMIT 100;

-- "All requests in this user's session"
SELECT * FROM logs
WHERE session_id = 'sess_xyz789'
ORDER BY timestamp;

-- "Trace this order across all services"
SELECT service, outcome, duration_ms, error.message
FROM logs
WHERE order_id = 'ord_555666'
ORDER BY timestamp;

-- "Which users are hitting this error?"
SELECT user_id, COUNT(*) as occurrences
FROM logs
WHERE error.code = 'PAYMENT_DECLINED'
  AND timestamp > NOW() - INTERVAL 1 HOUR
GROUP BY user_id
ORDER BY occurrences DESC;

High-Cardinality Fields to Always Include

Checklist

• [ ] Include all relevant entity IDs (user, account, order, session)
• [ ] Add trace/correlation IDs for distributed tracing
• [ ] Include business entity IDs (cart, invoice, subscription)
• [ ] Don't avoid high-cardinality—embrace it
• [ ] Use columnar database designed for high-cardinality queries

Required Context Fields

Impact: HIGH

Every wide event must include baseline context fields. These enable filtering, correlation, and debugging across your entire system.

Minimum Required Fields

interface RequiredEventFields {
  // Identity
  request_id: string;      // Unique per request
  trace_id?: string;       // Distributed tracing correlation

  // Timing
  timestamp: number;       // Unix epoch ms
  duration_ms: number;     // Request duration

  // Service metadata
  service: string;         // Service name
  version: string;         // Deployment version
  environment: string;     // prod/staging/dev
  region: string;          // Deployment region
  instance_id: string;     // Container/instance ID

  // Request details
  method: string;          // HTTP method or RPC name
  path: string;            // Endpoint path
  status_code: number;     // Response status

  // Outcome
  outcome: 'success' | 'failure';
}

Good Example: Complete Context

const event = {
  // === IDENTITY ===
  request_id: crypto.randomUUID(),
  trace_id: req.headers['x-trace-id'],
  span_id: crypto.randomUUID(),
  parent_span_id: req.headers['x-parent-span-id'],

  // === TIMING ===
  timestamp: Date.now(),
  duration_ms: 0, // Set at completion

  // === SERVICE METADATA ===
  service: "checkout-api",
  version: "2.3.1",
  git_sha: "abc123f",
  environment: "production",
  region: "us-east-1",
  availability_zone: "us-east-1a",
  instance_id: "i-0123456789",
  kubernetes_pod: "checkout-api-7f8d9-xk2lm",

  // === REQUEST ===
  method: "POST",
  path: "/v1/checkout",
  route_pattern: "/v1/checkout", // Normalized (no IDs)
  query_params: { expand: "items" },
  content_length: 1247,
  user_agent: req.headers['user-agent'],
  client_ip: req.ip, // Or anonymized

  // === RESPONSE ===
  status_code: 200,
  response_size_bytes: 892,

  // === OUTCOME ===
  outcome: "success"
};

Deployment Context (Often Missed)

// Capture at service startup, include in every event
const deploymentContext = {
  version: process.env.SERVICE_VERSION,
  git_sha: process.env.GIT_SHA,
  deploy_id: process.env.DEPLOY_ID,
  deployed_at: process.env.DEPLOY_TIMESTAMP,
  deployed_by: process.env.DEPLOY_USER,

  // Kubernetes context
  kubernetes_namespace: process.env.K8S_NAMESPACE,
  kubernetes_pod: process.env.K8S_POD_NAME,
  kubernetes_node: process.env.K8S_NODE_NAME,

  // Feature flags snapshot
  feature_flags: getActiveFlags()
};

// Merge into every event
const event = {
  ...deploymentContext,
  ...requestContext
};

Bad Example: Missing Critical Context

// ❌ Can't correlate, can't identify which version, can't filter by region
const event = {
  message: "checkout completed",
  userId: "123"
};

Context Categories

Checklist

• [ ] Every event has request_id and timestamp
• [ ] Service name, version, and region are included
• [ ] Deployment metadata (git SHA, deploy ID) captured
• [ ] Request method, path, and status code present
• [ ] Duration calculated and included
• [ ] Clear success/failure outcome field

Events to Never Sample Away

Impact: CRITICAL

Some events must have 100% retention regardless of volume or cost. Losing these during an outage is unacceptable.

Always Keep Categories

Implementation

const ALWAYS_KEEP_RULES: SamplingRule[] = [
  // === ERRORS ===
  {
    name: 'server_errors',
    condition: (e) => e.status_code >= 500,
    reason: 'Server errors must always be retained'
  },
  {
    name: 'client_errors_auth',
    condition: (e) => e.status_code === 401 || e.status_code === 403,
    reason: 'Auth failures are security-relevant'
  },
  {
    name: 'outcome_failure',
    condition: (e) => e.outcome === 'failure',
    reason: 'All business failures are signal'
  },

  // === PERFORMANCE ===
  {
    name: 'slow_requests',
    condition: (e) => e.duration_ms > getP99(e.path) * 1.5,
    reason: 'Slow requests indicate degradation'
  },
  {
    name: 'timeout',
    condition: (e) => e.error?.type === 'TIMEOUT',
    reason: 'Timeouts indicate system issues'
  },

  // === BUSINESS CRITICAL ===
  {
    name: 'vip_customers',
    condition: (e) => e.user?.tier === 'enterprise' || e.user?.is_vip,
    reason: 'Enterprise customers expect reliability'
  },
  {
    name: 'high_value_transactions',
    condition: (e) => (e.transaction?.amount_cents ?? 0) > 100000,
    reason: 'High-value transactions are revenue-critical'
  },
  {
    name: 'new_users',
    condition: (e) => (e.user?.account_age_days ?? Infinity) < 7,
    reason: 'New user experience is critical for retention'
  },

  // === DEBUGGING ===
  {
    name: 'feature_flagged',
    condition: (e) => (e.feature_flags?.length ?? 0) > 0,
    reason: 'Need full visibility during rollouts'
  },
  {
    name: 'canary_deployment',
    condition: (e) => e.deployment_ring === 'canary',
    reason: 'Canary deployments need 100% observability'
  },

  // === SECURITY ===
  {
    name: 'rate_limited',
    condition: (e) => e.is_rate_limited,
    reason: 'Rate limiting indicates abuse or capacity issues'
  },
  {
    name: 'suspicious_activity',
    condition: (e) => e.security?.risk_score > 0.7,
    reason: 'Security events require full audit trail'
  }
];

function mustKeep(event: WideEvent): { keep: boolean; reason?: string } {
  for (const rule of ALWAYS_KEEP_RULES) {
    if (rule.condition(event)) {
      return { keep: true, reason: rule.name };
    }
  }
  return { keep: false };
}

Cost Management

If always-keep volume becomes expensive:

1. Aggregate, don't drop: Keep 100% of error metadata, sample error details 2. Tiered storage: Move old always-keep events to cold storage 3. Fix the cause: High error volume = fix bugs, don't hide them

Bad Example: Dropping Errors

// ❌ NEVER DO THIS
if (Math.random() > 0.05) {
  return; // Might drop critical error!
}
logger.error(event);

Checklist

• [ ] All 5XX errors retained at 100%
• [ ] All auth failures (401/403) retained
• [ ] Slow requests above p99 retained
• [ ] VIP/enterprise customer requests retained
• [ ] Feature-flagged requests retained
• [ ] New user requests retained
• [ ] High-value transactions retained
• [ ] Security-relevant events retained

Tail Sampling Strategy

Impact: HIGH

Make sampling decisions after request completion based on outcomes. This ensures you keep 100% of important events (errors, slow requests, VIPs) while sampling routine traffic.

Head vs Tail Sampling

Tail Sampling Decision Logic

interface SamplingDecision {
  keep: boolean;
  reason: string;
  sample_rate: number;  // For weighted analytics
}

function shouldKeepEvent(event: WideEvent): SamplingDecision {
  // === ALWAYS KEEP (100% retention) ===

  // All errors
  if (event.outcome === 'failure' || event.status_code >= 500) {
    return { keep: true, reason: 'error', sample_rate: 1.0 };
  }

  // Slow requests (above p99)
  if (event.duration_ms > getP99Threshold(event.path)) {
    return { keep: true, reason: 'slow', sample_rate: 1.0 };
  }

  // VIP customers
  if (event.user?.tier === 'enterprise' || event.user?.is_vip) {
    return { keep: true, reason: 'vip', sample_rate: 1.0 };
  }

  // Feature-flagged requests (rollout debugging)
  if (event.feature_flags?.includes('new_checkout_v2')) {
    return { keep: true, reason: 'feature_flag', sample_rate: 1.0 };
  }

  // Recent signups (onboarding issues)
  if (event.user?.account_age_days < 7) {
    return { keep: true, reason: 'new_user', sample_rate: 1.0 };
  }

  // High-value transactions
  if (event.transaction?.amount_cents > 100000) { // > $1000
    return { keep: true, reason: 'high_value', sample_rate: 1.0 };
  }

  // === SAMPLE THE REST ===

  // 5% of successful, fast, routine requests
  const shouldSample = Math.random() < 0.05;
  return {
    keep: shouldSample,
    reason: 'random_sample',
    sample_rate: 0.05
  };
}

Implementation Pattern

// Final middleware applies sampling
app.use((req, res, next) => {
  res.on('finish', () => {
    const event = finalizeEvent(res.statusCode < 400 ? 'success' : 'failure');
    const decision = shouldKeepEvent(event);

    if (decision.keep) {
      // Include sample_rate for weighted analytics
      event._sampling = {
        kept: true,
        reason: decision.reason,
        rate: decision.sample_rate
      };
      logger.info(event);
    }
  });
  next();
});

Weighted Analytics

When analyzing sampled data, weight by inverse sample rate:

-- Correct: Weight sampled events
SELECT
  path,
  SUM(1 / _sampling.rate) as estimated_total_requests,
  COUNT(*) as sampled_count
FROM logs
WHERE timestamp > NOW() - INTERVAL 1 HOUR
GROUP BY path;

-- Wrong: Raw counts underrepresent sampled traffic
SELECT path, COUNT(*) FROM logs GROUP BY path; -- Misleading!

Bad Example: Head Sampling

// ❌ Decision at request START — might drop errors
app.use((req, res, next) => {
  // 5% random sample before we know if this will fail
  if (Math.random() > 0.05) {
    req.skipLogging = true;  // Might skip a critical error!
  }
  next();
});

Checklist

• [ ] Sampling decision made after request completion
• [ ] 100% retention for errors and 5XX responses
• [ ] 100% retention for slow requests (above p99)
• [ ] 100% retention for VIP/enterprise customers
• [ ] 100% retention for feature-flagged requests
• [ ] Sample rate included in event for weighted analytics
• [ ] 1-10% sampling for routine successful requests