Logging Best Practices

Logging Best Practices

Most logging code is written as if it were for a monolith circa 2010: scattered console.log / logger.info calls printing human-readable strings that tell a reader what the code is doing. In a distributed system under load that approach fails in predictable ways — you get 10k lines of noise per minute, no way to correlate across services, and when the incident hits you can answer "did it crash?" but not "which customers were affected, on which deploy, in which region". This skill exists to push code toward an observability primitive that actually works: one structured, context-rich event per unit of work.

The core ideas come from Stripe's canonical log lines, the wide-events / observability-2.0 lineage (Charity Majors et al.), and Boris Tane's loggingsucks.com manifesto and logging-best-practices skill. This skill adapts and extends that work with concrete guidance for Python and Cloudflare Workers.

When to apply

Any time logging code is being written, reviewed, or designed — whether that's a fresh service, a debug session, or "just adding a log line". The reframe matters even for single log additions: before adding logger.info("thing happened"), ask whether this should instead enrich a canonical event for the current request.

When triggered, do this first

A short orchestration recipe to avoid jumping straight to "write some logger code". Skip steps that are obviously already done.

1. Identify the runtime so you load the correct reference. Check package.json / pyproject.toml / wrangler.jsonc. If it is a multi-service repo, ask which service. 2. Audit what exists, before proposing changes:

• Where is the logger configured? (search for structlog.configure, pino(, winston.createLogger, logging.basicConfig, console.log)
• Is there middleware or a request hook? (search for app.use, add_middleware, Middleware)
• How many distinct call sites — console.log / logger.info / print / log.info? A rough count tells you whether this is greenfield, growing, or legacy-debt territory.
• Where do logs end up? (stdout? file? a vendor SDK? Workers Logs? Logpush?)

3. Identify the unit of work — HTTP request, queue handler, cron tick, CLI invocation. The canonical event is per unit of work; you need to know what that is before you can wrap it. 4. Plan the change in this order, and confirm with the user before implementing anything large:

• Logger configuration (one place, structured JSON to stdout, contextvars wired).
• Canonical middleware/wrapper around the unit of work.
• Helper for handlers to annotate the in-flight event.
• Migration plan for existing log calls (do not delete them all at once — see "A note on retrofitting" below).

5. Load the matching reference before writing platform-specific code. Then implement. 6. Verify: hit the service (or run the script), inspect a real emitted event, confirm it has identity / environment / business / perf+outcome fields populated. If any of the four categories is missing, the work is not done.

For tiny tasks ("our health check is spamming logs"), skip the audit and go straight to step 4 — the recipe is for the common "improve our logging" ask.

Pick the implementation reference — before writing code

Load the reference that matches the runtime before writing any code. The principles here are platform-agnostic; real correctness (API signatures, middleware wiring, platform quirks) lives in the references. Skipping this step is how you produce plausible-looking code that misuses the platform — waitUntil as a log sink on Workers, BaseHTTPMiddleware that silently loses contextvars in Python.

For stacks not yet covered (Node/Express, Go, Rust): load the closest reference as a template. The shape of the solution (structured logger + canonical middleware + finally-emit) is the same; substitute the idiomatic library (pino, slog, tracing) and framework hook.

The pattern in one place (pseudocode)

Every concrete implementation in the references is a variation of this shape. If you are tempted to skip loading a reference, re-read this block — the real code needs library-specific APIs, context-propagation mechanics, and platform pitfalls that pseudocode cannot capture.

# At service startup: configure the structured logger once (JSON to stdout).

# Per unit of work (HTTP request, queue message, cron tick):

on entry:
    event = {
        request_id:   new_or_inherited_id(),
        service:      "checkout",
        version:      BUILD_SHA,
        environment:  "prod",
        # + whatever environment/identity fields the platform gives you
    }
    bind_into_context(request_id = event.request_id)   # so nested logs carry it
    start = now()

    try:
        run_handler()      # handlers call annotate(user_id=..., tenant_id=..., cart_total_cents=...)
    except Exception as exc:
        event.error_class   = type(exc).__name__
        event.error_message = str(exc)
        raise              # re-raise; do NOT log-and-raise
    finally:
        event.duration_ms   = now() - start
        event.status_code   = response.status
        event.route         = route_template            # low-cardinality
        event.path          = raw_path                  # high-cardinality
        logger.info(event)                              # the one canonical event

Core principles

1. Emit one canonical wide event per unit of work

A "unit of work" is an HTTP request, a queue message, a cron tick, a CLI invocation. Allocate an event object at entry, let middleware and business logic annotate it as work happens, and emit it once in a finally block at exit. The finally matters — you want the event even when the handler throws, because that is the case you most need to debug.

The mental shift is from "log what the code is doing" (imperative, noisy, unqueryable) to "record what happened to this request" (declarative, one row, queryable). A good canonical event answers every reasonable question about a single request in one place: identity, input, business context, perf, outcome, error.

2. High cardinality and high dimensionality are features, not costs

Cardinality is the number of unique values a field can take. user_id is high cardinality (millions). http_method is low cardinality (a handful). Dimensionality is the number of fields per event; 20–100 is a healthy target. Both matter because you cannot predict at write-time which dimension will be the one you need to slice by during an incident.

The old objection — "high-cardinality data is too expensive" — was true in 2012 and is not true now. Columnar stores (ClickHouse, BigQuery, the backend of every modern observability vendor) handle it fine. Design the schema as if every field will be filterable and groupable, because it will need to be.

3. Include the four context categories on every event

Anything less and the event cannot stand alone.

• Identity: request_id / trace_id, span_id, service, version, deployment_id.
• Environment: region, instance_id, commit_sha, environment (prod/staging), runtime.
• Business: user_id, tenant_id, subscription_tier, feature_flags, cart_total_cents, whatever the domain cares about. Without this, you know an error occurred; you do not know whether it was a $5 customer or a $50k customer.
• Performance and outcome: duration_ms, sub-operation timings (db_ms, cache_ms), status_code, outcome (success/error/timeout), error.class, error.message. Include these on every event, not just errors — that's how you build p99 dashboards for free.

4. Wide events and OpenTelemetry spans are the same idea

A span is a wide event with a start time, end time, and parent/child relationships. If OTel is in play, the canonical middleware should correlate the event with the current span: inject trace_id and span_id into the event, use OTel Semantic Conventions for field names (http.method, http.route, http.status_code, db.system), and let the collector handle shipping. This means your logs and your traces share a primary key, which is the single biggest debugging-speed win available.

If OTel is not in play, a locally-generated request_id propagated via x-request-id is the fallback. Either way, every event must carry a correlation ID.

5. Infrastructure in middleware, business context in handlers

Middleware owns the boring parts: starting the event, binding request_id into context, timing, catching exceptions, emitting in finally. Handlers and business logic only add domain fields (user_id, feature_flag_X, cart_total). This separation is why the canonical pattern scales — individual handlers stay clean, and any new request automatically gets the full infrastructure envelope for free.

6. Stable field names, JSON to stdout, two levels

Pick field names once and never rename them: user_id everywhere, not userId in one service and uid in another. OTel Semantic Conventions is a good baseline schema.

Output is JSON on a single line, written to stdout. The 12-factor reason is real: the app should not know or care where logs end up; a runtime, sidecar, or platform ships them.

Two log levels (info for canonical events, error for things that need paging) are almost always enough. Debug/trace/warn/notice/critical proliferate without adding query power. If you find yourself wanting a debug log, add the data as a field on the wide event instead — it's queryable and survives beyond the current terminal session.

7. Emit once; do not log-and-raise

An exception should produce exactly one log event: the canonical event for that request, with error.class and error.message populated. log.error(e); raise produces two events for one failure and doubles the stack in your aggregator. Let the boundary (middleware) record the error once.

Anti-patterns worth rejecting on sight

Cloudflare Workers — logging-best-practices reference

Concrete guidance for implementing the wide-event / canonical-log-line pattern on Cloudflare Workers. Read SKILL.md first for the principles; this file covers how.

Retrieval first — trust live docs over this file

Cloudflare's observability surface changes weekly: limits, wrangler config shape, billing status of beta features (native tracing), and which OTLP transports are supported all shift. Before writing code or citing a specific number, fetch the current docs. Treat this reference as the shape of the solution, not the source of truth for API details.

If this file and the live docs disagree, trust the docs — especially for numeric limits, billing/beta status, supported OTLP formats, and exact config-key names. Numbers in this file are illustrative; use them to understand tradeoffs (e.g. "AE is sampled, Workers Logs is not"), not to hard-code thresholds.

The five observability primitives — pick deliberately

Workers offer distinct primitives that are often conflated. The right answer is usually "enable all five, use each for what it's good at".

Recommended default for a new Worker: Workers Logs + native Traces + one Analytics Engine dataset, enabled together in wrangler.jsonc.

Enable Workers Logs

Minimum Wrangler 3.78.6.

{
  "observability": {
    "enabled": true,
    "head_sampling_rate": 1,
    "logs": { "invocation_logs": true },
    "traces": {
      "enabled": true,
      "destinations": ["honeycomb-prod"],   // configured in CF dashboard
      "head_sampling_rate": 0.1
    }
  }
}

Per-environment: [env.staging.observability]. Head sampling decides at request entry and keeps all logs in that invocation or drops all of them — this preserves trace coherence.

The key behaviour: console.log auto-indexes JSON

Workers Logs detects when console.log is called with an object and indexes every field for dashboard filtering, aggregation, and alerting with unlimited cardinality. String-concatenated logs produce a single opaque blob that cannot be queried.

//  Bad — one indexed string, no queryable fields
console.log("user " + userId + " bought " + sku + " for " + amountCents);

//  Good — wide event, every field filterable in Query Builder
console.log({
  msg: "purchase",
  user_id: userId,
  tenant_id: tenantId,
  sku,
  amount_cents: amountCents,
  currency: "GBP",
  cart_size: cart.length,
  ab_variant: "checkout_v3",
  duration_ms: Date.now() - start,
});

This is the wide-event pattern on Workers — emit one of these per request at the end of the handler. Limits as of writing: 256 KB per log (then truncated with $cloudflare.truncated = true), 20M logs/month included on paid, 5B/day before account-wide sampling kicks in.

The canonical-event middleware (Hono)

This is the piece of code that, once added to a Worker, upgrades its observability from "bad" to "good". Attach it once; every request gets a wide event for free.

import { Hono } from "hono";

type Env = { Bindings: { AE: AnalyticsEngineDataset } };
const app = new Hono<Env>();

app.use("*", async (c, next) => {
  const start = Date.now();
  const cf = c.req.raw.cf ?? {};
  const rayId = c.req.header("cf-ray");
  c.set("ray_id", rayId);       // accessible downstream via c.get

  let err: unknown;
  try { await next(); }
  catch (e) { err = e; throw e; }
  finally {
    const duration_ms = Date.now() - start;
    const event = {
      msg: "http.request",
      ray_id: rayId,
      method: c.req.method,
      route:  c.req.routePath,                    // /users/:id — low cardinality
      path:   new URL(c.req.url).pathname,        // /users/123 — high cardinality
      status: c.res.status,
      duration_ms,
      colo: cf.colo,
      country: cf.country,
      asn: cf.asn,
      user_id:   c.get("user_id"),
      tenant_id: c.get("tenant_id"),
      error: err instanceof Error
        ? { name: err.name, message: err.message }
        : undefined,
    };
    console.log(event);                           // → Workers Logs

    // Wide-event metric → Analytics Engine (index by tenant for sampling fairness)
    c.env.AE.writeDataPoint({
      indexes: [c.get("tenant_id") ?? "anon"],
      blobs:   [c.req.routePath, c.req.method, String(c.res.status), cf.colo ?? "", cf.country ?? ""],
      doubles: [duration_ms],
    });
  }
});

Handlers annotate via c.set("user_id", ...) early in the request; the middleware picks them up in finally. Do not use Hono's built-in logger() middleware for production — it is dev-time sugar and emits an unstructured string.

Always-include Cloudflare context

Every wide event on Workers should carry these, because they are the fields that actually let you debug Cloudflare-specific issues (a bad POP, a noisy ASN, a specific region failing).

function cfContext(req: Request) {
  const cf = req.cf ?? {};
  return {
    ray_id:        req.headers.get("cf-ray"),          // canonical request ID
    client_ip:     req.headers.get("cf-connecting-ip"),
    colo:          cf.colo,                            // LHR, IAD — the POP
    country:       cf.country,
    asn:           cf.asn,
    as_org:        cf.asOrganization,
    tls_version:   cf.tlsVersion,
    http_protocol: cf.httpProtocol,                    // HTTP/2, HTTP/3
    bot_score:     cf.botManagement?.score,
  };
}

cf-ray is Cloudflare's native request ID (<hex>-<colo>). Use it as the correlation key when OTel is not available.

Analytics Engine — for high-cardinality metrics

Analytics Engine is the right place for per-tenant / per-user timing and count metrics. It uses weighted adaptive sampling per index value so rare tenants are preserved while hot ones get downsampled.

// wrangler.jsonc
{ "analytics_engine_datasets": [ { "binding": "AE", "dataset": "app_events" } ] }

// Worker
env.AE.writeDataPoint({
  indexes: [tenantId],                                    // EXACTLY ONE, ≤ 96 bytes
  blobs:   [route, method, country, colo, String(status), variant],
  doubles: [durationMs, cpuMs, bytesOut, subrequestCount],
});

Limits at time of writing: up to 20 blobs, up to 20 doubles, exactly one index (multiple = silently dropped), 16 KB total blob payload, 250 datapoints per invocation, 3-month retention.

Index choice matters. The index is what sampling fairness is keyed by. Pick a stable grouping column (tenant_id, customer_id, api_key_hash), not a per-request ID like request_id — that defeats the sampling benefit.

Query with `sum(_sample_interval)`, not `count()`. Every row carries _sample_interval (inverse of sample rate); ignoring it gives you wrong numbers for high-traffic indexes.

Native OpenTelemetry traces

Cloudflare ships automatic tracing as of 2025 (open beta, check current billing status). Spans are emitted for handler invocations, fetch(), cache, KV/R2/D1/Queues/DO bindings — no instrumentation code required. W3C traceparent propagates automatically across service bindings, subrequests, and Durable Objects.

Current limitations: OTLP/JSON only (not protobuf), so Datadog/Elastic APM do not work without an intermediary. When the workload needs custom spans or protobuf, use `@microlabs/otel-cf-workers` instead — it requires compatibility_flags = ["nodejs_compat"] and gives full OTel SDK control.

When OTel is configured, logs exported via OTLP share the trace ID automatically — backends like Honeycomb/Sentry/Axiom will link traces and logs for you.

Pitfalls unique to Workers

• `waitUntil` is unreliable for log flushing. It gives up to 30s of post-response runtime but is best-effort — if the Worker throws, queued work may be dropped. For billing-critical or audit logs, use a Tail Worker or push to Cloudflare Queues from inside the handler. console.log itself does not need waitUntil; invocation logs flush via the runtime lifecycle.
• No filesystem, no long-lived process. No rotating log files. Everything goes through console.log, Analytics Engine, Tail Worker, or Logpush.
• Isolate reuse. Module-scope state persists across requests in the same isolate. Per-request state at module scope leaks between users; always scope to the request.
• Subrequest budget. 50 free / 1000 paid-bundled / unlimited unbound. Direct HTTP log shipping from the Worker eats this — prefer Workers Logs, Analytics Engine, or Tail Workers which do not count.
• CPU time limit. Serialising huge objects into a log call can blow the CPU budget. Cap event size.
• WebSocket handlers. console.log during a long-lived WebSocket may not appear in wrangler tail until the socket closes; prefer Workers Logs or synchronous pushes for visibility.
• Field naming is forever. Workers Logs indexes by exact JSON path. Renaming userId → user_id mid-flight splits your dashboard. Pick once.

Tail Workers — when you need guaranteed delivery

A Tail Worker runs once per invocation of a producer Worker, after the producer finishes, and receives its logs/exceptions/outcome as input. It runs regardless of whether the producer threw — which makes it the right tool for guaranteed shipping and for centralised redaction before egress.

// tail-worker/src/index.ts
export default {
  async tail(events, env, ctx) {
    for (const e of events) {
      // e.scriptName, e.outcome ("ok" | "exception" | "exceededCpu" | ...)
      // e.logs: { level, message, timestamp }[]
      // e.exceptions: { name, message, timestamp }[]
      // e.event (FetchEventInfo etc.)
      await ship(redact(e), env);
    }
  }
} satisfies ExportedHandler;

// producer wrangler.jsonc
// { "tail_consumers": [{ "service": "tail-worker" }] }

Request URLs and headers are redacted by default — call getUnredacted() if the Tail Worker needs them.

Sampling on Workers

Head sampling via head_sampling_rate is the simplest lever. For outcome-based tail sampling (keep errors, slow requests, VIP tenants — see sampling.md), the pragmatic patterns on Workers are:

1. In-handler keep-rule: always emit to Analytics Engine; console.log only when status >= 400, duration_ms > threshold, or tenant is flagged important. 2. Tail Worker filter: emit everything from the producer, let the Tail Worker decide what to ship downstream. 3. For OTel traces, configure tail sampling in the collector (or at the backend like Honeycomb's Refinery) rather than the Worker.

Python — logging-best-practices reference

Concrete guidance for implementing the wide-event / canonical-log-line pattern in Python services. Read SKILL.md first for the principles; this file covers how.

Library choice

Default to `structlog`. It is the only mainstream Python logger built around structured events with first-class contextvars support. stdlib logging can be bridged through it for library logs.

• stdlib logging alone — OK for one-file scripts. Not enough for a service: producing JSON with contextvars requires significant custom scaffolding.
• loguru — pleasant for CLI tools, but its model is string-templates-with-extras rather than true structured events, and its contextvars story is weaker. Avoid for distributed services.
• python-json-logger — useful as a stdlib formatter if you cannot fully bridge through structlog (e.g. a library you cannot control).

Baseline structlog config

This is the config worth copying into a new service. It gives pretty console output in dev (auto-detected via isatty) and JSON-to-stdout in prod, bridges stdlib logging through the same pipeline (so uvicorn, sqlalchemy, etc. render consistently), and exposes contextvars for request-scoped binding.

# logging_setup.py
import logging.config, os, sys
import structlog
from structlog.types import EventDict, Processor

def _drop_color_message_key(_, __, event_dict: EventDict) -> EventDict:
    # Uvicorn duplicates the message under color_message; drop the duplicate.
    event_dict.pop("color_message", None)
    return event_dict

def setup_logging(env: str = os.getenv("ENV", "dev"), level: str = "INFO") -> None:
    shared: list[Processor] = [
        structlog.contextvars.merge_contextvars,          # must come first
        structlog.stdlib.add_logger_name,
        structlog.stdlib.add_log_level,
        structlog.processors.StackInfoRenderer(),
        structlog.processors.UnicodeDecoder(),
        _drop_color_message_key,
        structlog.processors.TimeStamper(fmt="iso", utc=True),
    ]

    if env == "dev" and sys.stderr.isatty():
        renderer: Processor = structlog.dev.ConsoleRenderer()
    else:
        shared += [structlog.processors.dict_tracebacks,
                   structlog.processors.EventRenamer("message")]
        renderer = structlog.processors.JSONRenderer()

    structlog.configure(
        processors=shared + [structlog.stdlib.ProcessorFormatter.wrap_for_formatter],
        wrapper_class=structlog.make_filtering_bound_logger(getattr(logging, level)),
        logger_factory=structlog.stdlib.LoggerFactory(),
        cache_logger_on_first_use=True,
    )

    logging.config.dictConfig({
        "version": 1,
        "disable_existing_loggers": False,
        "formatters": {"structlog": {
            "()": structlog.stdlib.ProcessorFormatter,
            "processor": renderer,
            "foreign_pre_chain": shared,
        }},
        "handlers": {"default": {
            "class": "logging.StreamHandler",
            "stream": "ext://sys.stdout",
            "formatter": "structlog",
        }},
        "root": {"handlers": ["default"], "level": level},
        "loggers": {
            "uvicorn.access": {"handlers": ["default"], "level": "INFO", "propagate": False},
            "sqlalchemy.engine": {"level": "WARNING"},
        },
    })

For very hot paths (>10k events/s), swap LoggerFactory for BytesLoggerFactory and the JSON renderer for JSONRenderer(serializer=orjson.dumps). This is the fastest known structlog configuration.

Context binding — bind once, read everywhere

structlog.contextvars is how request/job-scoped context reaches every log call without threading it through every function signature. It works correctly for both threads and asyncio tasks.

from structlog.contextvars import bind_contextvars, clear_contextvars

clear_contextvars()                                  # fresh slate per request
bind_contextvars(request_id=rid, user_id=uid, tenant_id=tid)

log = structlog.get_logger()
log.info("order.placed", order_id=oid, total_cents=total)
# -> {"event": "order.placed", "request_id": "...", "user_id": "...", "tenant_id": "...", "order_id": "...", ...}

Two rules that save hours of confusion:

1. merge_contextvars must be the first processor in the chain. Otherwise the bound vars are invisible to renderers. 2. clear_contextvars() at the start of each request/job. contextvars do not automatically reset between requests in async frameworks; stale context leaking between requests is a classic bug.

Canonical log line — ASGI middleware

The middleware is the leverage point: it creates a per-request event dict, binds request_id into contextvars, lets handlers annotate via annotate(...), and emits one event in finally. Works for FastAPI, Starlette, and anything speaking ASGI.

A vetted implementation ships with this skill at `scripts/canonical_asgi.py` — copy it into the target project rather than re-deriving it from scratch. The file handles the subtle cases (case-insensitive header lookup, latin-1 decoding, route template extraction, correct ContextVar reset, inheriting x-request-id if present). Import it and add to the app:

from canonical_asgi import CanonicalLogMiddleware, annotate

app = FastAPI()
app.add_middleware(CanonicalLogMiddleware)

Handlers then annotate without knowing anything about the middleware:

from canonical import annotate

@app.post("/checkout")
async def checkout(req, user=Depends(current_user)):
    annotate(user_id=user.id, subscription=user.tier, feature_flag_new_checkout=True)
    cart = await load_cart(user.id)
    annotate(cart_total_cents=cart.total, cart_item_count=len(cart.items))
    ...

For Celery/RQ/cron jobs, use the same shape as a context manager that initialises the dict on task start and emits in finally.

The BaseHTTPMiddleware gotcha

If you use starlette.middleware.base.BaseHTTPMiddleware instead of the raw ASGI class above, contextvars bound inside dependencies/handlers will not be visible in the middleware's finally. BaseHTTPMiddleware copies the context, so mutations happen on the copy and are discarded. This is FastAPI issue #4696 and hits people repeatedly.

Fix: use a pure ASGI middleware class (shown above). If you cannot, stash mutations on request.state and re-bind from there.

OpenTelemetry correlation

If OTel is running, inject the current span's IDs into every event. One processor does it:

from opentelemetry import trace

def add_otel_context(_, __, event_dict):
    span = trace.get_current_span()
    ctx = span.get_span_context() if span else None
    if ctx and ctx.is_valid:
        event_dict["trace_id"] = format(ctx.trace_id, "032x")
        event_dict["span_id"]  = format(ctx.span_id, "016x")
        event_dict["trace_sampled"] = ctx.trace_flags.sampled
    return event_dict

Insert it just before the renderer. Also add service.name, service.version, deployment.environment as static fields. Use OTel Semantic Conventions for field names (http.method, http.route, http.status_code, db.system, messaging.destination.name) so your events line up with whatever backend consumes them.

For auto-shipping logs via OTLP, opentelemetry-instrument --logs_exporter otlp ... and OTEL_PYTHON_LOG_CORRELATION=true handle the stdlib bridge. The processor above still matters for JSON-to-stdout flows.

Exceptions

• Inside except: log.exception("operation.failed", order_id=oid) — attaches exc_info automatically.
• Production chain should include structlog.processors.dict_tracebacks so stack frames serialise as a structured array (queryable by exception type, frame, line) rather than an opaque string.
• Do not log.error(exc); raise. Either log.exception and swallow, or just raise and let the boundary (canonical middleware) record error.class and error.message once. Two events for one failure double-counts in every downstream system.

Python-specific anti-patterns

The generic anti-patterns in SKILL.md apply; these are the ones that bite specifically in Python.

Sampling in Python

For head-based rate sampling or content-based drops, write a processor that raises structlog.DropEvent:

def drop_health_checks(logger, name, event_dict):
    if event_dict.get("http.path") == "/health":
        import random
        if random.random() > 0.01:   # keep 1%
            raise structlog.DropEvent
    return event_dict

Gotcha: do not raise DropEvent inside ProcessorFormatter (i.e. the stdlib-bridge path) — it crashes the formatter. Drop in structlog's own chain, before the formatter wrapper.

For tail sampling, see `sampling.md`. In short: the right home for tail sampling in Python is the OpenTelemetry Collector's tail_sampling processor keyed on trace_id, not the application.

Sampling — logging-sucks reference

Cross-platform sampling strategy. Read SKILL.md first. This file covers why naive random sampling is wrong and what to do instead.

The problem with head-based random sampling

Head sampling decides at request entry: flip a coin, if it's heads keep all logs for this request, otherwise drop them. At 1% head-sampling, you keep 1% of errors. You keep 1% of slow requests. You keep 1% of the requests that broke for your biggest customer. At scale, 1% is often enough for dashboards — but it is catastrophic for debugging, which is exactly when you need the logs.

Head sampling has one genuine virtue: it is cheap and it preserves trace coherence (every log in a kept request is kept together). Use it as a prefilter for bulk volume reduction, not as the whole strategy.

Tail sampling — decide based on outcome

Tail sampling makes the keep/drop decision after the request completes, when the outcome is known. The rules that matter in practice:

1. Always keep errors. 100% of status >= 500, 100% of unhandled exceptions, 100% of explicit outcome = "error". 2. Always keep slow requests. Requests above the p99 latency threshold — these are the ones that build the tail of your latency distribution and expose cascading slowness. 3. Always keep VIPs. Enterprise customers, internal staff, flagged debug users. One angry enterprise customer with missing logs costs more than a month of storage. 4. Always keep feature-flag-enabled requests. When a flag is at 1% rollout, you need 100% visibility into that 1% to evaluate it, not 1%-of-1%. 5. Randomly sample the rest at 1–5%. This fills in the normal-case baseline for dashboards.

This keeps cardinality where it matters (the failures, the outliers, the important users) and trims it where it does not (the boring successful p50).

A keep-rule sketch (language-agnostic)

function shouldKeep(event):
    if event.status_code >= 500:                     return true
    if event.outcome == "error":                     return true
    if event.duration_ms > P99_THRESHOLD_MS:         return true
    if event.user.tier in ("enterprise", "internal"):return true
    if event.feature_flags has any experimental:     return true
    return random() < 0.05

Apply this in the finally of the canonical middleware — that is the moment when you have the full event and can make an outcome-aware decision.

Where to implement it

Three sensible homes, from closest-to-the-code to furthest:

1. In-app, in the canonical middleware. The event is already assembled; gating the logger.info(event) call on a keep-rule function is cheap and works everywhere. Downside: you pay the CPU to assemble events you then drop. Usually fine.

2. In a sidecar / Tail Worker / OTel Collector. The application emits everything; a downstream process filters before shipping to the expensive tier. This is where outcome-based tail sampling actually belongs when you have distributed tracing — the OTel Collector's `tail_sampling` processor can key on trace_id and hold spans until all spans of a trace arrive, then make a whole-trace decision. On Cloudflare, a Tail Worker does the same job.

3. At the backend (Honeycomb Refinery, vendor-side rules). Useful when operational control of the collector is awkward. Same shape of rules, just run further from the app.

Combine freely: app-side drops the /health noise, the collector does outcome-based keep-rules, the backend does long-tail downsampling. The rules compose.

Python

Head-based / content-based drops belong in a structlog processor that raises structlog.DropEvent:

import random, structlog

def drop_health_checks(_, __, event_dict):
    if event_dict.get("http.path") == "/health" and random.random() > 0.01:
        raise structlog.DropEvent
    return event_dict

Place before the formatter-wrapper processor. Raising DropEvent inside ProcessorFormatter (the stdlib-bridge path) crashes the formatter; drop in structlog's own chain.

For tail sampling in Python, the real home is the OpenTelemetry Collector with tail_sampling keyed on trace_id. Let the app emit everything; push the decision downstream.

Cloudflare Workers

Three practical layers:

• `observability.head_sampling_rate` in wrangler.jsonc — platform-level head sampling. Whole invocations kept/dropped together, trace coherence preserved.
• In the canonical middleware — only console.log the wide event when a keep-rule matches; always writeDataPoint to Analytics Engine (AE's own sampling handles volume there with fairness).
• In a Tail Worker — producer emits everything; the Tail Worker filters before shipping to an external backend. This is the right home for expensive outcome-aware rules that shouldn't run on every hot-path invocation.

What you lose when sampling — and how to mitigate

The fundamental risk is losing the one event that would have told you what broke. Two mitigations that matter:

• Sample by trace, not by span. If you keep 5% of traces, you keep every span of those traces; you do not get fragmented half-traces that look like bugs. This is why the OTel Collector tail processor keys on trace_id.
• Always keep errors, always keep outliers, always keep VIPs. This is the single biggest difference between "sampling works" and "sampling silently breaks debugging".

Rule of thumb

Start without tail sampling — emit everything. Add tail sampling only when log volume or cost forces it. When you add it, add keep-rules first (errors, slow, VIPs, flags), then random sampling of the remainder. Never flip the order: a 5% random sampler without keep-rules is worse than no sampler at all, because it produces a false sense of observability.