Observability

Observability

If you cannot ask arbitrary questions about your system's behavior from the outside, your system is not observable — it is merely monitored.

Observability is the ability to understand internal system state from its external outputs. Three complementary signals make this possible: logs (discrete events), metrics (aggregate measurements), and traces (request-scoped causal chains). Each pillar answers different questions. Using the wrong pillar for a question wastes resources and hides the answer.

This skill covers high-level observability discipline — purposes, interconnection, and good practices for all three pillars. It is technology-agnostic: specific tools (Prometheus, StatsD, OpenTelemetry) have their own dedicated skills.

---

The Three Pillars

<pillars>

Logs — What Happened

Logs are timestamped, discrete event records. They capture what happened at a specific moment: an error thrown, a user action, a configuration loaded, a connection refused.

Use logs when you need:

• Rich diagnostic context for a specific event
• Debugging information with full error details and stack traces
• Audit trails of who did what and when
• Record of discrete state transitions

Logs are poor at:

• Showing aggregate system health (use metrics)
• Tracing request flow across services (use traces)
• High-frequency numeric trends (too expensive at volume)

Metrics — How Is It Doing

Metrics are numeric measurements aggregated over time. They capture how the system is performing as quantitative time series: request rates, error percentages, latencies, queue depths, resource utilization.

Use metrics when you need:

• Real-time health signals and alerting
• Trend analysis over hours, days, weeks
• Capacity planning and saturation monitoring
• Pre-aggregated data that scales cheaply regardless of traffic

Metrics are poor at:

• Explaining why something is broken (use logs)
• Showing the path of a single request (use traces)
• Storing per-event detail (cardinality explosion)

Traces — How Did It Flow

Traces record the causal chain of operations that make up a single request as it propagates through distributed components. A trace is a tree of spans, where each span represents one unit of work (an HTTP call, a database query, a queue publish).

Use traces when you need:

• End-to-end latency breakdown across services
• Dependency mapping and bottleneck identification
• Understanding the path a failing request took
• Correlating work across process and network boundaries

Traces are poor at:

• Aggregate health monitoring (use metrics)
• Detailed per-event diagnostics on a single node (use logs)
• Cheap, long-term trend storage (traces are expensive at 100% sampling)

</pillars>

Choosing the Right Signal

---

Structured Logging

<structured-logging>

Always Structured

Emit logs as structured records (JSON or equivalent key-value format) with a consistent schema. Unstructured string logs are acceptable only in local development. Structured logs are machine-parseable, indexable, and filterable at scale.

Log Levels

Use levels consistently. Every team member must agree on what each level means.

Rules:

• Production defaults to INFO or above. DEBUG/TRACE are off unless explicitly enabled

for a bounded investigation window.

• WARN is not a dumping ground. If it never leads to action, it is noise — downgrade

to DEBUG or remove it.

• ERROR means something is broken. Expected conditions (404 for missing resources,

validation failures from bad input) are not errors — they are INFO with a status field.

• Make log level configurable at runtime without restarts.

Structured Fields

Every log record should include these baseline fields:

Add contextual fields relevant to the event:

Sensitive Data

Never log:

• Passwords, tokens, API keys, secrets
• Full credit card numbers, SSNs, or equivalent PII
• Session tokens or authentication cookies
• Request/response bodies containing user-submitted personal data

When user identifiers are needed, log opaque IDs (user_id), not email addresses or names. If regulations (GDPR, HIPAA) apply, verify that logged fields comply. When in doubt, omit the field.

Logging at Boundaries

At application startup:

• INFO: service name, version, loaded configuration (without secrets), listen address
• WARN: degraded mode (e.g., fallback to local cache because Redis is unreachable)
• ERROR/FATAL: unrecoverable startup failures

Per incoming request:

• INFO: method, path (scrubbed of PII), status code, duration, request dimensions

(tenant, region)

• WARN/ERROR: only for unexpected exceptions; catch at the top-level handler

Per outgoing dependency call:

• INFO or DEBUG: target service, operation, status, duration
• ERROR: failures in dependent services (Redis, database, queue, etc.)

Log Once, at the Right Level

Log a raised exception once. Do not catch-log-rethrow at every layer. Let exceptions propagate to the top-level handler, which logs with full context. If you must log and rethrow, do it only when adding context that would otherwise be lost.

</structured-logging>

---

Metrics

<metrics>

Metric Types

Rules:

• Use counters for events that accumulate. Derive rates from counters (rate(),

increase()), never store pre-computed rates.

• Use gauges for current-state snapshots. Never rate() a gauge.
• Use histograms for latency and size distributions. Histograms enable percentile

calculation across instances; summaries do not aggregate.

• Export timestamps as Unix epoch seconds, not "time since" values.
• Initialize all metrics with a zero value at startup to avoid missing-metric problems.

What to Measure

The Four Golden Signals (Google SRE)

For every user-facing service, measure these four:

Distinguish successful latency from error latency. A fast 500 is not good latency. A slow error is worse than a fast error. Track both.

RED Method (Request-Centric)

For every microservice:

• Rate — requests per second
• Errors — failed requests per second
• Duration — distribution of request latency

RED is a focused subset of the golden signals, optimized for request-driven services.

USE Method (Resource-Centric)

For every resource (CPU, memory, disk, network, thread pool):

• Utilization — percentage of capacity in use
• Saturation — backlog / queue depth
• Errors — resource-level error count

RED tells you what is degraded from the user's perspective. USE tells you why at the infrastructure level. Use both together.

Service-Type Instrumentation

Metric Naming

Metric names should be self-documenting. Follow these conventions:

• Prefix with namespace. myapp_http_requests_total, not requests_total.
• Use base units. Seconds (not milliseconds), bytes (not megabytes),

ratio 0-1 (not percentage 0-100).

• Suffix with unit. _seconds, _bytes, _total (for unit-less counters).
• One metric, one unit, one quantity. Never mix request size with request duration

in the same metric.

• snake_case. http_request_duration_seconds, not httpRequestDurationSeconds.

Labels and Cardinality

Labels add dimensions to a metric. Every unique combination of label values creates a separate time series.

Good labels (bounded, low cardinality):

• method (GET, POST, PUT, DELETE)
• status_code (200, 404, 500 — or class: 2xx, 4xx, 5xx)
• service, region, version

Dangerous labels (unbounded, high cardinality):

• user_id, email, session_id
• request_path with dynamic segments (/users/12345)
• error_message (arbitrary strings)

Rules:

• Keep label cardinality below 10 values per label for most metrics.
• If a label can grow unbounded, it does not belong on a metric. Log it instead.
• Use labels instead of encoding dimensions in the metric name.

http_requests_total{method="GET"}, not http_get_requests_total.

• Ensure sum() or avg() across all label values is meaningful. If not,

split into separate metrics.

Percentiles and Tail Latency

Averages hide outliers. A service with 100ms average latency may have 1% of requests taking 5 seconds. That 1% tail can dominate user experience when users hit multiple services per page load.

• Always track p50, p90, p99 latency at minimum.
• Use histograms with exponentially distributed bucket boundaries

(e.g., 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s).

• Alert on p99, not mean. Mean latency alerts miss tail degradation.

</metrics>

---

Distributed Tracing

<tracing>

Core Concepts

Span Kinds

Context Propagation

Context propagation is the mechanism that connects spans across process boundaries into a single trace. Without it, you get disconnected spans, not traces.

Rules:

• Propagate context on every outgoing call. HTTP headers (W3C Trace Context or

B3), message metadata, gRPC metadata — every cross-process boundary must carry trace context.

• Extract context on every incoming call. The receiving service must extract

trace context and create a child span under the propagated parent.

• Use W3C Trace Context (`traceparent`/`tracestate`) as the default propagation

format unless the ecosystem requires otherwise (e.g., legacy B3).

• Never generate a new trace ID when you should be continuing an existing trace.

A new trace ID means a broken trace.

What to Trace

Instrument at meaningful boundaries:

Span Attributes

Attach attributes that enable filtering and analysis:

Use semantic conventions for attribute names rather than inventing custom ones. Consistent naming enables cross-service analysis.

Span Status

Leave status as Unset for normal success. Set Error only for actual failures. Do not set Error for client errors like 404 on a server span — the server operated correctly.

Sampling

At high traffic volumes, tracing 100% of requests is expensive. Sampling reduces cost while preserving signal.

Rules:

• Never drop error traces. If cost is a concern, sample successful traces at a lower

rate but keep 100% of error and high-latency traces.

• Sample at the entry point (head) and propagate the decision. Do not let each

service decide independently — this creates partial traces.

• Start with a low sampling rate (1-10%) and increase based on need, not the reverse.

</tracing>

---

Connecting the Pillars

<interconnection>

The three pillars become powerful when correlated. An alert fires on a metric → you find the offending trace → the trace points to a span → the span's logs reveal the root cause.

Correlation Keys

Rules:

• Embed trace_id and span_id in every log record emitted within a request context.

This is the primary bridge between logs and traces.

• Use a correlation/request ID that is assigned at the edge (API gateway, load

balancer) and propagated to all downstream services.

• Attach exemplars to metrics. An exemplar is a trace_id attached to a specific

metric observation, enabling drill-down from a metric spike to a representative trace.

The Correlation Workflow

Metrics dashboard shows error rate spike
  → Filter by service + time window
  → Find exemplar trace_id on the error counter
  → Open trace in tracing UI
  → Identify the failing span (database timeout)
  → Search logs by trace_id for full error details
  → Root cause: connection pool exhausted

Metric-to-Trace Exemplars

Exemplars attach a trace_id sample to a metric data point. When you see a latency spike on a histogram, the exemplar gives you a concrete trace to investigate rather than guessing.

• Attach exemplars to histogram observations for latency metrics.
• Attach exemplars to counter increments for error metrics.
• Not every metric point needs an exemplar — one per scrape interval is sufficient.

Trace-to-Log Linking

When viewing a trace, each span should link to its logs. When viewing a log, the trace_id should link back to the full trace. This bidirectional linking is the backbone of incident investigation.

</interconnection>

---

Anti-Patterns

---

Application

<application>

When Writing Code

• Instrument from the start. Add golden signal metrics, structured logging, and

trace context propagation before the first production deploy — not after the first incident.

• Follow the conventions silently. Apply structured logging, metric naming, and

tracing patterns without narrating each rule.

• If the codebase has existing patterns, follow them. Consistency within a codebase

beats theoretical correctness. Flag divergences from this skill's guidance once, then move on.

• Choose the right pillar. Before adding instrumentation, ask: "Is this a metric,

a log, or a span?" Use the decision table above.

• Connect the signals. Every log in a request context must carry trace_id and

span_id. Every error metric should have an exemplar.

When Reviewing Code

• Check that new endpoints/operations have golden signal coverage. Missing metrics

on a new endpoint is a review blocker.

• Verify structured logging. Unstructured log.Print("something happened") in

production code should be flagged with the fix inline.

• Check log levels. Expected client errors logged as ERROR, or debug noise left

on at INFO, are common mistakes.

• Verify trace context propagation. Any new outgoing HTTP/gRPC/queue call must

propagate trace context. Missing propagation breaks traces.

• Check label cardinality. New metric labels must be bounded. Flag unbounded labels

(user IDs, free-text) immediately.

• No sensitive data in logs or span attributes. Passwords, tokens, PII in telemetry

is a security and compliance defect.

Bad review comment:
  "According to observability best practices, you should consider
   adding structured logging with appropriate fields..."

Good review comment:
  "Missing trace_id in log context — requests through this handler
   won't correlate to traces. Add ctx.TraceID() to the logger fields."

</application>

---

Integration

This skill provides observability discipline alongside other skills:

• Coding skill — Discovery, planning, verification workflow
• Observability (this skill) — What to log, measure, and trace
• Tool-specific skills (Prometheus, StatsD, OTel) — How to implement with a

specific technology

The coding skill governs workflow. This skill governs observability design decisions. Tool-specific skills govern implementation details for their respective technologies.

Observability is not an afterthought. Instrument from day one. If you cannot observe it, you cannot operate it.