Oma Observability

Observability Agent - Intent-based Router

Scheduling

Goal

Route, design, tune, and review observability work across MELT+P signals, layers, boundaries, vendor categories, transport choices, meta-observability, and incident forensics.

Intent signature

• User asks for observability, telemetry, OTel, metrics, logs, traces, profiles, SLOs, RUM, APM, incident forensics, trace propagation, transport tuning, or observability-as-code.
• User needs vendor/category routing or observability architecture instead of a single vendor's already-covered setup.

When to use

• Setting up an observability pipeline (OTel SDK + Collector + vendor backend)
• Designing traceability across service and domain boundaries (W3C propagators, baggage, multi-tenant, multi-cloud)
• Tuning transport layer (UDP/MTU, OTLP gRPC vs HTTP, Collector DaemonSet vs sidecar topology)
• Running incident forensics (6-dimension localization: code / service / layer / host / region / infra)
• Selecting a vendor category (OSS full-stack vs commercial SaaS vs high-cardinality specialist vs profiling specialist)
• Implementing observability-as-code (Grafana Jsonnet dashboards, PrometheusRule CRD, OpenSLO YAML, SLO burn-rate alerts)
• Meta-observability (pipeline self-health, clock skew detection, cardinality guardrails, retention matrix)
• Covering the MELT+P signal set: metrics, logs, traces, profiles (OTEP 0239), cost (OpenCost), audit (SOC2/ISO), privacy (GDPR/PIPA)
• Migrating off deprecated tools (Fluentd → Fluent Bit or OTel Collector, per CNCF 2025-10 guide)

When NOT to use

• LLM ops (prompt versioning, evals, gen_ai span deep dive); use Langfuse, Arize Phoenix, LangSmith, or Braintrust directly
• Data pipeline lineage: use OpenLineage + Marquez, dbt test, or Airflow lineage backends
• IoT / hardware / datacenter physical-layer telemetry (IPMI, BMC, SNMP); use vendor DCIM tooling (Nlyte, Sunbird, Device42)
• Chaos engineering orchestration: use Chaos Mesh, Litmus, Gremlin, or ChaosToolkit (this skill consumes their telemetry; it does not orchestrate chaos)
• GPU / TPU infrastructure observability: use NVIDIA DCGM Exporter + Prometheus
• Software supply chain (SBOM, attestation): use sigstore (cosign / rekor), in-toto framework, SLSA level attestations
• Incident response workflow (on-call rotation, paging, escalation); use PagerDuty, OpsGenie, or Grafana OnCall
• Single-vendor setup already fully covered by that vendor's own published skill; invoke the vendor skill directly

Expected inputs

• Observability intent, target system, architecture boundary, signals, vendor context, and incident symptoms if any
• Existing OTel/collector/vendor configs, dashboards, SLOs, trace/log/metric examples, or deployment topology

Expected outputs

• Routed observability guidance, setup/migration/tuning plan, incident-forensics path, alerting/SLO guidance, or observability-as-code recommendations
• Transport, meta-observability, privacy, audit, and retention checks
• Vendor delegation target when appropriate

Dependencies

• OTel/W3C/CNCF references and resources under resources/
• Vendor categories, matrix, standards, incident forensics, meta-observability, transport, layers, boundaries, and signal guides

Control-flow features

• Branches by intent, vendor category, layer/boundary/signal matrix, transport topology, privacy/audit risk, and incident localization dimension
• May read/write observability config and docs; generally delegates vendor-specific implementation
• Requires live status verification for load-bearing CNCF/vendor currency

Structural Flow

Entry

1. Classify the intent: setup, migrate, investigate, alert, trace, tune, or route. 2. Identify layers, boundaries, signals, and vendor category. 3. Load only the relevant resource guide(s).

Scenes

1. PREPARE: Classify intent and matrix coverage. 2. ACQUIRE: Read configs, topology, telemetry examples, or incident signals. 3. REASON: Route vendor/category, tune transport, assess meta-observability, or localize incident. 4. ACT: Produce setup/migration/tuning/alert/trace/forensics guidance or config changes. 5. VERIFY: Check pipeline health, clock skew, cardinality, retention, privacy, and audit concerns. 6. FINALIZE: Report route, evidence, risks, and handoff references.

Transitions

• If a vendor-owned skill fully covers setup, delegate instead of duplicating docs.
• If Fluentd appears, recommend Fluent Bit or OTel Collector migration.
• If incident investigation is requested, use 6-dimensional localization.
• If transport tuning appears, load transport-specific resources.

Failure and recovery

• If live CNCF/vendor status is load-bearing, verify current status.
• If telemetry samples are missing, provide instrumentation/collection steps before analysis.
• If scope belongs to out-of-scope domains, route to external authoritative tools.

Exit

• Success: observability path is routed, evidence-backed, and checks are explicit.
• Partial success: missing telemetry, stale vendor status, or external-domain handoff is explicit.

Logical Operations

Actions

Tools and instruments

• OTel/CNCF/W3C standards references
• Vendor categories, matrix, incident forensics, meta-observability, transport and signal guides
• Optional CLI/config tooling from the target stack

Canonical workflow path

1. Classify intent: setup, migrate, investigate, alert, trace, tune, or route.
2. Select layer/boundary/signal coverage from `resources/matrix.md`.
3. Load the specific vendor, transport, incident, or signal guide before producing guidance.

When CNCF/vendor status is load-bearing, verify live state at https://landscape.cncf.io.

Resource scope

Preconditions

• Observability intent and system boundary are identifiable.
• Relevant telemetry/config evidence is available or missing evidence is stated.

Effects and side effects

• May recommend or modify observability config, dashboards, alerts, and instrumentation docs.
• May route to vendor-owned skills or external tools.

Guardrails

1. Classify intent before routing: every query goes through intent classification; setup | migrate | investigate | alert | trace | tune | route 2. Category-first, not vendor-registry: delegate to vendor-owned skills via resources/vendor-categories.md; do not duplicate their documentation 3. Transport tuning is the moat: UDP/MTU thresholds, OTLP protocol selection, Collector topology, and sampling recipes are in-skill depth that other skills do not cover 4. Meta-observability is non-negotiable: always validate pipeline self-health, clock sync (< 100 ms drift), cardinality, and retention before declaring setup complete 5. CNCF-first preference: Prometheus, Jaeger, Thanos, Fluent Bit, OpenFeature (Graduated 2024-11), Flagger, Falco (Graduated); OpenTelemetry, Cortex, OpenCost (Incubating) 6. Fluentd is deprecated: per CNCF 2025-10 migration guide, recommend Fluent Bit or OTel Collector for all new and migration work 7. W3C Trace Context as default propagator: translate per cloud (AWS X-Ray X-Amzn-Trace-Id, GCP Cloud Trace, Datadog, Cloudflare, Linkerd) via boundaries/cross-application.md 8. Privacy before features: PII redaction, sampling-aware baggage rules, and compliance (SOC2/ISO immutable audit + GDPR/PIPA erasure) are applied at collection, not only at storage 9. Domain-level trust: all vendor and tool references are timestamped as of 2026-Q2; verify live status at https://landscape.cncf.io 10. No stub in final deliverable: scaffolds are editing anchors only during build phase; remove before output

Out of Scope (use external tools)

The combinations below are outside this skill's boundary. The external tools listed are authoritative for each domain.

Architecture (4 x 4 x 7 matrix)

                  User / Other Skill Query
                            |
                            v
              +-----------------------------+
              |      Intent Classifier      |
              |  setup | migrate | investigate
              |  alert | trace | tune | route|
              +-----------------------------+
                            |
                            v
              +-----------------------------+
              |      Vendor Router          |
              |  category-first delegation  |
              +-----------------------------+
                            |
                            v
              +-----------------------------+
              |   vendor-categories.md      |
              |   (a) OSS Full-Stack        |
              |   (b) Commercial SaaS APM   |
              |   (c) High-Cardinality      |
              |   (d) Profiling Specialist  |
              |   (e) SIEM / Enterprise Logs|
              |   (f) FinOps / Cost         |
              |   (g) Feature Flags/Rollout |
              |   (h) Log Pipeline          |
              |   (i) Time Series Storage   |
              |   (j) Crash Analytics       |
              +-----------------------------+
                            |
                            v
              +-----------------------------+
              |  Matrix Coverage Selector   |
              |  4 Layers x 4 Boundaries    |
              |  x 7 Signals = 112 cells    |
              +-----------------------------+
                            |
                            v
              +-----------------------------+
              |  Transport Depth /          |
              |  Meta-observability         |
              |  UDP, OTLP, Collector,      |
              |  cardinality, clock skew    |
              +-----------------------------+
                            |
                            v
              +-----------------------------+
              |  Incident Forensics         |
              |  6-dim localization:        |
              |  code/service/layer/host/   |
              |  region/infra               |
              +-----------------------------+

Layers (4): L3-network, L4-transport, mesh, L7-application Boundaries (4): multi-tenant, cross-application, slo, release Signals (7): metrics, logs, traces, profiles, cost, audit, privacy

See resources/matrix.md for the full 112-cell coverage map with N/A markers for invalid combinations.

Routes (Intent)

Invocation

Standalone:

/oma-observability "set up OTel stack on Kubernetes"
/oma-observability --migrate "move from Fluentd to Fluent Bit"
/oma-observability --investigate "5xx spike in ap-northeast-2"
/oma-observability --alert "configure SLO burn-rate alert for checkout API"
/oma-observability --trace "W3C propagator across AWS + GCP boundary"
/oma-observability --tune "UDP statsd MTU throughput limit"
/oma-observability --route "multi-tenant log isolation with data residency"

Shared invocation (from other skills): 1. State intent: setup | migrate | investigate | alert | trace | tune | route 2. Pass the user query string 3. Receive routed guidance or a vendor-skill delegation target

How to Execute

Follow resources/execution-protocol.md step by step. See resources/examples.md for end-to-end walkthroughs. Use resources/intent-rules.md for intent classification reference. Use resources/matrix.md for coverage navigation across layers, boundaries, and signals. Use resources/vendor-categories.md for vendor delegation and category selection. Before submitting, run resources/checklist.md.

Integrations with OMA Ecosystem

Versioning & Deprecation

• Spec version pinning: otel_spec / otel_semconv keys in each file's frontmatter document the assumed version. If content depends on a specific attribute stability tier, the tier is stated inline.
• Update triggers (not scheduled):
• OTel semconv promotion (Development → RC → Stable) affecting attributes cited in this skill → update resources/standards.md and the affected file, bump minor version.
• Attribute deprecation → replace across all citing files; migration note in resources/standards.md.
• CNCF status change for a vendor/project named in vendor-categories.md (Graduated / Archived / acquired) → update the vendor table.
• Authoritative live state: https://landscape.cncf.io for CNCF project status. This skill does not promise to track it on any schedule; verify at use time if the information is load-bearing.
• No per-file review stamps: earlier drafts carried last_reviewed / next_review frontmatter. Those were removed because no automated enforcement exists; relying on voluntary manual review produces stale stamps that misrepresent currency. Git history (git log path/to/file) is the source of truth for when a file was last changed.

Contribution Protocol

• Do NOT pre-declare future OMA skill names in user-facing documentation. If OMA-native coverage becomes warranted for an out-of-scope domain, evaluate and name it at that point.
• File edits follow the ownership matrix in docs/plans/designs/005-oma-observability.md §Ownership. CTO co-signs changes to standards.md, matrix.md, anti-patterns.md.
• Run resources/checklist.md §1 Setup validation before merging.

References

• Execution steps: resources/execution-protocol.md
• Intent classification: resources/intent-rules.md
• Coverage matrix: resources/matrix.md
• Standards (OTel spec, W3C, ISO): resources/standards.md
• Vendor categories: resources/vendor-categories.md
• Incident forensics: resources/incident-forensics.md
• Meta-observability: resources/meta-observability.md
• Observability-as-code: resources/observability-as-code.md
• Anti-patterns catalog (72 entries): resources/anti-patterns.md
• Checklist: resources/checklist.md
• Examples: resources/examples.md
• Transport:
• resources/transport/udp-statsd-mtu.md
• resources/transport/otlp-grpc-vs-http.md
• resources/transport/collector-topology.md
• resources/transport/sampling-recipes.md
• Layers:
• resources/layers/L3-network.md
• resources/layers/L4-transport.md
• resources/layers/mesh.md
• resources/layers/L7-application/web-rum.md
• resources/layers/L7-application/mobile-rum.md
• resources/layers/L7-application/crash-analytics.md
• resources/layers/L7-application/waf.md
• Boundaries:
• resources/boundaries/multi-tenant.md
• resources/boundaries/cross-application.md
• resources/boundaries/slo.md
• resources/boundaries/release.md
• Signals:
• resources/signals/metrics.md
• resources/signals/logs.md
• resources/signals/traces.md
• resources/signals/profiles.md
• resources/signals/cost.md
• resources/signals/audit.md
• resources/signals/privacy.md

oma-observability Anti-Patterns

Consolidated catalog of patterns that break observability/traceability.

Each entry: pattern name, why-it-fails, remediation, severity.

Sources: all resource files in this skill tree. See individual "See also" references.

Severity legend

• CRITICAL: security breach / compliance violation / production data loss
• HIGH: blocks on-call ability to resolve incidents
• MEDIUM: degrades observability quality / cost / maintainability
• LOW: style or minor optimization

---

A: Privacy & Sensitive Data

A.1 Claiming "anonymization" for pseudonymized data

Severity: CRITICAL Why it fails: Pseudonymized data (reversible with a key) is still personal data under GDPR Article 4(1). Misclassifying it as anonymous data leads to non-compliant retention periods, missing erasure obligations, and exposes the organization to 4% global-turnover fines. Remediation: Apply the reversibility test from signals/privacy.md §3: "Could we recover the original value if compelled?" If yes, it is pseudonymization. Update your ROPA and re-classify accordingly. See also: signals/privacy.md §3 Anonymization vs Pseudonymization

A.2 OTel Collector hash action on low-entropy user IDs

Severity: CRITICAL Why it fails: The Collector's built-in hash action applies SHA-256 without a salt. Numeric user IDs (e.g., 6-digit codes) are reversible via rainbow tables, making the pseudonymization ineffective and constituting a GDPR breach. Remediation: Perform HMAC-SHA256 with a vault-managed key at the SDK layer before emitting spans. Store the salt in a separate region with independent IAM (GDPR Art. 32). Rotate quarterly. See also: signals/privacy.md §8 Salted Hashing Caveats

A.3 Baggage carrying PII across trust boundaries

Severity: CRITICAL Why it fails: W3C Baggage propagates to every downstream service, including external partners. Placing user.email, session tokens, or credentials in baggage violates W3C Baggage §Security and GDPR Article 5(1)(c) minimization, leaking PII to untrusted collectors. Remediation: Apply a baggage allowlist at every egress trust boundary (API gateway, external webhook). Allowed values: tenant.id, feature.variant, deployment.sha, region.hint. Strip all other keys. See also: boundaries/cross-application.md §4 Baggage Rules, signals/privacy.md §4 Common PII

A.4 PII in crash stack traces without redaction filter

Severity: CRITICAL Why it fails: Exception messages frequently capture SQL queries, HTTP headers, and URL query strings verbatim, exposing user.email, Authorization tokens, card numbers, and passwords in vendor SaaS storage; a GDPR and PIPA breach. Remediation: Implement a beforeSend / before_send allowlist hook in your crash SDK. Strip Authorization, Cookie, and any regex-matching email/card patterns before the crash report is serialized. Do not rely on server-side redaction alone. See also: layers/L7-application/crash-analytics.md §8 Privacy and PII, signals/privacy.md §7 SDK-Layer Redaction

A.5 Raw IP addresses retained without redaction

Severity: CRITICAL Why it fails: IP addresses linked to natural persons are personal data under GDPR Article 4(1) and PIPA. Storing raw srcaddr / client.address in long-retention backends violates the data minimization principle (GDPR Art. 5(1)(c)) and is grounds for regulatory action. Remediation: Truncate the last IPv4 octet (e.g., 203.0.113.0) or apply rotating-salt SHA-256 pseudonymization at pipeline ingestion. Never store raw IPs beyond the 7-day raw retention tier. See also: signals/privacy.md §5 PII Handling Rules, layers/L3-network.md §3.3 Privacy Note

A.6 db.query.text with untrimmed PII

Severity: CRITICAL Why it fails: SQL query text in db.query.text span attributes frequently contains WHERE email = 'user@example.com' or WHERE ssn = '...'. These surface verbatim in trace backends, violating GDPR minimization and leaking credentials. Remediation: Use the OTel Collector redaction processor with an allowlist; block free-text query content unless it matches safe patterns. Apply SQL parameterization at the SDK layer so user data never enters query text. See also: signals/privacy.md §6 OTel Collector Processors, signals/traces.md

A.7 Observability backend open to all engineers (no RBAC)

Severity: HIGH Why it fails: Production traces and logs containing even pseudonymized user data are sensitive. Unrestricted access violates the principle of least privilege, increases PII exposure risk, and fails SOC 2 CC7.2 and ISO/IEC 27001 A.8.15 access controls. Remediation: Implement role-scoped access: on-call engineers see own-service traces (24h); security analysts see all services (30d); auditors get read-only audit index. Use Grafana folder permissions or Datadog Teams scopes. See also: signals/privacy.md §12 Backend RBAC

A.8 Routing telemetry to 3rd-party vendor without DPA

Severity: HIGH Why it fails: Observability vendors receiving personal data are data processors under GDPR Article 28. Routing data before signing a Data Processing Agreement is a direct GDPR violation. Remediation: Block data flows to Datadog, Sentry, Grafana Cloud, Honeycomb, New Relic, or Elastic Cloud until a DPA is signed. Confirm storage region matches your compliance obligations. Reference the legal team's approved vendor list. See also: signals/privacy.md §13 Third-Party Processor Obligations

A.9 Session replay without client-side PII masking

Severity: HIGH Why it fails: Session replay captures DOM mutations including <input> fields. Without SDK-level masking, email addresses, credit card numbers, and passwords are captured in replay payloads before reaching the vendor; a GDPR Article 6 consent violation. Remediation: Enable input masking in the SDK config (Sentry, Datadog both support this). Wire replay consent to the cookie consent flow. Test via automated replay review that sensitive fields are masked. See also: layers/L7-application/web-rum.md §8 Session Replay, signals/privacy.md §5

A.10 Unencrypted telemetry queue on mobile device

Severity: HIGH Why it fails: Mobile telemetry queued to disk is PII at rest on user devices. If the device is lost, compromised, or forensically examined, unencrypted queue files expose personal data; violating GDPR Art. 32 and PIPA § 29 safety measures. Remediation: Encrypt the queue using platform-native key storage: iOS Keychain, Android Keystore. Apply field-level redaction before write, not before send. See also: layers/L7-application/mobile-rum.md §3 Offline-First Queuing

A.11 Cross-region telemetry routing without GDPR mechanism

Severity: HIGH Why it fails: GDPR Chapter V prohibits transfer of personal data to non-adequate countries without Standard Contractual Clauses or an adequacy decision. Routing EU telemetry through a US-hosted collector without SCC in place is a direct violation. Remediation: Implement routing connector per signals/privacy.md §10: route EU traffic to EU-region backend, KR traffic to KR-region backend. Confirm SCC or adequacy decision with legal before routing. See also: signals/privacy.md §10 Cross-Border Transfer, transport/collector-topology.md §7

---

B: Cardinality & Cost

B.1 user.id as metric label

Severity: CRITICAL Why it fails: Creates one TSDB time series per user. For a service with 1M users, this is 1M series; instant storage explosion, query latency degradation, and SaaS vendor bill spike. Additionally, user IDs are PII under GDPR Art. 4(1). Remediation: Replace with user.tier, user.cohort, or aggregated bucket labels. Never use any unbounded identifier as a metric label. Enforce via OTel SDK View attribute allow-list. See also: signals/metrics.md §9 Cardinality Budget, meta-observability.md §Section C

B.2 New metric name per tenant

Severity: HIGH Why it fails: Creating http_requests_total_tenant_acme for each tenant bypasses TSDB cardinality controls entirely and cannot be aggregated across tenants. It also disables cardinality budget alerting. Remediation: Use http_requests_total{tenant_id="acme"} with a top-N cap (≤ 1000 explicit tenants). Map overflow to label value "other". Apply Collector transform processor for normalization. See also: signals/metrics.md §9.3 Tenant Cap, signals/cost.md §10

B.3 Raw http.url as metric label

Severity: HIGH Why it fails: URL query strings are unbounded and may contain tokens or email addresses (?token=..., ?email=...). Using raw url.full as a label causes cardinality explosion and PII leakage in the TSDB. Remediation: Use http.route (normalized route) instead. Apply replace_pattern in OTel Collector transform processor to strip numeric IDs and UUIDs from route segments. See also: meta-observability.md §Section C, signals/metrics.md §3.3 Label Rules

B.4 Cost label at per-request metric granularity

Severity: HIGH Why it fails: Writing gen_ai.cost.total_usd as a metric label at request granularity creates one series per request; causing OOM on the TSDB ingestor and making cost attribution unusable. Remediation: Use gen_ai.cost.total_usd as a span attribute only. Aggregate cost metrics by tenant_id, namespace, and workload at the metric surface. Configure tail-sampler to always retain spans where cost exceeds $0.50. See also: signals/cost.md §10, transport/sampling-recipes.md §3

B.5 Summary instrument for cross-service aggregation

Severity: MEDIUM Why it fails: Prometheus Summary computes quantiles client-side per process. p99 from three replicas cannot be merged into a fleet-level p99; the values are mathematically incompatible. Remediation: Replace Summary with Histogram + histogram_quantile() at query time. Set explicit bucket boundaries matching the expected value range (e.g., seconds-scale for request duration, not the default millisecond buckets). See also: signals/metrics.md §2.5 Summary

B.6 Histogram with default bucket boundaries

Severity: MEDIUM Why it fails: The OTel SDK default buckets are millisecond-scale [0, 5, 10, 25, 50, ...]. For second-scale operations (database queries, LLM inference, file uploads), all measurements land in the last bucket, making histogram_quantile() meaningless. Remediation: Set explicit bucket boundaries per instrument view using View(instrument_name=..., aggregation=ExplicitBucketHistogramAggregation([0.01, 0.05, 0.1, 0.5, 1, 2, 5])). See also: signals/metrics.md §2.4 Histogram

B.7 LLM spans not tail-sampled on cost threshold

Severity: MEDIUM Why it fails: High-cost LLM spans (e.g., $2+ per trace) may be dropped by probabilistic sampling before they reach the backend. Silent budget blowup is invisible until the cloud billing invoice arrives. Remediation: Use a transform processor to set sampling.keep_reason = "high_cost" on spans where gen_ai.cost.total_usd > 0.50, then add a string_attribute policy in tail_sampling that always retains these spans. See also: signals/cost.md §6, transport/sampling-recipes.md §3

---

C: Pipeline & Collector

C.1 Missing memory_limiter processor

Severity: CRITICAL Why it fails: Without memory_limiter, a traffic spike or backend backpressure causes the Collector heap to grow unbounded until the process OOM-kills. All in-flight signals are lost and the pipeline is silent until the pod restarts. Remediation: Add memory_limiter as the first processor in every pipeline (traces, metrics, logs). Set limit_percentage: 75 and spike_limit_percentage: 20. Apply to both DaemonSet agents and gateway tiers. See also: meta-observability.md §Section A3, transport/collector-topology.md §2, transport/otlp-grpc-vs-http.md §4.3

C.2 Sidecar collectors on standard Kubernetes nodes

Severity: HIGH Why it fails: Sidecar mode injects one Collector per application pod. A 100-pod deployment runs 100 Collectors. CPU and memory cost scales linearly with pod count, overwhelming cluster resources. Additionally, sidecar collectors see only spans from their own pod, breaking tail sampling. Remediation: Use DaemonSet mode (one Collector per node) for standard Kubernetes. Reserve sidecar mode for AWS Fargate or GCP Cloud Run where DaemonSets are unavailable. See also: transport/collector-topology.md §3 When to Use Sidecar

C.3 Tail sampling in sidecar

Severity: HIGH Why it fails: A sidecar Collector only sees spans from its own pod. A trace spanning multiple services has spans on different pods, each with a different sidecar. No single sidecar has the complete trace; sampling decisions are based on incomplete data, producing systematically wrong retention. Remediation: Run tail_sampling processor in the gateway tier (Deployment mode) only, combined with a loadbalancing exporter using consistent hash by trace_id to ensure trace completeness. See also: transport/collector-topology.md §3, transport/sampling-recipes.md §2

C.4 Single gateway replica

Severity: HIGH Why it fails: A single gateway Collector is a single point of failure. Under backend backpressure or a rolling restart, all telemetry is lost for the duration. This eliminates observability exactly when it is most needed. Remediation: Deploy a minimum of 2–3 gateway replicas. Use a PodDisruptionBudget to prevent simultaneous eviction. Add a loadbalancing exporter upstream for trace-complete routing. See also: transport/collector-topology.md §6 High-Throughput Gateway Scaling

C.5 Fluentd as new log pipeline deployment in 2026+

Severity: HIGH Why it fails: CNCF announced Fluentd deprecation on 2025-10. Choosing Fluentd for a new deployment in 2026+ means adopting a deprecated tool with no future community investment, missing Fluent Bit's OTLP native output, and incurring Ruby runtime overhead (100+ MB vs Fluent Bit's 5–15 MB). Remediation: Use Fluent Bit (CNCF Graduated, C/Rust, native OTLP output) as the edge DaemonSet log agent. Use OTel Collector for gateway aggregation, PII redaction, and routing. See also: signals/logs.md §6 Collector and Agent Options

C.6 Prometheus receiver on multiple replicas without target allocator

Severity: MEDIUM Why it fails: When multiple gateway Collector replicas each run a prometheus receiver, every replica scrapes every target. This produces duplicate metrics in the TSDB backend and inflates series counts. Remediation: Enable the OTel Operator's spec.targetAllocator to distribute scrape targets across replicas. Each replica scrapes a non-overlapping subset of targets. See also: transport/collector-topology.md §4 Component Placement Reference

C.7 Collector self-metrics not scraped by independent instance

Severity: MEDIUM Why it fails: If the Collector scrapes its own :8888/metrics endpoint and the Collector fails, the metrics that would reveal the failure are also lost. The pipeline's health is invisible at the moment of failure. Remediation: Scrape Collector self-metrics from a separate Prometheus instance or second Collector instance. Expose otelcol_* metrics at service.telemetry.metrics.address: 0.0.0.0:8888 with level: detailed. See also: meta-observability.md §Section A1

---

D: Sampling & Retention

D.1 Head-based sampling on multi-service call paths

Severity: HIGH Why it fails: Head-based sampling makes a sampling decision at the trace root. If a downstream service independently samples out, its spans are dropped; the reconstructed trace has gaps. Tracing backends show incomplete traces that mislead incident investigation. Remediation: Use tail-based sampling in the gateway tier for multi-service systems. Propagate traceparent on every hop regardless of local sampling decision. The gateway buffers all spans and decides after the trace is complete. See also: transport/sampling-recipes.md §1, transport/sampling-recipes.md §6 Pitfalls

D.2 Missing loadbalancing exporter before tail sampler

Severity: HIGH Why it fails: Without consistent-hash routing, spans for the same trace arrive at different gateway replicas. Each replica's tail_sampling processor sees an incomplete trace and makes wrong retention decisions; high-value traces are dropped, low-value traces are retained. Remediation: Deploy loadbalancing exporter (with routing_key: traceID) in the tier upstream of tail_sampling. Use a headless Kubernetes Service so the exporter resolves per-pod DNS. See also: transport/sampling-recipes.md §2, transport/collector-topology.md §6

D.3 Audit logs not stored in WORM (mutable audit storage)

Severity: CRITICAL Why it fails: SOC 2 CC7.2, PCI DSS Requirement 10, and HIPAA §164.312(b) require tamper-evident, immutable audit storage. Mutable audit logs can be deleted or altered, nullifying compliance evidence and enabling concealment of unauthorized actions. Remediation: Apply S3 Object Lock in Compliance mode (not Governance), GCS retention policy with locked bucket, or Azure Immutable Blob Storage at bucket creation time. Set 7-year retention as a baseline. Do not use Governance mode; it allows privileged override. See also: signals/audit.md §5 Immutable WORM Storage

D.4 Audit log retention below regulatory minimum

Severity: HIGH Why it fails: HIPAA requires 6-year retention; PCI DSS requires 1 year online + offline; SOC 2 audit periods are typically 12 months. Flushing audit logs before the minimum period is a direct compliance violation discoverable during any audit. Remediation: Use a 7-year baseline with automated lifecycle policy (S3 → Glacier Deep Archive after 90 days). Set WORM Object Lock at write time; it cannot be applied retroactively. Monitor compliance via an audit_log_retention_days metric alert. See also: signals/audit.md §7 7-Year Retention Policy, meta-observability.md §Section F Alert 5

D.5 No tamper evidence on audit trail

Severity: HIGH Why it fails: WORM prevents deletion but does not prove that records were not silently modified or that records are not missing. An audit trail without a hash chain or Merkle root anchoring cannot satisfy forensic integrity requirements for SOC 2 or ISO/IEC 27001. Remediation: Implement per-event hash chain: event_N.hash = SHA256(event_N.payload + event_{N-1}.hash). Anchor daily Merkle root to an external transparency log (rekor/sigstore). Run automated chain verification weekly. See also: signals/audit.md §6 Tamper Evidence

D.6 Kubernetes audit logs routed to operational log store

Severity: HIGH Why it fails: Operational log stores (Loki, Elasticsearch) are mutable and have short retention policies. Kubernetes API audit logs; which record secret access, RBAC mutations, and cluster-admin actions; require WORM storage with multi-year retention to satisfy PCI DSS Requirement 10. Remediation: Route K8s audit logs via a separate pipeline to the WORM cold tier (S3 Glacier Deep Archive with Object Lock). Tag with source: k8s_apiserver. Do not co-mingle with operational logs. See also: signals/audit.md §9 Kubernetes Audit Logs

D.7 Decision wait too short in tail sampler

Severity: MEDIUM Why it fails: If decision_wait is shorter than the p99 inter-service latency, spans from slow downstream services arrive after the sampling decision is finalized. Those spans are dropped unconditionally, making tail sampling silently incomplete. Remediation: Set decision_wait to exceed your p99 cross-service latency (typically 30–60 s for synchronous calls). For cross-region deployments, account for additional propagation delay. Monitor otelcol_processor_queued_retry_send_queue_length for buffer pressure. See also: transport/sampling-recipes.md §6 Pitfalls

---

E: Release & Deployment

E.1 No release markers in telemetry

Severity: HIGH Why it fails: Without service.version on spans, metrics, and logs; and without a deployment event at release time; there is no way to correlate a metric anomaly or error spike to a specific deploy. Incident investigation degrades to git bisect guesswork. Remediation: Set service.version on the OTel Resource at SDK init (injected via CI as OTEL_RESOURCE_ATTRIBUTES). Emit a structured deployment event at release time and pipeline it to Grafana as a vertical annotation line. See also: boundaries/release.md §9 Release Markers

E.2 Canary analysis without SLI metric

Severity: HIGH Why it fails: Promoting a canary without a Flagger MetricTemplate or Argo Rollouts AnalysisTemplate means the promotion decision is manual and intuition-based. Regressions in error rate or latency ship silently to 100% of traffic. Remediation: Gate every canary on at minimum: error rate (max +0.5% allowed) and p99 latency (max +50ms allowed). Reference PromQL SLI queries from boundaries/slo.md as the metric source. See also: boundaries/release.md §10 Canary Analysis Metric Suite

E.3 Symbol upload not automated in CI (mobile / web)

Severity: HIGH Why it fails: Without automated dSYM / ProGuard / source map upload on every release, production crashes produce unreadable minified stack traces. Incident investigation time for mobile/web crashes increases by 30–60 minutes per incident. Remediation: Add symbol upload as a mandatory CI step, gated on the same pipeline step as container image push or app store submission. Use Sentry CLI, fastlane-plugin-sentry, or Gradle Sentry plugin. Never store symbols in git LFS. See also: layers/L7-application/crash-analytics.md §9 CI Integration

E.4 GitOps drift unalerted

Severity: MEDIUM Why it fails: When the cluster state diverges from the git manifest (OutOfSync in Argo CD, stalled Flux reconcile), the cluster is running unintended configuration. This silently breaks observability contracts; dashboards may reference metrics from a previous config version. Remediation: Alert on argocd_app_info{sync_status="OutOfSync"} == 1 and gotk_reconcile_condition{type="Ready",status="False"} == 1. Treat drift as an incident, not a warning. See also: boundaries/release.md §7 GitOps Engines

E.5 Feature flag evaluation not observed

Severity: MEDIUM Why it fails: Without feature_flag.* span attributes on every flag evaluation, A/B experiment effects are invisible. A latency regression introduced by a new feature variant cannot be attributed to the flag without telemetry. Remediation: Emit feature_flag.key, feature_flag.variant, and feature_flag.provider_name as span attributes on every evaluation (OTel feature_flag.* semconv, Experimental tier). Track feature_flag_evaluation_total and error rate delta per variant. See also: boundaries/release.md §6 Observing Feature Flag Evaluations

E.6 Rollback without post-mortem audit trail

Severity: MEDIUM Why it fails: A rollback event without an audit record severs the learning loop. The same deployment failure repeats because there is no evidence of what was deployed, who authorized it, and what metric triggered rollback. Remediation: Record every rollback in the audit log with: actor.type, action: rollback, service.version (both from and to), event.outcome, and trace_id. Cross-reference to the triggering alert. See also: signals/audit.md §3 Audit Event Categories, boundaries/release.md §11

---

F: Security & Compliance

F.0 Audit pipeline supply-chain integrity unverified

Severity: CRITICAL Why it fails: If the Collector binary, Falco rules, or sigstore/rekor clients shipped into the audit pipeline are tampered with, every downstream "immutable" audit record inherits the compromise. WORM storage guarantees nothing about the ingestor's trustworthiness. Remediation: Sign all audit-pipeline container images with cosign (sigstore). Pin Falco rule SHA digests in Helm values. Verify rekor transparency-log entries for rule provenance. Deploy an admission controller (e.g., Kyverno + verifyImages) that rejects unsigned images from the observability namespace. Cross-ref signals/audit.md §8 Falco and sigstore documentation. See also: signals/audit.md, sigstore.dev

F.1 traceparent stripped at service boundary

Severity: HIGH Why it fails: Stripping traceparent on outbound calls silently breaks distributed trace continuity. The downstream service starts a new root trace, making it impossible to correlate a customer-reported request across services. MTTR increases by 15–30 minutes per incident. Remediation: Every outbound HTTP/gRPC call MUST forward traceparent. Configure the OTel SDK auto-instrumentation to inject the header automatically. Test with an integration test that asserts the traceparent header on every outbound call. See also: standards.md §2.2 W3C Trace Context

F.2 Mixed propagators without normalization at ingress

Severity: HIGH Why it fails: When B3 (Zipkin), AWS X-Ray, and W3C headers coexist without a composite propagator at the ingress gateway, spans from different origins appear as disconnected root spans in the trace backend. Waterfall correlation is impossible across cloud or vendor boundaries. Remediation: Configure the ingress gateway with a composite propagator that extracts all known formats (tracecontext, b3multi, awsxray, datadog) and emits only W3C downstream. See also: boundaries/cross-application.md §3 Propagator Matrix

F.3 request_id not exposed to frontend users

Severity: MEDIUM Why it fails: Without a user-visible request_id in error banners, customer support cannot correlate a user-reported error to backend traces. Support agents must rely on user-provided timestamps and symptoms; escalations that should take 2 minutes take 20. Remediation: Return x-request-id in every HTTP response header. Display it in frontend error banners. Customer support uses this ID; engineers use it to pivot to trace_id in the log system. See also: boundaries/cross-application.md §7 request_id to trace_id Integration

F.4 service.namespace not set on services

Severity: MEDIUM Why it fails: Without service.namespace, a system with 80+ microservices presents a flat, unordered list in trace backends and dashboards. Domain-level KPIs (e.g., payment-domain error rate vs. inventory-domain error rate) are impossible to compute. Remediation: Assign service.namespace per DDD bounded context at deployment time via OTEL_RESOURCE_ATTRIBUTES or OTel Collector resource processor. Align namespaces with domain boundaries, not team names. See also: boundaries/cross-application.md §5 DDD Bounded Context

F.5 New trace_id generated on DLQ replay

Severity: MEDIUM Why it fails: Generating a new trace_id at dead letter queue replay severs the forensic chain from the original failure span to the replay event. Root cause analysis of repeated failures requires tracing back to the originating request. Remediation: Re-inject the original traceparent and causation_id from the failed message headers when replaying. Use a span link; not parent-child; to connect replay span to the original trace. See also: boundaries/cross-application.md §8 Idempotency and Event-Driven Trace Lineage

F.6 WAF rule hit / block rate unmonitored

Severity: HIGH Why it fails: When a WAF rule update over-blocks legitimate traffic, the edge returns 403 to users while backend services see request volume drop. Service-level error rate stays flat; the on-call investigates "missing traffic" rather than "WAF blocking my users", and the FP storm is discovered only via customer complaints. MTTR runs into hours. Remediation: Emit waf_action_total{action, rule_id, rule_set_version} from the edge tier. Alert when the block rate per rule.id step-changes more than 3σ above its rolling 24h baseline. Dashboard the action distribution alongside the application 5xx rate so block-vs-error is visible side-by-side. See also: layers/L7-application/waf.md §5 False-Positive Surge

F.7 WAF rule-set deploy without release marker

Severity: HIGH Why it fails: Without a release marker stream for ruleset rollouts, an edge 403 surge cannot be aligned with the rule change that caused it. The rollback decision is guesswork; engineers may roll back the application instead of the ruleset. Remediation: Emit a structured deployment event on every ruleset rollout carrying waf.rule.set.version, waf.vendor, and the rollout strategy (full vs canary). Treat WAF ruleset versions as a peer to service.version for release-correlation purposes per boundaries/release.md. See also: layers/L7-application/waf.md §3 Required Attributes, boundaries/release.md

F.8 WAF fail-closed without dependency-health signal

Severity: HIGH Why it fails: WAFs that consult external services (IP reputation, bot scoring, GeoIP, threat-intel API) inherit those dependencies' failure modes. A fail-closed policy without dependency telemetry means a third-party API outage silently blocks 100% of traffic; on-call has no signal that the security layer is the cause. Remediation: Emit waf_dependency_failure_total{dependency, policy} per external dependency. Alert when non-zero for more than 60 s. Document the fail-open vs fail-closed policy per rule and surface it on the WAF dashboard. See also: layers/L7-application/waf.md §6 Fail-Open vs Fail-Closed

F.9 WAF logs detached from application trace context

Severity: MEDIUM Why it fails: When the WAF blocks a request and emits a log without trace correlation, the application has no record of the attempt. On-call cannot answer "did this user even reach our service?" without manually joining vendor-specific identifiers (e.g., cf-ray, X-Amzn-Trace-Id) to the application trace. Remediation: Configure the WAF to propagate traceparent even on block actions. Map vendor headers (cf-ray, X-Amzn-Trace-Id, Akamai X-Akamai-*) to W3C baggage at the edge so application traces and WAF logs join cleanly on trace_id. See also: layers/L7-application/waf.md §7 Vendor Telemetry Surfaces, boundaries/cross-application.md §Propagators

F.10 WAF rules promoted to block without log-mode soak

Severity: MEDIUM Why it fails: A new rule promoted directly to block action carries unknown FP rate; first contact with production traffic doubles as the rule's correctness test. The failure mode is a mass-block incident affecting real customers. Remediation: Mandatory minimum 24 h log-mode (detect-only) soak per new rule with FP rate verified below a target ratio (commonly < 0.1% of matched traffic) before promotion to block. Encode the soak requirement as a CI gate on the ruleset repository so promotion bypass is auditable. See also: layers/L7-application/waf.md §2 WAF Action Model §5 False-Positive Surge

---

G: Frontend / Mobile

G.1 3rd-party script loaded without CSP monitoring

Severity: HIGH Why it fails: Third-party scripts loaded without a Content Security Policy are an unmonitored XSS and supply-chain attack vector. A compromised CDN script executes in the user's browser with full page access. LCP regressions from script load delays are also invisible without attribution. Remediation: Add Content-Security-Policy header with report-to endpoint. Pipe CSP violation reports to the log backend and alert on new blocked-uri origins. Pin script hashes with Subresource Integrity (integrity="sha384-..."). See also: layers/L7-application/web-rum.md §6 Third-Party Scripts and CSP

G.2 Source maps not uploaded to error vendor

Severity: HIGH Why it fails: Minified production bundles produce unreadable stack traces: at t.<anonymous> (bundle.min.js:1:74821). Without source maps, engineers cannot identify the failing line of code. Incident investigation for frontend crashes is impossible. Remediation: Upload source maps to the error tracking vendor on every CI release pipeline step, before the release is considered complete. Use Sentry CLI or equivalent. Never store source maps in git LFS; use vendor symbol storage. See also: layers/L7-application/web-rum.md §9 Error Tracking, layers/L7-application/crash-analytics.md §3

G.3 FID still reported in dashboards after March 2024

Severity: MEDIUM Why it fails: First Input Delay (FID) was removed from Core Web Vitals in March 2024 and replaced by Interaction to Next Paint (INP). Dashboards still reporting FID mislead SLO reviews; FID scores passing does not mean the INP SLO is met. Remediation: Replace FID with INP in all dashboards and OpenSLO definitions. SLI target: INP p75 ≤ 200 ms. Use web-vitals JS library v4.x which provides onINP. See also: layers/L7-application/web-rum.md §2 Core Web Vitals

G.4 propagateTraceHeaderCorsUrls / allowedTracingUrls not configured

Severity: MEDIUM Why it fails: Browser CORS preflight rejects injection of the traceparent header to origins not listed in the SDK allowlist. Client-to-server trace correlation silently breaks; frontend traces appear disconnected from backend traces. Remediation: Add all API origin patterns to FetchInstrumentation's propagateTraceHeaderCorsUrls (OTel JS) or Datadog RUM's allowedTracingUrls. Test in a browser network inspector to confirm the header is present. See also: layers/L7-application/web-rum.md §5 Client-to-Server Error Correlation

G.5 Missing traceparent injection on mobile outbound HTTP

Severity: MEDIUM Why it fails: Without traceparent on outbound requests from the mobile app, the mobile user session is invisible in backend distributed traces. Customer-reported mobile errors cannot be linked to backend spans. Remediation: Configure the mobile SDK's HTTP interceptor at initialization time (not at individual call sites). Verify with a network proxy tool (Charles, mitmproxy) that traceparent appears on all outbound API calls. See also: layers/L7-application/mobile-rum.md §9 W3C Trace Context Propagation

G.6 No event TTL on mobile offline queue

Severity: MEDIUM Why it fails: Stale events queued on-device for hours or days eventually upload when the network reconnects. Events with device timestamps that are 24+ hours old mislead dashboards and SLO calculations; a crash from two days ago appears as a current incident. Remediation: Set an event TTL of 24–72 hours on the mobile offline queue. Drop events that exceed the TTL before upload, not after. Log dropped event counts as a metric for monitoring queue health. See also: layers/L7-application/mobile-rum.md §3 Offline-First Queuing

G.7 No release marker for crash correlation on mobile

Severity: MEDIUM Why it fails: Without service.version set on every crash event and without a release marker event at deploy time, a crash rate spike cannot be attributed to a specific app version. Investigation requires manual version comparison across crash groups. Remediation: Set service.version as a custom key on every crash report. Automate release marker events at submission time. Use the release marker to draw vertical lines on CFR trend charts for before/after comparison. See also: layers/L7-application/crash-analytics.md §4 Release Tracking Integration

---

H: Network / BGP / Clock

H.1 NTP drift left unmonitored

Severity: HIGH Why it fails: Distributed traces depend on synchronized clocks. When two nodes diverge by more than 100 ms, waterfall charts show child spans starting before their parent; engineers chase phantom race conditions instead of real bugs. MTTR increases by hours. Remediation: Emit node_clock_drift_ms from every host (chrony textfile collector or node exporter). Alert when drift exceeds 100 ms for 5 minutes. Run chronyc makestep to force resync. For financial / telco workloads requiring sub-ms precision, use PTP (IEEE 1588). See also: standards.md §6 Clock Discipline, meta-observability.md §Section B

H.2 Own-ASN BGP hijack left unmonitored

Severity: HIGH Why it fails: A BGP prefix hijack diverts traffic to a rogue AS, which may cause a complete outage or a silent man-in-the-middle attack. Without MOAS detection, hijacks go undetected for hours or days while traffic is silently stolen. Remediation: Deploy BGPalerter or ARTEMIS with MOAS detection for your ASN prefixes. Subscribe to Cloudflare Radar alerts for your prefixes. Validate RPKI ROA records at your RIR and enable ROV enforcement on border routers. See also: layers/L3-network.md §6.4 Security Observability

H.3 PMTUD black hole left uncorrected

Severity: HIGH Why it fails: When firewalls block ICMP Type 3 Code 4 ("Fragmentation Needed"), PMTUD fails silently. Large TCP transfers stall while health checks (small packets) pass; masking the problem. Services appear healthy in monitors while bulk data transfers time out. Remediation: Enable MSS clamping at VPN/tunnel endpoints. Allow ICMP Type 3 Code 4 through security groups. Verify the fix with ping -M do -s 1472 <destination>. Cross-reference UDP MTU constraints for StatsD pipelines. See also: layers/L3-network.md §4 PMTUD, transport/udp-statsd-mtu.md §2

H.4 RPKI-ROV not configured on advertised prefixes

Severity: HIGH Why it fails: An IP prefix announced without a valid RPKI ROA record is marked "Not Found" by downstream validators; not "Invalid", but also not cryptographically anchored. Rogue AS announcements for your prefix are undetectable by validators, increasing hijack risk. Remediation: Create ROA records at your RIR (ARIN, RIPE NCC, APNIC) for all advertised prefixes. Enable ROV enforcement on border routers to drop or de-prefer RPKI Invalid routes. See also: layers/L3-network.md §6.4 RPKI-ROV

H.5 Connection pool observability absent

Severity: HIGH Why it fails: Pool queue saturation causes application latency spikes that are invisible in TCP metrics alone. Neither retransmit rate nor error rate spikes until connection timeouts fire; engineers investigate network issues while the actual problem is a saturated database pool. Remediation: Instrument connection pool size, wait time, and timeout counters at the application layer (not only at the TCP layer). Alert on pool utilization > 80% before timeouts occur. See also: layers/L4-transport.md §3.3 Common Pitfalls

H.6 QUIC adoption without HTTP/3 trace tooling validation

Severity: HIGH Why it fails: Enabling QUIC without verifying that OTel SDKs emit network.transport: quic and network.protocol.version: "3" creates a transport-layer blind spot. UDP-based QUIC flows are invisible in TCP metrics (/proc/net/tcp), leaving L4 observability dark. Remediation: Add a canary assertion in staging: verify network.transport: quic appears in spans and that Envoy access logs show QUIC connection IDs. Confirm tooling (Beyla QUIC uprobe) before production rollout. See also: layers/L4-transport.md §7 QUIC / HTTP3 Transport Semantics

H.7 eBPF agent deployed without kernel/capability preflight

Severity: HIGH Why it fails: Beyla or Pixie DaemonSets on incompatible kernels (< 4.14) or without CAP_BPF fail silently, providing no operator-visible error or metric. The observability gap is discovered only during an incident. Remediation: Add an initContainer that asserts kernel version (uname -r >= 4.14) and CAP_BPF presence. Exit non-zero if requirements are unmet. This surfaces the incompatibility during deployment, not during an incident. See also: layers/L4-transport.md §5.2 Kernel and Privilege Requirements

---

I: As-Code & GitOps

I.1 Production dashboards edited directly in UI

Severity: HIGH Why it fails: UI-edited dashboards have no version history, no rollback path, no peer review, and no audit trail. SOC 2 change-management controls require an audit-traceable change process for detection and response configuration. A dashboard overwritten by mistake cannot be recovered without a backup. Remediation: Version all dashboards in git using Grafonnet (Jsonnet) or Terraform Grafana provider. Apply via CI/CD only. Gate on PR review and linting (jsonnetfmt --test). Treat the git history as the audit trail. See also: observability-as-code.md §1 Why Observability-as-Code

I.2 Alert thresholds hardcoded per-environment without parameterization

Severity: MEDIUM Why it fails: Hardcoded thresholds in per-environment alert YAML files diverge silently over time. Production runs a different error-rate threshold than staging, masking regressions that staging was supposed to catch. Remediation: Parameterize alert thresholds as Jsonnet or Terraform variables. Derive environment-specific values from a shared defaults map. Apply the same alert code to all environments with environment-scoped variable overrides. See also: observability-as-code.md §2 Dashboards-as-Code

I.3 SLO definitions stored only in the vendor UI

Severity: MEDIUM Why it fails: SLO definitions stored only in Datadog, Grafana Cloud, or Honeycomb UIs are not version-controlled, not peer-reviewed, and cannot be reconstructed after a vendor migration. SLO drift goes undetected. Remediation: Define SLOs in OpenSLO YAML, committed to git. Apply via sloth or the vendor's Terraform provider. The git commit history is the audit trail for SLO changes. See also: observability-as-code.md, boundaries/slo.md

---

Z: Cross-cutting

Z.1 trace_id missing from log records

Severity: HIGH Why it fails: Without trace_id and span_id on every log record, log-trace join during incident forensics fails. Engineers cannot pivot from a log error to the distributed trace waterfall, adding 15–30 minutes to MTTR per incident. Remediation: Inject trace_id and span_id into every log record via the OTel SDK context hook (Python: structlog; Java: Logback OpenTelemetryAppender; Node.js: pino-opentelemetry-transport). Assert in integration tests. See also: signals/logs.md §7 Trace ID Injection Rules, incident-forensics.md §2.3

Z.2 service.version missing from resource attributes

Severity: HIGH Why it fails: Without service.version on every signal, before/after comparison across a release is impossible. Canary analysis, SLO delta calculation, and post-incident release attribution all fail. Remediation: Set service.version on the OTel Resource at SDK initialization. Inject via CI as OTEL_RESOURCE_ATTRIBUTES=service.version=${GIT_SHA}. Never set it per-signal; the Resource is the single source. See also: incident-forensics.md §2.1 Resource Attributes, boundaries/release.md §9

Z.3 Pipeline delivery ratio unmonitored

Severity: HIGH Why it fails: If the OTel Collector is silently dropping 10% of traces, every SLO dashboard and alert is built on incomplete data. The pipeline degradation is invisible until SLO violations appear; at which point on-call cannot distinguish real incidents from telemetry gaps. Remediation: Alert when sum(rate(otelcol_exporter_sent_spans[5m])) / sum(rate(otelcol_receiver_accepted_spans[5m])) < 0.99 for 5 minutes. This is the single most important meta-observability alert. See also: meta-observability.md §Section A6, meta-observability.md §Section F Alert 1

Z.4 Tenant ID absent from multi-tenant telemetry

Severity: HIGH Why it fails: In a multi-tenant system, telemetry without tenant.id makes per-tenant SLO computation, chargeback, and incident isolation impossible. All tenants are indistinguishable in dashboards; a single noisy tenant can mask SLO violations for the entire fleet. Remediation: Propagate tenant.id via W3C Baggage from the API gateway through all downstream services. Emit it on every span, log record, and metric data point. Apply top-N cap (≤ 1000) when used as a metric label. See also: incident-forensics.md §2.3, boundaries/multi-tenant.md

Z.5 Incident forensics without 6-dimension MRA attributes

Severity: HIGH Why it fails: Missing Minimum Required Attributes (service.name, service.namespace, service.version, deployment.environment, cloud.region, k8s.pod.name) break the 6-dimension narrowing flow; Code / Service / Layer / Host / Region / Infra pivots fail silently. Remediation: Enforce MRA completeness at CI via an OTel attribute coverage gate. Set all resource attributes via OTEL_RESOURCE_ATTRIBUTES or the OTel Collector resource processor. Validate with otelcol debug exporter in staging before production rollout. See also: incident-forensics.md §2 Minimum Required Attributes

Z.6 Cost dashboard accessible to all engineers without RBAC

Severity: MEDIUM Why it fails: Per-tenant cost data reveals revenue tier, contract value, and resource consumption patterns. Exposing this to all engineers violates least-privilege access principles and may constitute a data breach under GDPR if cost data is linked to identifiable customers. Remediation: Separate cost dashboards by role: Finance sees full cost by tenant; Platform Engineering sees cost by namespace/workload; Application Engineering sees only their own service. Apply Grafana folder permissions or OPA policies. See also: signals/cost.md §9 Privacy & Access Control

---

Contribution Protocol

When adding an entry from another doc:

1. Place it in the correct section (A–Z) by primary concern. 2. If the same anti-pattern appears in multiple source files, one canonical entry only; list all sources in "See also". 3. Use the format: ### {Section}.{n} {Pattern name} / **Severity** / **Why it fails** / **Remediation** / **See also**. 4. Order within each section: CRITICAL → HIGH → MEDIUM → LOW. 5. All cross-references use relative paths from the resources/ root. 6. No forward references to planned but unwritten files.

Cross-Application Observability Boundary

1. Scope

This file governs trace context, request correlation, causation chains, and baggage propagation across application boundaries; including service-to-service, cross-cloud, and cross-mesh calls.

Design decision D3 absorbs two formerly separate documents into this file:

• Propagators (formerly propagators.md): per-ecosystem header formats and normalization rules for W3C, B3, AWS X-Ray, GCP, Azure, Datadog, Cloudflare, Istio/Envoy, and Linkerd.
• DDD bounded context (formerly multi-domain.md): service.namespace as the domain grouping attribute for multi-service architectures.

Cross-references to other files in this skill use the ../ prefix convention.

---

2. 4-Layer Correlation Model

Distributed systems require four distinct correlation identifiers, each serving a different audience and cardinality need. Using only trace_id conflates concerns; using only request_id loses async chain visibility.

Causation vs. idempotency are distinct; causation_id expresses event ancestry (what triggered this event) and is often unique per edge in an event chain; idempotency_key is a business-level deduplication token shared by all retries of the same operation. Using a single header for both breaks dedup logic during retry if the causation chain diverges. Implement as two separate attributes / headers.

2.1 trace_id

Auto-generated by the OTel SDK at the trace root. 128-bit, encoded as 32 lowercase hex characters. Carried in traceparent. This is an internal platform identifier; do not expose it directly to end users because it reveals internal infrastructure topology. Its role is to be the carrier that links all spans in one distributed trace.

2.2 request_id

Human-facing correlation ID. Must appear in:

• Every outbound HTTP response header (x-request-id)
• Every log record as a structured field (http.request.id span attribute)
• User-visible error banners on the frontend (support ticket anchor)

The frontend generates a UUIDv4 request_id and sends it on every outbound request. Middleware sets the span attribute and logs the field. Customer support agents hand users a request_id from the error banner; platform engineers query logs to retrieve trace_id and pivot to the distributed trace waterfall.

Cross-ref ../incident-forensics.md §MRA for the support-to-trace lookup flow.

2.3 causation_id / idempotency_key

Event-driven and async workflows break the synchronous call chain. A Kafka consumer processing a payment event may run minutes after the HTTP request that produced it. causation_id links the new trace back to the originating event chain without forcing a parent-child span relationship.

• HTTP POST: set Idempotency-Key request header → record as span attribute
• Kafka producer: write causation_id into Kafka message header
• Kafka consumer: extract causation_id → create OTel span link (not parent-child) to the originating span

Span links are the correct OTel primitive for async causation. Cross-ref ../signals/traces.md §Messaging patterns.

2.4 Business context via Baggage

Business identifiers (user.id, tenant.id, order.id, feature.variant) propagate as W3C Baggage key-value pairs. They flow through every service on the call path and appear as span attributes and log fields, enabling cross-cutting dashboard filters (e.g., "all traces for tenant ACME in the last hour").

Baggage carries PII risk. See Section 4 for trust-boundary filtering rules.

---

3. Propagator Matrix

The recommended default for all new deployments is W3C Trace Context. At ingress or multi-cloud boundaries, extract all known propagator formats and emit W3C downstream; normalization at the edge prevents split-brain traces inside the cluster.

3.1 Multi-propagator strategy at ingress

Configure the ingress gateway or API gateway collector with a composite propagator that extracts all known formats. After extraction, emit only W3C downstream:

Composite extract order (ingress):
  1. tracecontext  (W3C — primary)
  2. b3multi       (Zipkin legacy)
  3. awsxray       (AWS Lambda / API Gateway)
  4. datadog       (Datadog-instrumented upstream)

Inject order (egress, internal):
  1. tracecontext  (W3C — only format emitted internally)
  2. baggage       (W3C Baggage — always)

This pattern ensures that a request arriving from an AWS Lambda (X-Ray header) or a Datadog-instrumented partner API is absorbed into the W3C trace context and propagated uniformly to all internal services.

---

4. Baggage Rules and PII Guidance

W3C Baggage propagates to every service on the call path. The W3C Baggage specification §Security states:

"Application owners should either ensure that no proprietary or confidential information is stored in baggage, or ensure baggage is not present in requests that cross trust boundaries."

The OTel Baggage API spec does not independently repeat this warning; the normative source is the W3C parent specification. Implementers MUST read both.

Cross-ref ../signals/privacy.md §Baggage rules for the enforcement implementation.

4.1 Trust-boundary rule

At every egress trust boundary (API gateway, external webhook, third-party vendor API), apply a baggage filter:

• Default: strip all baggage unless the receiving service is on the explicit allowlist.
• Allowlist: maintained per-environment; requires security team sign-off to add entries.
• Internal-to-internal: baggage passes through without filtering (intra-cluster calls within the same trust zone).

4.2 Allowed and prohibited baggage values

4.3 Enforcement point

The mesh ingress gateway is the primary enforcement point. Configure Envoy's Lua filter or an OTel Collector attributes processor at the gateway to drop prohibited keys. Cross-ref ../layers/mesh.md §Baggage scrubbing.

---

5. DDD Bounded Context via service.namespace

OpenTelemetry resource attributes service.name and service.namespace map directly to DDD concepts:

5.1 Why service.namespace matters

Without service.namespace, a system with 80+ microservices presents a flat list in trace backends and dashboards. Operators cannot distinguish domain-level KPIs from service-level noise.

With service.namespace, dashboards can be scoped:

• service.namespace = payments → filter all payment-domain spans, compute payment-domain SLI
• service.namespace = inventory → separate inventory burn-rate alert from payments alert

5.2 Configuration

Set service.namespace as an OTel resource attribute at deployment time, not at instrumentation time:

# OTel Collector resourcedetection / resource processor
processors:
  resource:
    attributes:
      - key: service.namespace
        value: payments
        action: upsert
      - key: service.name
        value: checkout-api
        action: upsert

Alternatively, set via environment variable that the SDK's Environment Resource Detector picks up:

OTEL_RESOURCE_ATTRIBUTES=service.namespace=payments,service.name=checkout-api

5.3 Anti-pattern: flat service registry

One hundred services without namespace grouping causes cognitive overload in the trace backend service map. Namespace assignment is a required step in any multi-service deployment. Assign namespaces aligned with DDD bounded contexts, not with team names or infrastructure regions.

---

6. Cross-Cloud Trace Continuity

When a request crosses cloud provider boundaries (e.g., App A on AWS calls App B on GCP calls App C on-premises), trace continuity requires W3C traceparent on every hop.

Collector bridge strategy: deploy an OTel Collector at each cloud egress point. The Collector normalizes incoming vendor headers to W3C before forwarding to the next cloud's ingress. This ensures that a trace rooted in AWS survives the GCP boundary and terminates in an on-premises Jaeger backend with an unbroken span chain.

Cross-ref ../transport/collector-topology.md §Multi-cluster/regional for Collector deployment patterns.

---

7. request_id to trace_id Integration

The support escalation flow depends on reliable linkage between the user-visible request_id and the internal trace_id.

Flow:

1. Frontend generates UUIDv4 request_id and sends it on every outbound request. 2. Backend middleware extracts request_id, sets span attribute http.request.id, and writes it as a structured log field. 3. On error, the frontend displays the request_id to the user (never trace_id; the latter leaks internal topology). 4. Customer support receives the request_id from the user's error screenshot. 5. Platform engineer queries the log system: http.request.id = "<request_id>" → retrieves trace_id from the matching log record. 6. Platform engineer opens the trace waterfall in the trace backend using trace_id.

Cross-ref ../incident-forensics.md §MRA for the full minimum reproducible artifact workflow.

---

8. Idempotency and Event-Driven Trace Lineage

Idempotency keys and causation IDs are first-class observability primitives in event-driven architectures.

HTTP pattern (Stripe-style):

• Client sends Idempotency-Key: <uuid> header on POST.
• Server records the key as span attribute http.idempotency_key.
• On duplicate detection, the existing response is returned; the trace_id of the original attempt is logged alongside the duplicate attempt's trace_id.

Kafka / SQS pattern:

• Producer writes causation_id (UUIDv4) and original traceparent into message headers.
• Consumer extracts both headers before creating the CONSUMER span.
• Consumer span links to the producer span via span.addLink(producerSpanContext).
• The link: not parent-child; preserves trace lineage without forcing synchronous ordering assumptions.

DLQ replay:

• When replaying from a dead letter queue, re-inject the original traceparent and causation_id from the failed message headers. Generating a new trace_id at replay severs the forensic chain.

Cross-ref ../signals/traces.md §Messaging patterns for span link implementation details.

---

9. Trust Boundary Patterns

---

10. Context Inheritance in Non-Interactive Sessions

CI/CD pipelines, agent workflows, and background jobs need trace context for end-to-end visibility across automated steps.

Use the TRACEPARENT environment variable for child process context inheritance. The OTel SDK's Environment Resource Detector reads TRACEPARENT and applies it as the parent context for the first span created in that process.

# CI/CD pipeline: propagate trace context to child build steps
export TRACEPARENT="00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"
# Child process (e.g., test runner) inherits the parent span context

This pattern enables a single trace to span from the CI trigger event through build, test, deploy, and smoke-test verification steps. Cross-ref ../SKILL.md for the TRACEPARENT env var description in the skill architecture.

---

11. Matrix Coverage

These cells from ../matrix.md are primarily driven by this file:

---

12. Anti-Patterns

The following extend ../anti-patterns.md §Cross-application.

---

References

Internal cross-references:

• ../standards.md: normative semconv stability tiers and W3C spec versions
• ../matrix.md: full 112-cell coverage map (cross-application boundary row)
• ../layers/mesh.md: propagator headers per mesh; baggage scrubbing at ingress gateway
• ../signals/traces.md: OTel span data model, messaging span links, baggage security
• ../signals/privacy.md: PII classification, baggage PII redaction rules
• ../signals/audit.md: audit event correlation across application boundaries
• ../transport/collector-topology.md: multi-cluster and multi-region Collector bridge patterns
• ../incident-forensics.md: request_id to trace_id lookup flow (MRA section)

Primary sources

• W3C Trace Context Level 1 Recommendation: <https://www.w3.org/TR/trace-context/>
• W3C Trace Context Level 2 Candidate Recommendation: <https://www.w3.org/TR/trace-context-2/>
• W3C Baggage Recommendation: <https://www.w3.org/TR/baggage/>
• OTel Baggage API specification: <https://opentelemetry.io/docs/specs/otel/baggage/api/>
• AWS X-Ray trace header: <https://docs.aws.amazon.com/xray/latest/devguide/xray-concepts.html#xray-concepts-tracingheader>
• GCP Cloud Trace context propagation: <https://cloud.google.com/trace/docs/trace-context>
• Datadog trace context propagation: <https://docs.datadoghq.com/tracing/trace_collection/trace_context_propagation/>
• Linkerd distributed tracing: <https://linkerd.io/2.15/features/distributed-tracing/>

Multi-Tenant Observability

1. Scope

Multi-tenant observability covers the collection, routing, isolation, attribution, and residency of telemetry signals in a B2B SaaS platform where multiple customer tenants share underlying infrastructure.

In scope:

• Tenant isolation strategy across four tiers (soft / routing / dedicated-collector / dedicated-backend)
• Tenant attribute propagation via W3C Baggage
• Per-tenant sampling policies
• Per-tenant retention schedules
• Cost attribution per tenant (chargeback / showback)
• Data residency routing for GDPR (EU) and PIPA (KR) tenants

Out of scope (related but distinct):

• Cross-service propagation mechanics: see cross-application.md (propagators, baggage rules)
• FinOps unit economics and OpenCost metric surface; see ../signals/cost.md
• PII redaction and anonymization rules per tenant; see ../signals/privacy.md

---

2. OTel Attribute Conventions for Tenant

OpenTelemetry Semantic Conventions do not include a stable tenant.* group as of semconv 1.27.0. The attributes below are custom application-defined attributes. Follow the naming convention from ../standards.md §3 and prefix with the domain noun to avoid collisions with future OTel semconv additions.

Naming rationale: dot-separated namespace (tenant.*) matches OTel semconv style and avoids the underscore ambiguity common in Prometheus label names. Do not use customer_id, org_id, or account_id for the same concept; pick one canonical key and propagate it everywhere.

---

3. Four-Tier Isolation Strategy

Most B2B SaaS organizations apply a mix of tiers: enterprise tenants get Tier 3 or 4, pro gets Tier 2, and free gets Tier 1. Select the highest tier required by the tenant's compliance obligations.

Routing connector alpha caveat: Tier 2 using routing_connector is subject to the alpha stability warning documented in ../transport/sampling-recipes.md §4. For production Tier 2 deployments, prefer tail_sampling with and sub-policies (stable) over routing_connector (alpha as of 2025).

---

4. Tenant ID Propagation

Tenant context must be carried across service boundaries so every span, metric, and log record emitted by any service is attributable to its originating tenant.

Mechanism: W3C Baggage (baggage header, Recommendation 2022-12-22). Set tenant.id and tenant.tier as baggage entries at the ingress gateway. All downstream services read from the OTel Baggage API and apply these values as resource or span attributes.

Ingress Gateway
  → set baggage: tenant.id=acme-corp, tenant.tier=enterprise
      ↓
Service A (reads baggage → sets span attribute tenant.id)
      ↓
Service B (reads baggage → sets span attribute tenant.id)

Trust-boundary warning: W3C Baggage is visible to every service in the propagation chain, including third-party or external services. The W3C Baggage specification (https://www.w3.org/TR/baggage/) explicitly notes that baggage values cross trust boundaries. Carrying tenant.id to external egress endpoints leaks customer account existence information.

Rule: strip or validate tenant.* baggage entries at the egress gateway before forwarding to any external third-party endpoint. Internal propagation only. Cross-ref ../signals/privacy.md §Common PII in Telemetry for baggage PII rules.

---

5. Per-Tenant Sampling

Different tenant tiers justify different sampling rates. Enterprise tenants have SLA obligations and full debugging requirements; free tenants justify only ambient visibility.

Per-tier retention targets:

Recommended configuration; `tail_sampling` with `and` sub-policies (stable, production-safe):

processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 100000
    expected_new_traces_per_sec: 1000
    policies:
      - name: enterprise
        type: and
        and:
          and_sub_policy:
            - name: tier-check
              type: string_attribute
              string_attribute:
                key: tenant.tier
                values: ["enterprise"]
            - name: probabilistic
              type: probabilistic
              probabilistic:
                sampling_percentage: 100

      - name: pro
        type: and
        and:
          and_sub_policy:
            - name: tier-check
              type: string_attribute
              string_attribute:
                key: tenant.tier
                values: ["pro"]
            - name: probabilistic
              type: probabilistic
              probabilistic:
                sampling_percentage: 20

      - name: free-baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 2   # catches free tier and unmatched traffic

Cross-ref ../transport/sampling-recipes.md §4 for the full tenant-aware sampling recipe including routing_connector Option A (alpha) and the combined error + cost + tenant four-policy example.

---

6. Per-Tenant Retention

Retention schedules must be enforced per tier. Hot storage is fast-query; warm is compressed but queryable; cold is archival with retrieval latency.

Implementation: Apply Kubernetes-native or backend-native lifecycle policies keyed on the tenant.id and tenant.tier labels. For shared backends (Tier 1–2), use label-based TTL rules or index lifecycle management (e.g., OpenSearch ISM policies, Loki retention rules, Thanos compactor retention). For dedicated backends (Tier 3–4), set backend project-level retention per tenant.

Cross-ref ../meta-observability.md §Retention Matrix for the full retention policy table covering all seven signals.

---

7. Cost Attribution

Tenant-level cost attribution enables chargeback (billing tenants for their resource consumption) and showback (internal reporting without billing).

Kubernetes workload labeling: Tag every pod at deploy time with tenant.id as a Kubernetes label. OpenCost reads workload labels and produces opencost_workload_cost_total{tenant_id="acme-corp"} automatically.

# Kubernetes Pod template label (applied via Helm values or admission webhook)
metadata:
  labels:
    tenant.id: "acme-corp"
    tenant.tier: "enterprise"

Telemetry cost attribution: The observability bill itself (e.g., Datadog RUM sessions per tenant, Honeycomb events per tenant) must also be attributed. Instrument the collector pipeline with throughput counters per tenant to apportion the observability bill proportionally.

Cardinality constraint: tenant.id used as a metric label must be capped at the top-N active tenant count (100–1,000 tenants is safe for most TSDBs). Beyond this threshold, bucket overflow tenants under an "other" label to prevent metric series explosion. Cross-ref ../meta-observability.md §Cardinality Guardrails.

Cross-ref ../signals/cost.md §4 Cost Attribution by Dimension for FinOps unit economics (per-tenant PromQL formulas, OpenCost architecture, and FOCUS spec).

---

8. Data Residency

GDPR Chapter V (https://gdpr-info.eu/chapter-5/) restricts transfers of personal data outside the EU/EEA to countries or organizations that provide adequate protection. Korean PIPA (https://www.pipa.go.kr) applies equivalent restrictions for Korean resident data.

Routing rules:

Topology: deploy per-region edge collectors that aggregate locally and export only to backends in the same region. No cross-region OTLP export for EU or KR tenants.

EU Tenants → EU Edge Collector → EU Backend (e.g., eu-west-1 Grafana Cloud)
KR Tenants → KR Edge Collector → KR Backend (e.g., ap-northeast-2 region)
US Tenants → US Edge Collector → US Backend or global aggregator

Route by tenant.region at the ingress gateway before data enters the collector pipeline. Do not allow EU or KR tenant telemetry to flow through a non-compliant region, even transiently.

Source-of-truth rule (critical): tenant.region MUST be resolved from an internal, server-side authoritative source; tenant registry service, organization metadata table, or IdP claim stamped at session start. It MUST NOT be trusted from client-supplied input (HTTP header, baggage, query string, or JWT claim the client itself controls). A misconfigured or malicious tenant could otherwise self-declare a non-EU/KR region and bypass residency routing. Enforce at the ingress gateway: reject requests where a client-declared tenant.region disagrees with the registry lookup keyed on tenant.id. In practice, strip any inbound tenant.region attribute and re-attach the registry-sourced value before the Collector pipeline accepts the span/log.

Cross-ref ../transport/collector-topology.md §7 Federated / Multi-Cluster for the multi-region edge topology diagram. Cross-ref ../signals/privacy.md §2 Regulatory Drivers for GDPR and PIPA penalty context and ../signals/privacy.md §Cross-border transfer for PII-specific cross-border rules.

---

9. Tenant Onboarding and Offboarding

Onboarding checklist: 1. Provision tenant.id and tenant.tier as Kubernetes labels on all tenant workloads (via Helm values or admission webhook). 2. Create per-tenant routing rule if Tier 2+. 3. Provision per-tenant collector namespace if Tier 3+. 4. Create per-tenant backend project or organization if Tier 4. 5. Create per-tenant dashboard folder with RBAC rules (see §10). 6. Register tenant in cardinality allowlist (top-N cap enforcement).

Automate steps 1–6 as code; cross-ref ../observability-as-code.md for provisioning patterns.

Offboarding; GDPR Art. 17 Right to Erasure:

When a tenant terminates their contract, all telemetry data containing tenant.id must be deleted across every storage tier (hot, warm, cold) and every signal (metrics, logs, traces, profiles, cost records, audit records). This is a legal obligation under GDPR Art. 17, not an engineering convenience.

Offboarding procedure: 1. Trigger deletion job across all backends scoped to tenant.id. 2. Remove tenant from cardinality allowlist and routing rules. 3. Deprovision collector namespace (Tier 3) or backend account (Tier 4). 4. Emit an audit event recording the erasure action, timestamp, and operator identity.

Cross-ref ../signals/audit.md for audit event schema for offboarding erasure events.

---

10. Dashboard Isolation

RBAC rule: a tenant admin identity claim must be mapped to a Grafana or backend role that scopes data access strictly to that tenant's tenant.id. Cross-org or cross-tenant data leakage via dashboard query is a compliance violation.

Cross-ref ../signals/privacy.md §Backend RBAC for Grafana RBAC configuration patterns and OPA policy rules for query-level tenant isolation.

---

11. Noisy Neighbor Protection

Shared infrastructure (Tier 1 and Tier 2) is vulnerable to one high-volume tenant degrading the observability pipeline for all other tenants.

Controls:

Apply rate limits at the first collector tier (agent or edge). A tenant exceeding its quota must receive a well-defined error (e.g., OTLP ResourceExhausted gRPC status) rather than silently dropping data.

---

12. Matrix Cells: Multi-Tenant Row

Quick navigation for multi-tenant boundary cells in ../matrix.md:

---

13. Anti-Patterns

The following are candidates for ../anti-patterns.md §Multi-Tenant:

---

Cross-References

Release Boundary

1. Scope

The release boundary is the temporal boundary between versions of a service; the window where new code enters production and either stabilizes or is rolled back.

Covers: progressive delivery strategies, Flagger canary analysis, Argo Rollouts, OpenFeature feature flags, GitOps engine reconcile observability, and Kubernetes operator reconcile metrics.

Out of scope: SLI/SLO definitions and error budget math (see slo.md); dashboard layout and code (see ../observability-as-code.md); post-incident timelines (see ../incident-forensics.md).

This boundary consumes SLI/SLO metrics as promotion gates; it does not define SLOs.

---

2. Progressive Delivery Strategies

Choose based on: blast radius tolerance × metric feedback speed. Canary is the default recommendation for services with Prometheus SLI coverage.

---

3. Flagger (CNCF Graduated, part of Flux)

Source: <https://flagger.app>

Flagger automates canary promotion and rollback decisions using Prometheus, Datadog, or NewRelic metric queries. It integrates natively with Flux and supports any service mesh or ingress (Istio, Linkerd, Nginx, Contour).

Feedback loop: Flagger samples metrics on a configurable interval (30s–60s). If the metric breaches the threshold for N consecutive samples (failureThreshold), it rolls back. Promotion requires all metric checks to pass for the full iterations count.

# Flagger Canary CR — canary analysis referencing PromQL SLI
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: checkout
  namespace: prod
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout
  progressDeadlineSeconds: 600
  service:
    port: 8080
  analysis:
    interval: 60s        # sample every 60 seconds
    threshold: 5         # max failed checks before rollback
    maxWeight: 50        # max canary traffic weight (%)
    stepWeight: 10       # increment per successful check
    metrics:
      - name: error-rate
        templateRef:
          name: error-rate
          namespace: flagger-system
        thresholdRange:
          max: 1         # fail if error rate > 1%
        interval: 60s
      - name: latency-p99
        templateRef:
          name: latency
          namespace: flagger-system
        thresholdRange:
          max: 500       # fail if p99 > 500ms
        interval: 60s
---
apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
  name: error-rate
  namespace: flagger-system
spec:
  provider:
    type: prometheus
    address: http://prometheus.monitoring:9090
  query: |
    sum(rate(http_requests_total{
      namespace="{{ namespace }}",
      service="{{ target }}",
      status=~"5.."
    }[{{ interval }}]))
    /
    sum(rate(http_requests_total{
      namespace="{{ namespace }}",
      service="{{ target }}"
    }[{{ interval }}])) * 100

Cross-ref slo.md §3 for the full SLI PromQL patterns used as the canary threshold source.

---

4. Argo Rollouts

Source: <https://argoproj.github.io/rollouts>

Argo Rollouts integrates with Argo CD for GitOps-native progressive delivery. Supports canary, blue-green, and experiment steps. AnalysisTemplate CRD gates promotion with metric-based pass/fail.

# Argo Rollout — canary with AnalysisTemplate reference
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
spec:
  replicas: 5
  strategy:
    canary:
      steps:
        - setWeight: 20
        - pause: { duration: 2m }
        - analysis:
            templates:
              - templateName: error-rate-check
        - setWeight: 50
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: error-rate-check
      canaryService: checkout-canary
      stableService: checkout-stable
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate-check
spec:
  metrics:
    - name: error-rate
      interval: 60s
      successCondition: result[0] < 1.0   # < 1% error rate
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{
              job="checkout-canary",
              status=~"5.."
            }[5m]))
            /
            sum(rate(http_requests_total{
              job="checkout-canary"
            }[5m])) * 100

---

5. Feature Flags via OpenFeature (CNCF Graduated, 2024-11)

Source: <https://openfeature.dev> | CNCF graduation: <https://www.cncf.io/projects/openfeature/>

OpenFeature defines a vendor-agnostic SDK specification for feature flag evaluation via the OFREP (OpenFeature Remote Evaluation Protocol). The SDK core is provider-agnostic; teams swap providers without changing application code.

Supported providers: LaunchDarkly, Flagsmith, GrowthBook, Unleash, ConfigCat, Harness FF.

Evaluation context carries targeting attributes:

Integration with Flagger/Rollouts: use feature flags to gate traffic routing at the application layer while Flagger controls infrastructure-level traffic weight.

---

6. Observing Feature Flag Evaluations

OTel semantic conventions: feature_flag.* (Experimental as of semconv 1.27.0).

Key span attributes emitted on each flag evaluation:

feature_flag.key          = "checkout-v2-enabled"
feature_flag.variant      = "on"
feature_flag.provider_name = "flagsmith"

Example span attributes (JSON log-compatible):

{
  "name": "feature_flag.evaluation",
  "attributes": {
    "feature_flag.key": "checkout-v2-enabled",
    "feature_flag.variant": "on",
    "feature_flag.provider_name": "flagsmith",
    "user.id": "usr_abc123",
    "tenant.id": "org_enterprise"
  }
}

Metrics to track per flag:

Dashboards cross-ref ../observability-as-code.md for the flag evaluation panel template.

---

7. GitOps Engines

7.1 Argo CD

Source: <https://argo-cd.readthedocs.io>

Declarative Kubernetes sync using app-of-apps pattern. Reconcile metrics exposed via /metrics:

Drift detection alert (cluster state ≠ git manifest):

argocd_app_info{sync_status="OutOfSync"} == 1

7.2 Flux (CNCF Graduated)

Source: <https://fluxcd.io>; Flagger ships as part of the Flux ecosystem.

Stalled resource alert:

gotk_reconcile_condition{type="Ready",status="False"} == 1

Cross-ref ../signals/metrics.md for Prometheus scrape config snippets for both engines.

---

8. Kubernetes Operator Reconcile Observability

Every operator built on controller-runtime exposes /metrics automatically.

Built-in metrics:

Monitor CRD readiness via .status.conditions[]:

# Alert on reconcile error spike
rate(controller_runtime_reconcile_errors_total[5m]) > 0.1

Custom CRDs must emit reconcile metrics. Pattern: wrap reconcile logic in ObservedGeneration guard and expose a status.conditions entry per managed resource.

---

9. Release Markers in Telemetry

• service.version (OTel Stable): set on every span, metric data point, and log record

via the OTel Resource at SDK initialization. Never patch per-signal.

• Emit a deployment event at release time:

{
  "event.name": "deployment",
  "service.name": "checkout",
  "service.version": "2.4.1",
  "deployment.environment": "production",
  "timestamp": "2026-04-21T10:00:00Z"
}

• Pipeline this event to Grafana → vertical annotation line on all timeseries dashboards.
• Correlation rule: incidents within ≤30 minutes of a deployment event are flagged as

release suspects. Cross-ref ../incident-forensics.md §Scenario C for the triage playbook.

---

10. Canary Analysis Metric Suite

---

11. Rollback and Hotfix Pipeline

Manual override always remains possible regardless of automated analysis state.

Post-mortem audit trail required for every rollback event. Cross-ref ../signals/audit.md for the required audit log fields.

---

12. Matrix.md Cells (release row)

---

13. Anti-Patterns (candidates for ../anti-patterns.md §Section E Release & Deployment)

---

References

• Flagger: <https://flagger.app>
• Argo Rollouts: <https://argoproj.github.io/rollouts>
• OpenFeature: <https://openfeature.dev>
• CNCF OpenFeature incubating announcement: <https://cncf.io/blog/2023/12/>
• Flux (includes Flagger): <https://fluxcd.io>
• Argo CD: <https://argo-cd.readthedocs.io>
• OTel feature_flag semconv: <https://opentelemetry.io/docs/specs/semconv/feature-flags/>
• controller-runtime metrics: <https://book.kubebuilder.io/reference/metrics>

SLO Boundary

1. Scope

SLO (Service Level Objective) is the contractual reliability target bounding acceptable error rates and response times. Covers: SLI definition, SLO math, error budget, burn-rate alerts, tool options.

Out of scope: dashboard config (observability-as-code.md); progressive delivery gates (release.md); pipeline SLOs (../meta-observability.md §Section F).

---

2. Terminology

Rule: SLA target < SLO target. Never set SLA = SLO (no operations buffer).

---

3. SLI Selection

Cross-ref ../signals/metrics.md §SLI for PromQL sketches per method.

---

4. SLO Math

Availability SLI (28-day rolling window):

sum(rate(http_requests_total{status=~"2..|3.."}[28d]))
  / sum(rate(http_requests_total[28d]))

Latency SLI (fraction of requests under 300ms):

sum(rate(http_request_duration_seconds_bucket{le="0.3"}[28d]))
  / sum(rate(http_request_duration_seconds_count[28d]))

---

5. OpenSLO Spec

Source: <https://openslo.com>; community-driven, not CNCF. Vendor-neutral YAML adopted by Sloth, Pyrra, Nobl9.

apiVersion: openslo.com/v1
kind: SLO
metadata:
  name: checkout-availability
spec:
  service: checkout
  sloType: Request-Based
  indicator:
    spec:
      ratioMetric:
        good:
          metricSource:
            type: Prometheus
            spec:
              query: sum(rate(http_requests_total{service="checkout",status=~"2..|3.."}[{{.Window}}]))
        total:
          metricSource:
            type: Prometheus
            spec:
              query: sum(rate(http_requests_total{service="checkout"}[{{.Window}}]))
  objectives:
    - target: 0.999
  timeWindow:
    - duration: 28d
      isRolling: true

---

6. Tool Options (as of 2026-Q2)

Cross-ref ../vendor-categories.md §OSS Full-Stack. Sloth and Pyrra are the recommended OSS path for Kubernetes-native GitOps deployments.

---

7. Burn-Rate Alert Design

7.1 Problem

A simple error-budget-exhausted alert fires after damage is done. Multi-window burn-rate alerts detect fast-burning incidents while they still have budget remaining.

Source: <https://sre.google/workbook/alerting-on-slos/>

7.2 Multi-Window Tiers (99.9% SLO)

Multiplier derivation: budget_fraction / (window / 720h)

• Fast: 2% / (1h/720h) = 14.4
• Slow: 5% / (6h/720h) = 6

7.3 Fast Burn PromQL

(
  (1 - sum(rate(http_requests_total{status=~"2..|3.."}[1h]))
         / sum(rate(http_requests_total[1h])))
  > (1 - 0.999) * 14.4
)
and
(
  (1 - sum(rate(http_requests_total{status=~"2..|3.."}[5m]))
         / sum(rate(http_requests_total[5m])))
  > (1 - 0.999) * 14.4
)

7.4 Slow Burn PromQL

(
  (1 - sum(rate(http_requests_total{status=~"2..|3.."}[6h]))
         / sum(rate(http_requests_total[6h])))
  > (1 - 0.999) * 6
)
and
(
  (1 - sum(rate(http_requests_total{status=~"2..|3.."}[30m]))
         / sum(rate(http_requests_total[30m])))
  > (1 - 0.999) * 6
)

The short window gates the long window: long window detects sustained burns; short window suppresses false positives from transient spikes.

---

8. Error Budget Policy

Cross-ref release.md; Flagger and Argo Rollouts use the SLI error rate as canary promotion gate; the same threshold enforces budget-aware deployment freeze.

---

9. Cross-Integration

---

10. Matrix Coverage (slo row)

---

11. Anti-Patterns (candidates for ../anti-patterns.md §Section D)

---

References

• Google SRE Workbook: Alerting on SLOs: <https://sre.google/workbook/alerting-on-slos/>
• OpenSLO specification: <https://openslo.com>
• Sloth: <https://github.com/slok/sloth>
• Pyrra: <https://github.com/pyrra-dev/pyrra>
• Grafana SLO plugin: <https://grafana.com/grafana/plugins/grafana-slo-app>

UDP StatsD MTU and Fragmentation Guide

1. Scope

This document covers StatsD metric payloads transported over UDP or Unix Domain Socket (UDS) into an OpenTelemetry Collector statsdreceiver or a DogStatsD agent. It does not address TCP or HTTP transports, or metrics emitted directly via OTLP.

Sources:

• OTel statsdreceiver: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/statsdreceiver
• DogStatsD high-throughput: docs.datadoghq.com/developers/dogstatsd/high_throughput/
• Prometheus statsd_exporter issue #35 (buffer sizing)

---

2. Why Fragmentation Matters for UDP

UDP is connectionless and provides no retransmission. When an IP datagram exceeds the path MTU:

• No retransmission: if any IP fragment is lost in transit, the kernel discards the entire

reassembled datagram silently. StatsD has no acknowledgement mechanism, so the loss is invisible.

• Middlebox drops: NAT gateways, stateful firewalls, and VPN concentrators frequently drop

IP fragments entirely, or track only the first fragment for NAT state and discard the rest.

• Reassembly CPU overhead: fragment reassembly at the receiver incurs kernel memory allocation

and timer management. Under high packet rate this degrades throughput measurably.

Rule: keep each datagram within the path MTU so it is transmitted as a single unfragmented packet.

---

3. Optimal Datagram Size Table

IPv6 minimum MTU is 1280 B (RFC 8200); on links with lower MTU, fragmentation is performed

by the source host only; not routers; making drops more likely. Always test your actual path.

---

4. Verification

4.1 PMTUD Probe (Linux / macOS)

Find the largest datagram that reaches a destination without fragmentation:

# Linux — DF bit set, vary -s until you find the boundary
ping -M do -s 1472 -c 3 <collector-host>

# macOS equivalent
ping -D -s 1472 -c 3 <collector-host>

Decrease -s in steps of 10 until you stop seeing "Frag needed" / "Message too long" errors. The largest passing value is your PMTU; set max_packet_size to that value minus 28 (IPv4) or 48 (IPv6) to leave room for IP and UDP headers.

4.2 Wireshark Fragment Inspection

Capture on the collector interface and filter for IP fragments:

ip.flags.mf == 1

The More Fragments (MF) bit set on any packet indicates fragmentation is occurring. A fragment offset > 0 identifies continuation fragments. If you see these in production traffic, your client is sending oversized datagrams.

For UDS traffic, capture on a loopback or use socat to proxy and inspect:

socat -v UNIX-RECV:/tmp/statsd.sock UDP:127.0.0.1:8126

---

5. Client-Side Batching

Most StatsD client libraries default to sending one metric per datagram, which is wasteful:

• A single counter line (my.counter:1|c) is ~20 bytes; one UDP send per metric at 1 M metrics/min

generates 16 K packets/sec of syscall overhead.

• Enable multi-metric batching in your client library.

Rules:

• For external UDP: keep buffered payload at or below the PMTU value found in section 4.1.
• For UDS (same-host): buffer up to 8192 bytes per send; larger values risk ENOBUFS under load.
• Never exceed 65507 bytes (IPv4 UDP maximum); the kernel will return EMSGSIZE.

---

6. OTel StatsD Receiver Configuration

Default UDP port is 8125. Switch to UDS for same-host agents to eliminate network stack overhead.

receivers:
  statsd:
    endpoint: "0.0.0.0:8125"      # UDP; change to unixgram path for UDS
    # endpoint: "/var/run/statsd.sock"
    transport: udp                  # udp | unixgram
    aggregation_interval: 60s       # flush interval to next processor
    enable_metric_type: true        # attach metric type as attribute
    is_monotonic_counter: false     # set true for always-increasing counters
    timer_histogram_mapping:
      - statsd_type: "timing"
        observer_type: "histogram"
      - statsd_type: "histogram"
        observer_type: "histogram"

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128
  batch:
    send_batch_size: 8192
    timeout: 10s

exporters:
  otlp:
    endpoint: "otel-collector-tier2:4317"
    compression: gzip

service:
  pipelines:
    metrics:
      receivers: [statsd]
      processors: [memory_limiter, batch]
      exporters: [otlp]

Bridge pattern: StatsD clients → UDP/UDS → OTel Collector statsdreceiver → OTLP gRPC → downstream Collector or backend. This decouples metric ingestion from processing and allows the statsdreceiver to be scaled or replaced without changing client configuration.

For UDS on Linux, set socket permissions to allow the application user:

chmod 0660 /var/run/statsd.sock
chown root:app /var/run/statsd.sock

Set SO_RCVBUF on the collector socket to handle burst traffic (Linux default is typically 208 KB; increase to 8–25 MB for high-throughput agents via net.core.rmem_max):

sysctl -w net.core.rmem_max=26214400