Monitoring Observability

Monitoring & Observability

Use this skill when the main question is "what packet do we have, what should this system notice, and what should interrupt a human?"

The job is not to dump a Prometheus / Grafana / Datadog tutorial. The job is to normalize the packet, pick one primary observability mode, define the smallest useful signal plan, and route adjacent work away before the skill turns into debugging, performance tuning, rollout execution, or analytics reporting.

Read references/intake-packets-and-route-outs.md before handling an unfamiliar packet. Read references/modes-and-boundaries.md before handling mixed requests that blur telemetry setup, incident diagnosis, or product analytics. Read references/alert-dashboard-checklist.md when reviewing dashboards, alerts, and ownership gaps. Read references/telemetry-rollout-matrix.md when choosing the smallest rollout slice.

When to use this skill

• New service, worker, API, or multi-service system needs health signals, alerts, dashboards, or SLO-style coverage before launch
• Existing stack has dashboards / alerts / telemetry, but trust is low and a keep/fix/delete/add audit is needed
• Team needs to decide what to instrument, correlate, retain, or sample before choosing vendors or backend specifics
• Data, marketing, analytics, or pipeline work needs freshness / schema / volume / lineage monitoring rather than another manual trust check
• Game or live-ops work needs crash, session, build, or launch-event visibility without turning into engine-profiler interpretation
• Cross-functional reliability asks span backend, product/ops, marketing pipelines, and game live-ops, and the next owner is still unclear

When not to use this skill

• The packet is mainly logs and the job is finding the first actionable failure → log-analysis
• The job is reproduce → isolate → verify for a code bug or regression → debugging
• Measurements already exist and the main job is naming a bottleneck or tuning it → performance-optimization
• The main job is release execution, promotion, rollback, or post-deploy sequencing → deployment-automation
• The work is LLM-specific traces, evals, or prompt-observability → langsmith
• The work is Unity/Unreal/Godot frame-time capture interpretation → game-performance-profiler
• The task is pure dashboard/report presentation on curated BigQuery data → looker-studio-bigquery

Instructions

Step 1: Frame the packet

Record the smallest useful intake statement before recommending tooling.

Capture:

• surface: service/API | worker/queue | data/pipeline | dashboard/audit | game/live-ops | mixed | unknown
• request type: new setup | review/audit | incident follow-up | migration | launch readiness | unknown
• current packet: architecture note | alert rules | dashboard inventory | incident summary | telemetry config | stale-report complaint | crash/session brief | none
• user/business impact: latency | errors | stale data | missing visibility | crash/session risk | noisy pages | unknown
• ownership: app team | platform/SRE | data/ops | live-ops | shared | unknown

Quick frame:

Surface: data/pipeline
Request type: review/audit
Current packet: stale dashboard complaint + job ownership notes
Impact: stale data and trust erosion
Ownership: data/ops + dashboard consumer owner

Step 2: Start from the intake packet

Use references/intake-packets-and-route-outs.md.

Choose the packet the user actually has now:

• service / reliability packet
• telemetry-foundation packet
• data / pipeline packet
• review / audit packet
• game / live-ops packet
• no usable packet yet

Output this step as:

## Intake Packet
- Current packet:
- Why it is enough (or not enough):
- Missing context to collect next:

Rule: do not force a vendor comparison or telemetry-stack rewrite if the current packet already narrows the next decision.

Step 3: Choose one primary observability mode

Pick one primary mode from references/modes-and-boundaries.md.

Primary modes:

• service-reliability
• telemetry-foundation
• data-pipeline-observability
• game-liveops-visibility
• review-gap-audit
• unknown-needs-better-packet

Rule: one primary mode, optional secondary mode. Do not blend launch telemetry, stale dashboard audits, crash visibility, and generic instrumentation into one answer.

Step 4: Name the core monitoring question

Before listing tools or metrics, state what the system must answer.

Good examples:

• “Would user-visible API pain page us before customers report it?”
• “Do we know when Monday’s growth dashboard is stale, why it is stale, and who owns the fix?”
• “Can launch-event crashes be grouped by build/platform and escalated before social reports spike?”
• “Do current alerts point responders to one useful dashboard/runbook instead of three noisy symptoms?”

Avoid vague statements like “set up better monitoring.”

Step 5: Build one smallest signal plan

Use references/telemetry-rollout-matrix.md.

For the chosen mode, define:

• primary questions the dashboard / alert path must answer
• evidence surfaces: metrics, logs, traces, black-box checks, crash tooling, freshness checks, lineage views
• page now vs ticket later vs dashboard-only thresholds
• owner for each alert or dashboard family
• the first 1–3 implementation slices only

Rules:

• Alert on symptoms before internal causes.
• Prefer one clear page over many stack-layer pages for the same incident.
• Keep labels bounded; push high-cardinality detail into logs/traces.
• Include runbook/dashboard links whenever an alert expects human action.
• Treat metamonitoring as first-class when alert delivery can fail silently.

Step 6: Make route-outs explicit

Hand work off when the job shifts.

Common route-outs:

• root-cause log forensics → log-analysis
• correctness-first regression hunt → debugging
• bottleneck diagnosis or tuning → performance-optimization
• release execution / post-deploy rollback path → deployment-automation
• LLM tracing / evals / prompt observability → langsmith
• KPI interpretation / stakeholder evidence summary → data-analysis
• BigQuery-backed dashboard presentation layer → looker-studio-bigquery
• engine-profiler interpretation → game-performance-profiler

Step 7: Return the observability brief

# Observability Brief

## Scope
- Surface:
- Request type:
- Intake packet:
- Primary mode:
- Confidence:

## Core Monitoring Question
- ...

## Signal Plan
- Metrics / checks:
- Logs / traces / crash context:
- Dashboards / views:
- Alert policy:

## Ownership
- Primary owner:
- Secondary owner(s):

## First Implementation Slice
1. ...
2. ...
3. ...

## Route-outs
- ...

Examples

Example 1: New API before launch

Input: “We’re launching a new API next week. Tell me what to instrument and what should page us.”

Expected shape: classify as service-reliability, use the current launch/readiness packet, define RED / golden-signals questions plus black-box coverage, page thresholds, and one smallest rollout slice.

Example 2: Growth dashboard keeps going stale

Input: “Our Monday morning growth dashboard is stale half the time. We need observability, not another spreadsheet check.”

Expected shape: classify as data-pipeline-observability, cover freshness/schema/volume/lineage/ownership, and distinguish dashboard trust checks from KPI interpretation work.

Example 3: Review / gap audit

Input: “We have tons of alerts and dashboards, but nobody trusts them. What should we keep versus delete?”

Expected shape: classify as review-gap-audit, use the alert/dashboard checklist, produce keep/fix/delete/add decisions, and call out noisy pages, ownerless panels, and missing metamonitoring.

Example 4: Route-out to rollout execution

Input: “We just deployed and need a step-by-step rollback/promotion checklist with health checks.”

Expected shape: route the execution workflow to deployment-automation, while optionally noting the few post-deploy observability questions that matter.

Example 5: Route-out to log forensics

Input: “Here are the outage logs. Find the root cause.”

Expected shape: do not use this as the main workflow; route to log-analysis and only propose observability follow-up after the first actionable failure is identified.

Best practices

1. Start with the packet and core monitoring question, not the vendor. 2. Keep one primary mode and one smallest implementation slice. 3. Alert on symptoms, not every possible cause. 4. Make ownership, runbooks, and dashboard links explicit. 5. Treat stale data / pipeline trust as first-class observability work. 6. Separate game live-ops visibility from engine-profiler interpretation. 7. Keep review/audit work honest: dead dashboards and noisy alerts should be deleted, not merely documented. 8. Sync compact discovery surfaces whenever the front-door boundary changes.

References

• Google SRE — Monitoring Distributed Systems
• OpenTelemetry — Observability primer
• Prometheus — Alerting
• Grafana — Dashboard best practices

Alert and Dashboard Checklist

Use this when reviewing an existing observability setup.

Alert quality

• What exact user-visible or business-visible symptom does this alert represent?
• Is there a clear owner who can act when it fires?
• Should this be a page, a ticket, or dashboard-only information?
• Does the alert tolerate short blips and self-healing behavior?
• Does it include a dashboard, runbook, or query link for first response?
• Are multiple stack-layer alerts duplicating the same incident?
• Would the team regret being woken up by this at 3am?

Dashboard quality

• What decision or question is this dashboard meant to support?
• Is there an owner who maintains thresholds and panel relevance?
• Are the key service questions visible without scrolling through noise?
• Does the dashboard separate symptom panels from deep-dive drilldown panels?
• Are deploy/build/version markers visible when changes often trigger incidents?
• Is stale or unused telemetry being graphed just because it exists?

Data / pipeline checks

• Freshness / lateness window defined?
• Schema-drift detection present?
• Volume / duplicate / null-rate anomalies covered?
• Lineage / downstream blast-radius view available?
• Dashboard consumers know which table/job owns each alert?

Game / live-ops checks

• Crash reports tagged by build, platform, region, and game mode?
• Session / matchmaking / purchase / backend health visible during events?
• Alerts separated between player-facing severity and internal investigation noise?
• Breadcrumbs, logs, or device context attached to critical issues?

Metamonitoring

• Do we know if the telemetry pipeline itself is down?
• Are alert delivery and black-box probes monitored independently?
• Could the current setup fail silently?

Intake Packets and Route-Outs

Start from the packet the user already has. Do not force a vendor or telemetry-stack discussion before classifying the evidence surface.

Service / reliability packet

Use when the packet is an architecture note, health check list, SLO draft, recent outage summary, or launch-readiness review for an API/app/worker.

Look for:

• user-visible journeys that matter
• one RED / golden-signals view per service
• paging thresholds vs ticket-only thresholds
• black-box checks and metamonitoring
• missing runbook / owner links

Route out when:

• the real task is release execution or rollback sequencing → deployment-automation
• the packet is already a bottleneck trace / query plan / profiler artifact → performance-optimization

Telemetry-foundation packet

Use when the packet is instrumentation debt, event naming drift, trace/log/metric gaps, or a migration between telemetry backends.

Look for:

• correlation fields (trace_id, request_id, job_id, build, region)
• bounded labels and cardinality risk
• retention and sampling defaults
• rollout slices by team / service / pipeline
• ownership for maintaining telemetry contracts

Route out when:

• the ask is AI/LLM trace quality, evals, or prompt observability → langsmith
• the task is generic environment/bootstrap setup rather than telemetry decisions → system-environment-setup

Data / pipeline packet

Use when the packet is a stale dashboard complaint, broken marketing report, warehouse job alert, freshness issue, or schema-drift concern.

Look for:

• freshness / lateness windows
• schema / contract drift
• volume, duplicate, or null-rate anomalies
• lineage / downstream blast radius
• which team owns the consumer-facing dashboard

Route out when:

• the main job is KPI interpretation, experiment readout, or stakeholder explanation → data-analysis
• the main job is report/dashboard presentation design on top of trusted BigQuery data → looker-studio-bigquery

Review / audit packet

Use when teams already have dashboards and alerts but trust is low.

Look for:

• noisy pages with no action
• dead dashboards or duplicated panels
• missing black-box coverage or metamonitoring
• alerts with no owner, runbook, or dashboard link
• keep / fix / delete / add decisions

Route out when:

• the packet is a pile of logs and the real ask is first actionable failure → log-analysis
• the packet is a code regression hunt rather than telemetry review → debugging

Game / live-ops packet

Use when the packet is launch-event readiness, crash visibility, backend health for a live event, or session/build/platform monitoring.

Look for:

• build / version / platform tags
• crash grouping and attached device/session context
• player-facing thresholds vs investigation-only thresholds
• session / matchmaking / purchase / backend health panels
• launch war-room dashboard and escalation ownership

Route out when:

• the main job is frame-time / CPU/GPU capture interpretation → game-performance-profiler
• the main job is Steam store timing / launch-page operations rather than telemetry → steam-store-launch-ops

Monitoring-Observability Modes and Boundaries

Primary modes

Neighboring skill boundaries

• log-analysis
• Use when logs already exist and the main task is finding the first actionable failure.
• Not for choosing instrumentation, retention, alert routing, or dashboard ownership.

• debugging
• Use when the likely bug is in code behavior and you need reproduce → isolate → verify.
• Not for deciding what telemetry should exist before the next incident.

• performance-optimization
• Use when measurements already show latency/throughput/resource bottlenecks and you need tuning.
• Not for building the telemetry foundation itself.

• deployment-automation
• Use when shipping a release safely through preview/staging/prod promotion and rollback planning.
• Monitoring-observability may define post-deploy telemetry questions, but not the release workflow.

• langsmith
• Use for LLM traces, evals, prompt versioning, and AI-specific production monitoring.
• Monitoring-observability covers general service / system / pipeline observability.

• game-performance-profiler
• Use when the real job is CPU/GPU/memory/frame-time bottleneck interpretation in Unity or Unreal.
• Monitoring-observability covers crash/session/backend/live-ops visibility around the game.

Anti-patterns

• One skill tries to do instrumentation setup, incident root-cause analysis, and code fixes.
• Dashboards are treated as the whole observability strategy.
• Alert volume grows because teams page on internal causes instead of end-user symptoms.
• Data pipeline reliability is ignored because the team only thinks in app/server terms.
• Game workflows are collapsed into generic backend examples with no build/platform/session context.

Telemetry Rollout Matrix

Use this matrix to choose the first implementation slice.

Retention / cardinality notes

• Keep labels bounded; avoid unbounded user IDs in metrics labels.
• Use logs or trace attributes for high-cardinality detail.
• Add correlation IDs consistently before chasing fancy dashboards.
• Sample expensive traces deliberately; do not silently destroy critical incident paths.

Ownership defaults

• Service reliability: app team + platform/SRE
• Data observability: data/ops owner + dashboard consumer owner
• Game live-ops: live-ops / backend owner + crash tooling owner
• Existing-stack review: whoever gets paged should be part of the ownership decision