Datadog Design Dashboard

Design Datadog Dashboard

Design a dashboard layout that tells a clear story — from high-level health signals down to granular diagnostics — using proper widget types, group organization, and template variables for reusability.

Important: Always check for existing dashboards first with pup dashboards list --agent. Do not create a new dashboard if one already exists for the same service or purpose — update the existing one instead. Only create a new dashboard when no relevant one exists or the user explicitly asks for a new one.

Philosophy: The frameworks, layouts, and widget guides in this skill are starting points — not rigid rules. Every product and business is different. Understand the domain first, then adapt the frameworks to fit. The best dashboards reflect how the business actually works, not how a generic template says they should.

---

Interview

First, determine the mode:

• Design mode — user wants to create or redesign a dashboard → ask design questions, then run the full workflow
• Audit mode — user wants to review an existing dashboard with no intent to redesign → skip to Audit

Design Questions

Skip if ALL of these are already specified: dashboard purpose, target audience, data sources, template variable needs, dashboard strategy.

1. Purpose — "What is this dashboard for? Service overview, infrastructure, executive KPIs, debugging, or SLO tracking?" 2. Audience — "Who will use this? On-call engineers, platform team, leadership, or mixed?" 3. Data Sources — "Which Datadog products are involved? Metrics only, APM + Metrics, Logs + Metrics, or full stack?" 4. Scope — "Is this for a single service, a group of services, or infrastructure-wide?" 5. Dashboard Strategy — "One dashboard per service, or a consolidated view?" — share the trade-offs from references/layouts.md to help them decide. If unsure, ask: "During an outage, does your team investigate one service at a time, or do they need to see all services simultaneously?" 6. Existing Dashboard — "Is there an existing dashboard to audit or redesign?" If yes, fetch with pup dashboards get <id> --agent before designing.

Audit Questions

Skip if ALL of these are already specified: dashboard ID or URL, service name or team context.

Always interview if: No dashboard ID is provided or multiple dashboards may be relevant.

1. Dashboard — "Which dashboard should I review? Provide a dashboard ID, URL, or service name to search for." 2. Business Context — "Can you tell me what this service does for customers? Are there codebases or docs I can read to understand the product?"

• Impact: Understanding the domain lets the review focus on whether the right metrics are being tracked, not just whether generic rules are followed

3. Focus — "Is there anything specific you want me to focus on? (A) Full review, (B) Alert thresholds only, (C) Business section, (D) Layout and readability"

• Default to full review if unspecified

---

Domain Discovery

Applies to design mode. Skip if auditing only.

Before designing, understand what you are building observability for. The metrics that matter depend entirely on the product and business context.

Ask the user:

• "Can you tell me about the product and what this service does for the business? What does a customer experience when they interact with it?"
• "What does a bad day look like for this service? What breaks, and how do customers feel it?"
• "Are there codebases, architecture docs, or README files I can read to understand the service and its dependencies?"

If the user points you to a codebase: Read it. Look at the entry points, the API routes, the database models, the queue consumers, the external service calls. Understanding the code gives you the context to choose metrics that actually matter — not just generic RED/USE signals.

If the user describes the business: Use that context to tailor the Business (B) section. An e-commerce service cares about checkout success rates. A messaging service cares about delivery latency. A payment service cares about transaction completion. Generic "request rate" and "error rate" are a starting point, but the real value comes from metrics that map to customer-visible outcomes.

Skip domain discovery if: You already have deep context about the service from prior conversations or the user has provided detailed specifications.

Gate: Before designing the Business group, you must be able to name at least 3 domain outcomes specific to this service — in plain language a product manager would recognize. Examples: "order placed", "payment completed", "message delivered". If you cannot name them, ask the user before proceeding. Do not substitute transport-layer metrics (gRPC error rate, HTTP request rate) as placeholders — those are P, not B. See the B trap in references/widgets.md.

---

Design

Skip to [Audit](#audit) if the user only wants to review an existing dashboard.

1. Gather existing context

pup dashboards list --agent
pup dashboards get <dashboard-id> --agent

If auditing an existing dashboard, fetch its definition first and analyze its current structure before redesigning.

2. Explore available telemetry

Before designing widgets, check what metrics and tag values actually exist for the service. This prevents designing around metrics that don't exist or using the wrong tag values in queries.

# See what metrics are available for the service
pup metrics list --filter="<service-name>.*" --agent

# Verify the service tag is active and see what metrics are flowing
pup metrics list --filter="trace.*" --tag-filter="service:<service-name>" --agent

Use the actual metric names and tag values you find here when writing widget queries — do not guess or invent them. If a metric you expect does not appear, flag it to the user before building widgets around it.

This applies to all query types: metric queries, APM span filters (operation_name, resource_name, span tags), and log filters. The **Configuration** sections in references/widgets.md describe JSON structure and field constraints only — they are not prescriptive queries. Always verify the actual filter values with pup before using them.

3. Choose a framework

Match the dashboard purpose to a framework. Read references/frameworks.md for detailed metric mappings and group structures.

4. Design the layout

Using your domain understanding and the chosen framework, design the group structure and select widgets. Read these references before designing:

• [layouts.md](references/layouts.md) — Template variable conventions, group structure patterns, dashboard strategy trade-offs, grid sizing, anti-patterns
• [widgets.md](references/widgets.md) — Widget selection guide, display options, sizing, naming conventions
• [thresholds.md](references/thresholds.md) — Alert threshold markers, threshold proximity, Y-axis configuration

5. Consider tabs

If the dashboard has 7+ top-level groups, evaluate whether tabs would reduce scroll fatigue. Read references/tabs.md for the organization pattern, JSON schema, and common mistakes.

Before adding tabs to an existing dashboard, audit all widget IDs for duplicates — tabs trigger strict uniqueness validation.

6. Write the design output

Present the design using this template:

# Dashboard Design: [Dashboard Title]

## Purpose
[1-2 sentences: what this monitors, who uses it]

## Template Variables
| Variable | Tag | Default |
|----------|-----|---------|
| ... | ... | `*` |

## Layout

### Group: [Group Title]
| Widget | Type | Query/Metric | Width | Alert Threshold |
|--------|------|-------------|-------|----------------|
| ... | ... | ... | ... | ... |

[Repeat for each group]

---

Audit

Applies to both modes. Run after design, or directly if auditing an existing dashboard.

The core principles are: graphs should earn their place with alert thresholds, thresholds should sit close to normal traffic, a business section should exist at the top, and the dashboard should be readable by someone with zero service knowledge.

These are guiding principles — not a rigid checklist. Apply judgment based on the product and business context. A context-providing metric (like deployment events) may earn its place without a threshold. A service with unusual traffic patterns may need different proximity rules.

1. Fetch the dashboard

# If given a service name, list all dashboards and identify the relevant one by title
pup dashboards list --agent

# If given a URL, extract the dashboard ID from the path (e.g., /dashboard/abc-def-ghi/...)

# Get the full dashboard definition (includes the dashboard URL in the response)
pup dashboards get <dashboard-id> --agent

# Verify real metric names exist
pup metrics list --filter="trace.http.request.*" --agent

Parse the response to build an inventory of all widgets, groups, and their configurations.

2. Build widget inventory

Read references/widgets.md for the full widget prefix system before cataloging.

Catalog every widget in the dashboard:

Focus on timeseries and query value widgets — these are the primary candidates for alert threshold markers.

3. Audit alert thresholds

Read references/thresholds.md for threshold marker principles, configuration details, and findings format.

For each timeseries widget, check:

• Does it have a marker/threshold line configured?
• Is the marker colored red for visibility?
• Does the threshold correspond to an actual monitor/alert?

4. Audit threshold proximity

Read references/thresholds.md for proximity guidance, Y-axis configuration rules, and findings format.

For each widget with a threshold, check:

• What is the typical (normal) value range?
• Where is the threshold set?
• Is there excessive whitespace between the normal line and the alert line?
• Is the Y-axis auto-scaled or explicitly set?

5. Audit business section

Principle: A dedicated Business (B) group should exist at the top of the dashboard with 5-8 key metrics for immediate outage identification. Business metrics are customer-visible outcomes — not infrastructure or domain internals. The specific metrics should reflect the product's business transactions, not generic traffic and error rates.

Check:

• Does a Business group exist (named "Business", "B", or equivalent)?
• Is it the first group on the dashboard?
• Do its widgets use the B0-N: prefix?
• Does it contain 5-8 metrics covering: customer-visible success rates, key transaction flows, and SLA-impacting latency?
• Can someone determine "are customers affected?" within 5 seconds of opening the dashboard?
• B trap check: For each B-prefixed widget, ask "Can a product manager interpret this without knowing the transport protocol?" If no — gRPC error rate, HTTP request rate, queue depth — it is P, not B, regardless of where it is placed. Flag and recommend moving to the appropriate platform group.

Findings format:

#### Business Section Audit

**Status**: MISSING / INCOMPLETE / OK

**Current state**: [Description of what exists]

**Recommended metrics** (if missing or incomplete):
1. B0: Key transaction success rate (are critical flows completing?)
2. B0: Customer-facing error rate (are requests failing for customers?)
3. B1: API p99 latency (are responses slow for customers?)
4. B1: Total request rate (are we receiving traffic?)
5. B2: Queue depth or processing lag (is async work backing up?)
6. B2: Key business event throughput (e.g. orders created, payments processed)

6. Apply zero-knowledge viewer test

Principle: Someone with zero knowledge of the service should be able to spot problems by looking for red indicators.

Evaluate:

• Can you identify a problem in under 10 seconds without reading widget titles?
• Are thresholds visible as red lines on every graph?
• Is conditional formatting applied to query value widgets (green/yellow/red)?
• Are group names self-explanatory?
• Is there a note widget with runbook links or team ownership?

Findings format:

#### Zero-Knowledge Readability Audit

| Check | Status | Finding |
|-------|--------|---------|
| Problems visible in <10s | FAIL | No red lines on 8 of 12 graphs |
| Conditional formatting on QV widgets | PARTIAL | 2 of 4 QV widgets have thresholds |
| Group names self-explanatory | OK | All groups use clear names |
| Runbook/ownership note | MISSING | No note widget with team info |

7. Audit tab organization

If the dashboard has 7+ top-level groups, check whether tabs are in use. See references/tabs.md.

• Are related groups consolidated into tabs by observability layer?
• Do tab names reflect the question they answer (Service Health, Platform, Infrastructure)?
• Are all top-level widgets assigned to exactly one tab?

8. Generate audit report

Compile all findings into a structured report:

# Dashboard Audit: [Dashboard Title]

**Dashboard ID**: [id]
**URL**: [url]
**Review date**: [date]

## Summary

[2-3 sentence summary: overall health of the dashboard, critical issues count]

## Critical Issues

[List issues that must be fixed before the dashboard is production-ready]

## Alert Threshold Audit
[From step 3]

## Threshold Proximity Audit
[From step 4]

## Business Section Audit
[From step 5]

## Zero-Knowledge Readability Audit
[From step 6]

## Recommended Actions

### Must Fix
1. [Action item with specific widget and group reference]

### Should Fix
1. [Action item]

### Nice to Have
1. [Action item]

---

Quality Principles

• [ ] Widget queries use real metric names verified via pup metrics list --agent — no invented metric names
• [ ] Dashboard reflects the actual product and business — metrics tailored to the domain
• [ ] Dashboard title follows [service] Purpose pattern — no "Dashboard" suffix, no environment in the title
• [ ] title field updated in the JSON (not just the filename) — redeploy after any title change
• [ ] Template variables match the dashboard type — see references/layouts.md
• [ ] Widget queries use template variable scopes verified via pup metrics list --agent — no hardcoded env, service, or host values; use the variable set appropriate for the dashboard type (see references/layouts.md)
• [ ] Business group with 5-8 B-prefixed metrics tailored to the service's customer-visible outcomes
• [ ] Groups ordered macro-to-micro (business → overview → details)
• [ ] Every widget title uses the layer-priority prefix (I0:, P1:, D0:, B0:, etc.) — see references/widgets.md
• [ ] Widget titles use sentence case, don't repeat group name
• [ ] Timeseries widgets have alert threshold markers (red lines) where the metric is alertable — see references/thresholds.md
• [ ] Thresholds close to normal traffic — no excessive whitespace
• [ ] Zero-knowledge readability — someone with no service knowledge can spot problems via red indicators
• [ ] Query Value widgets have conditional formatting (green/yellow/red)
• [ ] Every metric earns its place — if it spikes, someone can act on it
• [ ] All audit findings include specific widget names and group references
• [ ] Recommended actions categorized by priority (must/should/nice-to-have)
• [ ] Dashboard URL included in audit report

References

• [Observability Frameworks](references/frameworks.md) — RED, USE, Golden Signals, SLI/SLO with metric mappings
• [Layout & Structure](references/layouts.md) — Template variables, group patterns, dashboard strategy, grid sizing, anti-patterns
• [Widgets](references/widgets.md) — Widget prefix system, types, display options, sizing, naming conventions
• [Alert Thresholds](references/thresholds.md) — Threshold markers, proximity guide, Y-axis configuration
• [Tabs](references/tabs.md) — Dashboard tab organization, JSON schema, @N positional references, common mistakes

Observability Frameworks

Choose the framework that matches the dashboard purpose. Each framework defines which metrics to prioritize and how to organize them.

---

RED Method (Request-Driven Services)

Best for: microservices, APIs, web applications — anything that handles requests.

Group Structure

Overview → Rate → Errors → Duration → [Dependencies] → [Infrastructure]

When to Use RED

• Service receives external or internal HTTP/gRPC requests
• You care about user-facing latency and availability
• On-call engineers need to quickly identify degradation

Drill-Down Pattern

1. Overview: Query Value widgets showing current Rate, Error %, p99 Latency 2. Rate: Timeseries of request rate, Top List by endpoint 3. Errors: Timeseries of error rate, Top List of failing endpoints, Log Stream of errors 4. Duration: Timeseries of latency percentiles, Heatmap of latency distribution, Top List of slowest endpoints

---

USE Method (Resource-Oriented)

Best for: infrastructure, databases, queues — anything with finite capacity.

Group Structure

Overview → CPU → Memory → Disk → Network → [Application-specific]

When to Use USE

• Monitoring hosts, containers, or VMs
• Database or cache performance
• Queue/worker infrastructure
• Capacity planning

Resource Mapping

---

Golden Signals (SRE / Executive)

Best for: executive dashboards, cross-service views, SRE-level monitoring.

Group Structure

Executive Summary → Latency → Traffic → Errors → Saturation → [Business KPIs]

When to Use Golden Signals

• Cross-service or platform-wide view
• Executive or leadership audience
• SRE team situational awareness
• Incident command dashboards

Presentation Style

• Executive dashboards favor Query Value and Timeseries over detailed Top Lists
• Use wider time windows (1h, 4h, 1d) for trend visibility
• Include week-over-week comparisons where possible

---

SLI/SLO Tracking

Best for: SLO compliance, error budget monitoring, reliability reviews.

Group Structure

SLO Summary → Error Budget → Burn Rate Alerts → Historical Compliance

When to Use SLI/SLO

• Tracking reliability commitments
• Error budget-driven development decisions
• Reliability review meetings
• Incident post-mortem context

Widget Recommendations

• SLO widget: Native Datadog SLO widget for compliance tracking
• Query Value: Current error budget remaining (color-coded)
• Timeseries: Burn rate over time with threshold lines
• Change Widget: Week-over-week SLI comparison

Layout & Structure

Template variable conventions, group structure patterns, dashboard strategy trade-offs, and grid sizing.

---

Template Variables

Template variables make one dashboard serve many contexts. Define them before laying out widgets.

`env` is always required. The remaining variables depend on the dashboard type:

Optional variables (add when relevant to the dashboard type):

• For service dashboards (RED, Debugging): scope every widget query with {$service,$env}
• For infrastructure dashboards: scope queries with {$host,$env} or {$availability_zone,$env} instead
• For executive dashboards: scope queries with {$team,$env} or {$region,$env} instead
• Use * as the default value so dashboards load with full scope
• Never put environment or region in the dashboard title — that is what template variables are for
• Name variables after the tag they filter on

---

Dashboard Strategy

---

Group Structure

Organize widgets into collapsible groups. Groups are the primary navigation mechanism.

Recommended groups (in order):

1. Business — 5-8 B-prefixed metrics that answer "are customers affected?" within 5 seconds. Should be the first group. The specific metrics depend on the product. Design so someone with zero service knowledge can spot problems via red indicators. 2. Overview — Service checks, key health indicators, monitor summaries. 3. Domain-specific groups — Organized by the chosen framework (e.g., Rate / Errors / Duration for RED), adapted to the service's actual architecture and concerns.

Optional groups: Logs, Infrastructure, Dependencies, Deployment.

• Group titles use Title Case
• Keep groups to 4-8 widgets each (collapse if more)
• Order groups macro-to-micro (health overview → detailed diagnostics)
• Color-code group headers for visual scanning

---

Layout Templates

Standard group structures by dashboard type. Use these as starting points and customize based on specific needs.

Service Overview (RED)

The most common dashboard type. Monitors a single service's request-level health.

┌─────────────────────────────────────────────────────┐
│ Template Variables: env | service | region           │
├─────────────────────────────────────────────────────┤
│ Group: Business                                      │
│ ┌──────────┬──────────┬──────────┬──────────┐       │
│ │ B0:Req/s │ B0:Errs  │ B0:p99   │ B0:Apdex │       │
│ │ (QV+bg)  │ (QV+bg)  │ (QV+bg)  │ (QV+bg)  │       │
│ ├──────────────────────┬────────────────────┤       │
│ │ B0: Key txn success  │ B1: DB conn pool   │       │
│ │ (TS + red threshold) │ (TS + red thresh.) │       │
│ └──────────────────────┴────────────────────┘       │
│ 5-8 metrics · zero-knowledge readable               │
├─────────────────────────────────────────────────────┤
│ Group: Overview                                      │
│ ┌──────────┬──────────┬──────────┬──────────┐       │
│ │ P0:Req/s │ P0:Err % │ P0:p99   │ P0:Apdex │       │
│ │ (QV+bg)  │ (QV+bg)  │ (QV+bg)  │ (QV+bg)  │       │
│ └──────────┴──────────┴──────────┴──────────┘       │
├─────────────────────────────────────────────────────┤
│ Group: Rate                                          │
│ ┌────────────────────────┬──────────────────────┐   │
│ │ P0: Requests/s (TS)    │ P1: By endpoint (TL) │   │
│ │ ── red threshold ──    │                      │   │
│ └────────────────────────┴──────────────────────┘   │
├─────────────────────────────────────────────────────┤
│ Group: Errors                                        │
│ ┌────────────────────────┬──────────────────────┐   │
│ │ P0: Error rate (TS)    │ P1: Top errors (TL)  │   │
│ │ ── red threshold ──    │                      │   │
│ ├────────────────────────────────────────────────┤   │
│ │ P1: Error logs (log stream, full width)        │   │
│ └────────────────────────────────────────────────┘   │
├─────────────────────────────────────────────────────┤
│ Group: Duration                                      │
│ ┌────────────────────────┬──────────────────────┐   │
│ │ P0: Latency p50/90/99  │ P1: Latency heatmap  │   │
│ │ (TS + red threshold)   │ (heatmap)            │   │
│ ├────────────────────────────────────────────────┤   │
│ │ P1: Slowest endpoints (toplist, full width)    │   │
│ └────────────────────────────────────────────────┘   │
├─────────────────────────────────────────────────────┤
│ Group: Infrastructure (collapsed by default)         │
│ ┌────────────┬────────────┬────────────┐            │
│ │ I0: CPU    │ I0: Memory │ I1: Disk   │            │
│ │(TS + red)  │(TS + red)  │(TS + red)  │            │
│ └────────────┴────────────┴────────────┘            │
└─────────────────────────────────────────────────────┘

All widget titles use the layer-priority prefix system (I0:, P0:, D0:, B0:, etc.) — see widgets.md for details. All timeseries widgets include red alert threshold markers set close to normal traffic.

Widget count: 20-24 Target audience: On-call engineers, service owners

---

Infrastructure (USE)

Monitors host, container, or VM resource health.

┌─────────────────────────────────────────────────────┐
│ Template Variables: env | host | availability_zone   │
├─────────────────────────────────────────────────────┤
│ Group: Business                                      │
│ ┌──────────┬──────────┬──────────┬──────────┐       │
│ │B0:Svc Rq │B0:Svc Err│B0:Svc p99│B0:Apdex  │       │
│ │ (QV+bg)  │ (QV+bg)  │ (QV+bg)  │ (QV+bg)  │       │
│ ├──────────────────────┬────────────────────┤       │
│ │ B0: Host avail.      │ B1: Network errors │       │
│ │ (TS + red threshold) │ (TS + red thresh.) │       │
│ └──────────────────────┴────────────────────┘       │
│ 5-8 metrics · zero-knowledge readable               │
├─────────────────────────────────────────────────────┤
│ Group: Overview                                      │
│ ┌──────────┬──────────┬──────────┬──────────┐       │
│ │ I0:CPU % │ I0:Mem % │ I0:Disk% │ I1:NetBps│       │
│ │ (QV+bg)  │ (QV+bg)  │ (QV+bg)  │ (QV+bg)  │       │
│ └──────────┴──────────┴──────────┴──────────┘       │
├─────────────────────────────────────────────────────┤
│ Group: CPU                                           │
│ ┌────────────────────────┬──────────────────────┐   │
│ │ I0: CPU by core        │ I1: Load avg         │   │
│ │ (TS + red threshold)   │ (TS + red threshold) │   │
│ └────────────────────────┴──────────────────────┘   │
├─────────────────────────────────────────────────────┤
│ Group: Memory                                        │
│ ┌────────────────────────┬──────────────────────┐   │
│ │ I0: Memory usage       │ I1: Swap usage       │   │
│ │ (TS + red threshold)   │ (TS + red threshold) │   │
│ └────────────────────────┴──────────────────────┘   │
├─────────────────────────────────────────────────────┤
│ Group: Disk                                          │
│ ┌────────────────────────┬──────────────────────┐   │
│ │ I0: Disk utilization   │ I1: I/O wait         │   │
│ │ (TS + red threshold)   │ (TS + red threshold) │   │
│ └────────────────────────┴──────────────────────┘   │
├─────────────────────────────────────────────────────┤
│ Group: Network                                       │
│ ┌────────────────────────┬──────────────────────┐   │
│ │ I0: Bytes in/out       │ I0: Errors + drops   │   │
│ │ (TS + red threshold)   │ (TS + red threshold) │   │
│ └────────────────────────┴──────────────────────┘   │
└─────────────────────────────────────────────────────┘

All widget titles use the layer-priority prefix system — see widgets.md for details. All timeseries widgets include red alert threshold markers set close to normal traffic.

Widget count: 18-22 Target audience: Platform/SRE team

---

Executive / Golden Signals

High-level view across multiple services for leadership.

┌─────────────────────────────────────────────────────┐
│ Template Variables: env | team | region               │
├─────────────────────────────────────────────────────┤
│ Group: Business                                      │
│ ┌──────────┬──────────┬──────────┬──────────┐       │
│ │B0:Uptime │B0:Cst Err│B0:p99 Lat│B0:Traffic│       │
│ │ (QV+bg)  │ (QV+bg)  │ (QV+bg)  │ (QV+bg)  │       │
│ ├──────────────────────┬────────────────────┤       │
│ │ B0: Revenue txn succ │ B0: Checkout lat   │       │
│ │ (TS + red threshold) │ (TS + red thresh.) │       │
│ └──────────────────────┴────────────────────┘       │
│ 5-8 metrics · zero-knowledge readable               │
├─────────────────────────────────────────────────────┤
│ Group: Executive Summary                             │
│ ┌──────────┬──────────┬──────────┬──────────┐       │
│ │P0:Uptime │P1:Avg Lat│P1:Tot Rq │P0:Err %  │       │
│ │ (QV+bg)  │ (QV+bg)  │ (QV+bg)  │ (QV+bg)  │       │
│ └──────────┴──────────┴──────────┴──────────┘       │
├─────────────────────────────────────────────────────┤
│ Group: Traffic                                       │
│ ┌──────────────────────────────────────────────┐    │
│ │ P0: Request volume by svc (TS + red thresh.)  │    │
│ └──────────────────────────────────────────────┘    │
├─────────────────────────────────────────────────────┤
│ Group: Latency                                       │
│ ┌──────────────────────────────────────────────┐    │
│ │ P0: p99 latency by svc (TS + red threshold)   │    │
│ └──────────────────────────────────────────────┘    │
├─────────────────────────────────────────────────────┤
│ Group: Errors                                        │
│ ┌────────────────────────┬──────────────────────┐   │
│ │ P0: Err rate by svc    │ P1: Svcs w/ errors   │   │
│ │ (TS + red threshold)   │ (toplist)            │   │
│ └────────────────────────┴──────────────────────┘   │
├─────────────────────────────────────────────────────┤
│ Group: Saturation                                    │
│ ┌────────────────────────┬──────────────────────┐   │
│ │ I0: CPU across fleet   │ I0: Memory fleet     │   │
│ │ (TS + red threshold)   │ (TS + red threshold) │   │
│ └────────────────────────┴──────────────────────┘   │
└─────────────────────────────────────────────────────┘

All widget titles use the layer-priority prefix system — see widgets.md for details. All timeseries widgets include red alert threshold markers set close to normal traffic.

Widget count: 16-20 Target audience: Engineering leadership, SRE

---

Debugging / Investigation

Deep-dive dashboard for active incident investigation.

┌─────────────────────────────────────────────────────┐
│ Template Variables: env | service | host | endpoint   │
├─────────────────────────────────────────────────────┤
│ Group: Business                                      │
│ ┌──────────┬──────────┬──────────┬──────────┐       │
│ │B0:Cst Err│B0:p99 Lat│B0:Req/s  │B0:Apdex  │       │
│ │ (QV+bg)  │ (QV+bg)  │ (QV+bg)  │ (QV+bg)  │       │
│ ├──────────────────────┬────────────────────┤       │
│ │ B0: Key txn success  │ B1: DB conn pool   │       │
│ │ (TS + red threshold) │ (TS + red thresh.) │       │
│ └──────────────────────┴────────────────────┘       │
│ 5-8 metrics · zero-knowledge readable               │
├─────────────────────────────────────────────────────┤
│ Group: Current State                                 │
│ ┌──────────┬──────────┬──────────┬──────────┐       │
│ │P0:Err %  │P0:p99 ms │P1:Req/s  │I1:Hosts  │       │
│ │ (QV+bg)  │ (QV+bg)  │ (QV+bg)  │ (QV+bg)  │       │
│ └──────────┴──────────┴──────────┴──────────┘       │
├─────────────────────────────────────────────────────┤
│ Group: Error Analysis                                │
│ ┌────────────────────────┬──────────────────────┐   │
│ │ P0: Errors by type     │ P1: By endpoint (TL) │   │
│ │ (TS + red threshold)   │ (toplist)            │   │
│ ├────────────────────────────────────────────────┤   │
│ │ P1: Error logs (log stream, full width)        │   │
│ └────────────────────────────────────────────────┘   │
├─────────────────────────────────────────────────────┤
│ Group: Latency Breakdown                             │
│ ┌────────────────────────┬──────────────────────┐   │
│ │ P0: Latency by endpt   │ P1: Latency distrib. │   │
│ │ (TS + red threshold)   │ (heatmap)            │   │
│ └────────────────────────┴──────────────────────┘   │
├─────────────────────────────────────────────────────┤
│ Group: Dependencies                                  │
│ ┌────────────────────────┬──────────────────────┐   │
│ │ P0: Downstream latency │ P0: Downstream errs  │   │
│ │ (TS + red threshold)   │ (TS + red threshold) │   │
│ └────────────────────────┴──────────────────────┘   │
├─────────────────────────────────────────────────────┤
│ Group: Resource Pressure                             │
│ ┌────────────┬────────────┬────────────┐            │
│ │ I0: CPU    │ I0: Memory │ P1: Conns  │            │
│ │(TS + red)  │(TS + red)  │(TS + red)  │            │
│ └────────────┴────────────┴────────────┘            │
├─────────────────────────────────────────────────────┤
│ Group: Recent Events                                 │
│ ┌──────────────────────────────────────────────┐    │
│ │ Deploy + event stream (full width)            │    │
│ └──────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────┘

All widget titles use the layer-priority prefix system — see widgets.md for details. All timeseries widgets include red alert threshold markers set close to normal traffic.

Widget count: 24-30 Target audience: On-call engineers during incidents Note: More template variables for deeper filtering; endpoint variable enables per-route investigation

---

Grid System

Datadog uses a 12-column grid.

---

Anti-Patterns

Tabs

Dashboard tabs organize top-level widget groups into named sections. Tabs reduce scroll fatigue on large dashboards and let viewers navigate directly to the layer they care about during an incident.

---

When to Use Tabs

---

Tab Organization Pattern

Group tabs by observability layer, not by arbitrary concern. Each tab answers a different question during an incident.

Adapt the tab names and groupings to the service. A service with no CQRS layer does not need a Platform tab. A service with no load tests does not need a Testing tab.

---

JSON Schema

Tabs are a top-level dashboard field, sibling to widgets. Widgets stay in a flat array; tabs reference them by position.

{
  "tabs": [
    {
      "name": "Business",
      "widget_ids": ["@1", "@2", "@5"]
    },
    {
      "name": "Platform",
      "widget_ids": ["@3", "@4", "@8"]
    },
    {
      "name": "Infrastructure",
      "widget_ids": ["@6", "@7"]
    }
  ],
  "widgets": [ ... ]
}

Field reference

Positional references (@N)

Widget IDs use @N notation — 1-indexed strings referencing position in the widgets array.

• "@1" = first widget in the array
• "@15" = fifteenth widget in the array
• Must be strings ("@1"), not integers
• The Datadog API resolves @N to actual widget IDs server-side
• The Terraform provider converts API-returned integer IDs back to @N on read

Prefer `@N` over hardcoded integer IDs. Integer widget IDs are unstable — the API assigns them on creation and they change if widgets are recreated. @N references are positional and survive widget ID changes.

---

Common Mistakes

Wrong field names

The Datadog API documentation uses tab_name and tab_id. The actual API and Terraform provider expect name and id. Using the documented names returns a 400 error.

// WRONG — returns 400
{"tab_name": "Overview", "tab_id": "...", "widget_ids": [...]}

// CORRECT
{"name": "Overview", "id": "...", "widget_ids": [...]}

Duplicate widget IDs

Adding tabs triggers strict widget ID uniqueness validation that the API previously ignored. If your dashboard JSON has widgets with duplicate id fields (common when widgets were copy-pasted), the API will reject the update with "Duplicate widget IDs found".

Before adding tabs: audit all widget IDs for duplicates.

import json
from collections import Counter

with open("dashboard.json") as f:
    d = json.load(f)

all_ids = []
def collect_ids(widgets):
    for w in widgets:
        wid = w.get("id")
        if wid is not None:
            all_ids.append(wid)
        for nested in w.get("definition", {}).get("widgets", []):
            collect_ids([nested])

collect_ids(d["widgets"])
dupes = {k: v for k, v in Counter(all_ids).items() if v > 1}
print(f"Duplicates: {dupes}" if dupes else "No duplicates")

Every widget must be assigned

All top-level widgets must appear in exactly one tab. Missing a widget does not hide it — the API may reject the request or behave unpredictably.

Provider version

Tab support requires Terraform provider datadog/datadog >= 3.44.0. Earlier versions silently pass the tabs field through but cannot normalize @N references on read, causing perpetual diffs.

---

Terraform Provider Internals

Understanding the provider lifecycle prevents debugging dead ends.

On create/update (write path)

1. stripDeprecatedFields() removes is_read_only from the JSON 2. Same function injects deterministic UUIDs for any tab missing an id: uuid5(Nil, "tab-{index}") 3. The JSON — including @N references — is sent as-is to PUT /api/v1/dashboard/{id} 4. The API resolves @N to actual integer widget IDs server-side

On read (state path)

1. prepResource() strips computed fields (id, author_handle, author_name, created_at, modified_at, url) for diff comparison 2. Same function builds a widgetID → position map and converts integer widget_ids back to @N strings 3. Tab id fields are stripped from state to prevent diff noise

This means: the provider handles the @N ↔ integer conversion transparently. You write @N, the API receives @N, the API returns integers, the provider converts back to @N for state.

Alert Thresholds

Guidance for configuring alert threshold markers and Y-axis settings on timeseries widgets.

---

Threshold Markers

Timeseries widgets should have a red threshold marker line so anomalies are immediately visible.

• The threshold should correspond to an actionable alert — if the metric crosses this line, someone investigates
• Use markers in the widget definition with display_type: "error dashed" and value: "y = <threshold>"
• The threshold should match (or be derived from) an actual Datadog monitor
• Use red color for instant visual identification

Not every metric needs a threshold. Context-providing metrics (deployment markers, dependency traffic patterns) may earn their place on a dashboard without one. Use judgment based on the service's domain.

---

Threshold Proximity

Set thresholds close to normal traffic. Large gaps between normal values and the alert line create blind spots where anomalies go unnoticed.

Bad: Normal CPU is 20%, alert threshold at 95% — the graph is mostly empty space and a slow climb from 20% to 80% looks flat.

Good: Normal CPU is 20%, alert threshold at 45% — anomalies visually stand out immediately.

Design test: If the metric slowly climbs from normal to 2x normal, the change should be visually obvious on the graph. If the threshold is too far away, the graph compresses the normal range and the climb looks flat.

---

Y-Axis Configuration

Do not rely on Y-axis auto-scaling. Set yaxis.max explicitly to slightly above the alert threshold.

Why: Auto-scaling fits the entire value range (normal traffic + threshold), which compresses normal traffic into a flat band at the bottom of the graph. Anomalies become invisible because the Y-axis range is too wide.

Rule: The Y-axis max should frame the normal-to-threshold range so that deviations are visually obvious. The alert threshold should sit near the top of the graph, and normal traffic should occupy the visible area.

---

Audit Findings Format

Alert Threshold Audit

#### Alert Threshold Audit

| Widget | Group | Status | Finding |
|--------|-------|--------|---------|
| Requests/s | Rate | MISSING | No threshold marker — add alert line or remove widget |
| Error rate | Errors | OK | Red line at 5% |
| CPU usage | Infra | MISSING | No threshold — is this metric alertable? |

Threshold Proximity Audit

#### Threshold Proximity Audit

| Widget | Normal Range | Threshold | Gap | Y-Axis | Status |
|--------|-------------|-----------|-----|--------|--------|
| CPU usage | ~20% | 95% | 75% | auto | TOO FAR — lower to 40-50%, set Y-max to 55% |
| Error rate | ~0.1% | 5% | ~5% | auto | OK gap — but set Y-max to 6% |
| p99 latency | ~50ms | 500ms | 10x | auto | TOO FAR — lower to 100-150ms, set Y-max to 175ms |

---

Metrics Near Zero

Some metrics hover close to zero under normal conditions (e.g., error counts, retry rates, dead letter queue depth). For these:

• A threshold still makes sense — even a small absolute increase can indicate a problem
• Set the threshold at a level that represents "something changed" rather than a percentage of capacity
• The Y-axis max matters even more here — auto-scaling on a metric that's normally 0 and spikes to 5 will look the same as one that spikes to 5000

Widget Reference

Configuration details, display options, and best practices for each widget type.

---

Widget Selection

What are you showing?
│
├── Single current value? → Query Value (with timeseries bg)
├── Trend over time? → Timeseries
│   ├── Volume/composition? → area display
│   ├── Counts? → bars display
│   └── Multiple series? → lines display
├── Ranked comparison? → Top List
├── Distribution density? → Heatmap
├── Up/down status? → Check Status
├── Log entries? → Log Stream
├── Events/deploys? → Event Stream
├── Period comparison? → Change
├── SLO compliance? → SLO widget
└── Context/links? → Note

---

Query Value

Displays a single numeric value with optional conditional formatting and timeseries background.

Use for: Current metric values — error rate, latency, throughput, count.

Configuration:

• Enable timeseries background (a bare number without trend context is rarely useful)
• Use conditional formatting to color-code thresholds (green/yellow/red)
• Set appropriate precision (2 decimals for percentages, 0 for counts)

Sizing: 3 columns wide, 2 rows tall (standard), or 2 columns for compact layouts.

Conditional Formatting Example:

---

Timeseries

Time-based line, area, or bar chart. The most versatile widget.

Use for: Any metric over time — trends, comparisons, correlations.

Display Types:

Configuration:

• Add a legend (automatic mode for > 5 series)
• Alias formula expressions for readable legends
• Set y-axis minimum to 0 unless negative values are expected
• Set yaxis.max explicitly — see thresholds.md for Y-axis guidance
• Use markers for threshold lines — see thresholds.md for threshold configuration

Sizing: Minimum 4 columns, recommended 6 columns. Use 12 columns for high-detail single-metric views.

Multiple Queries: Overlay related metrics (e.g., p50, p90, p99 on one chart) rather than creating separate widgets.

---

Top List

Ranked list of values by a dimension.

Use for: Identifying outliers — busiest endpoints, highest error sources, slowest queries.

Configuration:

• Order descending by default (highest values first)
• Limit to top 10-25 entries
• Use conditional formatting to highlight problematic entries
• Show both absolute value and percentage where useful

Sizing: Minimum 4 columns, recommended 6 columns.

---

Heatmap

Color-coded density visualization.

Use for: Distribution data — latency distributions, request size distributions.

Configuration:

• Use for metrics with many unique values (percentile distributions)
• Color palette should represent density (lighter = fewer, darker = more)
• Works well paired with a timeseries showing the same metric's percentiles

Sizing: Minimum 4 columns, recommended 6 columns.

---

Group

Container widget that organizes other widgets into a collapsible section.

Use for: Organizing dashboard into logical sections.

Configuration:

• Title in Title Case
• Color-code headers for visual scanning (consistent across similar groups)
• Collapse non-critical groups by default (Infrastructure, Dependencies)
• Keep 4-8 widgets per group

---

Check Status

Color-coded grid showing service check results.

Use for: Binary health status — service up/down, integration connected/disconnected.

Configuration:

• One check per cell
• Green = OK, Red = Critical, Yellow = Warning, Grey = Unknown
• Group by host or service tag

Sizing: 2-3 columns wide.

---

Log Stream

Live-updating list of log entries matching a query.

Use for: Viewing actual log lines — error details, request traces, debug output.

Configuration:

• Filter to relevant log level (error, critical) or service
• Show timestamp, status, message columns at minimum
• Sort by timestamp descending (newest first)
• Link to Log Explorer for deeper investigation

Sizing: Minimum 6 columns, recommended 12 columns (full width). Needs horizontal space for message readability.

---

Event Stream

Timeline of Datadog events.

Use for: Deployment events, alert triggers, infrastructure changes.

Configuration:

• Filter by source (deploy, monitor, integration)
• Useful in debugging dashboards to correlate changes with metric shifts

Sizing: Minimum 6 columns, recommended 12 columns.

---

Note

Markdown text widget.

Use for: Context that is not metric data — runbook links, team ownership, on-call rotation, usage instructions.

Configuration:

• Keep concise — dashboards are for data, not documentation
• Include runbook links for on-call dashboards
• Use sparingly — if you need many notes, the dashboard structure might be unclear

Sizing: 2-4 columns wide.

---

List Stream (Trace / Log)

Live-updating list of trace spans or log entries. JSON widget type: list_stream.

Use for: Surfacing individual spans for investigation — failed jobs, slow commands, high-lag handler executions.

Data sources:

• trace_stream — APM trace spans
• logs_stream — Log entries

`trace_stream` query schema constraints — the query object for trace_stream accepts only these fields:

• data_source — must be "trace_stream"
• indexes — array, usually []
• query_string — the filter expression

Do NOT include sort, storage, compute, or any other fields inside the query object for trace_stream. The Datadog API will return a 400 validation error. These fields are valid for logs_stream but not trace_stream.

Sizing: Minimum 6 columns, recommended 12 columns (full width).

---

Change

Shows the change in a metric value over a time period.

Use for: Week-over-week or day-over-day comparisons.

Configuration:

• Compare against previous period (1d, 1w)
• Use conditional formatting for increase/decrease
• Pairs well with Query Value for current value + Change for trend

Sizing: 3-4 columns wide.

---

SLO

Native Datadog SLO tracking widget.

Use for: SLO compliance, error budget remaining.

Configuration:

• Reference an existing SLO definition
• Show target, current value, and error budget
• Time window matching SLO period (7d, 30d, 90d)

Sizing: 4-6 columns wide.

---

Widget Title Prefix System

Every widget title starts with a layer-priority prefix so anyone can immediately tell what layer the metric belongs to and how critical it is.

Layers

Priority Numbers

The number after the layer letter indicates priority within that layer. 0 is the most critical — the metric you look at first during an outage. Higher numbers are progressively less critical.

Examples

Group: "Business"
  B0: Checkout success rate
  B0: Order throughput
  B1: API p99 latency
  B1: Customer-visible error rate
  B2: Failed payment rate

Group: "Rate"
  P0: Requests per second
  P1: By endpoint

Group: "Errors"
  P0: Error rate over time
  D1: Order saga failures
  P2: Top errors by endpoint

Group: "Infrastructure"
  I0: CPU usage
  I0: Memory usage
  I1: Disk I/O
  I2: Network errors

Classification Guide

When assigning prefixes, use the domain discovery context:

• I (Infrastructure): Would this metric exist even if your code didn't? Load balancer, database engine, OS resources, network — things the ops team manages.
• P (Platform): Is this about how your code runs? Connection pools your code configures, gRPC channels your code opens, cache hit rates for caches your code uses — the technical platform layer.
• D (Domain): Is this technical health of a domain process? Saga step failures, aggregate timeouts, domain event processing lag — tech stuff that a domain engineer cares about.
• B (Business): Is this a business outcome? Payment success rate, checkout completion, on-time delivery — business stuff that a product manager or customer cares about.

The priority number comes from the ops review order: what do you look at first when paged at 3am? That's 0.

The B trap: transport metrics are not business metrics

The most common misclassification is putting transport-layer health metrics in the Business group. gRPC error rate, HTTP error rate, and request throughput are `P` — not `B` — even when they appear in the Business group and even when the service's only interface is gRPC or HTTP.

Ask: "Can a product manager interpret this without knowing what gRPC or HTTP is?" If no, it's P.

Rule: If you cannot complete the sentence "Customers are affected because ___" using only the metric name, it is not B.

Platform metric catalog

When you encounter metrics in a codebase or Datadog, use this to classify them correctly. This is not a list of widgets to add — it is a guide for recognising what layer a metric belongs to. Only include metrics that actually exist and are relevant to the service being observed.

The signals listed under each component type are examples of what tends to exist, not requirements. Every service is different.

Inbound protocol servers (HTTP, gRPC, GraphQL, WebSocket, etc.)

Any server that accepts requests or connections — regardless of protocol. Request rate, error rate, latency, and apdex are always P for these. They describe how the transport layer is performing, not what the business is doing.

Outbound clients (HTTP, gRPC, RPC, external APIs)

Calls the service makes to other services or third-party APIs.

Message queue consumers

Any process consuming from a queue or stream (regardless of broker).

Background job processors

Deferred or scheduled work — any job queue or scheduler.

Database and cache clients

The application's access layer — not the database engine or cache server (those are I).

Event-driven patterns (CQRS, event sourcing, pub/sub)

Command dispatch, aggregate execution, event handlers — when a service uses an event-driven architecture.

---

General Naming Rules

Dashboard title: Concise, purpose-driven. Example: Order Service, not Order Service Production US-East Dashboard v2.

Widget titles: Prefix + sentence case, concise, action-oriented.

• Always start with the layer-priority prefix (I0:, P1:, D0:, B0:, etc.)
• Do not repeat the group title after the prefix
• Do not repeat the integration name if it is obvious from context
• Alias all formulas so legends are readable