Dt Alerting

dt-alerting

Configure and understand the full alerting lifecycle in Dynatrace — from anomaly detector setup through Grail event storage, problem grouping, and workflow notification delivery.

The Alerting Lifecycle

┌─────────────────────────────────────────────────────────────────────┐
│  Alert Sources — five categories, each fires a DAVIS_EVENT          │
│  ─────────────────────────────────────────────────────────────────  │
│  1. DQL-based  · Grail-scheduled server-side detector               │
│  2. Edge       · OneAgent on monitored host or process              │
│  3. Pipeline   · OpenPipeline ingest-stream filter matcher          │
│  4. Synthetic  · Worldwide synthetic checker node                   │
│  5. External   · Events API, Workflow, or OneAgent local ingest     │
└──────────────────────────────┬──────────────────────────────────────┘
                               │ DAVIS_EVENT created per trigger per entity
                               ▼
             ┌─────────────────────────────────────┐
             │  Event stored in Grail              │  Persisted and queryable via DQL.
             └──────────────────┬──────────────────┘  One event per trigger per entity.
                                │ correlated by root-cause and impact graph
                                ▼
             ┌─────────────────────────────────────┐
             │  Problem (Denoising)                │  Events sharing the same root-cause
             └──────────────────┬──────────────────┘  and impact graph → one Problem.
                                │ problem event triggers workflow
                                ▼
             ┌─────────────────────────────────────┐
             │  Workflow Notification              │  Filters problems and routes to
             └─────────────────────────────────────┘  email, Slack, ServiceNow, webhook.

When to Use This Skill

• Detector setup — "How do I create an anomaly detector?", "What kind of

detector should I use?", "What is the difference between adaptive and seasonal?"

• Alert event history — "Query all alert events for this service", "Show me

which metrics triggered alerts last week"

• Problem denoising — "Why did these two alerts merge into one problem?",

"How does Davis group alerts?"

• Notification setup — "How do I send a Slack message when a problem opens?",

"Set up a ServiceNow ticket on critical problems"

• Best practices — "How do I avoid alert storms?", "Which sensitivity setting

should I use?"

• Over-alerting analysis — "Why am I getting too many alerts?", "How do I

reduce alert fatigue?", "Which detector is firing the most?", "How do I tune sensitivity or thresholds to avoid noise?"

• Notification routing — "How do I route alerts to the right team?", "Set up

scalable problem filters in workflows", "Send Slack notifications only to the team responsible for the affected service"

Agent Instructions

First step for any alerting setup request — Before recommending a specific detector or model, load references/anomaly-detectors.md and use its category and model decision guide to identify which detector category (DQL-based, Edge, Pipeline, Synthetic, External) and which model (Static, Adaptive, Seasonal) best fits the user's use-case. Only proceed with configuration guidance once the right detector type has been established.

Consolidate, don't multiply — When a user asks to alert on multiple entities of the same kind (e.g. "alert on services A, B, and C"), always recommend a single combined detector rather than one detector per entity. Use by: { <dimension> } in the DQL timeseries call to split results per entity, and a single filter: clause to scope to the relevant entities. Pair the combined detector with a single `dt.alert_group` tag shared across all alert conditions and the corresponding workflow notification filter. This keeps the number of detector configs small, ensures consistent routing, and makes the workflow notification channel reusable for future entities added to the same group.

Example for three services — one detector, one workflow:

timeseries avg(dt.service.request.response_time),
  by: { dt.smartscape.service },
  filter: { in(dt.smartscape.service, {toSmartscapeId("SERVICE-0000000000000001"), toSmartscapeId("SERVICE-0000000000000002"), toSmartscapeId("SERVICE-0000000000000003")}) }

Set dt.alert_group: "checkout-team" in the detector's event properties, then filter the notification workflow on matchesPhrase(dt.alert_group, "checkout-team"). If a new service must be covered, add it to the single filter: list — no new detector or workflow rule needed.

Intent Mapping

Analyzing existing problems — If the user wants to query or investigate

active/closed problems (root cause, impact, trending), load dt-obs-problems

instead. This skill covers configuration and flow, not problem query analytics.

Detector health monitoring — If the user asks whether detectors are

running or failing, load dt-platform (ANALYZER_EXECUTION_EVENT,

ANOMALY_DETECTOR_STATUS_EVENT). This skill covers setup, not operational health.

Prerequisites

• Access to a Dynatrace environment with Settings v2 write permissions for

detector configuration

• For querying alert history: DQL permissions on dt.davis.events
• Load dt-dql-essentials before writing DQL queries

Knowledge Base Structure

Key Concepts

Alert Source Categories

Five fundamental categories of anomaly detectors, distinguished by where detection runs and how the alert event reaches Dynatrace:

See references/anomaly-detectors.md for the full breakdown of each category, including trade-offs and configuration entry points.

Detector Models at a Glance

Davis Events vs. Problems

A single problem typically contains multiple events. Querying problems gives the operational view; querying events gives the raw alert history.

Problem Denoising

For questions about why alerts merged into a problem or how Davis groups events, load dt-obs-problems — the merge logic and rules are documented in dt-obs-problems/references/problem-merging.md. This skill covers alert configuration and flow only.

Quick Start

Check What Alerts Fired in the Last 24 Hours

fetch dt.davis.events, from: -24h
| filter event.status == "ACTIVE"
| summarize alert_count = count(), by: {event.name, event.category, dt.smartscape_source.id}
| sort alert_count desc
| limit 20

Check Alert Volume by Category

fetch dt.davis.events, from: -24h
| summarize count = count(), by: {event.category, event.status}
| sort count desc

See All Active Problems (→ load dt-obs-problems for full query patterns)

fetch dt.davis.problems, from: -24h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| fields event.start, display_id, event.name, event.category
| sort event.start desc
| limit 20

Best Practices

1. Match the model to the metric's behavior — Use static for hard SLO boundaries, adaptive for metrics without a natural fixed limit, seasonal for anything that follows business hours or weekly patterns. 2. Scope detectors narrowly — An entity selector that covers only relevant entities reduces noise and makes problems more actionable. 3. Tune sensitivity before going to production — Start with LOW sensitivity and move to MEDIUM or HIGH only after observing false-positive rates. 4. Let Davis denoise before notifying — Trigger workflow notifications on problems, not individual alert events. A problem groups correlated alerts so you notify once per incident, not once per metric. 5. Filter notifications by severity level — Route event.severity <= 2 problems to on-call channels immediately; route event.severity >= 3 problems to lower- urgency channels. Either set severity in the detector config or assign in a pipeline rule or workflow. 6. Use `dt.alert_group` event property for routing — Assign dt.alert_group to route alerts to the right team. Either set a static value in the detector config, use dynamic assignment through DQL query result mapping or assign in a pipeline rule. 7. Combine same-condition alerts into one detector and one workflow — When alerting on multiple entities with the same metric and threshold, merge them into a single DQL-based detector using by: { <dimension> } and a combined filter: clause. Assign the same dt.alert_group value to every condition in that detector and point the workflow notification channel at that single group. One detector + one workflow per logical alert group scales better than N detectors + N notification rules, and adding a new entity is a one-line filter change rather than a full detector/workflow addition.

Related Skills

• dt-obs-problems — Querying, analyzing, and trending detected problems
• dt-obs-predictive-analytics — Ad-hoc anomaly and novelty detection using

MCP analyzer tools (not persistent alert configs)

• dt-platform — Operational health of anomaly detectors (execution events,

failure rates)

• dt-platform-costs — Query costs generated by anomaly detector DQL
• dt-sdlc-quality-gates — Site Reliability Guardian for deployment gate alerting
• dt-dql-essentials — DQL syntax for writing detector queries and alert history

queries

Anomaly Detectors

Configure anomaly detectors to continuously evaluate metrics and fire Davis events when conditions are breached.

Contents

• Alert Source Categories
• DQL-Based Detector Variants
• Detector Models
• Built-in Davis AI Detection
• Davis Anomaly Detectors (Metric Key)
• Davis Anomaly Detectors (DQL-based)
• Runtime JSON structure
• Input key reference
• Example: Static threshold
• Example: Adaptive baseline
• Example: Seasonal baseline
• eventTemplate.properties reference
• DQL query notes
• Davis Anomaly Detectors (Record-based)
• How record-based detection works
• alertIdentityFields — per-row deduplication
• Example: Scalar alert — log error count above threshold
• Example: Per-entity alert — disk free space per host
• Example: Timeseries flattened to scalar per entity
• OneAgent Edge-Side Anomaly Detectors
• How Edge-Side Detection Works
• When to Prefer Edge Detection
• Disk Edge Alerts (`builtin:infrastructure.disk.edge.anomaly-detectors`)
• OS Service Monitoring (`os-services-monitoring`)
• Process Availability (`process-availability`)
• Choosing the Right Detector and Model
• Configuration Best Practices

---

Alert Source Categories

Dynatrace supports five fundamentally different categories of anomaly detectors, distinguished by where detection runs and how the alert event is generated. Understanding the category determines which tool to configure and what latency and data access trade-offs apply.

Category characteristics

1. DQL-based detectors Run entirely on the server side against data already stored in Grail. Evaluated on a configurable schedule (typically every minute). Covers the widest range of metrics — anything reachable via DQL, including calculated metrics, log-derived metrics, span metrics, and business events. Includes:

• Built-in Davis AI health alerts for services, hosts, databases, Kubernetes
• User-defined Davis anomaly detectors (builtin:davis.anomaly-detectors) — supports metric-key and DQL-based query definitions

2. Edge alerts Detected locally by the Dynatrace OneAgent running on the monitored host or process. The agent observes conditions (disk space, process crashes, network errors) that are known before data is ever sent to Grail, and pushes a Davis event directly. Examples: disk full alert, process crash alert, host availability alert. Near-real-time latency because no Grail read is required.

3. Pipeline alerts Evaluated by DQL filter matchers embedded directly in the OpenPipeline ingest path. Raw data (typically logs or events) is checked against the matcher rule as it flows through the pipeline, before it is indexed in Grail. This enables zero-latency alerting on log patterns — a matching log line raises an alert the moment it arrives, without waiting for a scheduled DQL query to run.

4. Synthetic alerts A specialized variant of edge alerting where the detection runs on Dynatrace synthetic checker nodes distributed across worldwide locations. Each node executes availability and performance checks (HTTP monitors, browser click paths, API tests) and raises a Davis event directly when a check fails or a latency threshold is exceeded. Like edge alerts, synthetic alerts bypass the Grail read path for near-real-time detection.

5. Externally ingested alert events The customer or an external tool owns the alert logic. Dynatrace acts as the alert ingestion, storage, and correlation layer rather than the detector. Alert events are pushed into Dynatrace using:

• Events API v2 (POST /api/v2/events/ingest) — direct REST call
• Workflow action — a workflow step that creates a Davis event
• OpenPipeline ingest APIs — event routed through the pipeline with alert classification
• OneAgent local event ingest — agent SDK used by custom application code

These events receive the same Davis AI problem grouping and workflow notification treatment as internally generated alerts. Common use cases: third-party monitoring tools forwarding alerts, application-level business KPI alerts, CI/CD deployment events that should trigger alerting.

Trade-offs by category

---

DQL-Based Detector Variants

Within the DQL-based category, Dynatrace offers three detector variants that differ in how the metric is specified and which models are available:

---

Detector Models

Static Threshold

Fires when a metric value exceeds (or falls below) a fixed value you define, for a minimum sustained duration.

Use when:

• You have a hard SLO or operational boundary (e.g., error rate > 5%, disk > 90%)
• The acceptable range does not change over time
• You want predictable, auditable alert conditions

Key parameters:

• threshold — the fixed boundary value
• violationType — ABOVE or BELOW
• violationDuration — how long the threshold must be breached before firing

Pitfall: Setting thresholds too tight on volatile metrics causes alert storms. Use adaptive baseline if the metric fluctuates naturally.

---

Adaptive Baseline

Learns normal behavior from recent historical data and fires when the metric deviates significantly from that learned baseline. The threshold is dynamic — it adjusts as behavior shifts.

Use when:

• The metric has no natural fixed limit but has a clear normal range
• Normal values differ between environments or time periods
• You want to detect anomalies without knowing the exact threshold upfront

Key parameters:

• sensitivity — LOW / MEDIUM / HIGH: controls how many standard deviations from

the baseline count as a violation (HIGH = fires sooner, more sensitive)

• referencePeriod — how much history to use for baseline learning (typically 7–30 days)

Pitfall: Running adaptive baseline on a metric that is genuinely trending upward will produce continuous false positives as the metric leaves its baseline. Use novelty detection (dt-obs-predictive-analytics) to detect trends instead.

---

Seasonal Baseline

Like adaptive baseline but explicitly models time-of-day and day-of-week patterns before deciding what is anomalous. Monday 9am traffic is compared to previous Monday 9am values, not to the overall average.

Use when:

• The metric follows clear business-hour or weekly patterns (request rate, active

users, transaction volume)

• Adaptive baseline fires too often during expected peaks and valleys
• You need to distinguish "high traffic Monday morning" from a genuine anomaly

Key parameters:

• Same sensitivity and reference period as adaptive
• Requires sufficient history (at least 2 full weekly cycles) before the seasonal

model stabilizes

Pitfall: Seasonal baseline needs at least 2 weeks of stable history to be reliable. Newly onboarded services or environments will have a learning period with higher false-positive rates.

---

Built-in Davis AI Detection

Dynatrace automatically detects anomalies for every monitored entity type (services, hosts, databases, Kubernetes workloads, synthetic monitors, etc.) without any configuration. Detection uses adaptive models tuned per entity.

What it covers

Tuning built-in detection

Navigate to Settings → Anomaly detection → [Entity type] to:

• Enable / disable specific detection categories (availability, performance, errors)
• Adjust detection sensitivity (LOW / MEDIUM / HIGH)
• Override sensitivity per individual entity

Built-in detection cannot be replaced with custom queries — use custom metric events or Davis anomaly detectors for metrics beyond the predefined catalog.

---

Davis Anomaly Detectors (Metric Key)

Settings schema: builtin:davis.anomaly-detectors

Note: The field names documented in this section (queryDefinition, modelProperties, eventTemplate.severity) reflect an older API representation. The current runtime format uses the analyzer-based structure described in the DQL-based section. When creating new detectors via dtctl apply or the Settings v2 API, use the analyzer structure with analyzer.name and analyzer.input key-value pairs.

Allows user-defined alert rules on any metric available in Dynatrace using a metric key selector. Each rule defines a metric key, an entity scope, a model, and a threshold. Use this variant when you do not need the full flexibility of a DQL query. The legacy builtin:anomaly-detection.metric-events schema served the same purpose but has been superseded by builtin:davis.anomaly-detectors and does not support DQL-based query definitions.

Configuration fields

Example: Static threshold on error rate

{
  "enabled": true,
  "queryDefinition": {
    "type": "METRIC_KEY",
    "metricKey": "builtin:service.errors.total",
    "aggregation": "AVG",
    "entityFilter": {
      "dimensionKey": "dt.entity.service",
      "conditions": [
        { "type": "TAG", "value": "env:production" }
      ]
    }
  },
  "modelProperties": {
    "type": "STATIC_THRESHOLD",
    "threshold": 5.0,
    "violationType": "ABOVE",
    "alertCondition": "ALL_SERIES_IN_VIOLATIONS",
    "violationWindowInMinutes": 5,
    "dealertingWindowInMinutes": 10
  },
  "eventTemplate": {
    "title": "Error rate above 5% in production",
    "severity": "AVAILABILITY"
  }
}

---

Davis Anomaly Detectors (DQL-based)

Settings schema: builtin:davis.anomaly-detectors

The most powerful and flexible custom detector type. You write a DQL timeseries query to define exactly which metric to evaluate. Davis evaluates it on a schedule and applies the chosen model.

Why use DQL-based detectors

• Any metric reachable via DQL — including calculated metrics, span metrics,

log-based metrics, and business events

• Flexible entity grouping via by:{} dimensions
• All three models available: static, adaptive, seasonal
• DQL lets you pre-aggregate, filter, or transform before anomaly evaluation

Runtime JSON structure

The actual runtime format for builtin:davis.anomaly-detectors uses an analyzer-based structure — not the field paths sometimes shown in older documentation. The top-level object to POST to PUT /api/v2/settings/objects (or apply via dtctl apply -f) is:

{
  "schemaId": "builtin:davis.anomaly-detectors",
  "scope": "tenant",
  "value": {
    "title": "...",
    "description": "...",
    "enabled": true,
    "source": "Davis Anomaly Detection",
    "analyzer": {
      "name": "<analyzer-class-name>",
      "input": [
        { "key": "<param>", "value": "<value>" }
      ]
    },
    "eventTemplate": {
      "properties": [
        { "key": "<property>", "value": "<value>" }
      ]
    },
    "executionSettings": {
      "actor": "<user-uuid>"
    }
  }
}

The `analyzer.name` selects the detection model. The `analyzer.input` array supplies all model parameters as key-value string pairs. The three supported analyzers and their input signatures are:

Input key reference

Shared across all three models:

Model-specific:

Example: Static threshold

Fires when average CPU load stays above 80% for 3 of 5 consecutive samples:

{
  "schemaId": "builtin:davis.anomaly-detectors",
  "scope": "tenant",
  "value": {
    "title": "CPU load above 80% — production hosts",
    "enabled": true,
    "source": "Davis Anomaly Detection",
    "analyzer": {
      "name": "dt.statistics.ui.anomaly_detection.StaticThresholdAnomalyDetectionAnalyzer",
      "input": [
        { "key": "query",              "value": "timeseries avg(dt.host.cpu.usage), by: {dt.entity.host}" },
        { "key": "threshold",          "value": "80" },
        { "key": "alertCondition",     "value": "ABOVE" },
        { "key": "alertOnMissingData", "value": "false" },
        { "key": "violatingSamples",   "value": "3" },
        { "key": "slidingWindow",      "value": "5" },
        { "key": "dealertingSamples",  "value": "5" }
      ]
    },
    "eventTemplate": {
      "properties": [
        { "key": "dt.source_entity", "value": "{dims:dt.entity.host}" },
        { "key": "event.type",       "value": "RESOURCE_CONTENTION_EVENT" },
        { "key": "event.name",       "value": "CPU load above 80% on {dims:dt.entity.host}" },
        { "key": "dt.alert_group",   "value": "ops-team" }
      ]
    },
    "executionSettings": { "actor": "<user-uuid>" }
  }
}

Example: Adaptive baseline

Fires when service response time rises significantly above its learned normal behavior (detects gradual degradation without a fixed threshold):

{
  "schemaId": "builtin:davis.anomaly-detectors",
  "scope": "tenant",
  "value": {
    "title": "Abnormal latency increase — JourneyService",
    "enabled": true,
    "source": "Davis Anomaly Detection",
    "analyzer": {
      "name": "dt.statistics.ui.anomaly_detection.AutoAdaptiveAnomalyDetectionAnalyzer",
      "input": [
        { "key": "query.expression",          "value": "timeseries avg_latency = avg(dt.service.request.response_time), by: {dt.smartscape.service} | filter dt.smartscape.service == toSmartscapeId(\"SERVICE-18AA85290DF3D5D2\")" },
        { "key": "numberOfSignalFluctuations", "value": "1" },
        { "key": "alertCondition",             "value": "ABOVE" },
        { "key": "alertOnMissingData",         "value": "false" },
        { "key": "violatingSamples",           "value": "3" },
        { "key": "slidingWindow",              "value": "5" },
        { "key": "dealertingSamples",          "value": "5" }
      ]
    },
    "eventTemplate": {
      "properties": [
        { "key": "dt.source_entity", "value": "{dims:dt.smartscape.service}" },
        { "key": "event.type",       "value": "PERFORMANCE_EVENT" },
        { "key": "event.name",       "value": "Abnormal latency increase on JourneyService" },
        { "key": "event.description","value": "Latency deviated above its adaptive baseline. Detected {violating_samples} violation samples within the evaluation window." },
        { "key": "dt.alert_group",   "value": "my-routing-group" }
      ]
    },
    "executionSettings": { "actor": "<user-uuid>" }
  }
}

Example: Seasonal baseline

Fires when a business metric deviates from its expected time-of-day / day-of-week pattern. Requires at least 2 weeks of stable history before the model stabilizes:

{
  "schemaId": "builtin:davis.anomaly-detectors",
  "scope": "tenant",
  "value": {
    "title": "Abnormal order rate — business hours pattern",
    "enabled": true,
    "source": "Davis Anomaly Detection",
    "analyzer": {
      "name": "dt.statistics.ui.anomaly_detection.SeasonalBaselineAnomalyDetectionAnalyzer",
      "input": [
        { "key": "query.expression",  "value": "timeseries orders = sum(orders_placed_count), by: {dt.entity.service}" },
        { "key": "tolerance",         "value": "4" },
        { "key": "alertCondition",    "value": "BELOW" },
        { "key": "alertOnMissingData","value": "false" },
        { "key": "violatingSamples",  "value": "3" },
        { "key": "slidingWindow",     "value": "5" },
        { "key": "dealertingSamples", "value": "5" }
      ]
    },
    "eventTemplate": {
      "properties": [
        { "key": "dt.source_entity", "value": "{dims:dt.entity.service}" },
        { "key": "event.type",       "value": "CUSTOM_ALERT" },
        { "key": "event.name",       "value": "Order rate below seasonal baseline" },
        { "key": "dt.alert_group",   "value": "my-routing-group" }
      ]
    },
    "executionSettings": { "actor": "<user-uuid>" }
  }
}

eventTemplate.properties reference

`event.type` → Davis problem category mapping:

DQL query notes

• Filter by Smartscape entity ID: Use toSmartscapeId("SERVICE-...") when

comparing a string literal to a Smartscape dimension — raw string comparison produces a warning and may not match correctly.

timeseries avg_latency = avg(dt.service.request.response_time), by: {dt.smartscape.service}
| filter dt.smartscape.service == toSmartscapeId("SERVICE-03F1F46B45BFA6C4")

• Multiple entities: Chain or conditions with toSmartscapeId() per entity.
• `query` vs `query.expression`: Both keys are accepted. query.expression is

the newer form and is required when also supplying query.filterSegments[0].id for filter segment scoping.

Davis evaluates the timeseries on a schedule, applies the chosen model to each series returned by the by:{} dimension independently, and fires a Davis event for any series that violates the threshold.

---

Davis Anomaly Detectors (Record-based)

Settings schema: builtin:davis.anomaly-detectors Analyzer class: dt.statistics.anomaly_detection.RecordAnomalyDetectionAnalyzer

The record-based detector is a fundamentally different kind of DQL detector. Where the timeseries-based analyzers continuously evaluate a metric signal against a model (static, adaptive, or seasonal), the record analyzer evaluates any DQL query on a schedule and treats each row of the result as a violation that triggers a Davis event.

The DQL query IS the alert condition. You write a query whose filter clauses define what constitutes a violation, and you structure the query so that it returns rows only when the condition is actually breached. When Davis evaluates the query:

• Zero rows returned → no alert fires
• N rows returned → N alert events fire, one per row

This approach supports alert conditions that are impossible to express as a timeseries threshold:

• Existence checks — fire when a specific record appears (or disappears) in a log, event, or entity list
• Scalar aggregates — fire when a summarized count exceeds a threshold (e.g., total error count in the last hour)
• Entity inventory conditions — fire for each entity matching a structural criterion (e.g., each host that has more than one disk mount, each service with zero throughput)
• Cross-signal joins — combine logs, traces, metrics, and entities in a single fetch pipeline

How record-based detection works

Scheduler tick
      │
      ▼
DQL query executes
      │
      ├─ 0 rows returned ──→ no alert, any open alert for this detector closes
      │
      └─ N rows returned ──→ one Davis event fires per row
                              │
                              ▼
                     alertIdentityFields determine deduplication:
                     - no alertIdentityFields: N independent events, no dedup
                     - with alertIdentityFields: each unique field-value
                       combination is one open alert; same combination on
                       the next tick updates the existing problem instead
                       of opening a new one; disappearing row closes the alert

alertIdentityFields — per-row deduplication

alertIdentityFields is an optional list of column names from the query result that together uniquely identify each violating entity. When set:

• Each distinct combination of those field values tracks as one open alert
• A row that appeared in the previous evaluation and appears again is treated

as "still violating" — Davis updates the existing problem rather than opening a second one for the same entity

• A row that appeared previously but no longer appears is treated as "recovered"

— Davis closes the alert for that combination

Without `alertIdentityFields`: every row on every evaluation tick creates a new independent event. This is appropriate for one-shot notification patterns but will generate duplicate problems if the condition persists across multiple evaluations.

With `alertIdentityFields`: alerts behave like persistent per-entity state, opening when a row appears, staying open while it persists, closing when it disappears. This is the correct pattern for per-entity violation tracking.

Input key format: alertIdentityFields[0], alertIdentityFields[1], … (zero-indexed array)

Event template field placeholders

In eventTemplate.properties, all column names from the query result are available as {column_name} placeholders in event.name and event.description. This is different from the timeseries analyzers where only a fixed set of {violating_samples}, {threshold}, etc. are available.

{ "key": "event.name",        "value": "High error count on {dt.service.name}: {error_count} errors" }
{ "key": "event.description", "value": "Service {dt.service.name} produced {error_count} errors in the last hour. Error rate: {error_rate_pct}%" }

Use {dims:<column>} (instead of {column}) when referencing an entity ID column that should resolve to the entity's display name.

To link the Davis event to a Smartscape entity:

• dt.source_entity: {dims:<entity_id_column>} — preferred; resolves entity ID to entity name
• dt.smartscape_source.id: {<smartscape_id_column>} — alternative when the result contains a Smartscape ID directly

Input key reference

Example: Scalar alert — log error count above threshold

Fires a single alert when the total number of ERROR log entries in the last hour across the entire environment exceeds 100. Returns at most one row, so one event fires. When the count drops below 100, the query returns no rows and the alert closes.

{
  "schemaId": "builtin:davis.anomaly-detectors",
  "scope": "tenant",
  "value": {
    "title": "High ERROR log volume — environment-wide",
    "enabled": true,
    "source": "Davis Anomaly Detection",
    "analyzer": {
      "name": "dt.statistics.anomaly_detection.RecordAnomalyDetectionAnalyzer",
      "input": [
        { "key": "query.expression", "value": "fetch logs, from: -1h | filter loglevel == \"ERROR\" | summarize error_count = count() | filter error_count > 100" }
      ]
    },
    "eventTemplate": {
      "properties": [
        { "key": "event.type",       "value": "ERROR_EVENT" },
        { "key": "event.name",       "value": "High ERROR log volume: {error_count} errors in last hour" },
        { "key": "event.description","value": "The environment produced {error_count} ERROR log lines in the last hour, exceeding the threshold of 100." },
        { "key": "dt.alert_group",   "value": "my-routing-group" }
      ]
    },
    "executionSettings": { "actor": "<user-uuid>" }
  }
}

Example: Per-entity alert — high CPU usage per host

Fires one alert per host where the maximum CPU usage has exceeded 80%. Uses alertIdentityFields so that each host tracks as a separate open alert that closes when CPU usage drops back below the threshold.

{
  "schemaId": "builtin:davis.anomaly-detectors",
  "scope": "tenant",
  "value": {
    "title": "High CPU usage — per host",
    "enabled": true,
    "source": "Davis Anomaly Detection",
    "analyzer": {
      "name": "dt.statistics.anomaly_detection.RecordAnomalyDetectionAnalyzer",
      "input": [
        { "key": "query.expression",   "value": "timeseries cpu=avg(dt.host.cpu.usage), by: { dt.smartscape.host }\n| fieldsAdd max_cpu = arrayMax(cpu)\n| fieldsRemove cpu\n| filter max_cpu > 80" },
        { "key": "alertIdentityFields[0]", "value": "dt.smartscape.host" }
      ]
    },
    "eventTemplate": {
      "properties": [
        { "key": "dt.source_entity", "value": "{dims:dt.smartscape.host}" },
        { "key": "event.type",       "value": "RESOURCE_CONTENTION_EVENT" },
        { "key": "event.name",       "value": "High CPU usage on {dims:dt.smartscape.host}: {max_cpu}%" },
        { "key": "event.description","value": "Host {dims:dt.smartscape.host} CPU usage reached {max_cpu}%, exceeding the 80% threshold." },
        { "key": "dt.alert_group",   "value": "my-routing-group" }
      ]
    },
    "executionSettings": { "actor": "<user-uuid>" }
  }
}

Example: Timeseries flattened to scalar per entity

A timeseries query can be used with the record analyzer by collapsing the array values to scalars with fieldsAdd ... = arrayAvg(...) and then filter-ing to the violating rows. This gives per-entity static threshold alerting without using the StaticThresholdAnomalyDetectionAnalyzer, and allows arbitrary post-processing (joins, renaming, calculated fields) before the threshold test.

{
  "schemaId": "builtin:davis.anomaly-detectors",
  "scope": "tenant",
  "value": {
    "title": "CPU usage above 80% — per host",
    "enabled": true,
    "source": "Davis Anomaly Detection",
    "analyzer": {
      "name": "dt.statistics.anomaly_detection.RecordAnomalyDetectionAnalyzer",
      "input": [
        { "key": "query.expression",       "value": "timeseries cpu_usage = avg(dt.host.cpu.usage), by: {dt.entity.host}\n| fieldsAdd cpu_usage = arrayAvg(cpu_usage)\n| filter cpu_usage > 80" },
        { "key": "alertIdentityFields[0]", "value": "dt.entity.host" }
      ]
    },
    "eventTemplate": {
      "properties": [
        { "key": "dt.source_entity", "value": "{dims:dt.entity.host}" },
        { "key": "event.type",       "value": "RESOURCE_CONTENTION_EVENT" },
        { "key": "event.name",       "value": "CPU above 80% on {dims:dt.entity.host}" },
        { "key": "event.description","value": "Average CPU usage is {cpu_usage}% on host {dims:dt.entity.host}, exceeding the 80% threshold." },
        { "key": "dt.alert_group",   "value": "my-routing-group" }
      ]
    },
    "executionSettings": { "actor": "<user-uuid>" }
  }
}

When to use record-based vs. timeseries-based detectors

---

OneAgent Edge-Side Anomaly Detectors

Edge-side detectors run entirely within the Dynatrace OneAgent process on the monitored host. The agent observes local system conditions — disk space, CPU, memory, process availability — and evaluates alert thresholds without sending data to Grail first and without executing any DQL query. When a threshold is breached, the agent emits a Davis event directly.

This makes edge detection fundamentally different from DQL-based detectors: there is no scheduled query, no Grail read latency, and no dependency on the Dynatrace cluster being reachable at the moment the condition occurs. Each agent evaluates its own host independently, so detection scales linearly with the monitored fleet at zero additional query load on Grail.

---

How Edge-Side Detection Works

1. The OneAgent process monitors the host operating system and collects raw infrastructure metrics locally (disk usage, inodes, process state, network interface stats, etc.). 2. Configured thresholds are evaluated in the agent's local evaluation loop, typically every 60 seconds, without any round-trip to the Dynatrace cluster. 3. When a threshold breach is detected, the agent creates a Davis event and forwards it to the Dynatrace ingest endpoint. If connectivity is interrupted, events are buffered locally and delivered when connectivity is restored. 4. Davis AI receives the event and applies the same problem grouping and noise reduction as it does for server-side alerts.

Because the entire detection pipeline lives on the host, latency between a real breach and the resulting Davis event is measured in seconds rather than the minutes a DQL query schedule would impose.

---

When to Prefer Edge Detection

Prefer edge-side detectors over DQL-based detectors when all of the following are true:

• A OneAgent is already deployed on the host (no additional instrumentation needed)
• The condition you want to alert on is a local infrastructure signal (disk, CPU,

memory, process, network interface) that the agent can observe directly

• You need fast detection — a DQL query schedule delay is unacceptable
• The fleet is large and you want to avoid Grail query fan-out costs at scale

Do not use edge detection when:

• The alert condition requires combining multiple signals or metrics from

different hosts or services (use DQL-based detector instead)

• The condition is derived from logs, spans, or business events (not observable

locally by the agent)

• No OneAgent is running on the target host (use DQL-based detector instead)

---

Disk Edge Alerts (builtin:infrastructure.disk.edge.anomaly-detectors)

Settings schema: builtin:infrastructure.disk.edge.anomaly-detectors

The disk edge detector is the recommended approach for alerting on disk-related conditions on any host where a OneAgent is running. It is fast, requires no DQL query, and scales to arbitrarily large host fleets without increasing Grail query load. Detection covers disk space exhaustion, inode exhaustion, and slow disk read/write performance.

Prefer `builtin:infrastructure.disk.edge.anomaly-detectors` over a DQL-based disk detector whenever a OneAgent is present. The agent observes disk metrics at the OS level in real time; a DQL query on builtin:host.disk.used.percent would only evaluate on a scheduler cadence and adds unnecessary Grail read overhead for a signal that is already available locally.

What it detects

Configuration fields

Example: Configure disk edge alerts via Settings API

The following payload applies disk edge alert thresholds to a specific host group. POST it to PUT /api/v2/settings/objects with schema builtin:infrastructure.disk.edge.anomaly-detectors.

{
  "schemaId": "builtin:infrastructure.disk.edge.anomaly-detectors",
  "scope": "HOST_GROUP-0000000000000001",
  "value": {
    "enabled": true,
    "diskLowSpaceDetection": {
      "enabled": true,
      "thresholds": {
        "high": { "freeSpacePercentage": 5 },
        "medium": { "freeSpacePercentage": 10 }
      }
    },
    "diskLowInodesDetection": {
      "enabled": true,
      "thresholds": {
        "high": { "freeInodesPercentage": 5 },
        "medium": { "freeInodesPercentage": 10 }
      }
    },
    "diskSlowWritesAndReadsDetection": {
      "enabled": true,
      "writeAndReadTime": {
        "slowDisk": 200
      }
    }
  }
}

Scope options

Settings cascade from environment → host group → host. A host-level setting always wins. This lets you set conservative defaults globally and tighten thresholds for critical hosts.

Scalability note

Each OneAgent evaluates the disk thresholds independently using its local OS metrics. Adding 1,000 more hosts to your environment does not increase Grail query load for disk alerting — each new agent simply runs its own evaluation loop. This is the primary scalability advantage over DQL-based disk detectors, which would require Grail to query and evaluate builtin:host.disk.used.percent across all hosts on every scheduler tick.

---

OS Service Monitoring (os-services-monitoring)

Settings schema: os-services-monitoring

The OS service monitoring detector instructs the OneAgent to continuously check whether selected operating system services (systemd units on Linux, Windows Services on Windows) are running on the host. When a monitored service stops or enters a failed state, the agent emits a Davis event directly without any DQL query or Grail round-trip.

Prefer `os-services-monitoring` over a DQL-based availability check whenever a OneAgent is present. The agent polls the OS service manager (systemd / Windows SCM) locally; a DQL-based approach would require the agent to first ship availability metrics to Grail and then wait for a scheduled query to evaluate them, adding minutes of latency for a condition the agent already knows about immediately.

What it detects

Configuration fields

Example: Monitor selected services via Settings API

The following payload configures OS service monitoring for a host group to watch nginx and postgresql. Apply it via PUT /api/v2/settings/objects with schema os-services-monitoring.

{
  "schemaId": "os-services-monitoring",
  "scope": "HOST_GROUP-0000000000000001",
  "value": {
    "enabled": true,
    "monitoringMode": "MONITOR_SELECTED_SERVICES",
    "serviceFilter": [
      { "serviceId": "nginx" },
      { "serviceId": "postgresql" }
    ],
    "statusCondition": "NOT_RUNNING",
    "alertActivationDuration": 1
  }
}

To monitor all services on every host in the environment and alert as soon as any service stops running:

{
  "schemaId": "os-services-monitoring",
  "scope": "environment",
  "value": {
    "enabled": true,
    "monitoringMode": "MONITOR_ALL_SERVICES",
    "statusCondition": "NOT_RUNNING",
    "alertActivationDuration": 0
  }
}

Scope options

Settings cascade from environment → host group → host, with the most specific scope winning. Use environment scope for a broad baseline and override at host group or host level where service lists differ.

Scalability note

The OneAgent polls the local OS service manager (systemd on Linux, Service Control Manager on Windows) directly. No metric is shipped to Grail until a violation is detected, and no DQL query is executed on any schedule. A fleet of 10,000 hosts each running OS service monitoring adds exactly zero additional Grail query load for availability checking. This is the preferred approach for any service availability use case on OneAgent-monitored hosts.

---

Process Availability (process-availability)

Settings schema: process-availability

The process availability detector instructs the OneAgent to watch whether selected processes are running on the host. The agent monitors the local process table directly and emits a Davis event the moment a watched process disappears, without any DQL query or Grail round-trip.

This is the successor to the deprecated server-side Process Group Availability detector (builtin:availability.process-group-alerting). That legacy schema evaluated process availability by querying Grail on a schedule, introducing minutes of detection latency and adding query load proportional to the monitored fleet. The process-availability schema moves the check to the agent, eliminating both problems.

Prefer `process-availability` over `builtin:availability.process-group-alerting` and over any DQL-based process check whenever a OneAgent is present. The deprecated schema should no longer be used for new configurations.

What it detects

Configuration fields

Example: Watch specific processes via Settings API

The following payload monitors nginx (at least 1 instance) and a Java application (at least 2 instances) on a host group. Apply it via PUT /api/v2/settings/objects with schema process-availability.

{
  "schemaId": "process-availability",
  "scope": "HOST_GROUP-0000000000000001",
  "value": {
    "enabled": true,
    "processAvailabilityRule": [
      {
        "name": "nginx must be running",
        "condition": "$eq(nginx)",
        "minimumInstances": 1
      },
      {
        "name": "Java application — minimum 2 instances",
        "condition": "$contains(java)",
        "minimumInstances": 2
      }
    ],
    "recoveryDetectionTime": 5
  }
}

Deprecation note: builtin:availability.process-group-alerting

The legacy Process Group Availability schema evaluated process state by reading process group metrics from Grail on a scheduled query. It is deprecated and should not be used for new configurations. Existing rules should be migrated to process-availability to gain:

• Seconds-level detection latency instead of minutes
• No additional Grail query load as the fleet grows
• Agent-local resilience — detection continues even when connectivity to the

Dynatrace cluster is temporarily interrupted

Scope options

Settings cascade from environment → host group → host, with the most specific scope winning.

Scalability note

The OneAgent scans the local process table on each evaluation cycle. No process metrics are shipped to Grail until a violation is detected, and no DQL query is ever executed. A fleet of 10,000 hosts each running process availability monitoring adds exactly zero additional Grail query load — the inverse of the deprecated builtin:availability.process-group-alerting schema, which would issue one or more Grail queries per host per scheduler tick.

---

Choosing the Right Detector and Model

Decision guide

Is the condition a local infrastructure signal (disk, OS service availability,
process availability) AND a OneAgent is running on the host?
  └─ YES → Use an edge-side detector
           Disk conditions          → builtin:infrastructure.disk.edge.anomaly-detectors
           OS service availability  → os-services-monitoring
           Process availability     → process-availability
           Fast, scalable, no DQL, no Grail dependency
  └─ NO ──────────────────────────────────────────────────────────┐
                                                                   │
Is the metric predefined (service response time, host CPU, etc.)? │
  └─ YES → Use built-in Davis AI detection, tune sensitivity only │
  └─ NO ──────────────────────────────────────────────────────────┤
                                                                   │
  Do you know the exact acceptable boundary (hard SLO/limit)?     │
  └─ YES → Static threshold (DQL-based)                          │
  └─ NO ──────────────────────────────────────────────────────────┤
                                                                   │
  Does the metric follow business-hour or weekly patterns?         │
  └─ YES → Seasonal baseline (DQL-based)                         │
  └─ NO  → Adaptive baseline (DQL-based)                         │

Model comparison

---

Configuration Best Practices

1. Start with LOW sensitivity — reduces false positives during the initial learning period. Move to MEDIUM or HIGH only after observing real baseline behavior over 1–2 weeks.

2. Use `violationWindow` / `dealertingWindow` — require a sustained breach before firing and a sustained recovery before clearing. Prevents flapping on spiky metrics. A window of 5–10 minutes is a good starting point.

3. Scope entity selectors tightly — a detector that covers all services will fire on every service simultaneously during a shared infrastructure event, creating dozens of Davis events. Scope to a management zone, tag, or specific entity list.

4. Test the DQL query before creating the detector — run the timeseries query in a Notebook or the Explore view and inspect the signal shape. Confirm it returns data, has the right granularity, and is not null for key entities.

5. Name events clearly — the eventTemplate.title becomes the Davis event name and the problem title. Use the pattern: [Metric] [condition] for [scope] — e.g., "Error rate above 5% — payment-service".

6. Avoid overlapping detector definitions — two detectors on the same metric and entity scope will each fire independently, creating two Davis events and potentially two problems for the same incident.

7. Use severity levels intentionally — AVAILABILITY and ERROR severity events are weighted higher in Davis problem prioritization than PERFORMANCE or CUSTOM_ALERT. Match severity to business impact, not technical magnitude.

8. Filter notification workflows by `smartscape.affected_entity.ids`, not `root_cause_entity_id` — when a problem trigger in a workflow needs to be scoped to a specific entity, do not filter on root_cause_entity_id. Davis may not populate a root cause, and the root cause assignment may shift during the problem lifecycle. Instead, filter on the smartscape.affected_entity.ids list and match the target entity ID:

   matchesPhrase(smartscape.affected_entity.ids, "SERVICE-0000000000000001")

This makes the filter resilient to root-cause re-assignments and ensures the workflow fires whenever the entity is involved in the problem, regardless of whether Davis considers it the root cause.

Davis Events in Grail

Every anomaly detector trigger produces a Davis event — a structured record stored in Grail that captures what was detected, on which entity, and for how long. Davis events are the raw material from which AI root-cause analysis builds problems.

Contents

• What is a Davis Event?
• Davis Event Categories
• Key Fields
• event.provider — Alert Source and Licensing
• Settings Reference — Tracing Alerts Back to Their Config
• DQL Query Patterns
• Relationship to Problems
• Davis Event Lifecycle

---

What is a Davis Event?

A Davis event is created when an anomaly detector (built-in (=health alert) or custom) determines that a metric has violated its threshold. Each Davis event:

• Represents one condition violation on one entity
• Is stored in Grail and queryable via DQL
• Has an active window (event.start to event.end)
• Is independently matched against other Davis events by Dynatrace root-cause analysis for problem grouping

One detector firing on 10 different service entities creates 10 separate Davis events — not one. Davis AI then decides which of those Davis events to group into a single problem.

---

Davis Event Categories

The event.category field classifies what kind of condition was detected:

---

Key Fields

Naming convention for event.name

event.name is the primary identifier humans and workflow filters see — it appears in problem titles, notification messages, and DQL results. Inconsistent names make it hard to filter, aggregate, or route alerts reliably.

Rule of thumb — use the pattern: `{Component} {Condition} {Direction/Threshold}`

Good examples

• Payment Service Error Rate High
• Kubernetes Node CPU Saturated
• PostgreSQL Connection Pool Exhausted
• Synthetic Check Availability Failed

Avoid

• Generic names like Alert, Threshold Exceeded, Metric Alert — these are

meaningless in aggregated views

• Including dynamic values (entity names, current metric values) in the title —

use event.description for those; titles should be stable so they can be used as filter keys in workflows and DQL queries

• Mixing naming styles across detectors — decide on one pattern and apply it

consistently so summarize ... by {event.name} produces clean groupings

Severity levels for event.severity

event.severity carries an ITIL-aligned integer that expresses how critical the condition is. The scale runs from 1 (most severe) to 5 (least severe):

Setting and overriding severity

The alert source (detector configuration) should always set an explicit, meaningful default severity (1–5) — do not rely on a platform default. A well-chosen default makes workflow filters and notification routing work without manual triage.

At the same time, the detector configuration should expose a user-facing override so that teams can adjust the default severity for their context without modifying the detector logic. This is especially important for shared or platform-managed detectors where the right severity differs by team or environment (e.g. the same CPU detector may warrant severity 2 in production but severity 4 in a dev environment).

---

event.provider — Alert Source and Licensing

event.provider identifies which Dynatrace subsystem or integration raised the Davis event. It is automatically assigned by the Dynatrace platform — a Davis event client (external caller, workflow, Events API) cannot set or override it.

Why it matters

1. Source traceability — event.provider tells you exactly where in the alerting pipeline a Davis event originated, which is the first step when investigating over-alerting or unexpected alert volume. 2. Licensing — Dynatrace uses event.provider to determine whether a Davis event is covered by your existing license or whether it is priced separately under the event rate card. Understanding which providers are active helps control costs.

Known provider values

This list is not exhaustive. Use the DQL query below to discover all providers

active in your environment.

Query: report active event.provider sources

fetch dt.davis.events, from: -24h
| filter event.status == "ACTIVE"
| summarize alert_count = count(), by: {event.provider}
| sort alert_count desc
| limit 100

Run this query to see which providers are generating the most alert volume — useful as a starting point for over-alerting analysis or license attribution.

---

Settings Reference — Tracing Alerts Back to Their Config

Most (but not all) Davis events carry two fields that link the Davis event back to the exact Dynatrace settings entry that produced it:

object_id vs. schema_id

• `dt.settings.object_id` identifies one specific detector configuration — the

individual rule you created or that Dynatrace auto-generated. Two detectors of the same type will have different object IDs.

• `dt.settings.schema_id` identifies the type of detector. Different detector

categories always use different schema tables, so the schema ID tells you which part of the alerting pipeline the setting belongs to (e.g. DQL-based metric events vs. built-in service health vs. synthetic).

Not all Davis events carry these fields. External Davis events pushed via the Events API and some built-in Davis events may have no settings reference.

One setting, many Davis events

A single settings object can be responsible for one Davis event or thousands, because a detector's entity selector or DQL query can match any number of entities. One alerting rule checking 5,000 Kubernetes pods will produce up to 5,000 simultaneous Davis events if all pods breach the threshold at once.

Implication for over-alerting analysis: a high Davis event count against one dt.settings.object_id does not automatically mean the setting is misconfigured — it may simply cover many entities. Always cross-reference the Davis event count with the number of distinct entities (dt.smartscape_source.id) the setting is actually firing on before concluding that a detector is too broad or too sensitive.

Query: alert volume per settings object

fetch dt.davis.events, from: -24h, to: now()
| filter isNotNull(dt.settings.object_id)
| summarize count = count(), by: {dt.settings.object_id, dt.settings.schema_id, event.name, event.category}
| sort count desc
| limit 100

This query is the starting point for identifying which detector configurations generate the most Davis events. To go deeper, add distinctCount(dt.smartscape_source.id) as entity_count to the summarize clause — a high count combined with a low entity_count (many Davis events from few entities) is a stronger signal of a noisy or over-sensitive detector than raw volume alone.

---

DQL Query Patterns

Active Davis Events

fetch dt.davis.events, from: -24h
| filter event.status == "ACTIVE"
| fields event.start, event.name, event.category, dt.smartscape_source.id, event.provider
| sort event.start desc
| limit 50

Alert Volume by Category and Status

fetch dt.davis.events, from: -24h
| summarize count = count(), by: {event.category, event.status}
| sort count desc

Davis Events for a Specific Entity

fetch dt.davis.events, from: -24h
| filter dt.smartscape_source.id == "SERVICE-XXXXXXXXXX"
| fields event.start, event.end, event.name, event.category, event.status
| sort event.start desc

Alert Frequency Over Time (hourly)

fetch dt.davis.events, from: -7d
| makeTimeseries alerts = count(), interval: 1h, by: {event.category}

Davis Events That Did NOT Merge Into a Problem

fetch dt.davis.events, from: -24h
| filter event.status == "CLOSED"
| filter dt.davis.is_merging_allowed == false
| fields event.start, event.end, event.name, dt.smartscape_source.id
| sort event.start desc
| limit 20

Long-Duration Active Davis Events (potential stuck alerts)

fetch dt.davis.events, from: -7d
| filter event.status == "ACTIVE"
| fieldsAdd duration_h = (now() - event.start) / 1h
| filter duration_h > 2
| fields event.start, event.name, event.category, dt.smartscape_source.id, duration_h
| sort duration_h desc
| limit 20

---

Relationship to Problems

Davis events and problems are separate but linked:

Davis event (fetch dt.davis.events)
    │
    │  Davis AI evaluates: same entity? same time window? same root-cause graph?
    │
    └──► Problem (dt.davis.problems) — one problem per correlated Davis event group

Key distinctions:

Merge logic summary

Davis merges Davis events into one problem when: 1. Active time windows overlap 2. Source entities are the same or topologically related (same host, same call chain) 3. dt.davis.is_merging_allowed is true on both Davis events

See dt-obs-problems/references/problem-merging.md for the full merge decision logic.

---

Davis Event Lifecycle

Detector threshold breached
    │
    ▼
event.status = "ACTIVE", event.start = now()
    │
    │  (if merging allowed and related Davis events exist)
    ▼
Davis creates or extends a Problem — problem groups this Davis event
    │
    │  (threshold no longer breached)
    ▼
event.status = "CLOSED", event.end = now()
    │
    │  (if all contributing Davis events are closed)
    ▼
Problem closes — event.status = "CLOSED" on the problem record

A Davis event stays ACTIVE as long as the metric remains in violation. If the metric briefly recovers and then re-violates within the dealertingWindow, the same Davis event stays ACTIVE rather than creating a new one — this prevents flapping.

Workflow Notifications

Send targeted notifications when Dynatrace detects a problem by configuring problem-triggered workflows. Workflows let you filter exactly which problems notify which channels, avoiding alert storms and routing incidents to the right team.

Simple Workflows vs. Normal Workflows

Dynatrace distinguishes two tiers of workflows with different licensing and capability boundaries.

Simple workflows

A simple workflow consists of exactly one trigger and one action. Its primary purpose is alert notification: react to a problem event and send a message to a channel (Slack, email, ServiceNow, webhook, etc.). Because simple workflows are included in the Dynatrace license at no additional cost, they are the right choice for all standard notification use cases.

Note on FaaS billing: Even though simple workflows are license-included, any execution that invokes a Function-as-a-Service (FaaS) action is billed according to the Dynatrace rate card. This applies consistently across both workflow tiers.

Normal workflows

A normal workflow can contain multiple actions, conditional branching, loops, and JavaScript code actions that execute arbitrary logic. Normal workflows are suited for automation scenarios that go beyond notification: creating and updating tickets, enriching problem context by calling external APIs, orchestrating remediation steps, or coordinating changes across multiple systems. Normal workflow executions are billed according to the Dynatrace rate card.

---

Contents

• Simple Workflows vs. Normal Workflows
• How Problem Notifications Work
• Trigger: Problem Events
• Filtering Which Problems Notify
• Notification Actions
• Routing Patterns
• Scalable Multi-Team Routing with `dt.alert_group`
• Best Practices

---

How Problem Notifications Work

Problem opens / updates / closes
         │
         ▼
Workflow trigger fires (event-driven)
         │
         ▼
Condition filter evaluated
   ├─ condition NOT met → workflow stops, no notification sent
   └─ condition met ──────────────────────────────────────────┐
                                                               │
                                                               ▼
                                                   Notification action executes
                                                   (Slack, email, ServiceNow, …)

The key design principle: notify on problems, not on single Davis events. A problem groups all related alerts into one incident record. Triggering on problems means one notification per incident, not one notification per detector firing.

---

Trigger: Problem Events

Configure the workflow trigger as "Problem" in the Workflows UI or via the Workflows API. The problem trigger ships with five built-in filter options that control which problems actually activate the workflow.

Trigger filters

Trigger payload fields

The workflow receives the full problem record as its trigger event. Key fields available for filtering and notification content:

---

Filtering Which Problems Notify

Apply a condition on the workflow to prevent every problem from triggering every notification channel. Conditions use the trigger event fields.

Filter by category

Send to on-call channel only for availability and error problems:

in(event.category, {"AVAILABILITY", "ERROR"})

Send to capacity team only for resource problems:

event.category == "RESOURCE"

Filter by affected entity

To route a problem to the team responsible for a specific entity, filter on smartscape.affected_entity.ids. This field is reliably populated for all problems and contains the entity IDs of every Smartscape entity directly affected.

matchesPhrase(smartscape.affected_entity.ids, "SERVICE-abc123def456")

Do not filter on `root_cause_entity_id` — Davis may not detect or populate a root cause for every problem (especially early in the problem lifecycle or for externally ingested events). Filtering on root_cause_entity_id will silently miss problems where the field is absent. Use smartscape.affected_entity.ids instead — it is always present when a problem has affected entities.

For team-level routing that does not depend on a specific entity ID, prefer filtering on dt.alert_group (see Scalable Multi-Team Routing):

matchesPhrase(dt.alert_group, "sev1_slack")

Combining conditions

On-call page only for high-impact availability problems in production:

event.category == "AVAILABILITY" AND event.severity == 1

---

Notification Actions

Connect the workflow to a notification connector. Dynatrace ships built-in connectors for common channels; additional channels are available via the HTTP request action.

Email

Action: Send email

Recommended fields to include in the email body:

• Problem title: {{event.name}}
• Category: {{event.category}}
• Start time: {{event.start}}
• Root cause: {{root_cause_entity_name}}
• Affected users: {{dt.davis.affected_users_count}}
• Direct link: https://<tenant>.apps.dynatrace.com/ui/problems/{{event.display_id}}

Slack

Action: Send Slack message via Slack connector

Structure the message for quick triage:

🔴 *{{event.category}} Problem Detected*
*{{event.name}}*
Root cause: {{root_cause_entity_name}}
Affected users: {{dt.davis.affected_users_count}}
Started: {{event.start}}
<https://<tenant>.apps.dynatrace.com/ui/problems/{{event.display_id}}|View in Dynatrace>

Use Slack blocks for richer formatting. Route to different channels by filtering on dt.alert_group or event.severity in the workflow condition rather than maintaining separate channel mappings in action configuration.

ServiceNow

Action: Create ServiceNow incident via ServiceNow connector

Map fields:

Add a resolve action triggered on PROBLEM_RESOLVED to automatically close or resolve the ServiceNow ticket.

Webhook / HTTP Request

Action: HTTP request

Use this for any system without a built-in connector (PagerDuty, OpsGenie, Jira, MS Teams, custom endpoints).

POST https://your-endpoint.example.com/alert
Content-Type: application/json

{
  "problemId": "{{event.display_id}}",
  "title": "{{event.name}}",
  "category": "{{event.category}}",
  "status": "{{event.status}}",
  "rootCause": "{{root_cause_entity_name}}",
  "affectedUsers": "{{dt.davis.affected_users_count}}",
  "url": "https://<tenant>.apps.dynatrace.com/ui/problems/{{event.display_id}}"
}

---

Scalable Multi-Team Routing with dt.alert_group

For environments with many teams and many detectors, maintaining per-team per-detector workflow conditions quickly becomes unmanageable. The recommended scalable alternative is to standardize routing on two fields: `dt.alert_group` for team-level routing and `event.severity` for urgency-based routing.

How it works

1. At the alert source, set the dt.alert_group field to a routing target identifier when the alert event is raised. The exact mechanism depends on the detector type:

• For Davis anomaly detectors (builtin:davis.anomaly-detectors): set

dt.alert_group as a custom property in the eventTemplate of the detector configuration

• For externally ingested events: include dt.alert_group in the event

payload sent to the Events API v2

• For OpenPipeline-sourced events: add a field enrichment rule that sets

dt.alert_group

2. Dynatrace stores the field: the dt.alert_group value is carried through to the Davis event and persisted in Grail as part of the event record. It is available as a filter field on the workflow trigger.

3. Each team's workflow filters on its own `dt.alert_group` value: the workflow condition simply checks matchesPhrase(dt.alert_group, "<target>"). Any alert event with that routing label activates the workflow; all others are ignored.

Field semantics: sets and problem-level merging

dt.alert_group is not a single string — it is a set of one or more group names. A single event can carry multiple routing targets simultaneously (e.g. ["slack_sev1", "pagerduty_oncall"]), and a workflow whose filter matches any member of that set will activate.

When Davis AI groups multiple contributing events into one problem, the dt.alert_group values from all constituent events are merged into a combined set at the problem level. The problem's dt.alert_group field is the union of every group name carried by any of its events. This means:

• A problem that merges events from two different detectors — each with its own

dt.alert_group value — will activate the workflows of both routing targets

• A team whose workflow filters on their group name receives the notification even

if their detector's event was not the root cause, only a contributing event

This merge behaviour is intentional: it ensures that every team whose detector contributed to a problem is notified, regardless of how Davis grouped or ranked the contributing events.

Example

A detector responsible for Sev 1 Slack notifications sets:

dt.alert_group = "slack_sev1"

The on-call team's workflow uses the Additional custom filter on the problem trigger:

matchesPhrase(dt.alert_group, "slack_sev1")

Every problem that contains an event carrying dt.alert_group = "slack_sev1" activates that workflow and is delivered to the team's Slack channel — without the workflow needing to know anything about which specific detector fired or which entity was affected.

Why this scales

Once a team has a workflow that filters on its dt.alert_group value, routing any new alert to that team requires only setting the correct dt.alert_group on the detector. The workflow is unchanged. This decouples detector authorship from notification routing and makes both independently maintainable.

---

Best Practices

1. Trigger on problems, not Davis events — Davis denoises multiple detector firings into one problem. Triggering on raw events bypasses denoising and floods channels.

2. Always filter by at least one condition — An unconditional workflow notifies on every problem in the environment. Start with dt.alert_group for team routing and event.severity for urgency filtering at minimum.

3. Separate workflows per team using `dt.alert_group` — One workflow per team filtering on their dt.alert_group value is easier to maintain and debug than one mega-workflow with complex branching.

4. Include the problem URL in every notification — {{event.display_id}} is not enough; include the direct deep link so recipients can navigate to the problem in one click.

5. Handle the resolution event — Always pair an open-notification workflow with a close-notification. Responders need to know when the incident is resolved, not just when it opened.

6. Test with a low-severity detector first — Create a CUSTOM category detector with a threshold that will fire in a test environment to validate the full workflow before connecting production alerts to on-call systems.

7. Never filter on `root_cause_entity_id` — Davis does not always detect or populate a root cause, especially for externally ingested events or early in the problem lifecycle. Use matchesPhrase(smartscape.affected_entity.ids, "<entity-id>") to target a specific entity, or matchesPhrase(dt.alert_group, "<group>") for team-based routing. Both fields are reliably present.

8. Use workflow execution history for debugging — Navigate to Automation → Workflows → [your workflow] → Executions to see the full payload, condition result, and action output for each triggered run.