Dt Obs Problems

Problem Analysis Skill

Analyze Dynatrace AI-detected problems including root cause identification, impact assessment, and correlation with logs and metrics.

---

Use Cases

1. Active Problem Triage

• Goal: List and prioritize currently active problems
• Trigger: "active problems", "what problems are open", "current issues", "availability issues"
• Done: Prioritized list of active problems with category, user impact, and display IDs

2. Root Cause Investigation

• Goal: Identify the root cause entity for a specific problem
• Trigger: "root cause of P-12345", "what caused this problem", "which entity is the root cause"
• Done: Root cause entity identified with affected entity list and blast radius

3. Problem Trending

• Goal: Analyze problem patterns over time to identify recurring issues
• Trigger: "recurring problems", "problem history", "problem trends last 30 days"
• Done: Trend data showing problem frequency, recurring root causes, and resolution times

---

Overview

Dynatrace automatically detects anomalies, performance degradations, and failures across your environment, creating problems that aggregate related alert, warning and info-level events and provide root cause and impact insights.

What are Problems?

Problems are automatically detected, software and infrastructure health and resilience issues that:

• Automatically correlate related alert, warning, and info-level events across services, infrastructure, frontend applications, and user sessions
• Identify root causes using causal analysis of Smartscape dependencies
• Assess business impact by tracking affected users and services
• Reduce alert noise by grouping related symptoms into single problems that share the same root cause and impact
• Track problem lifecycle from early detection through resolution

Event Kinds

The event.kind field (stable, permission) identifies the high-level event type:

event.provider (stable, permission) identifies the event source.

Problem Categories

Common event.category values:

Problem Lifecycle

Detection → ACTIVE → Under Investigation → CLOSED

• ACTIVE: Currently occurring issues requiring attention
• CLOSED: Resolved issues used for historical analysis

Essential Fields

Common Field Name Mistakes

Correct Status Values

// ✅ CORRECT: Use these status values
fetch dt.davis.problems
| filter event.status == "ACTIVE"   // Currently occurring problems
//     or event.status == "CLOSED"  // Resolved problems
// ❌ INCORRECT: event.status == "OPEN" does not exist!
| limit 1

Key Fields Reference

fetch dt.davis.problems, from:now() - 1h
| filter not(dt.davis.is_duplicate)
| fields
    event.start,                          // Problem start timestamp
    event.end,                            // Problem end timestamp (if closed)
    display_id,                           // Human-readable problem ID (P-XXXXX)
    event.name,                           // Problem title
    event.description,                    // Detailed description
    event.category,                       // Problem type
    event.status,                         // ACTIVE or CLOSED
    dt.smartscape_source.id,              // The smartscape ID for the affected resource
    dt.davis.affected_users_count,        // Number of affected users
    smartscape.affected_entity.ids,        // Array of affected entity IDs
    dt.smartscape.service,                // Affected services (may be array)
    dt.davis.root_cause_entity,           // Entity identified as root cause
    root_cause_entity_id,                 // Root cause entity ID
    root_cause_entity_name,               // Human-readable root cause name
    dt.davis.is_duplicate,                // Whether duplicate detection
    dt.davis.is_rootcause                 // Root cause vs. symptom
| limit 10

Standard Query Pattern

Always start problem queries with this foundation:

fetch dt.davis.problems, from:now() - 2h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| fields event.start, display_id, event.name, event.category
| sort event.start desc
| limit 20

Key components:

• fetch dt.davis.problems - The problems data source
• not(dt.davis.is_duplicate) - Filter out duplicate detections
• event.status == "ACTIVE" - Show only active problems
• Time range - Always specify a reasonable window

Common Query Patterns

Active Problems by Category

fetch dt.davis.problems
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| summarize problem_count = count(), by: {event.category}
| sort problem_count desc

High-Impact Active Problems (affecting many users)

fetch dt.davis.problems
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| filter dt.davis.affected_users_count > 100
| fields event.start, display_id, event.name, dt.davis.affected_users_count, event.category
| sort dt.davis.affected_users_count desc

High-Impact Active Problems (affecting many smartscape entities)

fetch dt.davis.problems
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| filter arraySize(affected_entity_ids) > 5
| fields event.start, display_id, event.name, affected_entity_ids, event.category, impacted_entity_count = arraySize(affected_entity_ids)
| sort impacted_entity_count desc

Specific Problem Details

fetch dt.davis.problems
| filter display_id == "P-XXXXXXXXXX"
| fields event.start, event.end, event.name, event.description, affected_entity_ids, dt.davis.affected_users_count, root_cause_entity_id, root_cause_entity_name

Service-Specific Problem History

fetch dt.davis.problems, from:now() - 7d
| filter not(dt.davis.is_duplicate)
| filter in(dt.smartscape.service, toSmartscapeId("SERVICE-XXXXXXXXX"))
| summarize problems = count(), by: {event.category, event.status}

Root Cause Analysis Patterns

Basic Root Cause Query

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| fields
    display_id,
    event.name,
    event.description,
    root_cause_entity_id,
    root_cause_entity_name,
    smartscape.affected_entity.ids

Root Cause by Entity Type

Identify which entity types most frequently cause problems:

fetch dt.davis.problems, from:now() - 7d
| filter not(dt.davis.is_duplicate)
| filter isNotNull(root_cause_entity_id)
| summarize problem_count = count(), by:{root_cause_entity_name}
| sort problem_count desc
| limit 20

Affected entity is an AWS resource

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| filter matchesPhrase(arrayToString(smartscape.affected_entity.types, delimiter:","), "AWS_")

Infrastructure Root Cause with Service Impact

fetch dt.davis.problems, from:now() - 30m
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| filter matchesPhrase(root_cause_entity_id, "HOST-")
| filter isNotNull(dt.smartscape.service)
| fields display_id, event.name, root_cause_entity_name, dt.smartscape.service

Problem Blast Radius

Calculate entity impact per root cause:

fetch dt.davis.problems, from:now() - 7d
| filter not(dt.davis.is_duplicate)
| filter isNotNull(root_cause_entity_id)
| fieldsAdd affected_count = arraySize(smartscape.affected_entity.ids)
| summarize
    avg_affected = avg(affected_count),
    max_affected = max(affected_count),
    problem_count = count(),
    by:{root_cause_entity_name}
| sort avg_affected desc

Recurring Root Causes

Identify entities repeatedly causing problems:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate)
| filter isNotNull(root_cause_entity_id)
| summarize
    problem_count = count(),
    first_occurrence = min(event.start),
    last_occurrence = max(event.start),
    by:{root_cause_entity_id, root_cause_entity_name}
| filter problem_count > 3
| sort problem_count desc

Cause Category vs. Root Cause Entity

These are different questions — pick the right approach:

• "What causes problems?" / "most common cause" → Summarize by event.category

(SLOWDOWN, ERROR, RESOURCE, AVAILABILITY, CUSTOM). Explain what triggers each category.

• "Which entity causes problems?" / "root cause entity" → Group by

root_cause_entity_name. Lists specific services, hosts, or apps.

Cause category breakdown (use when asked about common causes, patterns, or types):

fetch dt.davis.problems, from:now() - 30d
| filter not(dt.davis.is_duplicate)
| summarize problem_count = count(), by: {event.category}
| sort problem_count desc

Then for each category, explain what triggers it using the Problem Categories table and cite specific entities from the tenant data as examples.

Problem Trending and Pattern Analysis

Track problem trends over time, identify recurring issues, and analyze resolution performance.

Primary Files:

• references/problem-trending.md - Timeseries analysis and pattern detection

Common Use Cases:

• Active problems over time with makeTimeseries
• Problem creation rate by category
• Recurring problem detection by schedule
• Resolution time trends and P95 duration analysis

Key Techniques:

• `makeTimeseries` vs `bin()`: Choose the right approach for lifecycle spans vs discrete events
• NULL handling: Use coalesce(event.end, now()) for active problems
• Peak hours analysis: Identify when problems occur most frequently
• Impact trending: Track user impact changes over time

See references/problem-trending.md for complete query patterns and best practices.

Cross-Domain Problem Queries

Problems Associated with Kubernetes Clusters

Use affected_entity_ids or dt.smartscape_source.id to find problems related to Kubernetes:

fetch dt.davis.problems, from:now() - 7d
| filter not(dt.davis.is_duplicate)
| filter matchesPhrase(dt.smartscape_source.id, "KUBERNETES_CLUSTER")
    OR matchesPhrase(dt.smartscape_source.id, "K8S_")
| fields event.start, display_id, event.name, event.category, event.status,
    dt.smartscape_source.id, affected_entity_ids
| sort event.start desc

Alternative: expand affected entities and filter for K8s entity types:

fetch dt.davis.problems, from:now() - 7d
| filter not(dt.davis.is_duplicate)
| expand entity_id = affected_entity_ids
| filter matchesPhrase(entity_id, "KUBERNETES_CLUSTER")
    OR matchesPhrase(entity_id, "K8S_")
| fields event.start, display_id, event.name, event.category, entity_id
| sort event.start desc

Simple Problem Listing

List all problems from the last 24 hours (common request):

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate)
| fields event.start, event.end, display_id, event.name, event.category, event.status
| sort event.start desc

Response Construction

Problem Cause Summaries

When summarizing problem causes, categories, or patterns, provide a comprehensive breakdown across all standard categories present in the data: AVAILABILITY, ERROR, SLOWDOWN, RESOURCE, and CUSTOM. For each category found:

1. Category name and count of problems 2. What triggers it — brief explanation (e.g., RESOURCE = CPU/memory/disk threshold exceeded; AVAILABILITY = service or entity became unreachable) 3. Specific examples from the tenant's data (affected entity names, problem IDs)

Do not stop after the first two categories — users expect the full picture. Reference the Problem Categories table above for trigger descriptions.

Analysis Results

When presenting query results:

• Include entity names (not just IDs) — but choose the efficient method:
• Few entities (< 5): get-entity-name calls are fine
• Many entities: Use query-problems tool which returns names directly, or

include root_cause_entity_name / entityName() in the DQL query to resolve names inline. Avoid calling get-entity-name in a loop for 10+ entities — this can exhaust the tool call limit and return no answer at all.

• Provide actionable recommendations aligned to the identified causes
• Organize by frequency or impact for easy prioritization

Best Practices

Essential Rules

1. Always filter duplicates: Use not(dt.davis.is_duplicate) to avoid counting the same problem multiple times 2. Use correct status values: "ACTIVE" or "CLOSED", never "OPEN" 3. Specify time ranges: Always include time bounds to optimize performance 4. Include display_id: Essential for problem identification and linking 5. Test incrementally: Add one filter or field at a time when building queries 6. Filter early: Apply not(dt.davis.is_duplicate) immediately after fetch

Query Development

• Start simple: Begin with basic filtering, then add complexity
• Test fields first: Run with | limit 1 to verify field names exist
• Use meaningful time ranges: Too broad wastes resources, too narrow misses data
• Document problem IDs: Always capture and store display_id for reference

Root Cause Verification

• Always filter isNotNull(root_cause_entity_id) when required
• Cross-reference events using dt.davis.event_ids
• Consider time delays: root cause may appear in logs minutes before problem

Time Range Guidelines

// ✅ GOOD - Specific time range
fetch dt.davis.problems, from:now() - 4h

// ❌ BAD - Scans all historical data
fetch dt.davis.problems

Absolute Timeframes Require Double Quotes

When using absolute ISO 8601 timestamps for from and to in DQL queries, always wrap them in double quotes. Unquoted timestamps are a syntax error.

// ✅ CORRECT - absolute timestamps quoted
fetch dt.davis.problems, from: "2026-05-18T22:50:00Z", to: "2026-05-18T23:35:00Z"
| filter not(dt.davis.is_duplicate)
| fields event.start, display_id, event.name, event.category, event.status
| sort event.start desc

Troubleshooting

When to Load References

Load problem-trending.md when:

• Analyzing problem frequency over time
• Detecting recurring problems on a schedule
• Calculating resolution time trends and P95 durations
• Comparing problem creation rates by category

Load problem-correlation.md when:

• Correlating problems with logs or other telemetry
• Investigating events that preceded a problem
• Linking problems to deployment or config changes

Load impact-analysis.md when:

• Assessing business impact (affected users, services)
• Calculating blast radius for a root cause entity
• Prioritizing problems by technical and user impact

References

• problem-trending.md — Problem trending and timeseries analysis patterns
• problem-correlation.md — Correlating problems with logs and other telemetry
• impact-analysis.md — Business and technical impact assessment
• problem-merging.md — When and why DAVIS merges events into problems

Related Skills

• dt-dql-essentials - Core DQL syntax and query structure for problem queries
• dt-obs-logs - Correlate problems with application and infrastructure logs
• dt-obs-tracing - Investigate problems through distributed trace analysis

Problem Impact Analysis

Assess business and technical impact of DAVIS problems by analyzing affected users, entities, service dependencies, and problem scope to prioritize incident response.

Overview

Impact analysis helps answer critical questions:

• How many users are affected? (dt.davis.affected_users_count)
• Which services are impacted? (dt.smartscape.service, smartscape.affected_entity.ids)
• What is the blast radius? (count of affected entities)
• At what system layer? (dt.davis.impact_level)
• How critical is this? (combination of users + services + category)

Key Impact Fields

User Impact Analysis

High-Impact Problems

Problems affecting many users:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| filter dt.davis.affected_users_count > 0
| fields
    display_id,
    event.name,
    event.category,
    dt.davis.affected_users_count,
    dt.davis.impact_level
| sort dt.davis.affected_users_count desc

User Impact Threshold Categorization

Classify problems by user impact severity:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| filter isNotNull(dt.davis.affected_users_count)
| fieldsAdd impact_severity = if(dt.davis.affected_users_count > 1000, "CRITICAL",
    else: if(dt.davis.affected_users_count > 100, "HIGH",
    else: if(dt.davis.affected_users_count > 10, "MEDIUM", else: "LOW")))
| summarize problem_count = count(), by:{impact_severity, event.category}
| sort impact_severity asc

User Impact Trending

Track user impact over time:

fetch dt.davis.problems, from:now() - 14d
| filter not(dt.davis.is_duplicate)
| filter isNotNull(dt.davis.affected_users_count)
| summarize
    total_users_affected = sum(dt.davis.affected_users_count),
    avg_users_per_problem = avg(dt.davis.affected_users_count),
    max_users_in_problem = max(dt.davis.affected_users_count),
    by:{time_bucket = bin(event.start, 1h)}
| sort time_bucket asc

Active Problem Impact Dashboard

Real-time impact view:

fetch dt.davis.problems, from:now() - 2h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| summarize
    total_problems = count(),
    total_affected_users = sum(dt.davis.affected_users_count),
    total_affected_entities = sum(arraySize(smartscape.affected_entity.ids)),
    critical_problems = countIf(dt.davis.affected_users_count > 100),
    availability_issues = countIf(event.category == "AVAILABILITY"),
    error_issues = countIf(event.category == "ERROR")

Service Impact Analysis

Multi-Service Impact

Problems affecting multiple services:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| fieldsAdd service_count = arraySize(dt.smartscape.service)
| filter service_count > 1
| fields display_id, event.name, service_count, dt.smartscape.service
| sort service_count desc

Business-Critical Service Problems

Focus on specific critical services:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate)
| expand dt.smartscape.service
| filter dt.smartscape.service == toSmartscapeId("SERVICE-CRITICAL-APP")
| fields
    event.start,
    display_id,
    event.name,
    event.status,
    event.category,
    dt.davis.affected_users_count,
    dt.smartscape.service
| sort event.start desc

Service Problem History

Identify services with recurring problems:

fetch dt.davis.problems, from:now() - 30d
| filter not(dt.davis.is_duplicate)
| expand dt.smartscape.service
| summarize
    problem_count = count(),
    unique_categories = collectDistinct(event.category),
    total_users_affected = sum(dt.davis.affected_users_count),
    avg_users_per_problem = avg(dt.davis.affected_users_count),
    by:{dt.smartscape.service}
| filter problem_count > 5
| sort problem_count desc

Service Dependency Impact

When one service affects many others:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| filter isNotNull(root_cause_entity_id)
| fieldsAdd downstream_services = arraySize(dt.smartscape.service)
| filter downstream_services > 1
| fields
    display_id,
    event.name,
    root_cause_entity_name,
    downstream_services,
    dt.smartscape.service
| sort downstream_services desc

Entity Impact Analysis

Problem Blast Radius

Calculate how many entities each problem affects:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| fieldsAdd affected_count = arraySize(smartscape.affected_entity.ids)
| fields display_id, event.name, event.category, affected_count
| sort affected_count desc

Scope Categorization

Categorize by breadth of impact:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate)
| fieldsAdd affected_count = arraySize(smartscape.affected_entity.ids)
| fieldsAdd scope = if(affected_count == 1, "Single",
    else: if(affected_count <= 5, "Limited",
    else: if(affected_count <= 20, "Moderate", else: "Wide")))
| summarize problem_count = count(), by:{scope, event.category}
| sort scope asc

Entity Type Impact

Which entity types are most frequently affected:

fetch dt.davis.problems, from:now() - 7d
| filter not(dt.davis.is_duplicate)
| expand smartscape.affected_entity.types
| summarize
    problem_count = count(),
    avg_users_affected = avg(dt.davis.affected_users_count),
    by:{smartscape.affected_entity.types}
| sort problem_count desc

Cross-Entity Problems

Entities appearing in multiple problems:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate)
| expand smartscape.affected_entity.ids
| summarize
    problem_count = countDistinct(display_id),
    categories = collectDistinct(event.category),
    total_user_impact = sum(dt.davis.affected_users_count),
    by:{smartscape.affected_entity.ids}
| filter problem_count > 1
| sort problem_count desc

Impact Level Analysis

Distribution by Impact Level

Analyze problems by system layer:

fetch dt.davis.problems, from:now() - 7d
| filter not(dt.davis.is_duplicate)
| summarize
    problem_count = count(),
    avg_affected_users = avg(dt.davis.affected_users_count),
    total_affected_users = sum(dt.davis.affected_users_count),
    by:{dt.davis.impact_level, event.category}
| sort problem_count desc

Application-Level Problems

Focus on user-facing issues:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| filter dt.davis.impact_level == "Application"
| fields
    display_id,
    event.name,
    event.category,
    dt.davis.affected_users_count,
    dt.smartscape.service
| sort dt.davis.affected_users_count desc

Combined Impact Scoring

Priority Score Calculation

Create a composite priority score:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| fieldsAdd
    user_score = coalesce(dt.davis.affected_users_count, 0) / 10,
    entity_score = arraySize(smartscape.affected_entity.ids) * 5,
    category_score = if(event.category == "AVAILABILITY", 100,
        else: if(event.category == "ERROR", 50,
        else: if(event.category == "SLOWDOWN", 25, else: 10))),
    priority_score = user_score + entity_score + category_score
| fields display_id, event.name, priority_score, event.category, dt.davis.affected_users_count
| sort priority_score desc
| limit 20

Critical Problem Identification

Multi-criteria critical problem detection:

fetch dt.davis.problems, from:now() - 4h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| fieldsAdd
    is_high_user_impact = dt.davis.affected_users_count > 100,
    is_availability_issue = event.category == "AVAILABILITY",
    is_wide_scope = arraySize(smartscape.affected_entity.ids) > 10,
    is_critical = is_high_user_impact or is_availability_issue or is_wide_scope
| filter is_critical
| fields
    display_id,
    event.name,
    dt.davis.affected_users_count,
    event.category,
    affected_entity_count = arraySize(smartscape.affected_entity.ids)
| sort dt.davis.affected_users_count desc

Time-Based Impact Analysis

Problem Duration vs Impact

Analyze if high-impact problems take longer to resolve:

fetch dt.davis.problems, from:now() - 30d
| filter not(dt.davis.is_duplicate) and event.status == "CLOSED"
| filter isNotNull(dt.davis.affected_users_count)
| fieldsAdd
    duration_minutes = (event.end - event.start) / 60000000000,
    impact_category = if(dt.davis.affected_users_count > 100, "High",
        else: if(dt.davis.affected_users_count > 10, "Medium", else: "Low"))
| summarize
    avg_duration = avg(duration_minutes),
    p95_duration = percentile(duration_minutes, 95),
    problem_count = count(),
    by:{impact_category}

Peak Impact Hours

When do high-impact problems occur:

fetch dt.davis.problems, from:now() - 7d
| filter not(dt.davis.is_duplicate)
| filter dt.davis.affected_users_count > 50
| fieldsAdd hour_of_day = formatTimestamp(event.start, format:"HH")
| summarize
    problem_count = count(),
    avg_users_affected = avg(dt.davis.affected_users_count),
    by:{hour_of_day}
| sort hour_of_day asc

Best Practices

Prioritization Strategy

1. User-facing first: affected_users_count > 0 = highest priority 2. Impact level: Application level indicates end-user visibility 3. Scope: More affected entities = wider blast radius 4. Category: AVAILABILITY typically more urgent than SLOWDOWN 5. Root cause: Infrastructure problems may have cascading effects

Query Optimization

// ✅ GOOD - Filter early, calculate later
fetch dt.davis.problems, from:now() - 4h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| filter dt.davis.affected_users_count > 50
| fieldsAdd priority_score = dt.davis.affected_users_count * 10

// ❌ BAD - Calculating before filtering
fetch dt.davis.problems, from:now() - 4h
| fieldsAdd priority_score = dt.davis.affected_users_count * 10
| filter not(dt.davis.is_duplicate)

Handle Null Values

// ❌ WRONG - Nulls sort first, skew results
fetch dt.davis.problems
| sort dt.davis.affected_users_count desc

// ✅ CORRECT - Filter nulls first
fetch dt.davis.problems
| filter isNotNull(dt.davis.affected_users_count)
| sort dt.davis.affected_users_count desc

// ✅ CORRECT - Use coalesce for calculations
fetch dt.davis.problems
| fieldsAdd user_score = coalesce(dt.davis.affected_users_count, 0) * 10

Array Field Handling

// ✅ CORRECT - Check array size
fetch dt.davis.problems
| fieldsAdd service_count = arraySize(dt.smartscape.service)
| filter service_count > 0

// ✅ CORRECT - Check array contains
fetch dt.davis.problems
| filter in(toSmartscapeId("SERVICE-ABC"), dt.smartscape.service)

// ✅ CORRECT - Expand array for aggregation
fetch dt.davis.problems
| expand dt.smartscape.service
| summarize count(), by:{dt.smartscape.service}

Investigation Workflow

Step 1: Identify High-Impact Active Problems

fetch dt.davis.problems, from:now() - 2h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| filter dt.davis.affected_users_count > 10 or arraySize(smartscape.affected_entity.ids) > 5
| fields display_id, event.name, event.category, dt.davis.affected_users_count
| sort dt.davis.affected_users_count desc

Step 2: Assess Service Impact

fetch dt.davis.problems
| filter display_id == "P-XXXXXXXXXX"
| fields display_id, dt.smartscape.service, smartscape.affected_entity.ids, dt.davis.impact_level

Step 3: Calculate Blast Radius

fetch dt.davis.problems
| filter display_id == "P-XXXXXXXXXX"
| fieldsAdd
    total_entities = arraySize(smartscape.affected_entity.ids),
    total_services = arraySize(dt.smartscape.service)
| fields display_id, total_entities, total_services, smartscape.affected_entity.ids

Step 4: Correlate with Business Context

• Map affected services to business functions
• Check if critical services are impacted
• Consider time of day (business hours vs off-hours)
• Assess compliance/SLA implications

Common Pitfalls

Not Filtering Duplicates

// ❌ WRONG - Counts duplicate problems
fetch dt.davis.problems, from:now() - 24h
| summarize total_users = sum(dt.davis.affected_users_count)

// ✅ CORRECT
fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate)
| summarize total_users = sum(dt.davis.affected_users_count)

Ignoring Active vs Closed Status

// ❌ WRONG - Includes resolved problems in "current impact"
fetch dt.davis.problems, from:now() - 24h
| summarize current_impact = sum(dt.davis.affected_users_count)

// ✅ CORRECT
fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| summarize current_impact = sum(dt.davis.affected_users_count)

Not Considering Time Context

// ❌ BAD - Too broad, includes old problems
fetch dt.davis.problems
| filter event.status == "ACTIVE"

// ✅ GOOD - Recent problems only
fetch dt.davis.problems, from:now() - 4h
| filter event.status == "ACTIVE"

Related Documentation

• problem-correlation.md: Correlating problems with logs and telemetry
• ../SKILL.md: Core problem analysis concepts

Problem Correlation

Correlate DAVIS problems with logs, events, and other telemetry from affected entities to identify root causes through error messages, stack traces, and timeline analysis.

Overview

When DAVIS detects a problem, use smartscape.affected_entity.ids to query logs and telemetry from impacted entities. This correlation helps identify the specific error conditions, configuration changes, or resource constraints that triggered the problem.

Key Correlation Fields

Problem-to-Logs Correlation

Basic Pattern

fetch logs
| filter dt.smartscape_source.id in [
    fetch dt.davis.problems
    | filter display_id == "P-12345678"
    | fields smartscape.affected_entity.ids
  ] or dt.source_entity in [
    fetch dt.davis.problems
    | filter display_id == "P-12345678"
    | fields affected_entity_ids
  ]
| sort timestamp desc
| limit 100

Problem-to-Alert-Events Correlation

Basic Pattern

fetch dt.davis.events
| filter event.id in [ 
  fetch dt.davis.problems
  | filter display_id == "P-12345678"
  | fieldsKeep dt.davis.event_ids
  ]

Active Problems with Recent Logs

Find logs from entities affected by currently active problems:

fetch logs, from:now() - 1h
| filter dt.smartscape_source.id in [
    fetch dt.davis.problems, from:now() - 1h
    | filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
    | fields smartscape.affected_entity.ids
  ] or dt.source_entity in [
    fetch dt.davis.problems, from:now() - 1h
    | filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
    | fields affected_entity_ids
  ]
| filter in(loglevel, {"ERROR", "WARN"})
| fields timestamp, dt.source_entity, loglevel, content
| limit 200

Problem-Specific Error Analysis

Get error logs from a specific problem:

fetch logs, from:now() - 2h
| filter loglevel == "ERROR"
| filter dt.smartscape_source.id in [
   fetch dt.davis.problems
    | filter display_id == "P-12345678"
    | fields smartscape.affected_entity.ids
  ] or dt.source_entity in [
    fetch dt.davis.problems
    | filter display_id == "P-12345678"
    | fields affected_entity_ids
]
| sort timestamp desc
| limit 100

Log Pattern Detection

Identify common error patterns across problem-affected entities:

fetch logs, from:now() - 4h
| filter dt.smartscape_source.id in [
    fetch dt.davis.problems, from:now() - 4h
    | filter not(dt.davis.is_duplicate)
    | filter event.category == "ERROR"
    | fields smartscape.affected_entity.ids 
]
    or dt.source_entity in [
        fetch dt.davis.problems, from:now() - 4h
        | filter not(dt.davis.is_duplicate)
        | filter event.category == "ERROR"
        | fields affected_entity_ids
]
| filter loglevel == "ERROR"
| summarize error_count=count(), by:{content}
| sort error_count desc
| limit 20

Timeline Analysis

Logs Relative to Problem Occurrence

View logs in temporal context around problem detection:

fetch dt.davis.problems
| filter display_id == "P-12345678"
| fields problem_start=event.start, smartscape.affected_entity.ids, timestamp
| join [
    fetch logs
    | filter in(loglevel, {"ERROR", "WARN"})
    | fields content, timestamp, dt.source_entity, loglevel
    | limit 100
], on:{left[smartscape.affected_entity.ids] == right[dt.source_entity]}
| fieldsAdd time_offset = timestamp - problem_start
| sort timestamp asc
| fields timestamp, time_offset, right.loglevel, right.content

Before and After Problem Start

Query logs with expanded time window to see precursor events:

// Get problem start time
fetch dt.davis.problems
| filter display_id == "P-12345678"
| fields problem_start = event.start, problem_entities = smartscape.affected_entity.ids

// Query logs from 10 minutes before to 10 minutes after
fetch logs, from:now() - 1h
| filter dt.smartscape_source.id in [
    fetch dt.davis.problems
    | filter display_id == "P-12345678"
    | fields smartscape.affected_entity.ids
]
    or dt.source_entity in [
        fetch dt.davis.problems
        | filter display_id == "P-12345678"
        | fields affected_entity_ids
]
| filter timestamp >= (problem_start - 10m) and timestamp <= (problem_start + 10m)
| sort timestamp asc

Multiple Problems Correlation

Common Log Patterns Across Problems

Find shared error messages affecting multiple problems:

fetch logs, from:now() - 2h
| filter loglevel == "ERROR"
| filter dt.smartscape_source.id in [
    fetch dt.davis.problems, from:now() - 2h
    | filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
    | fields smartscape.affected_entity.ids
]
    or dt.source_entity in [
        fetch dt.davis.problems, from:now() - 2h
        | filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
        | fields affected_entity_ids
]
| summarize problems_affected = countDistinct(dt.source_entity), by:{content}
| filter problems_affected > 1
| sort problems_affected desc

Cross-Problem Entity Analysis

Identify entities appearing in multiple problems:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate)
| expand smartscape.affected_entity.ids
| summarize 
    problem_count = countDistinct(display_id),
    categories = collectDistinct(event.category),
    by:{smartscape.affected_entity.ids}
| filter problem_count > 1
| sort problem_count desc

Problem-to-Events Correlation

Underlying Davis Events

Retrieve Davis events contributing to the problem:

fetch dt.davis.events
| filter event.id in [
    fetch dt.davis.problems
    | filter display_id == "P-12345678"
    | fields dt.davis.event_ids
]
| fields event.start, event.name, event.description, dt.source_entity
| sort event.start asc

Deployment Correlation

Check if problems correlate with recent deployments:

fetch dt.davis.problems, from:now() - 2h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| fields problem_start = event.start, display_id, event.name, smartscape.affected_entity.ids, timestamp
| join [
    fetch events
    | filter event.type == "DEPLOYMENT"
], on:{left[smartscape.affected_entity.ids] == right[dt.smartscape.service]}
| fieldsAdd time_since_deployment = problem_start - timestamp
| filter time_since_deployment > 0m and time_since_deployment < 30m
| fields display_id, event.name, time_since_deployment

K8S or Technology Correlation

Check if active problems correlate with K8S deployment:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| fieldsAdd e = arraytoString(smartscape.affected_entity.ids, delimiter:",")
| filter matchesPhrase(arraytoString(smartscape.affected_entity.ids, delimiter:","), "K8S_DEPLOYMENT-")

Check if active problems correlate with AWS S3 buckets:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| fieldsAdd e = arraytoString(smartscape.affected_entity.ids, delimiter:",")
| filter matchesPhrase(arraytoString(smartscape.affected_entity.ids, delimiter:","), "AWS_S3_BUCKET-")

Root Cause Correlation

Root Cause Entity Logs

Focus on logs from the identified root cause entity:

fetch logs, from:now() - 1h
    | filter dt.smartscape_source.id in [
        fetch dt.davis.problems, from:now() - 1h
        | filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
        | filter isNotNull(root_cause_entity_id)
        | fields root_cause_entity_id
    ]
    or dt.source_entity in [
        fetch dt.davis.problems, from:now() - 1h
        | filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
        | filter isNotNull(root_cause_entity_id)
        | fields root_cause_entity_id
]
| filter loglevel == "ERROR"
| sort timestamp desc
| limit 50

Best Practices

Absolute Timeframes Require Double Quotes

When using absolute ISO 8601 timestamps for from and to in any DQL query, always wrap them in double quotes. This applies to all data sources including fetch logs and fetch dt.davis.problems.

// ✅ CORRECT - absolute timestamps quoted
fetch logs, from: "2026-05-18T22:50:00Z", to: "2026-05-18T23:35:00Z"
| filter dt.smartscape_source.id in [
    fetch dt.davis.problems, from: "2026-05-18T22:00:00Z", to: "2026-05-18T23:35:00Z"
    | filter display_id == "P-12345678"
    | fields smartscape.affected_entity.ids
  ] or dt.source_entity in [
    fetch dt.davis.problems, from: "2026-05-18T22:00:00Z", to: "2026-05-18T23:35:00Z"
    | filter display_id == "P-12345678"
    | fields affected_entity_ids
  ]
| filter loglevel == "ERROR"
| fields timestamp, dt.source_entity, loglevel, content
| sort timestamp desc
| limit 100

Query Optimization

1. Match time ranges: Use same time window for problems and logs

   // ✅ CORRECT - Time ranges aligned
   fetch logs, from:now() - 1h
   | filter dt.smartscape_source.id in [
       fetch dt.davis.problems, from:now() - 1h
       | fields smartscape.affected_entity.ids
   ] or dt.source_entity in [
       fetch dt.davis.problems, from:now() - 1h
       | fields affected_entity_ids
   ]
   | limit 100

2. Filter early: Apply loglevel filters before joins

   fetch logs, from:now() - 1h
   | filter loglevel == "ERROR"  // Filter before correlation
    | filter dt.smartscape_source.id in [...] or dt.source_entity in [...]

3. Limit results: Always use limit to prevent excessive data

   fetch logs
    | filter dt.smartscape_source.id in [...] or dt.source_entity in [...]
   | limit 200  // Reasonable limit

Investigation Workflow

1. Identify problem: Get display_id and smartscape.affected_entity.ids 2. Expand time window: Query logs from before problem start to after resolution 3. Filter by severity: Start with ERROR, expand to WARN if needed 4. Look for patterns: Use summarize to find recurring messages 5. Focus on root cause: If identified, query logs from root_cause_entity_id 6. Check timeline: Use joins to see temporal relationships 7. Correlate with events: Check for deployments, configuration changes

Common Pitfalls

// ❌ WRONG - Missing time range alignment
fetch logs, from:now() - 1h
| filter dt.smartscape_source.id in [
    fetch dt.davis.problems  // No time range
    | fields smartscape.affected_entity.ids
]
    or dt.source_entity in [
        fetch dt.davis.problems  // No time range
        | fields affected_entity_ids
]

// ❌ WRONG - Not filtering duplicates
fetch dt.davis.problems
| fields smartscape.affected_entity.ids  // Includes duplicates

// ✅ CORRECT - Time ranges aligned and duplicates filtered
fetch logs, from:now() - 1h
| filter dt.smartscape_source.id in [
    fetch dt.davis.problems, from:now() - 1h
    | filter not(dt.davis.is_duplicate)
    | fields smartscape.affected_entity.ids
]
    or dt.source_entity in [
        fetch dt.davis.problems, from:now() - 1h
        | filter not(dt.davis.is_duplicate)
        | fields affected_entity_ids
]

Handle Edge Cases

1. Active problems have NULL event.end: Use coalesce(event.end, now()) 2. Some problems have no root_cause_entity_id: Use isNotNull() check 3. smartscape.affected_entity.ids is an array: Use in operator for filtering 4. Log timestamps may be slightly off: Expand time window by 5-10 minutes

Advanced Patterns

Problem Frequency vs Log Volume

Check if log volume spikes correlate with problem frequency:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate)
| summarize problem_count = count(), by:{time_bucket = bin(event.start, 1h)}
| join [
    fetch logs, from:now() - 24h
    | filter loglevel == "ERROR"
    | summarize log_count = count(), by:{time_bucket = bin(timestamp, 1h)}
], on:{left[time_bucket] == right[time_bucket]}
| fieldsAdd log_count = right.log_count
| fields time_bucket, problem_count, log_count
| sort time_bucket asc

Related Documentation

• impact-analysis.md: Assessing business and technical impact
• ../SKILL.md: Core problem analysis concepts

Problem Merging Guidance

Understand when individual alarm events (dt.davis.event) are merged into one problem and why they may stay separate.

Overview

Dynatrace can merge multiple events into one problem when they look like the same incident context.

Merging decisions are mainly driven by:

• Time overlap of active event windows
• Shared source entity context
• Topology relationship (for example, vertical stack relationships, horizontal trace dependency relationship)
• Merge policy flags on the events

Events Usually Merge When

1. Active time windows overlap

• The active duration of both events overlaps from event.start to event.end.
• In practice, event start times should be close (roughly within about 3 minutes).

2. Source entity is the same

• Preferred field: dt.smartscape_source.id is the same.
• Backward-compatible/older field: dt.source_entity is the same.

3. Events are in the same vertical deployment stack

• Example: a process runs on a host.
• A CPU-high event on the process and a CPU-high event on the host can merge because both describe the same stack context.

Events Usually Do Not Merge When

1. Time windows do not overlap

• If active intervals are separate (no overlap), events typically remain in different problems.

2. Start times are too far apart

• If starts are not close (outside the rough ~3 minute proximity), events are less likely to merge.

3. Merging is explicitly disabled

• If an event has dt.davis.is_merging_allowed == false, Davis does not merge that event into other problems.

Field Checklist

Use these fields when investigating merge behavior:

• event.start
• event.end
• dt.smartscape_source.id
• dt.source_entity
• dt.davis.is_merging_allowed

Investigation Tips

1. Compare event intervals first

• Check if active windows overlap.

2. Compare entity identity next

• Check both dt.smartscape_source.id and dt.source_entity.

3. Validate topology relationship

• If entities differ, check whether they are still part of the same vertical stack (for example process on host).
• If entities differ, check whether they are still part of the same horizontal stack (for example service calls service or frontend calls service).

4. Check merge policy flags

• Confirm whether dt.davis.is_merging_allowed is false on any involved event.

Problem Trending and Timeseries Analysis

Analyze problem patterns over time to identify trends, detect recurring issues, and understand frequency changes.

Key Concepts

`makeTimeseries` vs `bin()`:

• makeTimeseries: Handles problem lifecycle spans (start to end)
• bin(): Groups discrete timestamps into intervals

Problem Lifecycle:

• event.start: When problem began
• event.end: When resolved (NULL if active)
• Use spread: timeframe() to show problems across duration

Active Problems Over Time

Chart currently active problem count, typically visualized as red bar chart:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| makeTimeseries
    active_problems = count(),
    interval: 1h,
    spread: timeframe(from:event.start, to:coalesce(event.end, now()))

Problem Creation Rate

Track when new problems are detected:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate)
| summarize new_problems = count(), by:{start_bin = bin(event.start, 1h)}
| sort start_bin asc

Category Trends

Track problem trends by category:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate)
| summarize problem_count = count(), by:{event.category, start_day = bin(event.start, 1d)}
| sort start_day asc

Service-Specific Trending

Monitor problem frequency for a service:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate)
| filter in(dt.smartscape.service, toSmartscapeId("SERVICE-00E66996F1555897"))
| summarize by:{start_day = bin(event.start, 24h)}, {
    total_problems = count(),
    active = countIf(event.status == "ACTIVE"),
    closed = countIf(event.status == "CLOSED")
}
| sort start_day asc

Peak Problem Hours

When problems occur most frequently:

fetch dt.davis.problems, from:now() - 24h
| filter not(dt.davis.is_duplicate)
| fieldsAdd hour_of_day = formatTimestamp(event.start, format:"HH")
| summarize problem_count = count(), by:{hour_of_day, event.category}
| sort hour_of_day asc

Problem Duration Trending

Analyze if problems take longer to resolve:

fetch dt.davis.problems, from:now() - 30d
| filter not(dt.davis.is_duplicate) and event.status == "CLOSED"
| fieldsAdd duration_minutes = (event.end - event.start) / 60000000000
| summarize by:{start_day = bin(event.start, 24h)}, {
    avg_duration = avg(duration_minutes),
    p95_duration = percentile(duration_minutes, 95)
}
| sort start_day asc

Recurring Problem Detection

Find problems repeating on similar schedules:

fetch dt.davis.problems, from:now() - 7d
| filter not(dt.davis.is_duplicate)
| fieldsAdd
    day_of_week = formatTimestamp(event.start, format:"EEEE"),
    hour = formatTimestamp(event.start, format:"HH")
| summarize problem_count = count(), by:{day_of_week, hour}
| filter problem_count > 3
| sort problem_count desc

Impact Trending

Track user impact over time:

fetch dt.davis.problems, from:now() - 7d
| filter not(dt.davis.is_duplicate)
| filter isNotNull(dt.davis.affected_users_count)
| summarize
    total_users_affected = sum(dt.davis.affected_users_count),
    avg_users_per_problem = avg(dt.davis.affected_users_count),
    by:{bin(event.start, 1h)}

Best Practices

makeTimeseries Guidelines

// ✅ GOOD - Show durations accurately
fetch dt.davis.problems, from:now() - 24h
| filter event.status == "ACTIVE"
| makeTimeseries
    count(),
    interval: 15m,
    spread: timeframe(from:event.start, to:coalesce(event.end, now()))

Common Pitfalls

// ❌ WRONG - Not filtering duplicates
fetch dt.davis.problems, from:now() - 7d
| summarize count(), by:{bin(event.start, 1h)}

// ✅ CORRECT - Always filter duplicates
fetch dt.davis.problems, from:now() - 7d
| filter not(dt.davis.is_duplicate)
| summarize count(), by:{bin(event.start, 1h)}

Handle Null Timestamps

Active problems have NULL event.end - use coalesce(event.end, now()) in spread.