Dt Obs Services

Application Services Skill

Monitor application service performance, health, and runtime-specific metrics using DQL.

---

Core Capabilities

1. Service Performance (RED Metrics)

Monitor service Rate, Errors, Duration using metrics-based timeseries queries.

Key Metrics:

• dt.service.request.response_time - Response time (microseconds)
• dt.service.request.count - Request count
• dt.service.request.failure_count - Failed request count

Common Use Cases:

• Response time monitoring (avg, p50, p95, p99)
• Error rate tracking and spike detection
• Traffic analysis (throughput, peaks, growth)
• Performance degradation detection
• Multi-cluster comparison

Quick Example:

timeseries {
  p95 = percentile(dt.service.request.response_time, 95),
  total_requests = sum(dt.service.request.count),
  failures = sum(dt.service.request.failure_count)
}, by: {dt.service.name}
| fieldsAdd p95_ms = p95[] / 1000, error_rate_pct = (failures[] * 100.0) / total_requests[]

→ For detailed queries: See references/service-metrics.md

2. Advanced Service Analysis

Span-based queries for complex scenarios requiring flexible filtering and custom aggregations.

Use Cases:

• SLA compliance tracking with custom thresholds
• Service health scoring (multi-dimensional)
• Operation/endpoint-level performance analysis
• Custom error classification
• Failure pattern detection with error details

Quick Example:

fetch spans, from: now() - 1h | filter request.is_root_span == true
| fieldsAdd meets_sla = if(request.is_failed == false AND duration < 3s, 1, else: 0)
| summarize total = count(), sla_compliant = sum(meets_sla), by: {dt.service.name}
| fieldsAdd sla_compliance_pct = (sla_compliant * 100.0) / total

→ For detailed queries: See references/service-metrics.md

3. Service Messaging Metrics

Monitor message-based service communication (queues, topics).

Key Metrics:

• dt.service.messaging.publish.count - Messages sent to queues or topics
• dt.service.messaging.receive.count - Messages received from queues or topics
• dt.service.messaging.process.count - Messages successfully processed
• dt.service.messaging.process.failure_count - Messages that failed processing

Use Cases:

• Message throughput monitoring (publish/receive rates)
• Message processing failure tracking
• Queue/topic health analysis
• Consumer lag detection (publish vs receive rate comparison)

Quick Example:

timeseries {
  published = sum(dt.service.messaging.publish.count),
  received = sum(dt.service.messaging.receive.count),
  processed = sum(dt.service.messaging.process.count),
  failed = sum(dt.service.messaging.process.failure_count)
}, by: {dt.service.name}

→ For detailed queries: See references/service-metrics.md

4. Service Mesh Monitoring

Monitor service mesh ingress performance and overhead.

Key Metrics:

• dt.service.request.service_mesh.response_time - Mesh response time (microseconds)
• dt.service.request.service_mesh.count - Mesh request count
• dt.service.request.service_mesh.failure_count - Mesh failure count

Use Cases:

• Mesh vs direct performance comparison
• Mesh overhead calculation
• Mesh failure analysis
• gRPC traffic monitoring
• Multi-cluster mesh performance

Quick Example:

timeseries {
  direct_p95 = percentile(dt.service.request.response_time, 95),
  mesh_p95 = percentile(dt.service.request.service_mesh.response_time, 95)
}, by: {dt.service.name}
| fieldsAdd mesh_overhead_ms = (mesh_p95[] - direct_p95[]) / 1000

→ For detailed queries: See references/service-metrics.md

5. Runtime-Specific Monitoring

Technology-specific runtime performance and resource usage metrics.

Java/JVM - references/java.md

• Memory: heap, pools, metaspace
• GC: impact, suspension, frequency, pause time
• Threads: count monitoring, leak detection
• Classes: loading, unloading, growth

Node.js - references/nodejs.md

• Event loop: utilization, active handles
• V8 heap: memory used, total
• GC: collection time, suspension
• Process: RSS memory

.NET CLR - references/dotnet.md

• Memory: consumption by generation
• GC: collection count, suspension time
• Thread pool: threads, queued work
• JIT: compilation time

Python - references/python.md

• Threads: active thread count
• Heap: allocated blocks
• GC: collection by generation, pause time
• Objects: collected, uncollectable

PHP - references/php.md

• OPcache: hit ratio, memory, restarts
• GC: effectiveness, duration
• JIT: buffer usage
• Interned strings: usage, buffer

Go - references/go.md

• Goroutines: count, leak detection
• GC: suspension, collection time
• Memory: heap by state, committed
• Scheduler: worker threads, queue size
• CGo: call frequency

---

When to Use This Skill

✅ Use for:

• Monitoring service performance (response time, errors, traffic)
• Calculating SLA compliance
• Analyzing service mesh performance
• Monitoring messaging throughput and processing failures
• Troubleshooting runtime-specific issues (GC, memory, threads)
• Multi-cluster service comparison
• Operation/endpoint-level analysis

❌ Don't use for:

• Infrastructure metrics (use infrastructure skills)
• Log analysis (use logs skills)
• Distributed tracing workflows (use traces/spans skills)
• Database performance (use database skills)
• Product documentation or how-to configuration questions → use ask-dynatrace-docs

---

Agent Instructions

Act First, Refine Later

When a user asks for analysis — threshold checks, anomaly detection, performance comparisons — proceed immediately with sensible defaults. Do not ask the user for parameter values you can reasonably assume.

Why this matters: analysis tools (e.g., static-threshold-analyzer) require specific inputs like threshold values and service scope. The user expects results, not a parameter interview. Pick reasonable defaults, state them clearly in the response, and let the user refine.

Default values when not specified:

Example: threshold violation request 1. Use create-dql to build a timeseries query for avg(dt.service.request.response_time) grouped by dt.smartscape.service 2. Pass the query to static-threshold-analyzer with threshold = 1000000 (µs), alertCondition = ABOVE 3. Resolve entity IDs to names using get-entity-name 4. Present violations with service names, timestamps, values, and duration

Reading user phrasing: Phrases like "the fixed threshold", "a threshold", or "the limit" name the type of analysis — static threshold check — not a specific number the user expects you to already know. "Fixed" distinguishes a static cutoff from a dynamic or seasonal baseline. When you see these phrases, apply the 1000 ms default from the table above and present results — the user can then refine if the default doesn't match their intent.

Scope Boundary

This skill covers service performance metrics and runtime monitoring only. If the user asks a product documentation or configuration question (e.g., "How do I add custom sensors?", "How do I configure service detection?"), use ask-dynatrace-docs instead — this skill does not contain configuration how-tos.

Understanding User Intent

Map user questions to capabilities:

Query Construction Patterns

1. Metrics-based (timeseries)

• Use for: Standard monitoring, dashboards, alerting
• Pattern: timeseries <metric> = <aggregation>(<metric_name>), by: {dimensions}
• Files: service-metrics.md, all runtime-specific files

2. Span-based (fetch spans)

• Use for: Complex filtering, custom logic, detailed analysis
• Pattern: fetch spans | filter request.is_root_span == true | fieldsAdd ... | summarize ...
• Files: service-metrics.md (Advanced Service Analysis section)

3. Comparison queries

• Use append for baseline comparison
• Use shift: -15m for time-shifted baselines
• Example: Performance degradation detection

Response Construction Guidelines

Always include: 1. Metric name(s) - Clear metric identifiers 2. Aggregation - How data is aggregated (avg, sum, percentile) 3. Grouping - Dimensions used (dt.service.name, k8s.workload.name, etc.) 4. Unit conversion - Convert microseconds to milliseconds where appropriate 5. Filtering - Relevant thresholds or conditions

When referencing runtime-specific content:

• Check user's technology stack first
• Provide only relevant runtime queries (don't overwhelm with all 6 runtimes)
• Explain runtime-specific metrics (e.g., "OPcache hit ratio" measures PHP opcode cache efficiency)

---

Common Workflows

Workflow: Service Health Check

1. Check response time (RED metrics)
2. Check error rate (RED metrics)
3. Check traffic patterns (RED metrics)
4. If runtime-specific issues suspected → Load runtime-specific reference

Workflow: SLA Monitoring

1. Define SLA criteria (e.g., < 3s response time AND < 1% error rate)
2. Use span-based query for custom SLA logic
3. Calculate compliance percentage
4. Filter non-compliant services

Workflow: Service Mesh Analysis

1. Check mesh response time
2. Compare mesh vs direct performance
3. Calculate mesh overhead
4. Analyze mesh failure rates

Workflow: Runtime Troubleshooting

1. Identify technology stack → Load runtime-specific reference 2. Check memory/GC metrics → threads/goroutines → runtime features

---

Troubleshooting

---

References

Core Service Monitoring:

• references/service-metrics.md - Complete RED metrics, SLA tracking, service mesh queries

Runtime-Specific Monitoring:

• references/java.md - Java/JVM monitoring
• references/nodejs.md - Node.js monitoring
• references/dotnet.md - .NET CLR monitoring
• references/python.md - Python monitoring
• references/php.md - PHP monitoring
• references/go.md - Go runtime monitoring

.NET CLR Performance Metrics

Technology-specific metrics for .NET Common Language Runtime monitoring, including garbage collection, memory consumption, JIT compilation, and thread pool management.

CLR Memory Consumption by Generation

Monitor memory consumption across GC generations:

timeseries memory_bytes = avg(dt.runtime.clr.memory.consumption),
          by: {dt.smartscape.process, dt.process_group.id, generation = clr.gc.generation},
          from: now() - 2h
| fieldsAdd
    memory_mb = memory_bytes[] / 1048576
| filter arrayAvg(memory_mb) > 1024

Use Case: Track memory consumption by generation to identify memory pressure.

Garbage Collection Count by Generation

Analyze GC invocations across generations:

timeseries gc_rate = avg(dt.runtime.clr.gc.collection_count, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id, generation = clr.gc.generation},
          from: now() - 1h
| filter arrayAvg(gc_rate) > 0.1
| sort gc_rate desc

Use Case: Monitor GC frequency per generation to detect excessive collections.

Thread Pool Monitoring

Monitor CLR thread pool threads and work queue:

timeseries thread_count = avg(dt.runtime.clr.threadpool.threads),
          queued_items = avg(dt.runtime.clr.threadpool.queued_work_items),
          by: {dt.smartscape.process, dt.process_group.id, thread_type = clr.threadpool.thread_type},
          from: now() - 30m
| fieldsAdd
    threads = thread_count,
    queue_depth = queued_items
| filter arrayAvg(queue_depth) > 50 or arrayAvg(thread_count) > 100

Use Case: Identify thread pool saturation and work item queuing.

GC Suspension Time Analysis

Monitor the percentage of time the runtime was suspended for GC:

timeseries gc_suspension = avg(dt.runtime.clr.gc.suspension_time),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    suspension_percent = gc_suspension
| filter arrayAvg(suspension_percent) > 10
| sort suspension_percent desc

Use Case: Identify excessive GC suspension time impacting application performance.

JIT Compilation Time Percentage

Monitor JIT compilation overhead:

timeseries jit_time_percent = avg(dt.runtime.clr.jit.time_percentage),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    jit_overhead = jit_time_percent
| filter arrayAvg(jit_overhead) > 5

Use Case: Identify JIT compilation overhead during application startup or code generation.

GC Time Percentage Analysis

Monitor the percentage of time spent in garbage collection:

timeseries gc_time_percent = avg(dt.runtime.clr.gc.time_percentage),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    gc_overhead = gc_time_percent
| filter arrayAvg(gc_overhead) > 10
| sort gc_overhead desc

Use Case: Detect excessive time spent in garbage collection.

Total GC Collection Time

Monitor accumulated garbage collection time:

timeseries gc_collection_time_us = avg(dt.runtime.clr.gc.collection_time),
          gc_time_per_sec_us = avg(dt.runtime.clr.gc.collection_time, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 2h
| fieldsAdd
    gc_time_ms = gc_collection_time_us[] / 1000,
    gc_time_rate = gc_time_per_sec_us[] / 1000
| filter arrayAvg(gc_time_rate) > 100

Use Case: Track total GC collection time to identify GC impact on performance.

Go Runtime Performance Metrics

Technology-specific metrics for Go runtime monitoring, including goroutines, garbage collection, memory management, and scheduler performance.

Goroutine Count Monitoring

Monitor goroutine count and identify leaks:

timeseries goroutine_count = avg(dt.runtime.go.scheduler.goroutine_count),
          by: {dt.smartscape.process, dt.process_group.id, goroutine_owner = go.goroutine.owner},
          from: now() - 4h
| fieldsAdd
    avg_goroutines = arrayAvg(goroutine_count),
    max_goroutines = arrayMax(goroutine_count)
| filter avg_goroutines > 10000 or max_goroutines > 15000

Use Case: Identify goroutine leaks causing resource exhaustion.

GC Suspension Time Analysis

Monitor garbage collection suspension time:

timeseries gc_suspension = avg(dt.runtime.go.gc.suspension_time),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    suspension_percent = gc_suspension
| filter arrayAvg(suspension_percent) > 5
| sort suspension_percent desc

Use Case: Track the proportion of time spent in GC pauses.

Go Heap Memory Analysis

Analyze Go heap size and state:

timeseries heap_bytes = avg(dt.runtime.go.memory.heap),
          by: {dt.smartscape.process, dt.process_group.id, heap_state = go.heap.state},
          from: now() - 2h
| fieldsAdd
    heap_mb = heap_bytes[] / 1048576
| filter arrayAvg(heap_mb) > 1024

Use Case: Track heap memory usage by state (allocated, idle, etc.).

Go Memory Usage by Type

Monitor Go memory used by type:

timeseries memory_used = avg(dt.runtime.go.memory.used),
          by: {dt.smartscape.process, dt.process_group.id, memory_type = go.memory_type},
          from: now() - 30m
| fieldsAdd
    memory_used_mb = memory_used[] / 1048576
| filter arrayAvg(memory_used_mb) > 512

Use Case: Track memory usage across different memory types.

Go Memory Committed

Monitor committed memory:

timeseries memory_committed = avg(dt.runtime.go.memory.committed),
          by: {dt.smartscape.process, dt.process_group.id, memory_type = go.memory_type},
          from: now() - 1h
| fieldsAdd
    memory_committed_mb = memory_committed[] / 1048576
| filter arrayAvg(memory_committed_mb) > 2048

Use Case: Track committed memory to identify memory growth.

GC Collection Count

Monitor garbage collection frequency:

timeseries gc_count_per_sec = avg(dt.runtime.go.gc.collection_count, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 30m
| fieldsAdd
    gc_per_minute = gc_count_per_sec[] * 60
| filter arrayAvg(gc_per_minute) > 60

Use Case: Monitor GC frequency to detect excessive collections.

GC Collection Time

Monitor total garbage collection time:

timeseries gc_time_sec = avg(dt.runtime.go.gc.collection_time),
          gc_time_per_sec = avg(dt.runtime.go.gc.collection_time, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    gc_time = gc_time_sec,
    gc_time_rate = gc_time_per_sec
| filter arrayAvg(gc_time_rate) > 0.1

Use Case: Track time spent in garbage collection.

CGo Calls Monitoring

Track Go to C (CGo) call frequency:

timeseries cgo_call_rate = avg(dt.runtime.go.cgo_calls, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| filter arrayAvg(cgo_call_rate) > 100

Use Case: Monitor CGo overhead for applications with C integration.

Memory Limit Monitoring

Monitor Go runtime memory limit:

timeseries memory_limit = avg(dt.runtime.go.memory.limit),
          memory_used = avg(dt.runtime.go.memory.used),
          by: {dt.smartscape.process, memory_type = go.memory_type},
          from: now() - 2h
| fieldsAdd
    limit_mb = memory_limit[] / 1048576,
    used_mb = memory_used[] / 1048576
| filter arrayAvg(memory_limit) > 0

Use Case: Track memory usage against configured memory limits.

Heap Object Count

Track allocated Go objects on the heap:

timeseries object_count = avg(dt.runtime.go.memory.heap.object_count),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 4h
| fieldsAdd
    avg_objects = arrayAvg(object_count),
    max_objects = arrayMax(object_count)
| filter avg_objects > 10000000 or max_objects > 15000000

Use Case: Monitor live object count for memory leak detection.

Worker Thread Monitoring

Monitor Go worker thread count:

timeseries worker_threads = avg(dt.runtime.go.scheduler.worker_thread_count),
          by: {dt.smartscape.process, dt.process_group.id, thread_state = go.thread.state},
          from: now() - 1h
| fieldsAdd
    threads = worker_threads
| filter arrayAvg(threads) > 100

Use Case: Monitor scheduler worker threads.

Global Goroutine Queue Size

Monitor the global goroutine run queue:

timeseries queue_size = avg(dt.runtime.go.scheduler.queue_size),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    global_queue = queue_size
| filter arrayAvg(global_queue) > 100

Use Case: Identify scheduler contention with large global queue.

Idle Scheduling Context Count

Monitor idle scheduling contexts:

timeseries idle_contexts = avg(dt.runtime.go.scheduler.context.idle_count),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    idle_count = idle_contexts

Use Case: Monitor idle scheduling contexts.

System Call Count

Monitor Go runtime system calls:

timeseries syscall_rate = avg(dt.runtime.go.sys_calls, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| filter arrayAvg(syscall_rate) > 1000

Use Case: Monitor system call frequency.

GC Goal Percentage

Monitor GC heap size target:

timeseries gc_goal = avg(dt.runtime.go.gc.goal),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    goal_percent = gc_goal

Use Case: Track GC heap size target percentage.

HTTP Requests (Go)

Monitor total HTTP requests:

timeseries request_rate = avg(dt.runtime.go.http.requests, rate:1s),
          by: {dt.smartscape.process, status_code = http.response.status_code},
          from: now() - 1h

Use Case: Track HTTP traffic flow.

HTTP Latency (Go)

Monitor HTTP response latency:

timeseries latency_sec = avg(dt.runtime.go.http.latency),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    latency_ms = latency_sec[] * 1000
| filter arrayAvg(latency_ms) > 100

Use Case: Monitor application response time to clients.

Java JVM Performance Metrics

Technology-specific metrics for Java Virtual Machine monitoring, including heap memory, garbage collection, threads, and JVM health analysis.

JVM Memory Analysis

Monitor JVM memory usage patterns:

timeseries memory_max = avg(dt.runtime.jvm.memory.max),
          memory_total = avg(dt.runtime.jvm.memory.total),
          memory_free = avg(dt.runtime.jvm.memory.free),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 2h
| fieldsAdd
    memory_used = memory_total[] - memory_free[],
    memory_usage_percent = ((memory_total[] - memory_free[]) / memory_max[]) * 100,
    memory_available_mb = memory_free[] / 1048576
| filter arrayAvg(memory_usage_percent) > 80

Use Case: Identify processes approaching memory limits for capacity planning.

Garbage Collection Impact Analysis

Analyze GC frequency and duration impact:

timeseries gc_count = avg(dt.runtime.jvm.gc.collection_count),
          gc_time_ms = avg(dt.runtime.jvm.gc.collection_time),
          gc_count_rate = avg(dt.runtime.jvm.gc.collection_count, rate:1s),
          gc_time_rate = avg(dt.runtime.jvm.gc.collection_time, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id, gc_type = jvm.gc.name},
          from: now() - 1h
| fieldsAdd
    avg_gc_duration_ms = if(gc_count_rate[] > 0, gc_time_rate[] / gc_count_rate[], else: 0)
| filter arrayAvg(gc_count_rate) > 0.17 or arrayAvg(avg_gc_duration_ms) > 100
| sort avg_gc_duration_ms desc

Use Case: Detect excessive GC activity causing application pauses.

GC Suspension Time Analysis

Monitor GC pause impact on application responsiveness:

timeseries gc_suspension_time = avg(dt.runtime.jvm.gc.suspension_time),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    suspension_percent = gc_suspension_time
| filter arrayAvg(suspension_percent) > 5
| sort suspension_percent desc

Use Case: Track the proportion of time spent in GC pauses relative to elapsed time.

Total GC Activity Monitoring

Monitor aggregate GC metrics across all pools:

timeseries total_gc_time = avg(dt.runtime.jvm.gc.total_collection_time),
          total_gc_count = avg(dt.runtime.jvm.gc.total_activation_count),
          gc_time_rate_ms = avg(dt.runtime.jvm.gc.total_collection_time, rate:1s),
          gc_count_rate = avg(dt.runtime.jvm.gc.total_activation_count, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 2h
| fieldsAdd
    avg_gc_duration = if(gc_count_rate[] > 0, gc_time_rate_ms[] / gc_count_rate[], else: 0)
| filter arrayAvg(gc_count_rate) > 0.1 or arrayAvg(avg_gc_duration) > 100

Use Case: Monitor overall GC behavior across all garbage collection pools.

Process Group CPU Time During GC Suspensions

Monitor CPU usage during GC suspensions at the process group level:

timeseries cpu_suspension_rate_us = avg(dt.runtime.jvm.pgi.cpu_time_suspension, rate:1s),
          by: {dt.smartscape.process},
          from: now() - 1h
| fieldsAdd
    cpu_suspension_rate_ms = cpu_suspension_rate_us[] / 1000
| filter arrayAvg(cpu_suspension_rate_ms) > 100
| sort cpu_suspension_rate_ms desc

Use Case: Track CPU time consumed during garbage collector suspensions for process groups.

JVM Thread Monitoring

Monitor thread count and identify thread growth:

timeseries thread_count = avg(dt.runtime.jvm.threads.count),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 4h
| fieldsAdd
    avg_thread_count = arrayAvg(thread_count),
    max_thread_count = arrayMax(thread_count)
| filter avg_thread_count > 500 or max_thread_count > 1000

Use Case: Identify thread leaks or approaching thread pool limits.

JVM Memory Pool Analysis

Analyze specific memory pool usage (Old Gen, Young Gen, Metaspace):

timeseries {
    pool_used = avg(dt.runtime.jvm.memory_pool.used),
    pool_committed = avg(dt.runtime.jvm.memory_pool.committed),
    pool_max = avg(dt.runtime.jvm.memory_pool.max)
},
  by: {dt.smartscape.process, pool_name = jvm.memory.pool.name},
  from: now() - 1h
| fieldsAdd
    pool_usage_percent = (pool_used[] / pool_max[]) * 100,
    pool_used_mb = pool_used[] / 1048576
| filter in(pool_name, "Tenured Gen", "Old Gen", "Metaspace", "PS Old Gen")
| filter arrayAvg(pool_usage_percent) > 85

Use Case: Monitor critical memory pools like Old Gen and Metaspace for capacity issues.

JVM Class Loading Anomalies

Detect class loading issues and memory leaks:

timeseries classes_loaded = avg(dt.runtime.jvm.classes.loaded),
          classes_total = avg(dt.runtime.jvm.classes.total),
          classes_unloaded = avg(dt.runtime.jvm.classes.unloaded),
          class_growth_rate = avg(dt.runtime.jvm.classes.loaded, rate:1s),
          unload_rate = avg(dt.runtime.jvm.classes.unloaded, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 2h
| filter arrayAvg(classes_loaded) > 20000 or arrayAvg(class_growth_rate) > 10

Use Case: Identify classloader leaks or excessive dynamic class generation.

Young Generation GC Pressure

Focus on Young Gen GC metrics for throughput issues:

timeseries {
    young_gc_count = avg(dt.runtime.jvm.gc.collection_count),
    young_gc_time_ms = avg(dt.runtime.jvm.gc.collection_time),
    young_gen_used = avg(dt.runtime.jvm.memory_pool.used)
},
  by: {dt.smartscape.process, gc_type = jvm.gc.name},
  from: now() - 30m
| filter in(gc_type, "PS Scavenge", "ParNew", "G1 Young Generation", "Copy")
| fieldsAdd
    gc_overhead_percent = (arrayAvg(young_gc_time_ms) / (30 * 60 * 1000)) * 100,
    young_gen_used_mb = young_gen_used[] / 1048576
| filter gc_overhead_percent > 10

Use Case: Monitor minor GC overhead affecting application throughput.

JVM Full GC Event Tracking

Track major GC events that cause significant pauses:

timeseries full_gc_count = avg(dt.runtime.jvm.gc.collection_count),
          full_gc_time_ms = avg(dt.runtime.jvm.gc.collection_time),
          memory_total = avg(dt.runtime.jvm.memory.total),
          memory_free = avg(dt.runtime.jvm.memory.free),
          gc_count_per_sec = avg(dt.runtime.jvm.gc.collection_count, rate:1s),
          gc_time_per_sec = avg(dt.runtime.jvm.gc.collection_time, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id, gc_type = jvm.gc.name},
          from: now() - 6h
| filter in(gc_type, "PS MarkSweep", "ConcurrentMarkSweep", "G1 Old Generation", "MarkSweepCompact")
| fieldsAdd
    memory_used = memory_total[] - memory_free[],
    gc_count_rate = gc_count_per_sec[] * 3600,
    gc_time_rate = gc_time_per_sec
| fieldsAdd
    avg_pause_duration_ms = if(gc_count_rate[] > 0, gc_time_rate[] / gc_count_rate[] * 3600, else: 0),
    memory_used_gb = memory_used[] / 1073741824
| filter arrayAvg(gc_count_rate) > 5 or arrayAvg(avg_pause_duration_ms) > 1000

Use Case: Alert on excessive Full GC activity indicating heap sizing issues.

JVM CPU vs GC Time Correlation

Correlate CPU usage with GC overhead:

timeseries {
    cpu_usage = avg(dt.process.cpu.usage),
    gc_time_ms = avg(dt.runtime.jvm.gc.collection_time),
    gc_count = avg(dt.runtime.jvm.gc.collection_count),
    memory_total = avg(dt.runtime.jvm.memory.total),
    memory_free = avg(dt.runtime.jvm.memory.free)
},
  by: {dt.smartscape.process, dt.process_group.id},
  from: now() - 1h
| fieldsAdd
    memory_usage = ((memory_total[] - memory_free[]) / memory_total[]) * 100,
    avg_gc_count = arrayAvg(gc_count)
| filter arrayAvg(cpu_usage) > 50 and avg_gc_count > 0.1

Use Case: Identify when high CPU usage is caused by garbage collection.

JVM Memory After GC Trend

Monitor memory usage after GC to detect memory leaks:

timeseries {
    memory_total = avg(dt.runtime.jvm.memory.total),
    memory_free = avg(dt.runtime.jvm.memory.free),
    memory_max = avg(dt.runtime.jvm.memory.max),
    gc_count = avg(dt.runtime.jvm.gc.collection_count)
},
  by: {dt.smartscape.process},
  from: now() - 24h
| fieldsAdd
    memory_used = memory_total[] - memory_free[],
    memory_usage_percent = ((memory_total[] - memory_free[]) / memory_max[]) * 100,
    memory_used_gb = (memory_total[] - memory_free[]) / 1073741824,
    avg_gc_count = arrayAvg(gc_count)
| filter avg_gc_count > 0.01

Use Case: Track post-GC memory trends to identify memory leaks over time.

JVM Metaspace Growth Monitoring

Detect Metaspace exhaustion (Java 8+):

timeseries metaspace_used = avg(dt.runtime.jvm.memory_pool.used),
          metaspace_committed = avg(dt.runtime.jvm.memory_pool.committed),
          metaspace_max = avg(dt.runtime.jvm.memory_pool.max),
          by: {dt.smartscape.process, pool_name = jvm.memory.pool.name},
          from: now() - 12h
| filter pool_name == "Metaspace"
| fieldsAdd
    metaspace_usage_percent = (metaspace_used[] / metaspace_max[]) * 100,
    metaspace_used_mb = metaspace_used[] / 1048576
| filter arrayAvg(metaspace_usage_percent) > 80 or arrayMax(metaspace_used_mb) > 512

Use Case: Prevent OutOfMemoryError for Metaspace with early detection.

Node.js Performance Metrics

Technology-specific metrics for Node.js runtime monitoring, including event loop performance, V8 heap memory, garbage collection, and process memory.

Event Loop Utilization

Monitor event loop utilization percentage:

timeseries event_loop_util = avg(dt.runtime.nodejs.eventloop.utilization),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    utilization_percent = event_loop_util
| filter arrayAvg(utilization_percent) > 70

Use Case: Identify event loop saturation causing poor responsiveness.

V8 Heap Memory Used

Monitor V8 heap memory usage:

timeseries heap_used = avg(dt.runtime.nodejs.memory.used),
          by: {dt.smartscape.process, dt.process_group.id, heap_space = v8.heap_space.name},
          from: now() - 2h
| fieldsAdd
    heap_used_mb = heap_used[] / 1048576
| filter arrayAvg(heap_used_mb) > 512

Use Case: Track V8 heap memory usage by heap space.

V8 Heap Total Memory

Monitor total V8 heap size:

timeseries heap_total = avg(dt.runtime.nodejs.memory.total),
          by: {dt.smartscape.process, dt.process_group.id, heap_space = v8.heap_space.name},
          from: now() - 30m
| fieldsAdd
    heap_total_mb = heap_total[] / 1048576

Use Case: Track total allocated heap size by space.

Event Loop Active Handles

Monitor active event loop handles:

timeseries active_handles = avg(dt.runtime.nodejs.eventloop.active_handles),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    handle_count = active_handles
| filter arrayAvg(handle_count) > 1000

Use Case: Detect handle leaks (unclosed connections, timers, file descriptors).

Process Resident Set Size (RSS)

Monitor process RSS memory:

timeseries rss_bytes = avg(dt.runtime.nodejs.memory.rss),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 2h
| fieldsAdd
    rss_mb = rss_bytes[] / 1048576
| filter arrayAvg(rss_mb) > 1024

Use Case: Track total process memory footprint.

Garbage Collection Time

Monitor GC collection time:

timeseries gc_time_us = avg(dt.runtime.nodejs.gc.collection_time),
          gc_time_per_sec_us = avg(dt.runtime.nodejs.gc.collection_time, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 30m
| fieldsAdd
    gc_time_ms = gc_time_us[] / 1000,
    gc_time_rate_ms = gc_time_per_sec_us[] / 1000
| filter arrayAvg(gc_time_rate_ms) > 100

Use Case: Track time spent in garbage collection.

GC Suspension Time

Monitor GC suspension time percentage:

timeseries gc_suspension = avg(dt.runtime.nodejs.gc.suspension_time),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 4h
| fieldsAdd
    suspension_percent = gc_suspension
| filter arrayAvg(suspension_percent) > 10

Use Case: Track the proportion of time spent in GC pauses.

Memory and Event Loop Overview

Combined Node.js performance view:

timeseries heap_used = avg(dt.runtime.nodejs.memory.used),
          rss = avg(dt.runtime.nodejs.memory.rss),
          event_loop_util = avg(dt.runtime.nodejs.eventloop.utilization),
          active_handles = avg(dt.runtime.nodejs.eventloop.active_handles),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    heap_mb = heap_used[] / 1048576,
    rss_mb = rss[] / 1048576,
    utilization_percent = event_loop_util,
    handles = active_handles

Use Case: Monitor overall Node.js health and resource usage.

Memory Growth Monitoring

Track memory growth over time:

timeseries rss = avg(dt.runtime.nodejs.memory.rss),
          heap_used = avg(dt.runtime.nodejs.memory.used),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 2h
| fieldsAdd
    rss_mb = rss[] / 1048576,
    heap_mb = heap_used[] / 1048576
| filter arrayAvg(rss_mb) > 100 or arrayAvg(heap_mb) > 50

Use Case: Monitor high memory usage that may indicate memory leaks.

PHP Runtime Performance Metrics

Technology-specific metrics for PHP runtime monitoring, including OPcache performance, JIT compilation, garbage collection, and interned strings management.

OPcache Hit Ratio Analysis

Monitor OPcache hit ratio:

timeseries opcache_hits = avg(dt.runtime.php.opcache.hits),
          opcache_misses = avg(dt.runtime.php.opcache.misses),
          hit_rate = avg(dt.runtime.php.opcache.hits, rate:1s),
          miss_rate = avg(dt.runtime.php.opcache.misses, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    hit_rate_percent = (hit_rate[] / (hit_rate[] + miss_rate[])) * 100
| filter arrayAvg(hit_rate_percent) < 95

Use Case: Monitor OPcache efficiency and identify cache misses.

OPcache Memory Usage

Monitor OPcache memory consumption:

timeseries memory_used = avg(dt.runtime.php.opcache.memory.used),
          memory_free = avg(dt.runtime.php.opcache.memory.free),
          memory_wasted = avg(dt.runtime.php.opcache.memory.wasted),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 30m
| fieldsAdd
    used_mb = memory_used[] / 1048576,
    free_mb = memory_free[] / 1048576,
    wasted_mb = memory_wasted[] / 1048576,
    total_mb = (memory_used[] + memory_free[] + memory_wasted[]) / 1048576,
    usage_percent = (memory_used[] / (memory_used[] + memory_free[] + memory_wasted[])) * 100
| filter arrayAvg(usage_percent) > 90

Use Case: Monitor OPcache memory utilization and fragmentation.

OPcache Cached Scripts and Keys

Monitor cached items in OPcache:

timeseries cached_scripts = avg(dt.runtime.php.opcache.cached_scripts),
          cached_keys = avg(dt.runtime.php.opcache.cached_keys),
          max_keys = avg(dt.runtime.php.opcache.max_cached_keys),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    scripts = cached_scripts,
    keys = cached_keys,
    max_capacity = max_keys,
    key_usage_percent = (cached_keys[] / max_keys[]) * 100
| filter arrayAvg(key_usage_percent) > 85

Use Case: Monitor OPcache capacity and approaching limits.

OPcache Restart Monitoring

Track OPcache restarts by type:

timeseries restarts_manual = avg(dt.runtime.php.opcache.restarts_manual),
          restarts_oom = avg(dt.runtime.php.opcache.restarts_out_of_memory),
          restarts_hash = avg(dt.runtime.php.opcache.restarts_has),
          manual_restart_rate = avg(dt.runtime.php.opcache.restarts_manual, rate:1s),
          oom_restart_rate = avg(dt.runtime.php.opcache.restarts_out_of_memory, rate:1s),
          hash_restart_rate = avg(dt.runtime.php.opcache.restarts_has, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 2h
| fieldsAdd
    total_restart_rate = manual_restart_rate[] + oom_restart_rate[] + hash_restart_rate[]
| filter arrayAvg(total_restart_rate) > 0.001

Use Case: Identify OPcache restart causes (OOM, hash collision, manual).

PHP Garbage Collection Metrics

Monitor PHP GC collected objects:

timeseries collected_rate = avg(dt.runtime.php.gc.collected_count, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 4h
| filter arrayAvg(collected_rate) > 1000

Use Case: Track garbage collection activity.

PHP GC Effectiveness

Monitor garbage collection effectiveness:

timeseries gc_effectiveness = avg(dt.runtime.php.gc.effectiveness),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    effectiveness_percent = gc_effectiveness
| filter arrayAvg(effectiveness_percent) < 50

Use Case: Identify inefficient garbage collection cycles.

PHP GC Duration

Monitor garbage collection duration:

timeseries gc_duration = avg(dt.runtime.php.gc.duration_ms),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 30m
| fieldsAdd
    duration_ms = gc_duration
| filter arrayAvg(duration_ms) > 100

Use Case: Identify long GC pauses impacting performance.

OPcache Interned Strings

Monitor interned string buffer usage:

timeseries strings_count = avg(dt.runtime.php.opcache.number_of_strings),
          strings_memory = avg(dt.runtime.php.opcache.strings_used_memory),
          strings_buffer = avg(dt.runtime.php.opcache.strings_buffer_size),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    string_count = strings_count,
    memory_used_mb = strings_memory[] / 1048576,
    buffer_size_mb = strings_buffer[] / 1048576,
    usage_percent = (strings_memory[] / strings_buffer[]) * 100
| filter arrayAvg(usage_percent) > 85

Use Case: Monitor interned string buffer utilization.

OPcache Blocklist Misses

Monitor blocklist miss rate:

timeseries miss_rate = avg(dt.runtime.php.opcache.blocklist_misses, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 2h
| filter arrayAvg(miss_rate) > 10

Use Case: Track blocklist efficiency.

JIT Buffer Usage

Monitor JIT buffer allocation:

timeseries jit_buffer_size = avg(dt.runtime.php.jit.buffer_size),
          jit_buffer_free = avg(dt.runtime.php.jit.buffer_free),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 30m
| fieldsAdd
    buffer_size_mb = jit_buffer_size[] / 1048576,
    buffer_free_mb = jit_buffer_free[] / 1048576,
    buffer_used_mb = (jit_buffer_size[] - jit_buffer_free[]) / 1048576,
    usage_percent = ((jit_buffer_size[] - jit_buffer_free[]) / jit_buffer_size[]) * 100
| filter arrayAvg(usage_percent) > 85

Use Case: Monitor JIT buffer capacity and usage.

OPcache Overall Performance

Combined OPcache metrics view:

timeseries hits = avg(dt.runtime.php.opcache.hits),
          misses = avg(dt.runtime.php.opcache.misses),
          memory_used = avg(dt.runtime.php.opcache.memory.used),
          cached_scripts = avg(dt.runtime.php.opcache.cached_scripts),
          hit_rate = avg(dt.runtime.php.opcache.hits, rate:1s),
          miss_rate = avg(dt.runtime.php.opcache.misses, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    hit_ratio = (hit_rate[] / (hit_rate[] + miss_rate[])) * 100,
    memory_mb = memory_used[] / 1048576,
    scripts = cached_scripts

Use Case: Monitor overall OPcache health and performance.

Python Runtime Performance Metrics

Technology-specific metrics for Python runtime monitoring, including garbage collection by generation, thread count, and memory block allocation.

Python Thread Count

Monitor active Python threads:

timeseries thread_count = avg(dt.runtime.python.threads),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 1h
| fieldsAdd
    threads = thread_count
| filter arrayAvg(thread_count) > 50

Use Case: Monitor Python thread count for multi-threaded applications.

Python Heap Allocated Blocks

Monitor number of allocated memory blocks:

timeseries allocated_blocks = avg(dt.runtime.python.heap.allocated_blocks),
          by: {dt.smartscape.process, dt.process_group.id},
          from: now() - 2h
| fieldsAdd
    blocks = allocated_blocks,
    block_growth = arrayDelta(allocated_blocks)
| filter arrayAvg(allocated_blocks) > 1000000 or arrayAvg(block_growth) > 100000

Use Case: Track memory block allocation for memory leak detection.

Python GC Collection Count by Generation

Analyze GC collections by generation:

timeseries gc_count = avg(dt.runtime.python.gc.collection_count),
          collection_rate = avg(dt.runtime.python.gc.collection_count, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id, generation = python.gc.generation},
          from: now() - 30m
| fieldsAdd
    collections = gc_count
| filter arrayAvg(collection_rate) > 1

Use Case: Monitor GC frequency per generation.

Python GC Collected Objects

Track objects collected by GC:

timeseries collected_objects = avg(dt.runtime.python.gc.collected_objects),
          collection_rate = avg(dt.runtime.python.gc.collected_objects, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id, generation = python.gc.generation},
          from: now() - 1h
| fieldsAdd
    objects_collected = collected_objects
| filter arrayAvg(collection_rate) > 1000

Use Case: Monitor objects collected during garbage collection.

Python GC Uncollectable Objects

Monitor uncollectable objects:

timeseries uncollectable = avg(dt.runtime.python.gc.uncollectable_objects),
          by: {dt.smartscape.process, dt.process_group.id, generation = python.gc.generation},
          from: now() - 30m
| fieldsAdd
    uncollectable_count = uncollectable
| filter arrayAvg(uncollectable_count) > 100

Use Case: Identify objects that cannot be collected due to circular references.

Python GC Collection Time

Monitor time spent in garbage collection:

timeseries gc_time_us = avg(dt.runtime.python.gc.collection_time),
          gc_time_per_sec_us = avg(dt.runtime.python.gc.collection_time, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id, generation = python.gc.generation},
          from: now() - 1h
| fieldsAdd
    gc_time_ms = gc_time_us[] / 1000,
    gc_time_rate = gc_time_per_sec_us[] / 1000
| filter arrayAvg(gc_time_rate) > 100

Use Case: Track GC pause times by generation.

Python GC Overview

Combined GC metrics view:

timeseries gc_count = avg(dt.runtime.python.gc.collection_count),
          gc_collected = avg(dt.runtime.python.gc.collected_objects),
          gc_time = avg(dt.runtime.python.gc.collection_time),
          collection_rate = avg(dt.runtime.python.gc.collection_count, rate:1s),
          object_rate = avg(dt.runtime.python.gc.collected_objects, rate:1s),
          by: {dt.smartscape.process, dt.process_group.id, generation = python.gc.generation},
          from: now() - 2h
| fieldsAdd
    collections = gc_count,
    objects_collected = gc_collected,
    time_us = gc_time

Use Case: Monitor overall Python garbage collection health.

Service Metrics Reference

Complete reference for service performance monitoring, including RED metrics, advanced service analysis, and service mesh monitoring.

---

Part 1: Service Metrics (RED Metrics)

Monitor service performance, failures, and traffic using metrics-based timeseries queries.

Response Time Analysis

Basic Response Time Monitoring

timeseries response_time = avg(dt.service.request.response_time), by: {dt.service.name}
| fieldsAdd avg_response_ms = arrayAvg(response_time) / 1000
| sort avg_response_ms desc

Key Metrics:

• dt.service.request.response_time: Server-side response time (microseconds)
• dt.service.request.count: Total request count
• dt.service.request.failure_count: Failed request count

Response Time Percentiles

timeseries {
  p50 = percentile(dt.service.request.response_time, 50),
  p95 = percentile(dt.service.request.response_time, 95),
  p99 = percentile(dt.service.request.response_time, 99)
}, by: {dt.service.name}
| fieldsAdd p50_ms = p50[] / 1000, p95_ms = p95[] / 1000, p99_ms = p99[] / 1000

Response Time by Endpoint

timeseries response_time = avg(dt.service.request.response_time),
  by: {dt.service.name, endpoint.name}
| fieldsAdd avg_response_ms = arrayAvg(response_time) / 1000
| filter avg_response_ms > 500
| sort avg_response_ms desc
| limit 20

Performance Degradation Detection

timeseries recent_avg = avg(dt.service.request.response_time), by: {dt.service.name}, from: now() - 15m
| fieldsAdd recent_avg_ms = arrayAvg(recent_avg) / 1000
| append [
  timeseries baseline_avg = avg(dt.service.request.response_time), by: {dt.service.name}, shift: -15m
  | fieldsAdd baseline_avg_ms = arrayAvg(baseline_avg) / 1000
]
| fieldsAdd degradation_pct = (recent_avg_ms - baseline_avg_ms) * 100 / baseline_avg_ms
| filter degradation_pct > 50
| sort degradation_pct desc

Failure Analysis

Error Rate Calculation

timeseries {
    total_requests = sum(dt.service.request.count),
    failures = sum(dt.service.request.failure_count)
  }, by: {dt.service.name}
| fieldsAdd error_rate_pct = (failures[] * 100.0) / total_requests[]
| filter arrayAvg(error_rate_pct) > 0

Failure Spikes

timeseries failures = sum(dt.service.request.failure_count), by: {dt.service.name}
| fieldsAdd {
    max_failures = arrayMax(failures),
    avg_failures = arrayAvg(failures),
    spike_ratio = arrayMax(failures) / arrayAvg(failures)
  }
| filter spike_ratio > 3 and arraySum(failures) > 20
| sort spike_ratio desc

Failures by HTTP Status

timeseries failures = sum(dt.service.request.failure_count),
  by: {dt.service.name, http.response.status_code}
| fieldsAdd total_failures = arraySum(failures)
| filter total_failures > 0
| sort total_failures desc

Traffic Analysis

Request Throughput

timeseries requests = sum(dt.service.request.count), by: {dt.service.name}, bins: 100
| fieldsAdd requests_per_second = requests[] / 60

Peak Traffic Detection

timeseries requests = sum(dt.service.request.count), by: {dt.service.name}
| fieldsAdd {
    max_requests = arrayMax(requests),
    avg_requests = arrayAvg(requests),
    peak_ratio = arrayMax(requests) / arrayAvg(requests)
  }
| filter peak_ratio > 2
| sort peak_ratio desc

Traffic Growth

timeseries recent_total = sum(dt.service.request.count, scalar: true), by: {dt.service.name}, from: -30m, to: now()
| append [
  timeseries baseline_total = sum(dt.service.request.count, scalar: true), by: {dt.service.name}, from: -60m, to: -30m
]
| fieldsAdd growth_pct = ((recent_total - baseline_total) * 100.0) / baseline_total
| filter baseline_total > 100
| sort growth_pct desc

Kubernetes Context

Service Performance by Workload

timeseries {
    response_time = avg(dt.service.request.response_time),
    requests = sum(dt.service.request.count),
    failures = sum(dt.service.request.failure_count)
  }, by: {k8s.workload.name, k8s.namespace.name}
| fieldsAdd response_time_ms = response_time[] / 1000

Multi-Cluster Comparison

timeseries {
    avg_response = avg(dt.service.request.response_time),
    total_requests = sum(dt.service.request.count),
    failures = sum(dt.service.request.failure_count)
  }, by: {k8s.cluster.name, dt.service.name}
| fieldsAdd avg_response_ms = avg_response[] / 1000, error_rate = failures[] * 100.0 / total_requests[]

---

Part 2: Service Messaging Metrics

Monitor message-based service communication including publishing, receiving, and processing of messages via queues and topics.

Key Metrics:

Message Throughput

Publish and Receive Rate

timeseries {
  published = sum(dt.service.messaging.publish.count),
  received = sum(dt.service.messaging.receive.count)
}, by: {dt.service.name}

Processing Success and Failure Rate

timeseries {
  processed = sum(dt.service.messaging.process.count),
  failed = sum(dt.service.messaging.process.failure_count)
}, by: {dt.service.name}
| fieldsAdd failure_rate_pct = (failed[] * 100.0) / (processed[] + failed[])

Message Processing Failures

Services with Highest Processing Failures

timeseries failures = sum(dt.service.messaging.process.failure_count), by: {dt.service.name}
| fieldsAdd total_failures = arraySum(failures)
| filter total_failures > 0
| sort total_failures desc

Processing Failure Spike Detection

timeseries failures = sum(dt.service.messaging.process.failure_count), by: {dt.service.name}
| fieldsAdd {
    max_failures = arrayMax(failures),
    avg_failures = arrayAvg(failures),
    spike_ratio = arrayMax(failures) / arrayAvg(failures)
  }
| filter spike_ratio > 3 and arraySum(failures) > 10
| sort spike_ratio desc

Consumer Lag Analysis

Publish vs Receive Rate Comparison

timeseries {
  published = sum(dt.service.messaging.publish.count),
  received = sum(dt.service.messaging.receive.count)
}, by: {dt.service.name}
| fieldsAdd lag_indicator = published[] - received[]
| filter arrayAvg(lag_indicator) > 0
| sort lag_indicator desc

Combined Messaging Overview

Full Messaging Pipeline Health

timeseries {
  published = sum(dt.service.messaging.publish.count),
  received = sum(dt.service.messaging.receive.count),
  processed = sum(dt.service.messaging.process.count),
  failed = sum(dt.service.messaging.process.failure_count)
}, by: {dt.service.name}
| fieldsAdd
    total_published = arraySum(published),
    total_received = arraySum(received),
    total_processed = arraySum(processed),
    total_failed = arraySum(failed)
| fieldsAdd processing_failure_rate = if(total_processed + total_failed > 0, (total_failed * 100.0) / (total_processed + total_failed), else: 0)
| sort total_failed desc

---

Part 3: Advanced Service Performance Analysis

Span-based queries for complex service analysis requiring flexible filtering and custom aggregations. For standard metric monitoring, use timeseries queries in Part 1.

SLA Compliance Tracking

Custom SLA calculation with complex conditions:

fetch spans, from: now() - 1h
| filter request.is_root_span == true
| fieldsAdd
    meets_sla = if(request.is_failed == false AND duration < 3000000000, 1, else: 0)
| summarize
    total_requests = count(),
    sla_compliant = sum(meets_sla),
    by: {dt.service.name}
| fieldsAdd sla_compliance_percent = (sla_compliant * 100.0) / total_requests
| filter sla_compliance_percent < 99.9
| sort sla_compliance_percent asc

Use Case: Custom SLA thresholds combining failure status and duration.

Service Health Scoring

Multi-dimensional health assessment:

fetch spans, from:now()-1h
| filter request.is_root_span == true
| summarize
    total = count(),
    errors = countIf(request.is_failed == true),
    slow = countIf(duration > 3s),
    p95_duration = percentile(duration, 95),
    by: {dt.service.name}
| fieldsAdd
    error_rate = (errors * 100.0) / total,
    slow_rate = (slow * 100.0) / total
| fieldsAdd
    health_status = if(
        error_rate < 1.0 and slow_rate < 5.0, "healthy",
        else: if(error_rate < 5.0, "degraded", else: "critical")
    )
| sort health_status, error_rate desc

Use Case: Combined health score using multiple conditions and thresholds.

Operation-Level Performance

Analyze performance by specific operations:

fetch spans, from: now() - 2h
| filter request.is_root_span == true
| summarize
    request_count = count(),
    avg_duration_ms = avg(duration) / 1000000,
    p95_duration_ms = percentile(duration, 95) / 1000000,
    error_count = countIf(request.is_failed == true),
    by: {dt.service.name, span.name}
| fieldsAdd error_rate = (error_count * 100.0) / request_count
| filter request_count > 10
| sort p95_duration_ms desc
| limit 30

Use Case: Detailed operation/endpoint analysis with span names.

Custom Error Classification

Categorize errors with complex logic:

fetch spans, from: now() - 1h
| filter request.is_root_span == true and request.is_failed == true
| fieldsAdd
    error_category = if(
        http.response.status_code >= 500, "server_error",
        else: if(http.response.status_code >= 400, "client_error",
        else: "other_failure")
    )
| summarize count = count(),
  by: {dt.service.name, error_category, http.response.status_code}
| sort count desc

Use Case: Custom error categorization beyond standard failure metrics.

Request Context Analysis

Analyze performance with additional span attributes:

fetch spans, from: now() - 1h
| filter request.is_root_span == true
| summarize
    request_count = count(),
    avg_duration_ms = avg(duration) / 1000000,
    p95_duration_ms = percentile(duration, 95) / 1000000,
    by: {dt.service.name, http.request.method, http.route}
| filter request_count > 5
| sort p95_duration_ms desc
| limit 50

Use Case: Performance analysis by HTTP method and route patterns.

Failure Pattern Detection

Identify failure patterns using Dynatrace failure detection results. This extracts structured failure reasons from dt.failure_detection.results, matching exception details from span events.

fetch spans, from: now() - 2h
| filter request.is_root_span == true and request.is_failed == true

// Extract failure reasons from failure detection results
| expand dt.failure_detection.results
| fieldsAdd reason = dt.failure_detection.results[reason], exception_ids = dt.failure_detection.results[exception_ids]
| fieldsAdd exceptionsFound = iAny(arrayIndexOf(exception_ids, span.events[][exception.id]) > -1)
| expand exception_ids = if(exceptionsFound, exception_ids, else: array(0))
| expand event = if(exceptionsFound, span.events, else: array(0))
| filter isFalseOrNull(exceptionsFound) OR isNull(reason) OR reason != "exception" OR event[exception.id] == exception_ids
| fieldsAdd exceptionName = event[exception.type]
| fieldsAdd failure_reason = if(reason == "span_status", concat("Span status ", span.status_code),
   else: if(reason == "grpc_code", concat("GRPC status ", rpc.grpc.status_code),
   else: if(reason == "http_code", concat("HTTP ", http.response.status_code),
   else: if(reason == "exception", coalesce(exceptionName, "Unknown exception"),
   else: if(isNull(reason), "<No failure reason>",
   else: concat("Unknown reason: ", reason))))))

| summarize
    failure_count = count(),
    unique_errors = countDistinctExact(failure_reason),
    avg_duration_ms = avg(duration) / 1000000,
    by: {dt.service.name, span.name}
| filter failure_count > 3
| sort failure_count desc
| limit 20

Use Case: Pattern analysis using Dynatrace failure detection to classify failures by HTTP status codes, gRPC codes, exceptions, and span status.

Note: The dt.failure_detection.results attribute contains structured failure analysis data. The query expands these results and matches exception IDs against span events to extract exception types. Failure reasons are classified into categories: http_code, grpc_code, exception, span_status.

---

Part 4: Service Mesh Metrics

Monitor service mesh ingress performance, failures, and traffic patterns.

Mesh Response Time

Basic Mesh Performance

timeseries response_time = avg(dt.service.request.service_mesh.response_time), by: {dt.service.name}
| fieldsAdd avg_response_ms = arrayAvg(response_time) / 1000
| sort avg_response_ms desc

Key Metrics:

• dt.service.request.service_mesh.response_time: Mesh ingress response time (microseconds)
• dt.service.request.service_mesh.count: Mesh request count
• dt.service.request.service_mesh.failure_count: Mesh failure count

Mesh vs Direct Overhead

timeseries {
    direct_p95 = percentile(dt.service.request.response_time, 95),
    mesh_p95 = percentile(dt.service.request.service_mesh.response_time, 95)
  }, by: {dt.service.name}
| fieldsAdd direct_p95_ms = direct_p95[] / 1000, mesh_p95_ms = mesh_p95[] / 1000
| fieldsAdd mesh_overhead = mesh_p95_ms[] - direct_p95_ms[]
| filter arrayAvg(mesh_overhead) > 0
| sort mesh_overhead desc

Mesh Performance Degradation

timeseries recent_avg = avg(dt.service.request.service_mesh.response_time), by: {dt.service.name}, from: now() - 15m
| fieldsAdd recent_avg_ms = arrayAvg(recent_avg) / 1000
| append [
  timeseries baseline_avg = avg(dt.service.request.service_mesh.response_time), by: {dt.service.name}, from: now() - 30m, to: now() - 15m
  | fieldsAdd baseline_avg_ms = arrayAvg(baseline_avg) / 1000
]
| fieldsAdd degradation_pct = (recent_avg_ms - baseline_avg_ms) * 100 / baseline_avg_ms
| filter degradation_pct > 30
| sort degradation_pct desc

Mesh Failures

Mesh Error Rate

timeseries {
    total_requests = sum(dt.service.request.service_mesh.count),
    failures = sum(dt.service.request.service_mesh.failure_count)
  }, by: {dt.service.name}
| fieldsAdd error_rate_pct = (failures[] * 100.0) / total_requests[]
| filter arrayAvg(error_rate_pct) > 0

Mesh Failures by Status Code

timeseries {
    requests = sum(dt.service.request.service_mesh.count),
    failures = sum(dt.service.request.service_mesh.failure_count)
  }, by: {dt.service.name, http.response.status_code}
| fieldsAdd failure_pct = (failures[] * 100.0) / requests[]

Mesh Traffic

Mesh Request Volume

timeseries requests = sum(dt.service.request.service_mesh.count), by: {dt.service.name}
| fieldsAdd total_requests = arraySum(requests)
| sort total_requests desc

Mesh gRPC Traffic

timeseries {
    requests = sum(dt.service.request.service_mesh.count),
    failures = sum(dt.service.request.service_mesh.failure_count)
  }, by: {dt.service.name, rpc.grpc.status_code}
| filter isNotNull(rpc.grpc.status_code)

Mesh Kubernetes Context

Mesh Performance by Workload

timeseries {
    response_time = avg(dt.service.request.service_mesh.response_time),
    failures = sum(dt.service.request.service_mesh.failure_count),
    total = sum(dt.service.request.service_mesh.count)
  }, by: {k8s.workload.name, k8s.namespace.name}
| fieldsAdd response_time_ms = response_time[] / 1000, error_rate = failures[] * 100.0 / total[]

Mesh Multi-Cluster Performance

timeseries {
    avg_response = avg(dt.service.request.service_mesh.response_time),
    p95_response = percentile(dt.service.request.service_mesh.response_time, 95)
  }, by: {k8s.cluster.name, dt.service.name}
| fieldsAdd avg_response_ms = avg_response[] / 1000, p95_response_ms = p95_response[] / 1000