Dt Obs Tracing

Application Tracing Skill

Overview

Distributed traces in Dynatrace consist of spans - building blocks representing units of work. With Traces in Grail, every span is accessible via DQL with full-text searchability on all attributes. This skill covers trace fundamentals, common analysis patterns, and span-type specific queries.

---

Use Cases

1. Investigate Slow Requests

• Goal: Find and diagnose requests exceeding a latency threshold
• Trigger: "slow requests", "high latency", "p99 response time", "find traces over 5 seconds"
• Done: List of slow traces with duration, endpoint, service, and trace IDs for drilldown

2. Analyze Request Failures

• Goal: Identify failed requests, failure reasons, and exception patterns
• Trigger: "failed spans", "HTTP 500 errors", "exception analysis", "failure rate by service"
• Done: Failure breakdown by reason (HTTP code, exception, gRPC status) with exemplar traces

3. Map Service Dependencies

• Goal: Understand service-to-service communication patterns and external API calls
• Trigger: "service dependencies", "what services does X call", "outgoing HTTP calls"
• Done: Dependency map showing call counts, latency, and error rates between services

---

Core Concepts

Understanding Traces and Spans

Spans represent logical units of work in distributed traces:

• HTTP requests, RPC calls, database operations
• Messaging system interactions
• Internal function invocations
• Custom instrumentation points

Span kinds:

• span.kind: server - Incoming call to a service
• span.kind: client - Outgoing call from a service
• span.kind: consumer - Incoming message consumption call to a service
• span.kind: producer - Outgoing message production call from a service
• span.kind: internal - Internal operation within a service

Root spans: A request root span (request.is_root_span == true) represents an incoming call to a service. Use this to analyze end-to-end request performance.

Key Trace Attributes

Essential attributes for trace analysis:

Service Context

Spans reference services via Smartscape node IDs and the detected service name dt.service.name which is also present on every span.

fetch spans
| summarize spans=count(), by: { dt.smartscape.service, dt.service.name }

Node functions:

• getNodeName(dt.smartscape.service) - Adds dt.smartscape.service.name field with the human-readable service name
• getNodeField(dt.smartscape.service, "attribute_name") - Access specific node attributes

📖 Learn more: See Entity Lookups for advanced entity selectors, infrastructure correlation, and hardware analysis.

Sampling and Extrapolation

One span can represent multiple real operations due to:

• Aggregation: Multiple operations in one span (aggregation.count)
• ATM (Adaptive Traffic Management): Head-based sampling by agent
• ALR (Adaptive Load Reduction): Server-side sampling
• Read Sampling: Query-time sampling via samplingRatio parameter

When to extrapolate: Always extrapolate when counting actual operations (not just spans). Use the multiplicity factor:

fetch spans
| fieldsAdd sampling.probability = (power(2, 56) - coalesce(sampling.threshold, 0)) * power(2, -56)
| fieldsAdd sampling.multiplicity = 1 / sampling.probability
| fieldsAdd multiplicity = coalesce(sampling.multiplicity, 1)
                         * coalesce(aggregation.count, 1)
                         * dt.system.sampling_ratio
| summarize operation_count = sum(multiplicity)

📖 Learn more: See Sampling and Extrapolation for detailed formulas and examples.

Common Query Patterns

Basic Span Access

Fetch spans and explore by type:

fetch spans | limit 1

Explore spans by function and type:

fetch spans
| summarize count(), by: { span.kind, code.namespace, code.function }

Request Root Filtering

List request root spans (incoming service calls):

fetch spans
| filter request.is_root_span == true
| fields trace.id, span.id, start_time, response_time = duration, endpoint.name
| limit 100

Service Performance Summary

Analyze service performance with error rates:

fetch spans
| filter request.is_root_span == true
| summarize
    total_requests = count(),
    failed_requests = countIf(request.is_failed == true),
    avg_duration = avg(duration),
    p95_duration = percentile(duration, 95),
    by: {dt.service.name}
| fieldsAdd error_rate = (failed_requests * 100.0) / total_requests
| sort error_rate desc

Trace ID Lookup

Find all spans in a specific trace:

fetch spans
| filter trace.id == toUid("abc123def456")
| fields span.name, duration, dt.service.name

Performance Analysis

Response Time Percentiles

Calculate percentiles by endpoint:

fetch spans
| filter request.is_root_span == true
| summarize {
    requests=count(),
    avg_duration=avg(duration),
    p95=percentile(duration, 95),
    p99=percentile(duration, 99)
  }, by: { endpoint.name }
| sort p99 desc

💡 Best practice: Use percentiles (p95, p99) over averages for performance insights.

Slow Trace Detection

Find requests exceeding a threshold:

fetch spans, from:now() - 2h
| filter request.is_root_span == true
| filter duration > 5s
| fields trace.id, span.name, dt.service.name, duration
| sort duration desc
| limit 50

Duration Buckets with Exemplars

fetch spans, from:now() - 24h
| filter http.route == "/api/v1/storage/findByISBN"
| summarize {
    spans=count(),
    trace=takeAny(record(start_time, trace.id))
  }, by: { bin(duration, 10ms) }
| fields `bin(duration, 10ms)`, spans, trace.id=trace[trace.id], start_time=trace[start_time]

Performance Timeseries

Extract response time as timeseries:

fetch spans, from:now() - 24h
| filter request.is_root_span == true
| makeTimeseries {
    requests=count(),
    avg_duration=avg(duration),
    p95=percentile(duration, 95),
    p99=percentile(duration, 99)
  }, by: { endpoint.name }

📖 Learn more: See Performance Analysis for advanced patterns and timeseries techniques.

Failure Investigation

Failed Request Summary

Summarize failures by service:

fetch spans
| filter request.is_root_span == true
| summarize
    total = count(),
    failed = countIf(request.is_failed == true),
  by: { dt.service.name }
| fieldsAdd failure_rate = (failed * 100.0) / total
| sort failure_rate desc

Failure Reason Analysis

Breakdown by failure detection reason:

fetch spans
| filter request.is_failed == true and isNotNull(dt.failure_detection.results)
| expand dt.failure_detection.results
| summarize count(), by: { dt.failure_detection.results[reason] }

Failure reasons:

• http_code - HTTP response code triggered failure
• grpc_code - gRPC status code triggered failure
• exception - Exception caused failure
• span_status - Span status indicated failure
• custom_rule - Custom failure detection rule matched

HTTP Code Failures

Find failures by HTTP status code:

fetch spans
| filter request.is_failed == true
| filter iAny(dt.failure_detection.results[][reason] == "http_code")
| summarize count(), by: { http.response.status_code, endpoint.name }
| sort `count()` desc

Recent Failed Requests

List recent failures with details:

fetch spans
| filter request.is_root_span == true and request.is_failed == true
| fields
    start_time,
    trace.id,
    endpoint.name,
    http.response.status_code,
    duration
| sort start_time desc
| limit 100

📖 Learn more: See Failure Detection for exception analysis and custom rule investigation.

Service Dependencies

Service-to-Service Analysis

Analyze service communication patterns:

fetch spans, from:now() - 1h
| filter isNotNull(server.address)
| fieldsAdd
    remote_side = server.address
| summarize
    call_count = count(),
    avg_duration = avg(duration),
    by: {dt.service.name, remote_side}
| sort call_count desc

Outgoing HTTP Calls

Identify external API dependencies:

fetch spans
| filter span.kind == "client" and isNotNull(http.request.method)
| summarize
    calls = count(),
    avg_latency = avg(duration),
    p99_latency = percentile(duration, 99),
  by: { dt.service.name, server.address, server.port }
| sort calls desc

Trace Aggregation

Complete Trace Analysis

Aggregate all spans in a trace to understand full request flow:

fetch spans, from:now() - 30m
| summarize {
    spans = count(),
    client_spans = countIf(span.kind == "client"),

    // Endpoints involved in the trace
    endpoints = toString(arrayRemoveNulls(collectDistinct(endpoint.name))),

    // Extract the first request root in the trace
    trace_root = takeMin(record(
        root_detection_helper = coalesce(
            if(request.is_root_span, 1),
            if(isNull(span.parent_id), 2),
            3),
        start_time, endpoint.name, duration
      ))
}, by: { trace.id }

| fieldsFlatten trace_root
| fieldsRemove trace_root.root_detection_helper, trace_root

| fields
    start_time = trace_root.start_time,
    endpoint = trace_root.endpoint.name,
    response_time = trace_root.duration,
    spans,
    client_spans,
    endpoints,
    trace.id
| sort start_time
| limit 100

Root detection strategy: Use takeMin(record(...)) with a detection helper to reliably find the root request: 1. Priority 1: Spans with request.is_root_span == true 2. Priority 2: Spans without parent (root spans) 3. Priority 3: All other spans

Multi-Service Traces

Find traces spanning multiple services:

fetch spans, from:now() - 1h
| summarize {
    services = collectDistinct(dt.service.name),
    trace_root = takeMin(record(root_detection_helper = coalesce(if(request.is_root_span, 1), 2), endpoint.name))
}, by: { trace.id }
| fieldsAdd service_count = arraySize(services)
| filter service_count > 1
| fields endpoint = trace_root[endpoint.name], service_count, services = toString(services), trace.id
| sort service_count desc
| limit 50

Request-Level Analysis

Request Attributes

Access custom request attributes captured by OneAgent on request root spans:

fetch spans
| filter request.is_root_span == true
| filter isNotNull(request_attribute.PaidAmount)
| makeTimeseries sum(request_attribute.PaidAmount)

Field patterns: request_attribute.<name>, captured_attribute.<name> (always arrays)

→ Request Attributes — full patterns for request attributes, captured attributes, and request ID aggregation

Span Types

⚠️ Database spans: Can be aggregated (one span = multiple calls). Always use aggregation.count extrapolation for accurate operation counts.

📖 Detailed patterns per span type: See the reference files above.

Advanced Topics

Exception Analysis

Exceptions are stored as span.events within spans:

fetch spans
| filter iAny(span.events[][span_event.name] == "exception")
| expand span.events
| fieldsFlatten span.events, fields: { exception.type }
| summarize {
    count(),
    trace=takeAny(record(start_time, trace.id))
  }, by: { exception.type }
| fields exception.type, `count()`, trace.id=trace[trace.id], start_time=trace[start_time]

💡 Tip: Use iAny() to check conditions within span event arrays.

→ Logs Correlation — joining logs and traces, filtering traces by log content → Network Analysis — client IPs, DNS resolution, subnet analysis

Best Practices

---

Troubleshooting

Related Skills

• dt-dql-essentials — Core DQL syntax for querying trace data
• dt-app-dashboards — Embed trace queries in dashboards
• dt-migration — Smartscape entity model and relationship navigation

---

References

Detailed documentation for specific topics:

• [Performance Analysis](references/performance-analysis.md) - Advanced timeseries, duration buckets, endpoint ranking
• [Failure Detection](references/failure-detection.md) - Failure reasons, exception investigation, custom rules
• [Sampling and Extrapolation](references/sampling-extrapolation.md) - Multiplicity calculation, database extrapolation
• [Request Attributes](references/request-attributes.md) - Request attributes, captured attributes, request ID aggregation
• [Entity Lookups](references/entity-lookups.md) - Advanced node lookups, infrastructure correlation, hardware analysis
• [HTTP Span Analysis](references/http-spans.md) - Status codes, payload analysis, client IPs
• [Database Span Analysis](references/database-spans.md) - Extrapolated counts, slow queries, statement analysis
• [Messaging Span Analysis](references/messaging-spans.md) - Kafka, RabbitMQ, SQS throughput and latency
• [RPC Span Analysis](references/rpc-spans.md) - gRPC, SOAP, service dependencies
• [Serverless Span Analysis](references/serverless-spans.md) - Lambda, Azure Functions, cold start analysis
• [Logs Correlation](references/logs-correlation.md) - Joining logs and traces, correlation patterns
• [Network Analysis](references/networking-analysis.md) - IP addresses, DNS resolution, communication mapping

Database Span Analysis

Database operations in traces appear as client spans with database-specific attributes. Database spans can be aggregated (one span representing multiple calls).

Basic Database Queries

List Database Operations

Query database activity:

fetch spans
| filter isNotNull(db.query.text)
| summarize spans=count(), by: { db.system, db.operation.name, db.collection.name }

Database Spans by Service

Identify which services make database calls:

fetch spans
| filter span.kind == "client" and isNotNull(db.namespace)
| fieldsAdd getNodeName(dt.smartscape.service)
| summarize {
    spans=count(),
    avg_duration=avg(duration)
  }, by: { dt.smartscape.service.name, db.system, db.namespace }
| sort spans desc

Top Database Statements

Extrapolated Statement Counts

Count actual database calls (not just spans) per service:

fetch spans
| filter span.kind == "client" and isNotNull(db.namespace)
| fieldsAdd getNodeName(dt.smartscape.service)

// Calculate multiplicity for extrapolation
| fieldsAdd sampling.probability = (power(2, 56) - coalesce(sampling.threshold, 0)) * power(2, -56)
| fieldsAdd sampling.multiplicity = 1/sampling.probability
| fieldsAdd multiplicity = coalesce(sampling.multiplicity, 1)
                         * coalesce(aggregation.count, 1)
                         * dt.system.sampling_ratio

| summarize {
    db_calls = sum(multiplicity)
  }, by: {
    dt.smartscape.service.name,
    code.function,
    db.system,
    db.namespace,
    db.query.text
  }

| sort db_calls desc
| limit 100

Database Performance

Statement Duration Analysis

Analyze database call durations with aggregation awareness:

fetch spans
| filter span.kind == "client" and isNotNull(db.query.text)

// Calculate average duration for aggregated spans
| fieldsAdd aggregation.duration_avg = coalesce(
    aggregation.duration_sum / aggregation.count,
    duration
)

// Calculate multiplicity
| fieldsAdd sampling.probability = (power(2, 56) - coalesce(sampling.threshold, 0)) * power(2, -56)
| fieldsAdd sampling.multiplicity = 1/sampling.probability
| fieldsAdd multiplicity = coalesce(sampling.multiplicity, 1)
                         * coalesce(aggregation.count, 1)
                         * dt.system.sampling_ratio

| summarize {
    operation_count = sum(multiplicity),
    avg_duration = sum(aggregation.duration_avg * multiplicity) / sum(multiplicity),
    p95_duration = percentile(aggregation.duration_avg, 95)
  }, by: { db.system, db.operation.name, db.collection.name }
| sort operation_count desc
| limit 50

Slow Database Queries

Find slowest database statements:

fetch spans
| filter span.kind == "client" and isNotNull(db.query.text)
| fieldsAdd aggregation.duration_avg = coalesce(
    aggregation.duration_sum / aggregation.count,
    duration
)
| filter aggregation.duration_avg > 100ms
| fields
    trace.id,
    db.system,
    db.query.text,
    duration=aggregation.duration_avg,
    aggregated_calls=aggregation.count
| sort duration desc
| limit 50

Database Attributes

Common database span attributes:

• db.system - Database type (e.g., postgresql, mysql, mongodb)
• db.namespace - Database name
• db.query.text - SQL/query statement
• db.operation.name - Operation type (SELECT, INSERT, UPDATE, etc.)
• db.collection.name - Table/collection name
• db.affected_item_count - Number of rows/documents affected

Best Practices

• Always extrapolate - Use multiplicity factor when counting database operations
• Use `aggregation.duration_avg` - Calculate aggregation.duration_sum / aggregation.count for accurate durations
• Filter by `span.kind == "client"` and isNotNull(db.namespace) to identify database spans
• Check `aggregation.count` - Indicates one span represents multiple operations
• Consider read sampling - Use samplingRatio for better performance on large datasets
• Include service context - Add getNodeName(dt.smartscape.service) to identify which service makes calls

Related Topics

• Sampling and Extrapolation - Detailed extrapolation formulas
• Performance Analysis - Duration analysis techniques

---

← Back to: Application Tracing Skill

Entity Lookups and Service Context

Enrich traces with Smartscape node metadata, correlate with infrastructure, and analyze performance by hardware.

Overview

Spans reference services via Smartscape node IDs. Service names and attributes are available through Dynatrace Smartscape (Entity Model) using node lookups. This enables enriching trace data with infrastructure context, analyzing performance by hardware, and correlating across multiple node types.

Service Node Basics

List Services

Fetch service nodes:

smartscapeNodes SERVICE | limit 100

Add Service Name to Spans

Enrich spans with service names:

fetch spans
| fieldsAdd getNodeName(dt.smartscape.service)
| summarize spans=count(), by: { dt.smartscape.service, dt.smartscape.service.name }
| sort spans desc

Filtering by Service

Using Smartscape Node Filters

Filter spans for specific service (Smartscape node filters and traverse should be used instead of the legacy classicEntitySelector()):

fetch spans
// Smartscape node filters and traverse should be used instead of classicEntitySelector

// Via subquery (recommended)
| filter dt.smartscape.service in [
    smartscapeNodes SERVICE
    | filter name == "BookStore.Books.dev"
    | fields id
]
| filter request.is_root_span == true
| summarize { requests=count(), avg(duration) }, by: { url.path }

Service Performance

Service and Endpoint Response Times

Analyze performance by service and endpoint:

fetch spans
| filter request.is_root_span == true
| fieldsAdd getNodeName(dt.smartscape.service)
| summarize {
    requests=count(),
    avg_response_time=avg(duration)
  }, by: { dt.smartscape.service, dt.smartscape.service.name, endpoint.name }
| sort dt.smartscape.service.name, endpoint.name
| sort requests desc

Advanced Node Lookups

Host, Process Group, and Service Attributes

Lookup multiple node types with attributes:

fetch spans
| filter isNotNull(dt.smartscape.host) and isNotNull(dt.smartscape.service) and isNotNull(dt.process_group.id)

// Add host information including hardware
| fieldsAdd host_name = getNodeName(dt.smartscape.host),
            getNodeField(dt.smartscape.host, "bitness"),
            getNodeField(dt.smartscape.host, "additionalSystemInfo")

// Flatten system info
| fieldsFlatten dt.smartscape.host.additionalSystemInfo

| fieldsAdd service_name = getNodeName(dt.smartscape.service)

// Add process group detected name and technologies (process_group is not a Smartscape node)
| fieldsAdd dt.process_group.detected_name

// Note: process_group attributes like softwareTechnologies have no Smartscape equivalent
// The dt.process_group.detected_name field is available directly on spans

| limit 1

Performance by Hardware

Analyze response times split by CPU type:

fetch spans
| filter request.is_root_span == true

// Add host information including hardware
| fieldsAdd getNodeName(dt.smartscape.host),
            getNodeField(dt.smartscape.host, "bitness"),
            getNodeField(dt.smartscape.host, "additionalSystemInfo")

// Extract CPU information
| fieldsAdd host.cpu = dt.smartscape.host.additionalSystemInfo[system.processor.model]

| filter isNotNull(host.cpu)

| summarize {
    count(),
    avg(duration)
  }, by: { dt.smartscape.service, endpoint.name, bitness=dt.smartscape.host.bitness, host.cpu }

Best Practices

• Use `getNodeName()` to add node names: getNodeName(dt.smartscape.service) adds dt.smartscape.service.name
• Use `getNodeField()` to access specific node attributes
• Use Smartscape node filters and `traverse` instead of classicEntitySelector() for better performance and readability
• Filter early - apply node filters before expensive operations
• Access nested attributes using bracket notation after fieldsFlatten
• Parse complex attributes when node attributes contain structured text (e.g., software technologies)

Failure Detection Analysis

Dynatrace applies failure detection rules to identify failed requests. Use these queries to analyze failure patterns and investigate root causes.

Key Attributes

• request.is_failed: true if request is considered failed
• dt.failure_detection.results[]: array of detection results
• dt.failure_detection.results[].reason: why failure was detected
• dt.failure_detection.results[].verdict: failure or success

Failure Reasons

Failure Overview

Failed Request Summary

fetch spans
| filter request.is_root_span == true
| summarize
    total = count(),
    failed = countIf(request.is_failed == true),
  by: { getNodeName(dt.smartscape.service) }
| fieldsAdd failure_rate = (failed * 100.0) / total
| sort failure_rate desc

Failures by Endpoint

fetch spans
| filter request.is_root_span == true and request.is_failed == true
| summarize failures = count(), by: { endpoint.name, http.route }
| sort failures desc

Failure Reason Analysis

Breakdown by Reason

fetch spans
| filter request.is_failed == true and isNotNull(dt.failure_detection.results)
| expand dt.failure_detection.results
| summarize count(), by: { dt.failure_detection.results[reason] }

HTTP Code Failures

fetch spans
| filter request.is_failed == true
| filter iAny(dt.failure_detection.results[][reason] == "http_code")
| summarize count(), by: { http.response.status_code, endpoint.name }
| sort `count()` desc

Exception-Based Failures

fetch spans
| filter request.is_failed == true
| filter iAny(dt.failure_detection.results[][reason] == "exception")
| expand span.events
| filter span.events[span_event.name] == "exception"
| summarize count(), by: { span.events[exception.type] }

Failure Patterns

Failure Rate Over Time

fetch spans
| filter request.is_root_span == true
| makeTimeseries
    total = count(),
    failed = countIf(request.is_failed == true),
  by: { getNodeName(dt.smartscape.service) }

Correlate Failures with Response Codes

fetch spans
| filter request.is_root_span == true
| summarize
    total = count(),
    failed = countIf(request.is_failed == true),
  by: { http.response.status_code }
| sort failed desc

Custom Rule Investigation

Custom Rule Matches

fetch spans
| filter request.is_failed == true
| filter iAny(dt.failure_detection.results[][reason] == "custom_rule")
| expand dt.failure_detection.results
| filter dt.failure_detection.results[reason] == "custom_rule"
| summarize count(), by: { dt.failure_detection.results[custom_rule_name] }

Failure Investigation

Recent Failed Requests

fetch spans
| filter request.is_root_span == true and request.is_failed == true
| fields
    start_time,
    trace.id,
    endpoint.name,
    http.response.status_code,
    duration
| sort start_time desc
| limit 100

Failed Requests with Verdict Details

fetch spans
| filter request.is_failed == true
| expand dt.failure_detection.results
| fields
    trace.id,
    endpoint.name,
    reason = dt.failure_detection.results[reason],
    verdict = dt.failure_detection.results[verdict]
| limit 50

Exception Analysis

Exceptions in distributed traces are stored as span.events within individual spans. DQL provides powerful ways to query, filter, and analyze exceptions including full-text search on messages and stack traces.

Finding Exceptions

Filter for spans containing exceptions:

fetch spans
| filter iAny(span.events[][span_event.name] == "exception")
| limit 100

Exclude Specific Exceptions

Filter out known exceptions:

fetch spans
| filter iAny(span.events[][span_event.name] == "exception")
| filter iAny(not contains(span.events[][exception.message], "404"))
| expand span.events
| fields span.events
| fieldsFlatten span.events
| fieldsRemove span.events
| limit 1

Stack Trace Analysis

Search for specific patterns in stack traces:

fetch spans
| filter iAny(contains(span.events[][exception.stack_trace], "invoke"))
| expand span.events
| fields span.events
| fieldsFlatten span.events
| fieldsRemove span.events
| limit 1

Exception Aggregations

Count by exception type:

fetch spans
| filter iAny(span.events[][span_event.name] == "exception")
| expand span.events
| fieldsFlatten span.events, fields: { exception.type }
| summarize count(), by: { exception.type }

Exception Count with Trace Exemplars

Include example trace IDs for investigation:

fetch spans
| filter iAny(span.events[][span_event.name] == "exception")
| expand span.events
| fieldsFlatten span.events, fields: { exception.type }
| summarize {
    count(),
    trace=takeAny(record(start_time, trace.id))
  }, by: { exception.type }
| fields exception.type, `count()`, trace.id=trace[trace.id], start_time=trace[start_time]

Exception Timeseries

Chart exception frequency over time:

fetch spans, from:now() - 24h
| filter iAny(span.events[][span_event.name] == "exception")
| expand span.events
| fieldsFlatten span.events, fields: { exception.type }
| makeTimeseries count(), by: { exception.type }

Parsing Exception Messages

Extract structured data from exception messages:

fetch spans, from:now() - 2h
| filter iAny(contains(span.events[][exception.message], "Book in Storage is not found by isbn"))
| expand span.events
| fields span.events
| fieldsFlatten span.events
| fieldsRemove span.events
// Parse code and ISBN from message like: "404 NOT_FOUND \"Book in Storage is not found by isbn: 9999999998823\""
| parse span.events.exception.message, "INT:code LD 'not found by isbn:' LD:isbn '\"'"
| summarize count(), by: { isbn, code }
| sort `count()` desc
| limit 10

Best Practices

• Use `iAny()` to check for conditions within failure detection arrays and span events
• Expand and flatten dt.failure_detection.results and span.events to access attributes
• Full-text search works on both exception.message and exception.stack_trace
• Include trace exemplars using takeAny(record(start_time, trace.id)) for drilldown
• Parse exception messages with DQL parse command to extract structured information
• Monitor failure rates by calculating percentage: (failed_requests / total_requests) * 100

---

← Back to: Application Tracing Skill

HTTP Span Analysis

HTTP spans capture web requests and API calls. Server spans represent incoming requests; client spans represent outgoing calls to external services.

Key Attributes

Server-Side Analysis

Incoming Request Volume

fetch spans
| filter span.kind == "server" and isNotNull(http.request.method)
| summarize
    requests = count(),
    avg_duration = avg(duration),
  by: { http.request.method, http.route }
| sort requests desc

Response Code Distribution

fetch spans
| filter span.kind == "server" and isNotNull(http.response.status_code)
| summarize count(), by: { http.response.status_code }
| sort http.response.status_code asc

Error Rates by Endpoint

fetch spans
| filter span.kind == "server" and isNotNull(http.route)
| summarize
    total = count(),
    errors = countIf(http.response.status_code >= 400),
  by: { http.route }
| fieldsAdd error_rate = (errors * 100.0) / total
| sort error_rate desc

5xx Server Errors

fetch spans
| filter span.kind == "server" and http.response.status_code >= 500
| summarize
    errors = count(),
    example_trace = takeAny(trace.id),
  by: { http.route, http.response.status_code }
| sort errors desc

Client-Side Analysis

Outgoing HTTP Calls

fetch spans
| filter span.kind == "client" and isNotNull(http.request.method)
| summarize
    calls = count(),
    avg_duration = avg(duration),
  by: { server.address, http.request.method }
| sort calls desc

External API Dependencies

fetch spans
| filter span.kind == "client" and isNotNull(url.full)
| fieldsAdd caller_service = getNodeName(dt.smartscape.service)
| summarize
    calls = count(),
    avg_latency = avg(duration),
    p99_latency = percentile(duration, 99),
  by: { caller_service, server.address, server.port }
| sort calls desc

Failed Outgoing Calls

fetch spans
| filter span.kind == "client" and http.response.status_code >= 400
| summarize count(), by: { server.address, http.response.status_code }
| sort `count()` desc

Performance Analysis

Endpoint Latency Percentiles

fetch spans
| filter span.kind == "server" and isNotNull(http.route)
| summarize
    p50 = median(duration),
    p95 = percentile(duration, 95),
    p99 = percentile(duration, 99),
  by: { http.route }
| sort p99 desc

Slow Requests (>1s)

fetch spans
| filter span.kind == "server" and duration > 1000000000
| fields start_time, trace.id, http.route, http.request.method, duration
| sort duration desc
| limit 50

Latency Timeseries

fetch spans
| filter span.kind == "server" and isNotNull(http.route)
| makeTimeseries
    avg_duration = avg(duration),
    p99_duration = percentile(duration, 99),
  by: { http.route }

Payload Analysis

Request/Response Sizes

fetch spans
| filter span.kind == "server"
| summarize
    avg_request_size = avg(http.request.body.size),
    avg_response_size = avg(http.response.body.size),
    max_response_size = max(http.response.body.size),
  by: { http.route }
| sort max_response_size desc

Large Responses

fetch spans
| filter http.response.body.size > 1000000
| fields trace.id, http.route, http.response.body.size
| sort http.response.body.size desc

Client IP Analysis

Requests by Client

fetch spans
| filter span.kind == "server" and isNotNull(client.ip)
| summarize requests = count(), by: { client.ip, client.isp }
| sort requests desc
| limit 100

Public vs Private Clients

fetch spans
| filter span.kind == "server" and isNotNull(client.ip)
| summarize count(), by: { client.ip.is_public }

Protocol Versions

HTTP Version Distribution

fetch spans
| filter isNotNull(network.protocol.version)
| summarize count(), by: { network.protocol.name, network.protocol.version }

Related Topics

• Performance Analysis - Response time analysis
• Failure Detection - Failure investigation

---

← Back to: Application Tracing Skill

Logs and Traces Correlation

Logs can be enriched with trace_id and span_id to enable correlation with distributed traces. OneAgent can automatically enrich logs with trace context.

Finding Logs with Trace Context

Logs with Trace IDs

List logs containing trace context:

fetch logs, from:now() - 2h
| filter isNotNull(trace_id)
| limit 10

Filtering Traces by Log Content

Find Traces from Log Search

Find traces containing specific log messages:

fetch spans, from:now() - 30m
| filter trace.id in [
    fetch logs
    | filter isNotNull(trace_id)
    | filter contains(content, "books returned")
    | fields toUid(trace_id)
]
| limit 1

Note: Subqueries in in statements have size limits. If the subquery result is too large, you'll get an IN_KEYWORD_TABLE_SIZE DQL error.

Analyze Spans Emitting Specific Logs

Find performance of spans that emitted logs with specific content:

fetch spans, from:now() - 30m
| filter span.id in [
    fetch logs
    | filter isNotNull(span_id)
    | filter contains(content, "J. K. Rowling")
    | fields toUid(span_id)
]

// Pick span name or code location
| fieldsAdd name = coalesce(span.name, concat(code.namespace, ".", code.function))

| summarize {
    count(),
    avg(duration),
    p99=percentile(duration, 99),
    trace.id=takeAny(trace.id)
  }, by: { k8s.pod.name, name }

Joining Spans and Logs

Basic Join

Join spans with logs on trace ID:

fetch spans, from:now() - 30m
| join [ fetch logs | fieldsAdd trace.id = toUid(trace_id) ]
  , on: { trace.id }
  , fields: { content, loglevel }
| fields start_time, trace.id, span.id, code=concat(code.namespace, ".", code.function), loglevel, content
| limit 100

Left-First Execution

Control join execution order for performance:

fetch spans, from:now() - 30m
| join [ fetch logs | fieldsAdd trace.id = toUid(trace_id) ]
  , on:{ trace.id }
  , fields: { content, loglevel }
  , executionOrder:leftFirst
| fields start_time, trace.id, span.id, code=concat(code.namespace, ".", code.function), loglevel, content
| limit 100

Correlation Patterns

Logs for Failed Requests

Find logs associated with failed requests:

fetch logs, from:now() - 1h
| filter isNotNull(trace_id)
| filter trace_id in [
    fetch spans
    | filter request.is_root_span == true
    | filter request.is_failed == true
    | fields toString(trace.id)
]
| fields timestamp, loglevel, content, trace_id
| sort timestamp desc
| limit 100

Exception Logs with Trace Context

Correlate error logs with their traces:

fetch logs, from:now() - 1h
| filter loglevel == "ERROR"
| filter isNotNull(trace_id)
| fieldsAdd trace_id_uid = toUid(trace_id)
| join [
    fetch spans
    | filter request.is_root_span == true
    | fields trace.id, endpoint.name, duration
  ]
  , on: { left[trace_id_uid] == right[trace.id] }
  , fields: { endpoint.name, duration }
| fields timestamp, content, endpoint.name, duration, trace_id
| sort timestamp desc
| limit 50

Best Practices

• Convert trace_id from string to UID using toUid() when joining
• Be aware of subquery limits - large in subqueries may fail with IN_KEYWORD_TABLE_SIZE error
• Use `executionOrder:leftFirst` to optimize join performance when left side is smaller
• Filter logs early - apply log filters before joining with spans
• Include `trace_id` and `span_id` in log output for troubleshooting
• OneAgent auto-enrichment - configure OneAgent to automatically add trace context to logs

Related Topics

• Failure Detection - Investigate failures using logs and traces

---

← Back to: Application Tracing Skill

Messaging Span Analysis

Messaging spans capture async communication via Kafka, RabbitMQ, SQS, and other message brokers. Use these queries to monitor message processing health.

Key Attributes

• messaging.system: broker type (kafka, rabbitmq, aws_sqs, etc.)
• messaging.destination.name: queue/topic name
• messaging.operation.type: publish, receive, process
• messaging.batch.message_count: messages in batch operations
• span.kind: producer (send) or consumer (receive/process)

Message Throughput

Messages by Destination

fetch spans
| filter isNotNull(messaging.system)
| summarize
    spans = count(),
    messages = sum(coalesce(messaging.batch.message_count, 1)),
  by: { messaging.system, messaging.destination.name, messaging.operation.type }
| sort messages desc

Producer vs Consumer Volume

fetch spans
| filter isNotNull(messaging.system)
| summarize messages = sum(coalesce(messaging.batch.message_count, 1)),
  by: { span.kind, messaging.system }

Kafka Analysis

Consumer Group Lag Indicators

Identify slow consumer groups:

fetch spans
| filter messaging.system == "kafka" and messaging.operation.type == "process"
| summarize
    processed = count(),
    avg_duration = avg(duration),
    p99_duration = percentile(duration, 99),
  by: { messaging.consumer.group.name, messaging.destination.name }
| sort p99_duration desc

Partition Distribution

fetch spans
| filter messaging.system == "kafka"
| summarize count(), by: { messaging.destination.name, messaging.destination.partition.id }
| sort `count()` desc

Message Processing Health

Failed Message Processing

fetch spans
| filter messaging.operation.type == "process"
| summarize
    total = count(),
    failed = countIf(messaging.is_failed == true),
  by: { messaging.destination.name }
| fieldsAdd failure_rate = (failed * 100.0) / total
| sort failure_rate desc

Batch Processing Failures

fetch spans
| filter isNotNull(messaging.batch.message_count) and messaging.batch.failed_count > 0
| fields
    start_time,
    trace.id,
    messaging.destination.name,
    messaging.batch.message_count,
    messaging.batch.failed_count,
    messaging.batch.failure_codes

Processing Latency

End-to-End Message Latency

fetch spans
| filter messaging.operation.type == "process"
| summarize
    avg_latency = avg(duration),
    p95_latency = percentile(duration, 95),
    p99_latency = percentile(duration, 99),
  by: { messaging.system, messaging.destination.name }
| sort p99_latency desc

Latency Timeseries

fetch spans
| filter messaging.operation.type == "process"
| makeTimeseries
    avg_duration = avg(duration),
    p99_duration = percentile(duration, 99),
  by: { messaging.destination.name }

Broker Connectivity

Messages by Broker Server

fetch spans
| filter isNotNull(messaging.system)
| summarize
    spans = count(),
    services = countDistinct(dt.smartscape.service),
  by: { server.address, server.port, messaging.system }

Best Practices

• Use `messaging.batch.message_count` for accurate message counts (defaults to 1)
• Filter by `messaging.operation.type` to distinguish publish vs process operations
• Monitor p99 latency for consumer groups to detect lag
• Track failure rates by destination to identify problematic queues
• Analyze partition distribution for Kafka to ensure balanced consumption

Related Topics

• Performance Analysis - Latency analysis techniques
• Failure Detection - Error investigation

---

← Back to: Application Tracing Skill

Networking and IP Analysis

Trace data contains networking information including IP addresses, server addresses, and DNS resolution results.

Server Addresses and IPs

Outgoing Request Destinations

Collect all server addresses and resolved IPs:

fetch spans, from:now() - 30m
| filter isNotNull(server.resolved_ips)

// Collect all server IP addresses
| summarize {
    ips=collectDistinct(arraySort(server.resolved_ips))
  }, by: { server.address, server.port }

// Double expand: ips is array of arrays, flatten to list
| expand ips
| expand ips
| sort ips

Note: server.resolved_ips contains array of IP addresses from DNS lookup at request time.

Outgoing Requests by Service

Show which services connect to which servers:

fetch spans, from:now() - 30m
| filter isNotNull(server.resolved_ips)
| fieldsAdd getNodeName(dt.smartscape.service)

| summarize {
    count(),
    ips=collectDistinct(server.resolved_ips)
  }, by: {
    k8s.namespace.name,
    dt.smartscape.service.name,
    server.address,
    operation=coalesce(span.name, concat(code.namespace, ".", code.function)),
    span.kind
  }
| expand ips

Client IP Analysis

Client IP Count by Service

Analyze incoming client IPs with masking:

fetch spans, from:now() - 24h
| filter isNotNull(client.ip)

// Convert to structured IP type
| fieldsAdd client.ip = toIp(client.ip)

// Mask to not expose full IP
| fieldsAdd client.ip.masked = ipMask(client.ip, 16)

| fieldsAdd getNodeName(dt.smartscape.service)

| summarize {
    distinct_clients = countDistinct(client.ip),
    masked_client_ips=toString(arraySort(collectDistinct(client.ip.masked)))
  }, by: { k8s.namespace.name, dt.smartscape.service.name, span.kind }
| sort distinct_clients desc

Geographic Distribution

Analyze client IPs by subnet:

fetch spans, from:now() - 6h
| filter isNotNull(client.ip)
| filter request.is_root_span == true

// Convert and mask IPs
| fieldsAdd client.ip = toIp(client.ip)
| fieldsAdd client.subnet = ipMask(client.ip, 24)

| summarize {
    requests=count(),
    unique_clients=countDistinct(client.ip)
  }, by: { client.subnet, endpoint.name }
| sort requests desc
| limit 100

IP Functions

Working with IP Addresses

DQL provides IP manipulation functions:

fetch spans
| filter isNotNull(client.ip)

// Convert string to IP type
| fieldsAdd client.ip = toIp(client.ip)

// Mask IP to subnet (bits to keep)
| fieldsAdd subnet_24 = ipMask(client.ip, 24)  // /24 network
| fieldsAdd subnet_16 = ipMask(client.ip, 16)  // /16 network

| fields client.ip, subnet_24, subnet_16
| limit 10

Network Attributes

Common networking attributes:

• server.address - Server hostname/address for outgoing calls
• server.port - Server port
• server.resolved_ips - Array of IPs from DNS resolution
• client.ip - Client IP address for incoming calls
• client.port - Client port
• network.protocol.name - Protocol name (e.g., http, amqp)
• network.transport - Transport layer (tcp, udp, pipe)

Service Communication Map

Identify Service Dependencies

Map service-to-service communication:

fetch spans, from:now() - 1h
| filter span.kind == "client"
| filter isNotNull(server.address)

| fieldsAdd caller_service = getNodeName(dt.smartscape.service)

| summarize {
    calls=count(),
    avg_duration=avg(duration),
    resolved_ips=toString(collectDistinct(server.resolved_ips))
  }, by: {
    caller_service,
    server.address,
    server.port
  }
| sort calls desc
| limit 50

Best Practices

• Use `toIp()` to convert string IPs to structured IP type for IP functions
• Use `ipMask()` to mask IPs for privacy (specify bits to keep)
• `server.resolved_ips` is an array - use expand or collectDistinct() to work with values
• Double expand array of arrays: | expand ips | expand ips
• Filter by `span.kind` - "client" spans have server.address, "server" spans have client.ip
• Convert to string for display: toString(arraySort(collectDistinct(...)))
• Combine with service context using getNodeName(dt.smartscape.service)

---

← Back to: Application Tracing Skill

Performance Analysis

Analyze response times, percentiles, and create timeseries from span data using DQL aggregation functions.

Response Time Analysis

Service Response Times with Percentiles

Calculate average, median, and percentiles:

fetch spans
| filter contains(http.route, "storage")
| summarize {
    count(),
    avg=avg(duration),
    p50=median(duration),
    p99=percentile(duration, 99)
  }, by: { http.request.method, http.route }

Response Time Buckets

Group requests into duration buckets with trace exemplars:

fetch spans, from:now() - 24h
| filter http.route == "/api/v1/storage/findByISBN"
| summarize {
    spans=count(),
    trace=takeAny(record(start_time, trace.id))
  }, by: { bin(duration, 10ms) }
| fields `bin(duration, 10ms)`, spans, trace.id=trace[trace.id], start_time=trace[start_time]

This creates 10ms buckets and captures an example trace from each bucket for investigation.

Timeseries Extraction

Basic Timeseries

Extract average response time as timeseries:

fetch spans, from:now() - 24h
| filter http.route == "/api/v1/storage/findByISBN"
| makeTimeseries { avg=avg(duration) }, by: { http.route }, bins:250

Multi-Metric Timeseries

Create timeseries with multiple metrics:

fetch spans, from:now() - 24h
| filter request.is_root_span == true
| makeTimeseries {
    requests=count(),
    avg_duration=avg(duration),
    p95=percentile(duration, 95),
    p99=percentile(duration, 99)
  }, by: { endpoint.name }

Failed Requests Over Time

Chart failure rates as timeseries:

fetch spans, from:now() - 7d
| filter request.is_root_span == true
| makeTimeseries {
    failed_requests=countIf(request.is_failed == true)
  }, by: {endpoint.name}

Endpoint Performance

Top Slow Endpoints

Identify slowest endpoints:

fetch spans, from:now() - 1h
| filter request.is_root_span == true
| summarize {
    requests=count(),
    avg_duration=avg(duration),
    p95=percentile(duration, 95),
    p99=percentile(duration, 99)
  }, by: { endpoint.name }
| sort p99 desc
| limit 10

Performance by Service and Endpoint

Break down performance by service:

fetch spans
| filter request.is_root_span == true
| fieldsAdd getNodeName(dt.smartscape.service)
| summarize {
    requests=count(),
    avg_response_time=avg(duration)
  }, by: { dt.smartscape.service, dt.smartscape.service.name, endpoint.name }
| sort requests desc

Best Practices

• Use percentiles (p50, p95, p99) over averages for better performance insights
• Include trace exemplars with takeAny(record(start_time, trace.id)) for drilldown capability
• Use `bin()` to create duration buckets for distribution analysis
• Set appropriate `bins` parameter in makeTimeseries (default varies, 250 is common)
• Filter by `request.is_root_span` when analyzing end-to-end request performance
• Combine with service context using getNodeName(dt.smartscape.service) for service-level analysis

---

← Back to: Application Tracing Skill

Request Attributes and Request-Level Analysis

Query request attributes, captured attributes, and aggregate spans by request.

Overview

Requests in Dynatrace are represented by spans marked as request roots (request.is_root_span: true). These represent incoming calls to services. Request attributes and captured attributes provide custom metadata for request-level analysis, while request.id enables aggregation of all spans within a single request.

Request Root Spans

List Request Roots

Fetch individual request root spans:

fetch spans
| filter request.is_root_span == true
| fields trace.id, span.id, start_time, response_time = duration, endpoint.name
| limit 100

Failed Request Analysis

Chart failed requests over time:

fetch spans, from:now() - 7d
| filter request.is_root_span == true
| makeTimeseries {
    failed_requests=countIf(request.is_failed == true)
  }, by: {endpoint.name}

Filter for specific endpoint:

fetch spans
| filter request.is_root_span == true
| filter endpoint.name == "/api/v1/payment"
| filter request.is_failed == true
| fields trace.id, endpoint.name, duration, start_time
| limit 100

Request Aggregation

Aggregate all spans belonging to a request using request.id. All spans in a request carry this ID.

Note: request.id is only available for OneAgent-based traces, not API-ingested traces.

Aggregated Request Metrics

Calculate metrics across all spans in each request:

fetch spans
| filter isNotNull(request.id)
| summarize {
    spans = count(),
    client_spans = countIf(span.kind == "client"),
    span_events = sum(arraySize(span.events)),

    // Select the request root span
    request_root = takeMin(record(
        root_detection_helper = coalesce(if(request.is_root_span, 1), 2),
        start_time, endpoint.name, duration
      ))
}, by: { trace.id, request.id }

// Reset to NULL if root not found
| fieldsAdd request_root=if(request_root[root_detection_helper] < 2, request_root)
| fieldsFlatten request_root
| fieldsRemove request_root.root_detection_helper, request_root

| fields
    start_time = request_root.start_time,
    endpoint = request_root.endpoint.name,
    response_time = request_root.duration,
    spans,
    client_spans,
    span_events,
    trace.id

| limit 100

This query:

• Counts total spans per request
• Counts outgoing calls (client spans)
• Sums span events (e.g., exceptions)
• Extracts request root span details

Request Performance by Endpoint

Response Time Statistics

Analyze endpoint performance:

fetch spans
| filter request.is_root_span == true
| summarize {
    requests=count(),
    avg_duration=avg(duration),
    p95=percentile(duration, 95),
    p99=percentile(duration, 99),
    failed=countIf(request.is_failed == true)
  }, by: { endpoint.name }
| fieldsAdd failure_rate = (failed * 100.0) / requests
| sort p99 desc

Service Request Breakdown

Requests by service and endpoint:

fetch spans
| filter request.is_root_span == true
| fieldsAdd getNodeName(dt.smartscape.service)
| summarize {
    requests=count(),
    failed=countIf(request.is_failed == true),
    avg_duration=avg(duration)
  }, by: { dt.smartscape.service.name, endpoint.name }
| fieldsAdd failure_rate = (failed * 100.0) / requests
| sort requests desc

Request Attributes

Request attributes appear on request root spans with the key request_attribute.<name>:

fetch spans
| filter request.is_root_span == true
| filter isNotNull(request_attribute.PaidAmount)
| makeTimeseries sum(request_attribute.PaidAmount)

For attributes with special characters, use backticks:

fetch spans
| filter isNotNull(`request_attribute.My Customer ID`)

Captured Attributes

Attributes captured from method parameters appear as captured_attribute.<name> (always as arrays):

fetch spans
| filter isNotNull(captured_attribute.BookID_purchased)
| fields trace.id, span.id, code.namespace, code.function, captured_attribute.BookID_purchased
| limit 1

Best Practices

• Filter for request roots using request.is_root_span == true
• Check `request.is_failed` to identify failed requests
• Use `request.id` to aggregate spans within a single request (OneAgent traces only)
• Use `takeMin(record(...))` with detection helper to reliably extract request root from aggregation
• Monitor failure rates by calculating percentage: (failed_requests / total_requests) * 100
• Include endpoint name for meaningful breakdowns
• Request attributes data types depend on configuration ("All values" creates arrays)

RPC Span Analysis

Remote Procedure Call (RPC) spans cover gRPC, SOAP, Java RMI, and other RPC frameworks. These queries help monitor inter-service communication.

Key Attributes

• rpc.system: framework (grpc, jax_ws, dotnet_wcf, etc.)
• rpc.service: service name being called
• rpc.method: method invoked
• rpc.grpc.status_code: gRPC-specific status
• network.protocol.name: protocol (grpc, soap, rest_http)

RPC Traffic Overview

Calls by Service and Method

fetch spans
| filter isNotNull(rpc.system)
| summarize
    calls = count(),
    avg_duration = avg(duration),
  by: { rpc.system, rpc.service, rpc.method }
| sort calls desc

Client vs Server Spans

fetch spans
| filter isNotNull(rpc.system)
| summarize count(), by: { span.kind, rpc.system }

gRPC Analysis

gRPC Status Code Distribution

fetch spans
| filter rpc.system == "grpc"
| summarize count(), by: { rpc.grpc.status_code, rpc.service, rpc.method }
| sort `count()` desc

Common gRPC status codes:

• 0: OK
• 2: UNKNOWN
• 4: DEADLINE_EXCEEDED
• 13: INTERNAL
• 14: UNAVAILABLE

gRPC Errors

fetch spans
| filter rpc.system == "grpc" and rpc.grpc.status_code != 0
| summarize
    errors = count(),
    example_trace = takeAny(trace.id),
  by: { rpc.service, rpc.method, rpc.grpc.status_code }
| sort errors desc

gRPC Latency by Method

fetch spans
| filter rpc.system == "grpc" and span.kind == "server"
| summarize
    calls = count(),
    avg_latency = avg(duration),
    p99_latency = percentile(duration, 99),
  by: { rpc.service, rpc.method }
| sort p99_latency desc

SOAP/Web Services

SOAP Operations

fetch spans
| filter contains(toString(rpc.system), "ws") or network.protocol.name == "soap"
| summarize count(), by: { rpc.service, rpc.method, rpc.namespace }

Service Dependencies

RPC Call Graph

Identify service-to-service RPC dependencies:

fetch spans
| filter isNotNull(rpc.system) and span.kind == "client"
| fieldsAdd caller = getNodeName(dt.smartscape.service)
| summarize
    calls = count(),
    avg_latency = avg(duration),
  by: { caller, server.address, rpc.service }
| sort calls desc

Server Endpoints

fetch spans
| filter isNotNull(rpc.system) and span.kind == "server"
| fieldsAdd service = getNodeName(dt.smartscape.service)
| summarize
    requests = count(),
    p95_latency = percentile(duration, 95),
  by: { service, rpc.service, rpc.method }

Performance Monitoring

RPC Latency Timeseries

fetch spans
| filter isNotNull(rpc.system) and span.kind == "server"
| makeTimeseries
    avg_duration = avg(duration),
    p99_duration = percentile(duration, 99),
    calls = count(),
  by: { rpc.service }

Slow RPC Calls

fetch spans
| filter isNotNull(rpc.system) and duration > 1000000000
| fields start_time, trace.id, rpc.service, rpc.method, duration, server.address
| sort duration desc
| limit 50

Best Practices

• Filter by `span.kind` - "client" spans show outgoing calls, "server" spans show incoming
• Monitor gRPC status codes - Non-zero codes indicate errors
• Track latency by method - Identify slow RPC operations
• Map service dependencies - Use client spans to understand service call graphs
• Include service context - Add getNodeName(dt.smartscape.service) for clarity

Related Topics

• Performance Analysis - Latency analysis
• Failure Detection - Error investigation

---

← Back to: Application Tracing Skill

Sampling and Extrapolation

One span can represent multiple real operations due to aggregation or sampling. Extrapolation is needed to accurately count operations.

Types of Sampling

Aggregation

Certain operations (especially database calls) can be aggregated into a single span. Aggregated spans contain aggregation.count attribute.

Adaptive Traffic Management (ATM)

Rate-limiting, head-based sampling that adaptively reacts to request rates. Decision made at trace start by the agent.

Adaptive Load Reduction (ALR)

Server-side sampling to protect backend infrastructure from overload.

Read Sampling

Control data volume read in queries via samplingRatio parameter:

• Available rates: 1, 10, 100, 1000, 10000, 100000
• 1 = 100% of data, 100 = 1% of data
• Actual ratio accessible via dt.system.sampling_ratio
• Sampling is trace-aware: either all spans of a trace or none

Extrapolation Formula

Calculate multiplicity factor to extrapolate span counts to actual operation counts:

fetch spans, from:now() - 1h
| filter request.is_root_span == true
| fieldsAdd sampling.probability = (power(2, 56) - coalesce(sampling.threshold, 0)) * power(2, -56)
| fieldsAdd sampling.multiplicity = 1 / sampling.probability
| fieldsAdd multiplicity = coalesce(sampling.multiplicity, 1)
                         * coalesce(aggregation.count, 1)
                         * dt.system.sampling_ratio
| limit 10

Request Count Extrapolation

Extrapolated Request Counting

Count requests with proper extrapolation:

fetch spans
, samplingRatio:100  // Read only 1% of data

| filter request.is_root_span == true

// Calculate multiplicity factor
| fieldsAdd sampling.probability = (power(2, 56) - coalesce(sampling.threshold, 0)) * power(2, -56)
| fieldsAdd sampling.multiplicity = 1/sampling.probability
| fieldsAdd multiplicity = coalesce(sampling.multiplicity, 1)
                         * coalesce(aggregation.count, 1)
                         * dt.system.sampling_ratio

| summarize
    span_count=count(),
    request_count_extrapolated = sum(multiplicity)

Database Call Extrapolation

Database Operations with Extrapolation

Count and time database calls accurately:

fetch spans
, samplingRatio:100  // Read only 1% of data

| filter isNotNull(db.query.text)

// Calculate multiplicity factor
| fieldsAdd sampling.probability = (power(2, 56) - coalesce(sampling.threshold, 0)) * power(2, -56)
| fieldsAdd sampling.multiplicity = 1 / sampling.probability
| fieldsAdd multiplicity = coalesce(sampling.multiplicity, 1)
                         * coalesce(aggregation.count, 1)
                         * dt.system.sampling_ratio

// Calculate average duration for aggregated spans
| fieldsAdd aggregation.duration_avg = coalesce(aggregation.duration_sum / aggregation.count, duration)

| summarize {
    operation_count_extrapolated = sum(multiplicity),
    operation_duration_extrapolated = sum(aggregation.duration_avg * multiplicity) / sum(multiplicity)
}

Working with Aggregated Spans

Duration Calculation

For aggregated database spans, calculate average duration:

fetch spans, from:now() - 1h
| filter isNotNull(db.query.text)
| fieldsAdd aggregation.duration_avg = coalesce(
    aggregation.duration_sum / aggregation.count,
    duration
)
| limit 10

Database Analysis by Statement

Extrapolated database calls per service:

fetch spans
| filter span.kind == "client" and isNotNull(db.namespace)
| fieldsAdd getNodeName(dt.smartscape.service)

// Calculate multiplicity
| fieldsAdd sampling.probability = (power(2, 56) - coalesce(sampling.threshold, 0)) * power(2, -56)
| fieldsAdd sampling.multiplicity = 1/sampling.probability
| fieldsAdd multiplicity = coalesce(sampling.multiplicity, 1)
                         * coalesce(aggregation.count, 1)
                         * dt.system.sampling_ratio

| summarize {
    db_calls = sum(multiplicity)
  }, by: { dt.smartscape.service.name, code.function, db.system, db.namespace, db.query.text }
| sort db_calls desc
| limit 100

Best Practices

• Always extrapolate when counting operations (not just spans)
• Use `samplingRatio` parameter to reduce data read for better performance
• Check for `aggregation.count` to identify aggregated spans
• Calculate `multiplicity` as product of: sampling multiplicity × aggregation count × read sampling ratio
• Use `aggregation.duration_avg` for duration analysis on aggregated spans
• Fallback to `duration` when aggregation.duration_sum is not present
• Read sampling is trace-aware - you get complete traces or none at a given ratio

---

← Back to: Application Tracing Skill

Serverless (FaaS) Span Analysis

Function-as-a-Service spans capture AWS Lambda, Azure Functions, and GCP Cloud Functions execution. Monitor cold starts, invocations, and performance.

Key Attributes

• faas.name: function name
• faas.coldstart: true if cold start
• faas.trigger: invocation trigger type
• cloud.provider: aws, azure, gcp
• aws.region / azure.location / gcp.region: deployment region

Function Invocations

Invocations by Function

fetch spans
| filter isNotNull(faas.name) and span.kind == "server"
| summarize
    invocations = count(),
    avg_duration = avg(duration),
    p99_duration = percentile(duration, 99),
  by: { faas.name, cloud.provider }
| sort invocations desc

Invocations by Trigger Type

fetch spans
| filter isNotNull(faas.name)
| summarize count(), by: { faas.trigger, faas.name }

Common triggers: http, pubsub, datasource, timer, other

Cold Start Analysis

Cold Start Rate

fetch spans
| filter isNotNull(faas.name) and span.kind == "server"
| summarize
    total = count(),
    cold_starts = countIf(faas.coldstart == true),
  by: { faas.name }
| fieldsAdd cold_start_rate = (cold_starts * 100.0) / total
| sort cold_start_rate desc

Cold Start Duration Impact

fetch spans
| filter isNotNull(faas.name)
| fieldsAdd
    cold_duration = if(faas.coldstart == true, duration, else: null),
    warm_duration = if(faas.coldstart != true, duration, else: null)
| summarize
    avg_cold = avg(cold_duration),
    avg_warm = avg(warm_duration),
  by: { faas.name }
| fieldsAdd cold_start_overhead = avg_cold - avg_warm

Cold Starts Over Time

fetch spans
| filter isNotNull(faas.name)
| makeTimeseries
    cold_starts = countIf(faas.coldstart == true),
    warm_starts = countIf(faas.coldstart != true),
  by: { faas.name }

Cloud Provider Analysis

AWS Lambda

fetch spans
| filter isNotNull(aws.arn) and contains(aws.arn, ":function:")
| summarize
    invocations = count(),
    avg_duration = avg(duration),
  by: { faas.name, aws.region }

Azure Functions

fetch spans
| filter isNotNull(azure.site_name)
| summarize count(), by: { azure.site_name, azure.location }

GCP Cloud Functions

fetch spans
| filter isNotNull(gcp.resource.name) and contains(gcp.resource.name, "cloudfunctions")
| summarize count(), by: { faas.name, gcp.region }

Performance Monitoring

Function Duration Percentiles

fetch spans
| filter isNotNull(faas.name) and span.kind == "server"
| summarize
    p50 = median(duration),
    p95 = percentile(duration, 95),
    p99 = percentile(duration, 99),
  by: { faas.name }
| sort p99 desc

Memory Configuration

fetch spans
| filter isNotNull(faas.name) and isNotNull(faas.max_memory)
| summarize
    avg_duration = avg(duration),
    max_memory_mb = max(faas.max_memory) / 1048576,
  by: { faas.name }

Event Source Tracing

Event-Triggered Functions

fetch spans
| filter isNotNull(faas.event_source)
| summarize count(), by: { faas.event_source, faas.event_name, faas.name }

Document/Data Triggers

fetch spans
| filter faas.trigger == "datasource"
| summarize count(), by: { faas.document.operation, faas.document.collection }

Error Analysis

Failed Function Invocations

fetch spans
| filter isNotNull(faas.name) and request.is_failed == true
| summarize failures = count(), by: { faas.name, faas.trigger }
| sort failures desc

Best Practices

• Monitor cold start rates - High rates indicate configuration issues
• Compare cold vs warm duration - Quantify cold start overhead
• Track invocations by trigger - Understand function usage patterns
• Filter by `span.kind == "server"` for function entry points
• Analyze by region - Identify geographic performance differences
• Monitor failure rates - Track function reliability

Related Topics

• Performance Analysis - Duration analysis
• Failure Detection - Error investigation

---

← Back to: Application Tracing Skill