Datadog

Datadog Observability

Overview

Datadog is a SaaS observability platform providing unified monitoring across infrastructure, applications, logs, and user experience. It offers AI-powered anomaly detection, 1000+ integrations, and OpenTelemetry compatibility.

Core Capabilities:

• APM: Distributed tracing with automatic instrumentation for 8+ languages
• Infrastructure: Host, container, and cloud service monitoring
• Logs: Centralized collection with processing pipelines and 15-month retention
• Metrics: Custom metrics via DogStatsD with cardinality management
• Synthetics: Proactive API and browser testing from 29+ global locations
• RUM: Frontend performance with Core Web Vitals and session replay

When to Use This Skill

Activate when:

• Setting up production monitoring and observability
• Implementing distributed tracing across microservices
• Configuring log aggregation and analysis pipelines
• Creating custom metrics and dashboards
• Setting up alerting and anomaly detection
• Optimizing Datadog costs

Do not use when:

• Building with open-source stack (use Prometheus/Grafana instead)
• Cost is primary concern and budget is limited
• Need maximum customization over managed solution

Quick Start

1. Install Datadog Agent

Docker (simplest):

docker run -d --name dd-agent \
  -e DD_API_KEY=<YOUR_API_KEY> \
  -e DD_SITE="datadoghq.com" \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /proc/:/host/proc/:ro \
  -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
  gcr.io/datadoghq/agent:7

Kubernetes (Helm):

helm repo add datadog https://helm.datadoghq.com
helm install datadog-agent datadog/datadog \
  --set datadog.apiKey=<YOUR_API_KEY> \
  --set datadog.apm.enabled=true \
  --set datadog.logs.enabled=true

2. Instrument Your Application

Python:

from ddtrace import tracer, patch_all

# Automatic instrumentation for common libraries
patch_all()

# Manual span for custom operations
with tracer.trace("custom.operation", service="my-service") as span:
    span.set_tag("user.id", user_id)
    # your code here

Node.js:

// Must be first import
const tracer = require('dd-trace').init({
  service: 'my-service',
  env: 'production',
  version: '1.0.0',
});

3. Verify in Datadog UI

1. Go to Infrastructure > Host Map to verify agent 2. Go to APM > Services to see traced services 3. Go to Logs > Search to verify log collection

Core Concepts

Tagging Strategy

Tags enable filtering, aggregation, and cost attribution. Use consistent tags across all telemetry.

Required Tags:

Avoid High-Cardinality Tags:

• User IDs, request IDs, timestamps
• Pod IDs in Kubernetes
• Build numbers, commit hashes

Unified Observability

Datadog correlates metrics, traces, and logs automatically:

• Traces include span tags that link to metrics
• Logs inject trace IDs for correlation
• Dashboards combine all data sources

Best Practices

Start Simple

1. Install Agent with basic configuration 2. Enable automatic instrumentation 3. Verify data in Datadog UI 4. Add custom spans/metrics as needed

Progressive Enhancement

Basic → APM tracing → Custom spans → Custom metrics → Profiling → RUM

Key Instrumentation Points

• HTTP entry/exit points
• Database queries
• External service calls
• Message queue operations
• Business-critical flows

Common Mistakes

1. High-cardinality tags: Using user IDs or request IDs as tags creates millions of unique metrics 2. Missing log index quotas: Leads to unexpected bills from log volume spikes 3. Over-alerting: Creates alert fatigue; alert on symptoms, not causes 4. Missing service tags: Prevents correlation between metrics, traces, and logs 5. No sampling for high-volume traces: Ingests everything, causing cost explosion

Navigation

For detailed implementation:

• [Agent Installation](references/agent-installation.md): Docker, Kubernetes, Linux, Windows, and cloud-specific setup
• [APM Instrumentation](references/apm-instrumentation.md): Python, Node.js, Go, Java instrumentation with code examples
• [Log Management](references/log-management.md): Pipelines, Grok parsing, standard attributes, archives
• [Custom Metrics](references/custom-metrics.md): DogStatsD patterns, metric types, tagging best practices
• [Alerting](references/alerting.md): Monitor types, anomaly detection, alert hygiene
• [Cost Optimization](references/cost-optimization.md): Metrics without Limits, sampling, index quotas
• [Kubernetes](references/kubernetes.md): DaemonSet, Cluster Agent, autodiscovery

Complementary Skills

When using this skill, consider these related skills (if deployed):

• docker: Container instrumentation patterns
• kubernetes: K8s-native monitoring patterns
• python/nodejs/go: Language-specific APM setup

Resources

Official Documentation:

• APM: https://docs.datadoghq.com/tracing/
• Logs: https://docs.datadoghq.com/logs/
• Metrics: https://docs.datadoghq.com/metrics/
• DogStatsD: https://docs.datadoghq.com/developers/dogstatsd/

Cost Management:

• Billing: https://docs.datadoghq.com/account_management/billing/
• Usage Attribution: https://docs.datadoghq.com/account_management/billing/usage_attribution/

Agent Installation

The Datadog Agent collects metrics, traces, and logs from your infrastructure and applications.

Docker Installation

Basic Setup:

docker run -d --name dd-agent \
  -e DD_API_KEY=<YOUR_API_KEY> \
  -e DD_SITE="datadoghq.com" \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /proc/:/host/proc/:ro \
  -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
  gcr.io/datadoghq/agent:7

With APM and Logs:

docker run -d --name dd-agent \
  -e DD_API_KEY=<YOUR_API_KEY> \
  -e DD_SITE="datadoghq.com" \
  -e DD_APM_ENABLED=true \
  -e DD_APM_NON_LOCAL_TRAFFIC=true \
  -e DD_LOGS_ENABLED=true \
  -e DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /proc/:/host/proc/:ro \
  -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
  -p 8126:8126/tcp \
  -p 8125:8125/udp \
  gcr.io/datadoghq/agent:7

Docker Compose:

version: '3.8'
services:
  datadog-agent:
    image: gcr.io/datadoghq/agent:7
    environment:
      - DD_API_KEY=${DD_API_KEY}
      - DD_SITE=datadoghq.com
      - DD_APM_ENABLED=true
      - DD_APM_NON_LOCAL_TRAFFIC=true
      - DD_LOGS_ENABLED=true
      - DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true
      - DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
    ports:
      - "8126:8126"  # APM traces
      - "8125:8125/udp"  # DogStatsD metrics

Kubernetes Installation (Helm)

Add Helm Repository:

helm repo add datadog https://helm.datadoghq.com
helm repo update

Basic Installation:

helm install datadog-agent datadog/datadog \
  --set datadog.apiKey=<YOUR_API_KEY> \
  --set datadog.site=datadoghq.com

Full-Featured Installation:

helm install datadog-agent datadog/datadog \
  --set datadog.apiKey=<YOUR_API_KEY> \
  --set datadog.site=datadoghq.com \
  --set datadog.apm.portEnabled=true \
  --set datadog.logs.enabled=true \
  --set datadog.logs.containerCollectAll=true \
  --set datadog.processAgent.enabled=true \
  --set datadog.networkMonitoring.enabled=true \
  --set clusterAgent.enabled=true \
  --set clusterAgent.metricsProvider.enabled=true

Using values.yaml:

# datadog-values.yaml
datadog:
  apiKey: <YOUR_API_KEY>
  site: datadoghq.com

  # APM
  apm:
    portEnabled: true
    socketEnabled: true

  # Logs
  logs:
    enabled: true
    containerCollectAll: true

  # Process monitoring
  processAgent:
    enabled: true
    processCollection: true

  # Network monitoring
  networkMonitoring:
    enabled: true

# Cluster Agent for Kubernetes metrics
clusterAgent:
  enabled: true
  metricsProvider:
    enabled: true

# Node Agent resources
agents:
  resources:
    requests:
      cpu: 200m
      memory: 256Mi
    limits:
      cpu: 200m
      memory: 256Mi

helm install datadog-agent datadog/datadog -f datadog-values.yaml

Linux Package Installation

Debian/Ubuntu:

# Add Datadog repository
DD_API_KEY=<YOUR_API_KEY> DD_SITE="datadoghq.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)"

RHEL/CentOS/Amazon Linux:

DD_API_KEY=<YOUR_API_KEY> DD_SITE="datadoghq.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh)"

Manual Installation (Debian):

# Add repository
echo "deb [signed-by=/usr/share/keyrings/datadog-archive-keyring.gpg] https://apt.datadoghq.com/ stable 7" | sudo tee /etc/apt/sources.list.d/datadog.list

# Install
sudo apt-get update
sudo apt-get install datadog-agent

# Configure
sudo cp /etc/datadog-agent/datadog.yaml.example /etc/datadog-agent/datadog.yaml
sudo sed -i "s/api_key:.*/api_key: <YOUR_API_KEY>/" /etc/datadog-agent/datadog.yaml

# Start
sudo systemctl start datadog-agent
sudo systemctl enable datadog-agent

Windows Installation

PowerShell (Administrator):

$env:DD_API_KEY = "<YOUR_API_KEY>"
$env:DD_SITE = "datadoghq.com"

# Download and install
Start-Process -Wait msiexec -ArgumentList '/qn /i https://s3.amazonaws.com/ddagent-windows-stable/datadog-agent-7-latest.amd64.msi APIKEY="<YOUR_API_KEY>" SITE="datadoghq.com"'

Cloud-Specific Installation

AWS EKS

# Create secret for API key
kubectl create secret generic datadog-secret \
  --from-literal api-key=<YOUR_API_KEY>

# Install with EKS-specific settings
helm install datadog-agent datadog/datadog \
  --set datadog.apiKeyExistingSecret=datadog-secret \
  --set datadog.site=datadoghq.com \
  --set datadog.apm.portEnabled=true \
  --set datadog.logs.enabled=true \
  --set clusterAgent.enabled=true \
  --set clusterAgent.metricsProvider.enabled=true \
  --set clusterAgent.metricsProvider.useDatadogMetrics=true

Google GKE

helm install datadog-agent datadog/datadog \
  --set datadog.apiKey=<YOUR_API_KEY> \
  --set datadog.site=datadoghq.com \
  --set datadog.apm.portEnabled=true \
  --set datadog.logs.enabled=true \
  --set clusterAgent.enabled=true \
  --set providers.gke.autopilot=true  # For GKE Autopilot

Azure AKS

helm install datadog-agent datadog/datadog \
  --set datadog.apiKey=<YOUR_API_KEY> \
  --set datadog.site=datadoghq.com \
  --set datadog.apm.portEnabled=true \
  --set datadog.logs.enabled=true \
  --set clusterAgent.enabled=true \
  --set agents.tolerations[0].operator=Exists

Agent Configuration (datadog.yaml)

Core Configuration:

# /etc/datadog-agent/datadog.yaml
api_key: <YOUR_API_KEY>
site: datadoghq.com

# Global tags applied to all telemetry
tags:
  - env:production
  - team:platform

# APM configuration
apm_config:
  enabled: true
  env: production
  # Sampling rules
  apm_dd_url: https://trace.agent.datadoghq.com

# Log collection
logs_enabled: true
logs_config:
  container_collect_all: true
  # Processing rules
  processing_rules:
    - type: exclude_at_match
      name: exclude_healthchecks
      pattern: '"path":"/health"'

# DogStatsD for custom metrics
dogstatsd_non_local_traffic: true
dogstatsd_port: 8125

# Process monitoring
process_config:
  enabled: true
  process_collection:
    enabled: true

# Network monitoring
network_config:
  enabled: true

Verification

Check Agent Status:

# Linux
sudo datadog-agent status

# Docker
docker exec dd-agent agent status

# Kubernetes
kubectl exec -it <agent-pod> -- agent status

Verify in Datadog UI: 1. Infrastructure > Host Map: Agent should appear 2. Infrastructure > Containers: Container metrics visible 3. APM > Services: If APM enabled and instrumented

Common Issues

Agent not reporting:

• Verify API key is correct
• Check network connectivity to *.datadoghq.com
• Review agent logs: journalctl -u datadog-agent or docker logs dd-agent

Missing container metrics:

• Ensure Docker socket is mounted
• Verify cgroup paths are mounted correctly

APM traces not appearing:

• Confirm port 8126 is accessible
• Check DD_APM_ENABLED=true is set
• Verify application is instrumented

Alerting

Datadog monitors provide proactive alerting on metrics, logs, traces, and synthetic tests.

Monitor Types

Metric Monitor Examples

Basic Threshold Monitor

{
  "name": "High CPU Usage on {{host.name}}",
  "type": "metric alert",
  "query": "avg(last_5m):avg:system.cpu.user{env:production} by {host} > 90",
  "message": "CPU usage is above 90% on {{host.name}}.\n\nCurrent value: {{value}}%\n\n@slack-platform-alerts",
  "tags": ["team:platform", "severity:warning"],
  "priority": 3,
  "options": {
    "thresholds": {
      "critical": 90,
      "warning": 80
    },
    "notify_no_data": true,
    "no_data_timeframe": 10,
    "notify_audit": false,
    "require_full_window": false,
    "include_tags": true
  }
}

Error Rate Monitor

{
  "name": "High Error Rate - {{service.name}}",
  "type": "metric alert",
  "query": "sum(last_5m):sum:http.requests.errors{env:production} by {service}.as_rate() / sum:http.requests{env:production} by {service}.as_rate() * 100 > 5",
  "message": "Error rate exceeds 5% for {{service.name}}.\n\nCurrent error rate: {{value}}%\n\nRunbook: https://wiki/runbook/high-errors\n\n@pagerduty-platform",
  "tags": ["team:platform", "severity:critical"],
  "priority": 1,
  "options": {
    "thresholds": {
      "critical": 5,
      "warning": 2
    },
    "notify_no_data": false,
    "renotify_interval": 60,
    "escalation_message": "Error rate still elevated after 1 hour. Escalating."
  }
}

P99 Latency Monitor

{
  "name": "High P99 Latency - {{service.name}}",
  "type": "metric alert",
  "query": "percentile(last_5m):p99:trace.http.request.duration{env:production} by {service} > 1000000000",
  "message": "P99 latency exceeds 1 second for {{service.name}}.\n\nCurrent P99: {{value}}ns\n\n@slack-platform-alerts",
  "tags": ["team:platform", "severity:warning"],
  "options": {
    "thresholds": {
      "critical": 1000000000,
      "warning": 500000000
    }
  }
}

Anomaly Detection

Anomaly detection identifies deviations from historical patterns.

Anomaly Detection Algorithms

Anomaly Monitor Example

{
  "name": "Anomalous Request Rate - {{service.name}}",
  "type": "metric alert",
  "query": "avg(last_4h):anomalies(avg:http.requests{env:production} by {service}, 'agile', 3, direction='both', interval=60, alert_window='last_15m', count_default_zero='true') >= 1",
  "message": "Request rate is abnormally high/low for {{service.name}}.\n\nThis could indicate:\n- Traffic spike\n- Outage upstream\n- Bot activity\n\n@slack-platform-alerts",
  "tags": ["team:platform", "severity:warning"],
  "options": {
    "thresholds": {
      "critical": 1,
      "warning": 0.8
    },
    "threshold_windows": {
      "trigger_window": "last_15m",
      "recovery_window": "last_15m"
    }
  }
}

Anomaly Detection Best Practices

1. Require historical data: Anomaly detection needs baseline (weeks/months) 2. Use appropriate algorithm:

• Basic: Seasonal patterns (daily/weekly)
• Agile: Fast-changing metrics
• Robust: Noisy metrics with outliers

3. Set evaluation delay: 300+ seconds for cloud metrics 4. Start with wider bounds: Tighten after observing behavior 5. Avoid for new metrics: No baseline = false positives

Composite Monitors

Combine multiple monitors with boolean logic.

{
  "name": "Critical Service Degradation",
  "type": "composite",
  "query": "( high_error_rate && high_latency ) || service_down",
  "message": "Critical service degradation detected.\n\nMultiple signals indicate a major issue:\n- High errors: {{#is_alert}}YES{{/is_alert}}\n- High latency: {{#is_alert}}YES{{/is_alert}}\n- Service down: {{#is_alert}}YES{{/is_alert}}\n\n@pagerduty-critical",
  "tags": ["team:platform", "severity:critical"],
  "options": {
    "notify_no_data": false
  }
}

Forecast Monitors

Predict future threshold breaches.

{
  "name": "Disk Space Forecast - {{host.name}}",
  "type": "metric alert",
  "query": "max(next_1w):forecast(avg:system.disk.used{env:production} by {host}, 'linear', 1, interval='60m', history='1w', model='default') > 90",
  "message": "Disk usage predicted to exceed 90% within 1 week on {{host.name}}.\n\nCurrent usage: {{value}}%\n\nConsider:\n- Cleaning up old logs\n- Expanding storage\n- Archiving data\n\n@slack-infrastructure",
  "tags": ["team:infrastructure", "severity:warning"]
}

Log Monitors

Alert on log patterns and counts.

{
  "name": "High Error Log Volume",
  "type": "log alert",
  "query": "logs(\"status:error env:production\").index(\"main\").rollup(\"count\").last(\"5m\") > 100",
  "message": "More than 100 error logs in the last 5 minutes.\n\nSearch logs: https://app.datadoghq.com/logs?query=status:error%20env:production\n\n@slack-platform-alerts",
  "tags": ["team:platform", "severity:warning"],
  "options": {
    "thresholds": {
      "critical": 100,
      "warning": 50
    },
    "enable_logs_sample": true
  }
}

Alert Configuration Best Practices

Message Templates

{{#is_alert}}
ALERT: {{name}} triggered.

Current Value: {{value}}
Threshold: {{threshold}}
Host: {{host.name}}
Service: {{service.name}}
Environment: {{env.name}}

Runbook: https://wiki/runbook/{{name}}
Dashboard: https://app.datadoghq.com/dashboard/xxx

@pagerduty-{{team.name}}
{{/is_alert}}

{{#is_recovery}}
RECOVERED: {{name}} is back to normal.

Previous Value: {{value}}
{{/is_recovery}}

Thresholds

"options": {
  "thresholds": {
    "critical": 95,
    "critical_recovery": 85,
    "warning": 80,
    "warning_recovery": 70
  }
}

Recovery thresholds prevent flapping (rapid alert/recovery cycles).

Evaluation Windows

No Data Handling

"options": {
  "notify_no_data": true,
  "no_data_timeframe": 10
}

Use notify_no_data: true for:

• Critical services that should always emit data
• Synthetic monitors

Use notify_no_data: false for:

• Metrics that legitimately stop (batch jobs)
• Sparse event-based metrics

Alert Hygiene

Priority Levels

Alert on Symptoms, Not Causes

Good (symptoms):

• Error rate > 5%
• P99 latency > 1s
• Checkout failures > 10/min

Avoid (causes):

• CPU > 90% (may not affect users)
• Memory > 80% (system may handle it)
• Deployment started (not actionable)

Review Cadence

1. Weekly: Review all firing alerts 2. Monthly: Audit alert noise (alerts with no action taken) 3. Quarterly: Delete or tune unused monitors

Downtime Management

Scheduled Downtime

{
  "scope": "env:production AND service:api-gateway",
  "start": 1706400000,
  "end": 1706403600,
  "message": "Scheduled maintenance window for API gateway upgrade",
  "timezone": "America/New_York",
  "monitor_id": null,
  "recurrence": null
}

Recurring Downtime

{
  "scope": "env:staging",
  "message": "Weekly staging environment reset",
  "timezone": "UTC",
  "recurrence": {
    "type": "weeks",
    "period": 1,
    "week_days": ["Sun"],
    "until_date": null
  }
}

Runbook Integration

Runbook Template

Include in alert messages:

## Runbook: {{name}}

### Symptoms
- [What the alert indicates]

### Impact
- [User/business impact]

### Investigation Steps
1. Check dashboard: [link]
2. Check logs: [query link]
3. Check traces: [APM link]

### Mitigation
1. [Immediate actions]
2. [Rollback steps if needed]

### Escalation
- L1: @slack-platform
- L2: @pagerduty-platform (after 15 min)
- L3: @pagerduty-oncall-manager (after 1 hour)

Notification Integrations

Slack

@slack-channel-name
@slack-private-channel-name

PagerDuty

@pagerduty-service-name

Email

@user@company.com
@team-distribution-list@company.com

Webhooks

Configure in Integrations > Webhooks, then reference:

@webhook-my-webhook

APM Instrumentation

Datadog APM provides distributed tracing with automatic instrumentation for 8+ languages and 176+ integrations.

Python Instrumentation

Automatic Instrumentation

Installation:

pip install ddtrace

Option 1: ddtrace-run (Recommended)

# Automatically patches all supported libraries
DD_SERVICE=my-service \
DD_ENV=production \
DD_VERSION=1.0.0 \
ddtrace-run python app.py

Option 2: Programmatic patching

from ddtrace import tracer, patch_all

# Patch all supported libraries
patch_all()

# Or selective patching
from ddtrace import patch
patch(requests=True, flask=True, sqlalchemy=True, redis=True)

Manual Instrumentation

from ddtrace import tracer

# Basic span
with tracer.trace("custom.operation", service="my-service") as span:
    span.set_tag("user.id", user_id)
    span.set_tag("order.total", order_total)
    result = perform_operation()
    span.set_tag("result.status", result.status)

# Decorator
@tracer.wrap(service="my-service", resource="process_order")
def process_order(order_id):
    # Automatically traced
    pass

# Async support
async def async_operation():
    with tracer.trace("async.operation"):
        await some_async_call()

Flask Example

from flask import Flask
from ddtrace import tracer, patch_all

patch_all()

app = Flask(__name__)

@app.route('/api/users/<user_id>')
def get_user(user_id):
    # Automatically traced
    with tracer.trace("db.query", service="postgres") as span:
        span.set_tag("sql.query", "SELECT * FROM users WHERE id = ?")
        user = db.query_user(user_id)
    return jsonify(user)

FastAPI Example

from fastapi import FastAPI
from ddtrace import tracer, patch_all
from ddtrace.contrib.asgi import TraceMiddleware

patch_all()

app = FastAPI()
app = TraceMiddleware(app, tracer, service="my-fastapi-app")

@app.get("/items/{item_id}")
async def read_item(item_id: int):
    with tracer.trace("fetch.item"):
        item = await get_item(item_id)
    return item

Node.js Instrumentation

Automatic Instrumentation

Installation:

npm install dd-trace

Initialization (must be first import):

// tracer.js - import this FIRST in your app
const tracer = require('dd-trace').init({
  service: 'my-service',
  env: 'production',
  version: '1.0.0',
  logInjection: true,      // Inject trace IDs into logs
  runtimeMetrics: true,    // Collect runtime metrics
  profiling: true,         // Enable continuous profiler
});

module.exports = tracer;

// app.js
require('./tracer');  // Must be first!
const express = require('express');
const app = express();
// ... rest of app

Manual Instrumentation

const tracer = require('dd-trace');

// Basic span
const span = tracer.startSpan('custom.operation', {
  tags: {
    'service.name': 'my-service',
    'user.id': userId,
  }
});

try {
  const result = await performOperation();
  span.setTag('result.status', result.status);
} catch (error) {
  span.setTag('error', true);
  span.setTag('error.message', error.message);
  throw error;
} finally {
  span.finish();
}

// Using scope
tracer.trace('parent.operation', (span) => {
  // Child spans are automatically linked
  return tracer.trace('child.operation', (childSpan) => {
    return doWork();
  });
});

Express Example

require('./tracer');
const express = require('express');
const tracer = require('dd-trace');

const app = express();

app.get('/api/users/:id', async (req, res) => {
  const span = tracer.scope().active();
  span.setTag('user.id', req.params.id);

  const user = await tracer.trace('db.query', async (dbSpan) => {
    dbSpan.setTag('db.type', 'postgresql');
    return await db.getUser(req.params.id);
  });

  res.json(user);
});

Go Instrumentation

Installation

go get gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer
go get gopkg.in/DataDog/dd-trace-go.v1/contrib/...

Basic Setup

package main

import (
    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func main() {
    // Start tracer
    tracer.Start(
        tracer.WithService("my-service"),
        tracer.WithEnv("production"),
        tracer.WithServiceVersion("1.0.0"),
    )
    defer tracer.Stop()

    // Your application code
}

Manual Instrumentation

import (
    "context"
    "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func processOrder(ctx context.Context, orderID string) error {
    span, ctx := tracer.StartSpanFromContext(ctx, "process.order",
        tracer.ResourceName("ProcessOrder"),
        tracer.Tag("order.id", orderID),
    )
    defer span.Finish()

    // Child span
    childSpan, _ := tracer.StartSpanFromContext(ctx, "db.query")
    result, err := db.Query(ctx, "SELECT * FROM orders WHERE id = ?", orderID)
    if err != nil {
        childSpan.SetTag("error", true)
        childSpan.SetTag("error.message", err.Error())
    }
    childSpan.Finish()

    return err
}

HTTP Server Example

import (
    "net/http"
    httptrace "gopkg.in/DataDog/dd-trace-go.v1/contrib/net/http"
)

func main() {
    tracer.Start(tracer.WithService("my-api"))
    defer tracer.Stop()

    mux := httptrace.NewServeMux()
    mux.HandleFunc("/api/users", handleUsers)

    http.ListenAndServe(":8080", mux)
}

Java Instrumentation

Automatic Instrumentation (Agent)

Download Agent:

wget -O dd-java-agent.jar https://dtdg.co/latest-java-tracer

Run with Agent:

java -javaagent:dd-java-agent.jar \
  -Ddd.service=my-service \
  -Ddd.env=production \
  -Ddd.version=1.0.0 \
  -jar my-app.jar

Spring Boot Example

import datadog.trace.api.Trace;
import datadog.trace.api.DDTags;
import io.opentracing.Span;
import io.opentracing.util.GlobalTracer;

@RestController
public class UserController {

    @GetMapping("/api/users/{id}")
    @Trace(operationName = "get.user", resourceName = "UserController.getUser")
    public User getUser(@PathVariable String id) {
        Span span = GlobalTracer.get().activeSpan();
        span.setTag("user.id", id);

        return userService.findById(id);
    }
}

Manual Instrumentation

import datadog.trace.api.DDTags;
import io.opentracing.Scope;
import io.opentracing.Span;
import io.opentracing.Tracer;
import io.opentracing.util.GlobalTracer;

public class OrderService {
    private final Tracer tracer = GlobalTracer.get();

    public Order processOrder(String orderId) {
        Span span = tracer.buildSpan("process.order")
            .withTag(DDTags.SERVICE_NAME, "order-service")
            .withTag(DDTags.RESOURCE_NAME, "processOrder")
            .withTag("order.id", orderId)
            .start();

        try (Scope scope = tracer.activateSpan(span)) {
            // Your code here
            return doProcessOrder(orderId);
        } catch (Exception e) {
            span.setTag("error", true);
            span.setTag("error.message", e.getMessage());
            throw e;
        } finally {
            span.finish();
        }
    }
}

OpenTelemetry Integration

Datadog supports OpenTelemetry for vendor-neutral instrumentation.

Python with OpenTelemetry

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import Resource

# Configure to send to Datadog Agent OTLP endpoint
resource = Resource.create({
    "service.name": "my-otel-service",
    "deployment.environment": "production",
})

otlp_exporter = OTLPSpanExporter(
    endpoint="http://localhost:4317",
    insecure=True
)

provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(otlp_exporter))
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("my-service")

# Use OpenTelemetry API
with tracer.start_as_current_span("my-operation") as span:
    span.set_attribute("user.id", user_id)
    # Your code

Node.js with OpenTelemetry

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { BatchSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { Resource } = require('@opentelemetry/resources');

const provider = new NodeTracerProvider({
  resource: new Resource({
    'service.name': 'my-otel-service',
    'deployment.environment': 'production',
  }),
});

const exporter = new OTLPTraceExporter({
  url: 'http://localhost:4317',
});

provider.addSpanProcessor(new BatchSpanProcessor(exporter));
provider.register();

Enable OTLP in Datadog Agent

# datadog.yaml
otlp_config:
  receiver:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

Environment Variables

Common environment variables for all languages:

Best Practices

1. Always set service, env, version: Enables correlation and filtering 2. Use automatic instrumentation first: Add manual spans only when needed 3. Add meaningful tags: Business context, user IDs (if low cardinality), request metadata 4. Handle errors properly: Set error tags and capture stack traces 5. Propagate context: Pass context through async operations and service calls

Cost Optimization

Datadog costs can grow quickly without proper controls. This guide covers strategies to manage and optimize spending.

Primary Cost Drivers

Metrics Without Limits

Decouple metric ingestion from queryable tags to control custom metric costs.

How It Works

1. Ingest all tags: Full cardinality sent to Datadog 2. Configure allowlist: Select which tags are queryable 3. Pay for queryable: Only queryable tag combinations count as custom metrics

Configuration

Via UI: 1. Go to Metrics > Summary 2. Select your metric 3. Click "Configure Tags" 4. Select only needed tags

Via API:

from datadog_api_client.v2.api.metrics_api import MetricsApi
from datadog_api_client import Configuration, ApiClient

configuration = Configuration()
with ApiClient(configuration) as api_client:
    api = MetricsApi(api_client)

    api.update_tag_configuration(
        metric_name="myapp.http.requests",
        body={
            "data": {
                "type": "manage_tags",
                "id": "myapp.http.requests",
                "attributes": {
                    "tags": ["env", "service", "endpoint", "status_class"],
                    "include_percentiles": True,
                    "metric_type": "count"
                }
            }
        }
    )

Example Savings

Before (all tags queryable):

3 envs x 10 services x 100 endpoints x 10 status codes x 50 hosts
= 1,500,000 custom metrics

After (limited tags):

3 envs x 10 services x 100 endpoints x 5 status_classes
= 15,000 custom metrics (99% reduction)

Log Index Quotas

Prevent unexpected log volume spikes from causing cost overruns.

Set Daily Quota

{
  "name": "production-logs",
  "filter": {
    "query": "env:production"
  },
  "daily_limit": 5000000000,
  "daily_limit_reset": {
    "reset_time": "00:00",
    "reset_utc_offset": "+00:00"
  },
  "daily_limit_warning_threshold_percentage": 80
}

Quota Alerting

Create a monitor for quota warnings:

{
  "name": "Log Index Quota Warning",
  "type": "logs alert",
  "query": "logs(\"*\").index(\"production-logs\").rollup(\"count\").last(\"1d\") > 4000000000",
  "message": "Log index nearing daily quota (80% threshold).\n\nConsider:\n- Adding exclusion filters\n- Reducing log verbosity\n- Increasing quota\n\n@slack-platform-alerts"
}

Exclusion Filters

Drop logs before indexing to reduce costs.

Common Exclusion Patterns

{
  "exclusion_filters": [
    {
      "name": "Exclude health checks",
      "filter": {
        "query": "http.url:(\"/health\" OR \"/ready\" OR \"/live\")"
      },
      "is_enabled": true
    },
    {
      "name": "Exclude debug logs",
      "filter": {
        "query": "level:debug"
      },
      "is_enabled": true
    },
    {
      "name": "Sample verbose service (10%)",
      "filter": {
        "query": "service:verbose-service",
        "sample_rate": 0.1
      },
      "is_enabled": true
    },
    {
      "name": "Exclude bot traffic",
      "filter": {
        "query": "@http.useragent:(*bot* OR *crawler* OR *spider*)"
      },
      "is_enabled": true
    }
  ]
}

Agent-Side Exclusion (More Efficient)

Drop logs before sending to Datadog:

# /etc/datadog-agent/conf.d/app.d/conf.yaml
logs:
  - type: file
    path: /var/log/app/*.log
    source: myapp
    log_processing_rules:
      - type: exclude_at_match
        name: exclude_healthchecks
        pattern: GET /health

      - type: exclude_at_match
        name: exclude_debug
        pattern: '"level":"debug"'

APM Sampling Strategies

Ingestion Controls

Configure sampling to reduce trace volume while maintaining visibility.

Head-Based Sampling (Agent):

# datadog.yaml
apm_config:
  max_traces_per_second: 100  # Per agent

  # Rule-based sampling
  filter_tags:
    require:
      - env:production
    reject:
      - http.url:/health

Library-Level Sampling:

from ddtrace import tracer

tracer.configure(
    sampler=DatadogSampler(
        default_sample_rate=0.1,  # 10% default
        rules=[
            SamplingRule(sample_rate=1.0, service="critical-service"),
            SamplingRule(sample_rate=0.01, service="high-volume-service"),
        ]
    )
)

Retention Filters

Keep only valuable traces in long-term storage:

Via UI: 1. Go to APM > Setup & Configuration > Data Retention 2. Create retention filters

Recommended Filters:

[
  {
    "name": "Errors",
    "query": "@_top_level:1 @error.type:*",
    "rate": 1.0
  },
  {
    "name": "Slow requests (>1s)",
    "query": "@duration:>1000000000",
    "rate": 1.0
  },
  {
    "name": "Sample successful requests",
    "query": "@http.status_code:2* @duration:<1000000000",
    "rate": 0.1
  }
]

Custom Metrics Cardinality Control

Audit High-Cardinality Metrics

from datadog_api_client.v1.api.metrics_api import MetricsApi

api = MetricsApi(api_client)

# List all custom metrics
metrics = api.list_active_metrics(from_time=int(time.time()) - 86400)

# Find high-cardinality culprits
for metric in metrics.metrics:
    details = api.get_metric_metadata(metric)
    print(f"{metric}: {details}")

Dashboard for Cardinality Monitoring

Create widget with query:

count:datadog.estimated_usage.metrics.custom{*} by {metric_name}

Common Cardinality Issues

Usage Attribution API

Track costs by team, service, or project.

Query Usage

from datadog_api_client.v1.api.usage_metering_api import UsageMeteringApi
from datetime import datetime, timedelta

api = UsageMeteringApi(api_client)

# Get monthly usage summary
usage = api.get_usage_summary(
    start_month=datetime(2026, 1, 1),
    end_month=datetime(2026, 1, 31)
)

print(f"Hosts: {usage.usage[0].host_count}")
print(f"APM Hosts: {usage.usage[0].apm_host_top99p}")
print(f"Custom Metrics: {usage.usage[0].custom_ts_avg}")
print(f"Indexed Logs: {usage.usage[0].indexed_events_count}")

Usage Attribution by Tag

# Get usage attributed by 'team' tag
attribution = api.get_monthly_usage_attribution(
    start_month=datetime(2026, 1, 1),
    fields="*",
    tag_breakdown_keys="team"
)

for org in attribution.usage:
    print(f"Team: {org.tags.get('team')}")
    print(f"  Infra hosts: {org.infra_host_percentage}%")
    print(f"  Custom metrics: {org.custom_ts_percentage}%")

Create Cost Allocation Dashboard

{
  "title": "Datadog Cost Attribution",
  "widgets": [
    {
      "title": "Custom Metrics by Team",
      "type": "toplist",
      "query": "sum:datadog.estimated_usage.metrics.custom{*} by {team}"
    },
    {
      "title": "Log Volume by Service",
      "type": "toplist",
      "query": "sum:datadog.estimated_usage.logs.ingested_bytes{*} by {service}"
    },
    {
      "title": "APM Spans by Service",
      "type": "toplist",
      "query": "sum:datadog.estimated_usage.apm.indexed_spans{*} by {service}"
    }
  ]
}

Committed Contracts

Potential Savings

When to Commit

Good candidates:

• Stable host count (predictable infrastructure)
• Consistent log volume
• Established custom metrics footprint

Wait before committing:

• New to Datadog (need baseline)
• Rapid growth phase
• Major architecture changes planned

Negotiation Tips

1. Bundle products: Combine APM, logs, infra for better rates 2. Use current spend: Leverage existing usage as baseline 3. Multi-year: Longer terms = bigger discounts 4. Volume tiers: Higher volume = better per-unit pricing 5. Timing: End of quarter/year often has best deals

Cost Monitoring Alerts

Budget Alerting

{
  "name": "Datadog Daily Cost Alert",
  "type": "metric alert",
  "query": "sum(last_1d):sum:datadog.estimated_usage.hosts{*} * 15 + sum:datadog.estimated_usage.logs.ingested_bytes{*} / 1000000000 * 0.10 > 1000",
  "message": "Estimated daily Datadog cost exceeds $1000.\n\nBreakdown:\n- Hosts: {{value}}\n- Logs: {{value}} GB\n\nReview usage: https://app.datadoghq.com/billing/usage\n\n@finance-alerts"
}

Anomaly on Usage

{
  "name": "Anomalous Custom Metric Growth",
  "type": "metric alert",
  "query": "avg(last_1d):anomalies(sum:datadog.estimated_usage.metrics.custom{*}, 'basic', 3) >= 1",
  "message": "Unusual growth in custom metrics detected.\n\nThis may indicate:\n- New high-cardinality tags\n- New service with many metrics\n- Misconfigured instrumentation\n\n@platform-team"
}

Optimization Checklist

Weekly Review

• [ ] Check current month usage vs. budget
• [ ] Review top 10 custom metrics by cardinality
• [ ] Check log exclusion filter effectiveness
• [ ] Review any new high-volume services

Monthly Review

• [ ] Analyze usage attribution by team
• [ ] Audit unused monitors and dashboards
• [ ] Review APM sampling effectiveness
• [ ] Check for orphaned metrics (no queries)

Quarterly Review

• [ ] Evaluate committed contract vs. actual usage
• [ ] Review Metrics without Limits configurations
• [ ] Assess new product needs
• [ ] Plan capacity for next quarter

Quick Wins

Custom Metrics

DogStatsD enables submitting custom metrics from applications to Datadog.

DogStatsD Setup

Agent Configuration

DogStatsD runs on port 8125 by default. Enable non-local traffic for containerized apps:

# datadog.yaml
dogstatsd_non_local_traffic: true
dogstatsd_port: 8125
dogstatsd_origin_detection: true  # For container tagging

Docker:

docker run -d --name dd-agent \
  -e DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true \
  -p 8125:8125/udp \
  gcr.io/datadoghq/agent:7

Kubernetes:

# Helm values
datadog:
  dogstatsd:
    nonLocalTraffic: true
    useHostPort: true

Metric Types

Python Examples

Installation

pip install datadog

Basic Usage

from datadog import DogStatsd

statsd = DogStatsd(host="localhost", port=8125)

# COUNT - increment a counter
statsd.increment('page.views', tags=['page:home', 'env:production'])
statsd.increment('api.requests', value=1, tags=['endpoint:/api/users', 'method:GET'])
statsd.decrement('queue.jobs')

# GAUGE - set a point-in-time value
statsd.gauge('queue.size', 42, tags=['queue:orders'])
statsd.gauge('cache.hit_ratio', 0.85, tags=['cache:redis'])
statsd.gauge('system.memory_used_percent', 73.5)

# HISTOGRAM - track value distribution
statsd.histogram('request.latency', 0.123, tags=['endpoint:/api/users'])
statsd.histogram('payload.size', 4096, tags=['type:upload'])

# DISTRIBUTION - global percentile aggregation
statsd.distribution('payment.amount', 99.99, tags=['currency:usd'])
statsd.distribution('response.time', 45.2, tags=['service:api'])

# SET - count unique values
statsd.set('users.uniques', 'user123', tags=['source:web'])
statsd.set('sessions.active', session_id)

Context Manager for Timing

from datadog import DogStatsd

statsd = DogStatsd()

# Time a code block
with statsd.timed('database.query.duration', tags=['db:postgres']):
    result = db.execute(query)

# Time a function
@statsd.timed('api.endpoint.latency')
def process_request(request):
    return handle(request)

Flask Integration

from flask import Flask, request, g
from datadog import DogStatsd
import time

app = Flask(__name__)
statsd = DogStatsd()

@app.before_request
def before_request():
    g.start_time = time.time()

@app.after_request
def after_request(response):
    latency = time.time() - g.start_time

    statsd.increment('http.requests', tags=[
        f'method:{request.method}',
        f'endpoint:{request.endpoint}',
        f'status:{response.status_code}',
    ])

    statsd.histogram('http.latency', latency, tags=[
        f'method:{request.method}',
        f'endpoint:{request.endpoint}',
    ])

    return response

Node.js Examples

Installation

npm install hot-shots

Basic Usage

const StatsD = require('hot-shots');

const statsd = new StatsD({
  host: 'localhost',
  port: 8125,
  prefix: 'myapp.',
  globalTags: ['env:production'],
});

// COUNT
statsd.increment('page.views', ['page:home']);
statsd.increment('api.requests', 1, ['endpoint:/users', 'method:GET']);
statsd.decrement('queue.jobs');

// GAUGE
statsd.gauge('queue.size', 42, ['queue:orders']);
statsd.gauge('cache.hit_ratio', 0.85);

// HISTOGRAM
statsd.histogram('request.latency', 123, ['endpoint:/users']);

// DISTRIBUTION
statsd.distribution('payment.amount', 99.99, ['currency:usd']);

// SET
statsd.set('users.uniques', 'user123');

// TIMING
statsd.timing('db.query.time', 45, ['query:select']);

Express Middleware

const StatsD = require('hot-shots');
const statsd = new StatsD();

function metricsMiddleware(req, res, next) {
  const start = Date.now();

  res.on('finish', () => {
    const duration = Date.now() - start;
    const tags = [
      `method:${req.method}`,
      `path:${req.route?.path || req.path}`,
      `status:${res.statusCode}`,
    ];

    statsd.increment('http.requests', tags);
    statsd.histogram('http.latency', duration, tags);
  });

  next();
}

app.use(metricsMiddleware);

Go Examples

Installation

go get github.com/DataDog/datadog-go/v5/statsd

Basic Usage

package main

import (
    "github.com/DataDog/datadog-go/v5/statsd"
    "time"
)

func main() {
    client, err := statsd.New("localhost:8125",
        statsd.WithNamespace("myapp."),
        statsd.WithTags([]string{"env:production"}),
    )
    if err != nil {
        log.Fatal(err)
    }
    defer client.Close()

    // COUNT
    client.Incr("page.views", []string{"page:home"}, 1)
    client.Count("api.requests", 1, []string{"endpoint:/users"}, 1)
    client.Decr("queue.jobs", []string{}, 1)

    // GAUGE
    client.Gauge("queue.size", 42, []string{"queue:orders"}, 1)

    // HISTOGRAM
    client.Histogram("request.latency", 0.123, []string{"endpoint:/users"}, 1)

    // DISTRIBUTION
    client.Distribution("payment.amount", 99.99, []string{"currency:usd"}, 1)

    // SET
    client.Set("users.uniques", "user123", []string{}, 1)

    // TIMING
    start := time.Now()
    // ... operation
    client.Timing("operation.duration", time.Since(start), []string{}, 1)
}

HTTP Middleware

func metricsMiddleware(statsd *statsd.Client) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            start := time.Now()

            // Wrap response writer to capture status
            wrapped := &statusResponseWriter{ResponseWriter: w, status: 200}

            next.ServeHTTP(wrapped, r)

            tags := []string{
                fmt.Sprintf("method:%s", r.Method),
                fmt.Sprintf("path:%s", r.URL.Path),
                fmt.Sprintf("status:%d", wrapped.status),
            }

            statsd.Incr("http.requests", tags, 1)
            statsd.Timing("http.latency", time.Since(start), tags, 1)
        })
    }
}

Naming Conventions

Rules

1. Must start with a letter 2. Alphanumeric, underscores, periods only (ASCII) 3. Maximum 200 characters (prefer under 100) 4. Case-sensitive 5. No spaces

Recommended Pattern

<namespace>.<category>.<metric_name>

Examples:

myapp.api.request.count
myapp.api.request.latency
myapp.db.query.duration
myapp.cache.hit_ratio
myapp.queue.size
myapp.payment.amount

Avoid

MyApp.API.Request.Count     # Wrong: uppercase, dots instead of underscores
my-app.api.request-count    # Wrong: hyphens
1_request_count             # Wrong: starts with number
request count               # Wrong: spaces

Tagging Best Practices

Good Tags (Low Cardinality)

# Environment and service context
tags = [
    'env:production',
    'service:api-gateway',
    'version:1.2.3',
    'team:platform',
    'region:us-east-1',
]

# Request metadata
tags = [
    'method:GET',
    'endpoint:/api/users',
    'status_class:2xx',
]

# Business context (bounded values)
tags = [
    'plan:enterprise',
    'feature:checkout',
    'payment_method:credit_card',
]

Bad Tags (High Cardinality)

# AVOID - creates millions of unique metrics
tags = [
    f'user_id:{user_id}',           # Unbounded user IDs
    f'request_id:{request_id}',     # Unique per request
    f'timestamp:{timestamp}',        # Always unique
    f'pod:{pod_name}',              # Ephemeral in K8s
    f'transaction_id:{tx_id}',      # Unbounded
    f'session_id:{session_id}',     # Unbounded
]

Cardinality Management

Understanding Cardinality

Each unique combination of metric name + tag values = 1 custom metric.

Example:

# 3 endpoints x 3 methods x 5 status classes = 45 custom metrics
statsd.increment('http.requests', tags=[
    f'endpoint:{endpoint}',   # 3 values
    f'method:{method}',       # 3 values
    f'status_class:{class}',  # 5 values
])

Cardinality Explosion Example

# BAD: 1M users x 100 endpoints = 100M custom metrics = $$$
statsd.increment('requests', tags=[
    f'user_id:{user_id}',
    f'endpoint:{endpoint}',
])

# GOOD: 100 endpoints x 10 plans = 1000 custom metrics
statsd.increment('requests', tags=[
    f'plan:{user.plan}',
    f'endpoint:{endpoint}',
])

Metrics Without Limits

Configure tag allowlists to control cardinality at the Datadog level:

1. Go to Metrics > Summary 2. Select metric 3. Click "Configure Tags" 4. Add only tags you want to query by

API Configuration:

from datadog_api_client.v2.api.metrics_api import MetricsApi

api = MetricsApi(api_client)
api.update_tag_configuration(
    metric_name="myapp.http.requests",
    body={
        "data": {
            "type": "manage_tags",
            "id": "myapp.http.requests",
            "attributes": {
                "tags": ["env", "service", "endpoint"],
                "include_percentiles": True
            }
        }
    }
)

Cost Monitoring

Track Custom Metric Usage

from datadog_api_client.v1.api.usage_metering_api import UsageMeteringApi
from datetime import datetime

api = UsageMeteringApi(api_client)

# Get hourly custom metrics usage
usage = api.get_usage_top_avg_metrics(
    month=datetime(2026, 1, 1),
    names=["myapp.*"]
)

for metric in usage.usage:
    print(f"{metric.metric_name}: {metric.avg_metric_hour} avg/hour")

Dashboard for Metric Cardinality

Create a dashboard widget with:

count:datadog.estimated_usage.metrics.custom{*} by {metric_name}

Best Practices Summary

1. Use consistent naming: namespace.category.metric 2. Keep cardinality low: Bounded tag values only 3. Add business context: Plan, feature, region 4. Avoid user-level tags: Use aggregations instead 5. Monitor usage: Track custom metric counts 6. Use Metrics without Limits: Control queryable tags 7. Prefer DISTRIBUTION over HISTOGRAM: Global percentiles

Kubernetes Monitoring

Datadog provides comprehensive Kubernetes monitoring through the DaemonSet Agent and Cluster Agent.

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                     Kubernetes Cluster                       │
│                                                              │
│  ┌──────────────┐     ┌──────────────────────────────────┐  │
│  │ Cluster Agent│────▶│ Kubernetes API Server            │  │
│  │  (Deployment)│     │ - Cluster-level metrics          │  │
│  └──────────────┘     │ - Events, HPA metrics            │  │
│         │             └──────────────────────────────────┘  │
│         │                                                    │
│         ▼                                                    │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │  Node Agent  │  │  Node Agent  │  │  Node Agent  │      │
│  │  (DaemonSet) │  │  (DaemonSet) │  │  (DaemonSet) │      │
│  │              │  │              │  │              │      │
│  │ - Pod metrics│  │ - Pod metrics│  │ - Pod metrics│      │
│  │ - Container  │  │ - Container  │  │ - Container  │      │
│  │ - Logs       │  │ - Logs       │  │ - Logs       │      │
│  │ - Traces     │  │ - Traces     │  │ - Traces     │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
│                                                              │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
                       ┌──────────────┐
                       │   Datadog    │
                       │   Backend    │
                       └──────────────┘

DaemonSet Deployment

Helm Installation

Add Datadog Helm repository:

helm repo add datadog https://helm.datadoghq.com
helm repo update

Create values file:

# datadog-values.yaml
datadog:
  apiKey: <YOUR_API_KEY>
  appKey: <YOUR_APP_KEY>  # Optional, for Cluster Agent
  site: datadoghq.com

  # Cluster name for grouping
  clusterName: my-production-cluster

  # Global tags
  tags:
    - env:production
    - team:platform

  # Collect Kubernetes events
  collectEvents: true

  # Leader election for cluster-level checks
  leaderElection: true

  # APM configuration
  apm:
    portEnabled: true
    socketEnabled: true

  # Log collection
  logs:
    enabled: true
    containerCollectAll: true

  # Process monitoring
  processAgent:
    enabled: true
    processCollection: true

  # Network Performance Monitoring
  networkMonitoring:
    enabled: true

  # Orchestrator Explorer
  orchestratorExplorer:
    enabled: true

# Cluster Agent
clusterAgent:
  enabled: true
  replicas: 2

  # HPA metrics
  metricsProvider:
    enabled: true
    useDatadogMetrics: true

  # Admission Controller for auto-instrumentation
  admissionController:
    enabled: true
    mutateUnlabelled: false

# Node Agent resources
agents:
  image:
    repository: gcr.io/datadoghq/agent
    tag: 7

  resources:
    requests:
      cpu: 200m
      memory: 256Mi
    limits:
      cpu: 200m
      memory: 256Mi

  # Tolerations for running on all nodes
  tolerations:
    - operator: Exists

Install:

helm install datadog-agent datadog/datadog -f datadog-values.yaml -n datadog --create-namespace

Verify Installation

# Check DaemonSet
kubectl get daemonset -n datadog
kubectl get pods -n datadog -l app=datadog

# Check Cluster Agent
kubectl get deployment -n datadog datadog-cluster-agent
kubectl get pods -n datadog -l app=datadog-cluster-agent

# Check agent status
kubectl exec -it $(kubectl get pods -n datadog -l app=datadog -o jsonpath='{.items[0].metadata.name}') -n datadog -- agent status

Cluster Agent Setup

The Cluster Agent provides:

• Cluster-level metrics collection
• HPA custom metrics
• Kubernetes events
• Reduced API server load

Configuration

clusterAgent:
  enabled: true
  replicas: 2  # HA for production

  # Token for agent-cluster communication
  token: ""  # Auto-generated if empty

  # RBAC
  rbac:
    create: true
    serviceAccountName: datadog-cluster-agent

  # Metrics provider for HPA
  metricsProvider:
    enabled: true
    useDatadogMetrics: true
    wpaController: true  # Watermark Pod Autoscaler

  # Admission Controller
  admissionController:
    enabled: true
    mutateUnlabelled: false  # Only annotated pods

  resources:
    requests:
      cpu: 200m
      memory: 256Mi

Custom Metrics HPA

Use Datadog metrics for Horizontal Pod Autoscaler:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: External
      external:
        metric:
          name: datadogmetric@my-namespace:my-app-requests-per-second
          selector:
            matchLabels:
              service: my-app
        target:
          type: AverageValue
          averageValue: 100

Create DatadogMetric resource:

apiVersion: datadoghq.com/v1alpha1
kind: DatadogMetric
metadata:
  name: my-app-requests-per-second
  namespace: my-namespace
spec:
  query: sum:http.requests{service:my-app}.rollup(avg, 60)

Autodiscovery Annotations

Autodiscovery configures integrations using pod annotations.

Basic Pattern

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  annotations:
    # Log collection
    ad.datadoghq.com/my-container.logs: '[{"source": "python", "service": "my-app"}]'

    # Check integration
    ad.datadoghq.com/my-container.checks: |
      {
        "http_check": {
          "instances": [{"url": "http://%%host%%:%%port%%/health"}]
        }
      }

    # Custom tags
    ad.datadoghq.com/tags: '{"team": "platform", "version": "1.2.3"}'
spec:
  containers:
    - name: my-container
      image: my-app:latest

Log Collection

annotations:
  # Basic log collection
  ad.datadoghq.com/my-container.logs: '[{"source": "python", "service": "my-app"}]'

  # With processing rules
  ad.datadoghq.com/my-container.logs: |
    [{
      "source": "python",
      "service": "my-app",
      "log_processing_rules": [{
        "type": "exclude_at_match",
        "name": "exclude_healthchecks",
        "pattern": "GET /health"
      }]
    }]

  # Multi-container pod
  ad.datadoghq.com/app.logs: '[{"source": "python", "service": "my-app"}]'
  ad.datadoghq.com/sidecar.logs: '[{"source": "nginx", "service": "my-app-proxy"}]'

APM Trace Collection

annotations:
  # Enable APM for container
  ad.datadoghq.com/my-container.tags: '{"service": "my-app", "env": "production"}'

Integration Checks

annotations:
  # Redis check
  ad.datadoghq.com/redis.checks: |
    {
      "redisdb": {
        "instances": [{
          "host": "%%host%%",
          "port": "%%port%%",
          "password": "%%env_REDIS_PASSWORD%%"
        }]
      }
    }

  # PostgreSQL check
  ad.datadoghq.com/postgres.checks: |
    {
      "postgres": {
        "instances": [{
          "host": "%%host%%",
          "port": "5432",
          "username": "datadog",
          "password": "%%env_PG_PASSWORD%%",
          "dbname": "mydb"
        }]
      }
    }

Template Variables

Container Tagging

Automatic Tags

Datadog automatically adds:

• kube_namespace
• kube_deployment
• kube_daemon_set
• kube_replica_set
• kube_stateful_set
• kube_job
• kube_cronjob
• pod_name
• container_name
• container_id
• image_name
• image_tag

Label-Based Tags

Extract tags from Kubernetes labels:

# datadog-values.yaml
datadog:
  # Convert labels to tags
  podLabelsAsTags:
    app.kubernetes.io/name: kube_app_name
    app.kubernetes.io/version: kube_app_version
    app.kubernetes.io/component: kube_app_component

  # Convert annotations to tags
  podAnnotationsAsTags:
    custom.company.com/team: team

Unified Service Tagging

Use standard labels for automatic service tagging:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    tags.datadoghq.com/env: production
    tags.datadoghq.com/service: my-app
    tags.datadoghq.com/version: "1.2.3"
spec:
  template:
    metadata:
      labels:
        tags.datadoghq.com/env: production
        tags.datadoghq.com/service: my-app
        tags.datadoghq.com/version: "1.2.3"
    spec:
      containers:
        - name: my-app
          env:
            - name: DD_ENV
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['tags.datadoghq.com/env']
            - name: DD_SERVICE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['tags.datadoghq.com/service']
            - name: DD_VERSION
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['tags.datadoghq.com/version']

Pod-Level Metrics

Key Metrics

Resource Utilization Queries

# CPU utilization (actual vs request)
kubernetes.cpu.usage.total{kube_deployment:my-app} / kubernetes.cpu.requests{kube_deployment:my-app} * 100

# Memory utilization (actual vs limit)
kubernetes.memory.usage{kube_deployment:my-app} / kubernetes.memory.limits{kube_deployment:my-app} * 100

# Pod restart rate
sum:kubernetes.containers.restarts{kube_namespace:production}.as_rate()

Helm Chart Configuration Reference

Complete Production Values

# datadog-values.yaml - Production configuration
datadog:
  apiKey: <YOUR_API_KEY>
  appKey: <YOUR_APP_KEY>
  site: datadoghq.com
  clusterName: production-cluster

  tags:
    - env:production
    - team:platform

  # Features
  collectEvents: true
  leaderElection: true
  criSocketPath: /var/run/containerd/containerd.sock

  # APM
  apm:
    portEnabled: true
    socketEnabled: true
    socketPath: /var/run/datadog/apm.socket

  # Logs
  logs:
    enabled: true
    containerCollectAll: true

  # Process Agent
  processAgent:
    enabled: true
    processCollection: true

  # NPM
  networkMonitoring:
    enabled: true

  # Orchestrator Explorer
  orchestratorExplorer:
    enabled: true

  # Security
  securityAgent:
    compliance:
      enabled: false
    runtime:
      enabled: false

  # Label/annotation to tag mapping
  podLabelsAsTags:
    app.kubernetes.io/name: kube_app_name
    app.kubernetes.io/version: kube_app_version

# Cluster Agent
clusterAgent:
  enabled: true
  replicas: 2
  createPodDisruptionBudget: true

  metricsProvider:
    enabled: true
    useDatadogMetrics: true

  admissionController:
    enabled: true
    mutateUnlabelled: false

  resources:
    requests:
      cpu: 200m
      memory: 256Mi
    limits:
      cpu: 200m
      memory: 256Mi

# Node Agent
agents:
  image:
    repository: gcr.io/datadoghq/agent
    tag: 7

  rbac:
    create: true

  tolerations:
    - operator: Exists

  podSecurity:
    seccompProfiles:
      - runtime/default

  resources:
    requests:
      cpu: 200m
      memory: 256Mi
    limits:
      cpu: 200m
      memory: 256Mi

  # Volume mounts for logs
  volumes:
    - name: varlog
      hostPath:
        path: /var/log
    - name: containerlog
      hostPath:
        path: /var/lib/docker/containers

  volumeMounts:
    - name: varlog
      mountPath: /var/log
      readOnly: true
    - name: containerlog
      mountPath: /var/lib/docker/containers
      readOnly: true

# Kube State Metrics (if not already installed)
providers:
  kubeStateMetrics:
    enabled: true

Troubleshooting

Common Issues

Agent not collecting pod metrics:

# Check kubelet connectivity
kubectl exec -it <agent-pod> -n datadog -- agent check kubelet

Missing container logs:

# Verify log paths
kubectl exec -it <agent-pod> -n datadog -- ls /var/log/containers/

# Check agent log collection status
kubectl exec -it <agent-pod> -n datadog -- agent status | grep -A 20 "Logs Agent"

Cluster Agent not syncing:

# Check cluster agent logs
kubectl logs -n datadog -l app=datadog-cluster-agent

# Verify RBAC permissions
kubectl auth can-i list pods --as=system:serviceaccount:datadog:datadog-cluster-agent

Diagnostic Commands

# Full agent status
kubectl exec -it <agent-pod> -n datadog -- agent status

# Check specific integration
kubectl exec -it <agent-pod> -n datadog -- agent check kubernetes

# Agent configuration
kubectl exec -it <agent-pod> -n datadog -- agent configcheck

# Cluster Agent status
kubectl exec -it <cluster-agent-pod> -n datadog -- datadog-cluster-agent status

Log Management

Datadog Log Management provides centralized collection, processing, and analysis with 15-month retention options.

Agent Log Collection

Docker Container Logs

Enable container log collection:

# docker-compose.yaml
datadog-agent:
  environment:
    - DD_LOGS_ENABLED=true
    - DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock:ro

Application container labels:

services:
  my-app:
    labels:
      com.datadoghq.ad.logs: '[{"source": "python", "service": "my-app"}]'

File-Based Collection

Configuration in conf.d:

# /etc/datadog-agent/conf.d/my-app.d/conf.yaml
logs:
  - type: file
    path: /var/log/my-app/*.log
    source: my-app
    service: my-app
    tags:
      - env:production
      - team:platform

Multiple log files:

logs:
  - type: file
    path: /var/log/my-app/app.log
    source: my-app
    service: my-app

  - type: file
    path: /var/log/my-app/error.log
    source: my-app
    service: my-app
    log_processing_rules:
      - type: include_at_match
        name: only_errors
        pattern: (ERROR|CRITICAL)

TCP/UDP Log Collection

logs:
  - type: tcp
    port: 10514
    source: syslog
    service: my-app

  - type: udp
    port: 10515
    source: syslog
    service: my-app

Pipeline Configuration

Pipelines process logs before indexing, enabling parsing, enrichment, and transformation.

Creating a Pipeline

Via Datadog UI: 1. Go to Logs > Configuration > Pipelines 2. Click "New Pipeline" 3. Set filter query (e.g., source:nginx) 4. Add processors

Pipeline Example (JSON API):

{
  "name": "NGINX Access Logs",
  "filter": {
    "query": "source:nginx"
  },
  "processors": [
    {
      "type": "grok-parser",
      "name": "Parse NGINX logs",
      "source": "message",
      "samples": [
        "192.168.1.1 - - [27/Jan/2026:10:00:00 +0000] \"GET /api/users HTTP/1.1\" 200 1234"
      ],
      "grok": {
        "matchRules": "%{_client_ip} - - \\[%{_date_access}\\] \"%{_method} %{_url} HTTP/%{_version}\" %{_status_code} %{_bytes_sent}"
      }
    },
    {
      "type": "status-remapper",
      "name": "Set log status",
      "sources": ["http.status_code"]
    }
  ]
}

Grok Parsing Patterns

Common Log Formats

NGINX Access Log:

%{_client_ip:network.client.ip} - %{_ident} \[%{_date_access}\] "%{_method:http.method} %{_url:http.url} HTTP/%{_version:http.version}" %{_status_code:http.status_code} %{_bytes_sent:network.bytes_written}

Apache Combined Log:

%{_client_ip:network.client.ip} %{_ident} %{_auth} \[%{_date_access}\] "%{_method:http.method} %{_url:http.url} HTTP/%{_version}" %{_status_code:http.status_code} %{_bytes_sent:network.bytes_written} "%{_referer:http.referer}" "%{_user_agent:http.useragent}"

JSON Logs:

%{data::json}

Python Exception:

%{word:level} %{date("yyyy-MM-dd HH:mm:ss,SSS"):timestamp} %{notSpace:logger} - %{data:message}

Helper Patterns

Grok Parser Configuration

processors:
  - type: grok-parser
    name: Parse application logs
    source: message
    grok:
      supportRules: |
        _timestamp %{date("yyyy-MM-dd HH:mm:ss.SSS"):timestamp}
        _level %{word:level}
        _logger %{notSpace:logger}
      matchRules: |
        app_log %{_timestamp} %{_level} %{_logger} - %{data:message}
    samples:
      - "2026-01-27 10:30:45.123 INFO my.app.handler - Request processed successfully"

Standard Attributes

Map parsed fields to Datadog standard attributes for consistent querying and correlation.

Core Attributes

Attribute Remapper

processors:
  - type: attribute-remapper
    name: Remap to standard attributes
    sources:
      - request_method
    target: http.method
    targetType: string
    preserveSource: false

  - type: attribute-remapper
    name: Remap status code
    sources:
      - status
      - response_code
    target: http.status_code
    targetType: number

Processing Rules (Agent-Side)

Filter and transform logs before sending to Datadog.

Exclude Logs

# conf.d/my-app.d/conf.yaml
logs:
  - type: file
    path: /var/log/my-app/*.log
    source: my-app
    log_processing_rules:
      # Exclude health checks
      - type: exclude_at_match
        name: exclude_healthchecks
        pattern: '"path":"/health"'

      # Exclude debug logs in production
      - type: exclude_at_match
        name: exclude_debug
        pattern: DEBUG

Include Only Specific Logs

log_processing_rules:
  - type: include_at_match
    name: only_errors
    pattern: (ERROR|CRITICAL|FATAL)

Mask Sensitive Data

log_processing_rules:
  - type: mask_sequences
    name: mask_credit_cards
    pattern: \d{4}-\d{4}-\d{4}-\d{4}
    replace_placeholder: "[MASKED_CC]"

  - type: mask_sequences
    name: mask_ssn
    pattern: \d{3}-\d{2}-\d{4}
    replace_placeholder: "[MASKED_SSN]"

Multi-line Aggregation

log_processing_rules:
  # Java stack traces
  - type: multi_line
    name: java_stacktrace
    pattern: ^\d{4}-\d{2}-\d{2}

Archive and Rehydration

Configure Log Archive

Archive to S3:

{
  "type": "archives",
  "data": {
    "type": "archives",
    "attributes": {
      "name": "production-logs-archive",
      "query": "env:production",
      "destination": {
        "type": "s3",
        "bucket": "my-datadog-logs-archive",
        "path": "/logs",
        "region": "us-east-1",
        "integration": {
          "account_id": "123456789012",
          "role_name": "DatadogLogsArchiveRole"
        }
      },
      "rehydration_max_scan_size_in_gb": 100,
      "rehydration_tags": ["team:platform", "archived:true"]
    }
  }
}

Rehydrate Archived Logs

1. Go to Logs > Configuration > Rehydrate from Archives 2. Select archive and time range 3. Set rehydration query filter 4. Configure destination index 5. Submit rehydration request

Index Configuration

Create Index with Quotas

{
  "name": "production-index",
  "filter": {
    "query": "env:production"
  },
  "daily_limit": 5000000000,
  "daily_limit_reset": {
    "reset_time": "14:00",
    "reset_utc_offset": "-05:00"
  },
  "exclusion_filters": [
    {
      "name": "Exclude debug logs",
      "filter": {
        "query": "level:debug"
      },
      "is_enabled": true
    },
    {
      "name": "Sample high-volume service",
      "filter": {
        "query": "service:high-volume-svc",
        "sample_rate": 0.1
      },
      "is_enabled": true
    }
  ],
  "retention_days": 15
}

Best Practices

Pipeline Design

1. Limit processors: Maximum 20 processors per pipeline 2. Limit parsing rules: Maximum 10 rules per Grok processor 3. Use specific filters: Narrow pipeline scope with precise queries 4. Test with samples: Always include sample logs in Grok parsers

Log Size

1. Keep logs under 25KB: Larger logs are truncated 2. Avoid logging large payloads: Use references/IDs instead 3. Structure JSON logs: Easier to parse and query

Cost Control

1. Set daily quotas: Prevent unexpected volume spikes 2. Use exclusion filters: Drop unnecessary logs before indexing 3. Archive for compliance: Use archives instead of long retention 4. Sample high-volume logs: Reduce storage for noisy sources

Correlation

1. Inject trace IDs: Enable log-to-trace linking 2. Use standard attributes: Consistent querying across services 3. Add service/env/version tags: Match APM telemetry