Opentelemetry

OpenTelemetry Implementation Guide

Overview

OpenTelemetry (OTel) is a vendor-neutral observability framework for instrumenting, generating, collecting, and exporting telemetry data (traces, metrics, logs). This skill provides guidance for implementing OTEL in Kubernetes environments.

Quick Start

Deploy OTEL Collector on Kubernetes

# Add Helm repo
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

# Install with basic config
helm install otel-collector open-telemetry/opentelemetry-collector \
  --namespace monitoring --create-namespace \
  --set mode=daemonset

Send Test Data via OTLP

# gRPC endpoint: 4317, HTTP endpoint: 4318
curl -X POST http://otel-collector:4318/v1/traces \
  -H "Content-Type: application/json" \
  -d '{"resourceSpans":[]}'

Core Concepts

Signals: Three types of telemetry data:

• Traces: Distributed request flows across services
• Metrics: Numerical measurements (counters, gauges, histograms)
• Logs: Event records with structured/unstructured data

Collector Components:

• Receivers: Accept data (OTLP, Prometheus, Jaeger, Zipkin)
• Processors: Transform data (batch, memory_limiter, k8sattributes)
• Exporters: Send data (prometheusremotewrite, loki, otlp)
• Extensions: Add capabilities (health_check, pprof, zpages)

Collector Configuration

Basic Pipeline Structure

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: ${env:MY_POD_IP}:4317
        http:
          endpoint: ${env:MY_POD_IP}:4318

  processors:
    batch:
      timeout: 10s
      send_batch_size: 1024
    memory_limiter:
      check_interval: 5s
      limit_percentage: 80
      spike_limit_percentage: 25

  exporters:
    prometheusremotewrite:
      endpoint: "http://prometheus:9090/api/v1/write"
    loki:
      endpoint: "http://loki:3100/loki/api/v1/push"

  service:
    pipelines:
      metrics:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [prometheusremotewrite]
      logs:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [loki]
      traces:
        receivers: [otlp]
        processors: [memory_limiter, batch]
        exporters: [otlp/tempo]

Kubernetes Attributes Enrichment

processors:
  k8sattributes:
    auth_type: "serviceAccount"
    passthrough: false
    filter:
      node_from_env_var: ${env:K8S_NODE_NAME}
    extract:
      metadata:
        - k8s.pod.name
        - k8s.namespace.name
        - k8s.deployment.name
        - k8s.node.name

Deployment Modes

Common Patterns

Development Environment

• Enable debug exporter for visibility
• Lower resource limits (250m CPU, 512Mi memory)
• Include spot instance tolerations for cost savings

Production Environment

• Implement sampling (10-50% for traces)
• Higher batch sizes (2048-4096)
• Enable autoscaling and PodDisruptionBudget
• Use TLS for all endpoints

Detailed References

For in-depth guidance, see:

• Collector Configuration: COLLECTOR.md
• Kubernetes Deployment: KUBERNETES.md
• Troubleshooting: TROUBLESHOOTING.md
• Instrumentation: INSTRUMENTATION.md

Validation Commands

# Check collector pods
kubectl get pods -n monitoring -l app.kubernetes.io/name=otel-collector

# View collector logs
kubectl logs -n monitoring -l app.kubernetes.io/name=otel-collector --tail=100

# Test OTLP endpoint
kubectl run test-otlp --image=curlimages/curl:latest --rm -it -- \
  curl -v http://otel-collector.monitoring:4318/v1/traces

# Validate config syntax
otelcol validate --config=config.yaml

Key Helm Chart Values

mode: "daemonset"  # or "deployment"
presets:
  logsCollection:
    enabled: true
  hostMetrics:
    enabled: true
  kubernetesAttributes:
    enabled: true
  kubeletMetrics:
    enabled: true
useGOMEMLIMIT: true
resources:
  limits:
    cpu: 500m
    memory: 1Gi
  requests:
    cpu: 100m
    memory: 256Mi

---

Gotchas

• Collector tail_sampling at end of pipeline doesn't release inflight buffer — drop decisions made late still hold memory; OOM under load.
• Auto-instrumentation + manual instrumentation can double-count traces — pick one strategy per service or de-dupe explicitly via sampler.
• Context propagation: HTTP B3 vs W3C traceparent headers don't auto-convert; mixed environments silently break trace continuity.
• OTel SDK vs Collector protocol versions: minor version mismatches can drop attributes silently (especially semantic-convention attributes).
• Resource detection adds platform attributes (cloud.account.id, host.id) that bloat traces — disable when not used for routing.
• Batch processor + memory limiter ordering: limiter must precede batch in the pipeline or memory pressure causes batch drops without backpressure.

OpenTelemetry Collector Configuration Reference

Receivers

OTLP Receiver (Primary)

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: ${env:MY_POD_IP}:4317
        max_recv_msg_size_mib: 4
      http:
        endpoint: ${env:MY_POD_IP}:4318
        cors:
          allowed_origins: ["*"]

Prometheus Receiver

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'kubernetes-pods'
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              action: keep
              regex: true
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
              action: replace
              target_label: __metrics_path__
              regex: (.+)
            - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
              action: replace
              regex: ([^:]+)(?::\d+)?;(\d+)
              replacement: $1:$2
              target_label: __address__

Filelog Receiver (Container Logs)

receivers:
  filelog:
    include:
      - /var/log/pods/*/*/*.log
    exclude:
      - /var/log/pods/*/otel-collector/*.log
    start_at: beginning
    include_file_path: true
    include_file_name: false
    operators:
      - type: router
        id: get-format
        routes:
          - output: parser-docker
            expr: 'body matches "^\\{"'
          - output: parser-crio
            expr: 'body matches "^[^ Z]+ "'
          - output: parser-containerd
            expr: 'body matches "^[^ Z]+Z"'

Jaeger Receiver (Legacy Support)

receivers:
  jaeger:
    protocols:
      grpc:
        endpoint: ${env:MY_POD_IP}:14250
      thrift_http:
        endpoint: ${env:MY_POD_IP}:14268
      thrift_compact:
        endpoint: ${env:MY_POD_IP}:6831

Zipkin Receiver

receivers:
  zipkin:
    endpoint: ${env:MY_POD_IP}:9411

Hostmetrics Receiver

receivers:
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
      disk: {}
      filesystem: {}
      network: {}
      load: {}

Processors

Batch Processor

processors:
  batch:
    timeout: 10s           # Max time before sending
    send_batch_size: 1024  # Target batch size
    send_batch_max_size: 2048  # Hard limit

Production Tuning:

processors:
  batch:
    timeout: 30s
    send_batch_size: 2048
    send_batch_max_size: 4096

Memory Limiter Processor

processors:
  memory_limiter:
    check_interval: 5s
    limit_percentage: 80        # % of total memory
    spike_limit_percentage: 25  # % buffer for spikes

Resource Processor

processors:
  resource:
    attributes:
      - key: cluster.name
        value: ${env:CLUSTER_NAME}
        action: upsert
      - key: deployment.environment
        value: ${env:ENVIRONMENT}
        action: upsert

K8s Attributes Processor

processors:
  k8sattributes:
    auth_type: "serviceAccount"
    passthrough: false
    filter:
      node_from_env_var: ${env:K8S_NODE_NAME}
    extract:
      metadata:
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.deployment.name
        - k8s.namespace.name
        - k8s.node.name
        - k8s.pod.start_time
      annotations:
        - tag_name: service.name
          key: app.kubernetes.io/name
          from: pod
        - tag_name: service.version
          key: app.kubernetes.io/version
          from: pod
      labels:
        - tag_name: app
          key: app
          from: pod

Probabilistic Sampler (Traces)

processors:
  probabilistic_sampler:
    hash_seed: 22
    sampling_percentage: 10  # Sample 10% of traces

Filter Processor

processors:
  filter/logs:
    logs:
      exclude:
        match_type: regexp
        bodies:
          - "health.*check"
          - "GET /healthz"

Transform Processor

processors:
  transform:
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["cluster"], "my-cluster")

Exporters

Prometheus Remote Write

exporters:
  prometheusremotewrite:
    endpoint: "http://prometheus:9090/api/v1/write"
    tls:
      insecure: true  # Use false with proper certs in production
    resource_to_telemetry_conversion:
      enabled: true
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 120s

Loki Exporter

exporters:
  loki:
    endpoint: "http://loki-gateway:3100/loki/api/v1/push"
    tenant_id: "default"
    labels:
      attributes:
        cluster: ""
        namespace: ""
        pod: ""
        container: ""
      resource:
        cluster.name: "cluster"
        k8s.namespace.name: "namespace"
        k8s.pod.name: "pod"
        k8s.container.name: "container"

OTLP Exporter

exporters:
  otlp:
    endpoint: "tempo-distributor:4317"
    tls:
      insecure: true

  otlp/secure:
    endpoint: "otel-backend.example.com:4317"
    tls:
      cert_file: /etc/otel/certs/client.crt
      key_file: /etc/otel/certs/client.key
      ca_file: /etc/otel/certs/ca.crt

Debug Exporter

exporters:
  debug:
    verbosity: detailed  # basic, normal, detailed
    sampling_initial: 5
    sampling_thereafter: 200

Extensions

Health Check

extensions:
  health_check:
    endpoint: ${env:MY_POD_IP}:13133
    path: "/health"

Memory Ballast

extensions:
  memory_ballast:
    size_in_percentage: 20  # Reserve 20% for GC optimization

zPages (Debug)

extensions:
  zpages:
    endpoint: localhost:55679

pprof (Profiling)

extensions:
  pprof:
    endpoint: localhost:1777

Service Configuration

Complete Pipeline Example

service:
  extensions: [health_check, memory_ballast]

  telemetry:
    logs:
      level: info  # debug, info, warn, error
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: ${env:MY_POD_IP}
                port: 8888

  pipelines:
    metrics:
      receivers: [prometheus, otlp]
      processors: [memory_limiter, k8sattributes, resource, batch]
      exporters: [prometheusremotewrite]

    logs:
      receivers: [otlp, filelog]
      processors: [memory_limiter, k8sattributes, resource, batch]
      exporters: [loki]

    traces:
      receivers: [otlp, jaeger, zipkin]
      processors: [memory_limiter, k8sattributes, resource, batch]
      exporters: [otlp/tempo]

Environment Variables

Common environment variables for Helm deployments:

extraEnvs:
  - name: K8S_NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName
  - name: K8S_POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name
  - name: K8S_POD_NAMESPACE
    valueFrom:
      fieldRef:
        fieldPath: metadata.namespace
  - name: K8S_POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP
  - name: MY_POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP
  - name: CLUSTER_NAME
    value: "my-cluster"
  - name: ENVIRONMENT
    value: "production"

Configuration Syntax

Use ${env:VAR_NAME} for environment variable substitution:

endpoint: ${env:MY_POD_IP}:4317

With default values:

endpoint: ${env:MY_POD_IP:-0.0.0.0}:4317

Validate Configuration

# Validate config file
otelcol validate --config=config.yaml

# List available components
otelcol components

OpenTelemetry Instrumentation Guide

Overview

Instrumentation is how applications generate telemetry data. OpenTelemetry supports:

• Zero-code/Auto-instrumentation: Automatic via agents/libraries
• Code-based instrumentation: Explicit SDK usage

Environment Variables (Universal)

All OpenTelemetry SDKs respect these environment variables:

# Exporter endpoint
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317

# Protocol (grpc or http/protobuf)
OTEL_EXPORTER_OTLP_PROTOCOL=grpc

# Service identification
OTEL_SERVICE_NAME=my-service
OTEL_SERVICE_VERSION=1.0.0

# Resource attributes
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,service.namespace=my-ns

# Trace sampling
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

# Exporter timeouts
OTEL_EXPORTER_OTLP_TIMEOUT=30000

Kubernetes Pod Configuration

Basic OTLP Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
      - name: app
        env:
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "http://otel-collector.monitoring.svc.cluster.local:4317"
        - name: OTEL_SERVICE_NAME
          value: "my-app"
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "service.namespace=my-namespace,deployment.environment=production"

With Pod Metadata Injection

env:
- name: OTEL_SERVICE_NAME
  valueFrom:
    fieldRef:
      fieldPath: metadata.labels['app.kubernetes.io/name']
- name: OTEL_RESOURCE_ATTRIBUTES
  value: "k8s.pod.name=$(POD_NAME),k8s.namespace.name=$(POD_NAMESPACE)"
- name: POD_NAME
  valueFrom:
    fieldRef:
      fieldPath: metadata.name
- name: POD_NAMESPACE
  valueFrom:
    fieldRef:
      fieldPath: metadata.namespace

Language-Specific Instrumentation

Java

Auto-instrumentation (Agent):

FROM eclipse-temurin:17-jre
COPY opentelemetry-javaagent.jar /opt/
ENV JAVA_TOOL_OPTIONS="-javaagent:/opt/opentelemetry-javaagent.jar"
ENV OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
ENV OTEL_SERVICE_NAME=my-java-app

Kubernetes deployment:

env:
- name: JAVA_TOOL_OPTIONS
  value: "-javaagent:/opt/opentelemetry-javaagent.jar"
- name: OTEL_EXPORTER_OTLP_ENDPOINT
  value: "http://otel-collector.monitoring:4317"
- name: OTEL_SERVICE_NAME
  value: "my-java-app"
- name: OTEL_TRACES_EXPORTER
  value: "otlp"
- name: OTEL_METRICS_EXPORTER
  value: "otlp"
- name: OTEL_LOGS_EXPORTER
  value: "otlp"

Manual instrumentation:

// Add dependencies
// io.opentelemetry:opentelemetry-api
// io.opentelemetry:opentelemetry-sdk
// io.opentelemetry:opentelemetry-exporter-otlp

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.api.trace.Span;

Tracer tracer = GlobalOpenTelemetry.getTracer("my-instrumentation");
Span span = tracer.spanBuilder("my-operation").startSpan();
try {
    // Do work
} finally {
    span.end();
}

Python

Auto-instrumentation:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
opentelemetry-instrument python app.py

Kubernetes deployment:

env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
  value: "http://otel-collector.monitoring:4317"
- name: OTEL_SERVICE_NAME
  value: "my-python-app"
- name: OTEL_PYTHON_LOG_CORRELATION
  value: "true"
command: ["opentelemetry-instrument", "python", "app.py"]

Manual instrumentation:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Setup
trace.set_tracer_provider(TracerProvider())
otlp_exporter = OTLPSpanExporter(endpoint="otel-collector:4317", insecure=True)
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(otlp_exporter))

# Usage
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("my-operation") as span:
    span.set_attribute("key", "value")
    # Do work

Node.js/JavaScript

Auto-instrumentation:

npm install @opentelemetry/auto-instrumentations-node

// tracing.js - load before app
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://otel-collector:4317',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

Kubernetes deployment:

env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
  value: "http://otel-collector.monitoring:4317"
- name: OTEL_SERVICE_NAME
  value: "my-node-app"
- name: NODE_OPTIONS
  value: "--require ./tracing.js"

Go

Manual instrumentation:

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/trace"
)

func initTracer() (*trace.TracerProvider, error) {
    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("otel-collector:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }

    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
    )
    otel.SetTracerProvider(tp)
    return tp, nil
}

// Usage
tracer := otel.Tracer("my-instrumentation")
ctx, span := tracer.Start(ctx, "my-operation")
defer span.End()

.NET

Auto-instrumentation:

// Add packages
// OpenTelemetry.Extensions.Hosting
// OpenTelemetry.Instrumentation.AspNetCore
// OpenTelemetry.Exporter.OpenTelemetryProtocol

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddOtlpExporter(options =>
        {
            options.Endpoint = new Uri("http://otel-collector:4317");
        }));

Kubernetes Operator (Auto-Instrumentation)

Install Operator

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

Create Instrumentation Resource

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
  namespace: my-namespace
spec:
  exporter:
    endpoint: http://otel-collector.monitoring:4317
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "0.1"
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
  dotnet:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:latest

Annotate Pods for Auto-Instrumentation

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    metadata:
      annotations:
        # Choose one based on language
        instrumentation.opentelemetry.io/inject-java: "true"
        # instrumentation.opentelemetry.io/inject-python: "true"
        # instrumentation.opentelemetry.io/inject-nodejs: "true"
        # instrumentation.opentelemetry.io/inject-dotnet: "true"

Semantic Conventions

Use standard attribute names for interoperability:

Service Attributes

service.name
service.version
service.namespace
service.instance.id

Kubernetes Attributes

k8s.cluster.name
k8s.namespace.name
k8s.pod.name
k8s.pod.uid
k8s.deployment.name
k8s.node.name
k8s.container.name

HTTP Attributes

http.method
http.url
http.status_code
http.route
http.user_agent

Database Attributes

db.system
db.name
db.operation
db.statement

Testing Instrumentation

Verify Traces

# Check collector logs for received spans
kubectl logs -n monitoring -l app.kubernetes.io/name=otel-collector | grep -i span

# Use debug exporter
kubectl logs -n monitoring -l app.kubernetes.io/name=otel-collector | grep -A 20 "ResourceSpans"

Generate Test Traffic

# Simple curl test
for i in {1..10}; do
  curl -s http://my-app.default/api/test
  sleep 1
done

Verify in Backend

# Grafana Tempo query
kubectl port-forward -n monitoring svc/tempo 3100:3100
curl http://localhost:3100/api/search?q=service.name=my-app

# Jaeger UI
kubectl port-forward -n monitoring svc/jaeger-query 16686:16686
# Open http://localhost:16686

OpenTelemetry Kubernetes Deployment Reference

Helm Chart Installation

Add Repository

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

Basic Installation

helm install otel-collector open-telemetry/opentelemetry-collector \
  --namespace monitoring \
  --create-namespace \
  --set mode=daemonset

Installation with Values File

helm install otel-collector open-telemetry/opentelemetry-collector \
  --namespace monitoring \
  --create-namespace \
  -f values.yaml

Deployment Modes

DaemonSet Mode (Node-level Collection)

mode: "daemonset"

# Required for host metrics
hostNetwork: false

# Mount host paths for log collection
presets:
  logsCollection:
    enabled: true
  hostMetrics:
    enabled: true

Use DaemonSet when:

• Collecting host-level metrics
• Collecting container logs from nodes
• Need data from every node in cluster

Deployment Mode (Centralized Gateway)

mode: "deployment"
replicaCount: 3

# Enable cluster-level metrics
presets:
  kubernetesEvents:
    enabled: true
  clusterMetrics:
    enabled: true

# Autoscaling
autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70

# PodDisruptionBudget
podDisruptionBudget:
  enabled: true
  minAvailable: 1

Use Deployment when:

• Acting as a central gateway
• Need horizontal scaling
• Collecting cluster-wide metrics/events

Sidecar Mode (Per-pod Collection)

mode: "sidecar"

Use Sidecar when:

• Need per-application isolation
• Specific apps require custom processing

Helm Values Reference

Complete Production Values

nameOverride: "otel-collector"
mode: "daemonset"
namespaceOverride: "monitoring"

# Presets
presets:
  logsCollection:
    enabled: true
    includeCollectorLogs: false
    storeCheckpoints: true
    maxRecombineLogSize: 102400
  hostMetrics:
    enabled: true
  kubernetesAttributes:
    enabled: true
    extractAllPodLabels: true
    extractAllPodAnnotations: false
  kubeletMetrics:
    enabled: true
  kubernetesEvents:
    enabled: false  # Enable in deployment mode
  clusterMetrics:
    enabled: false  # Enable in deployment mode

# Go Memory Management
useGOMEMLIMIT: true

# Resources
resources:
  limits:
    cpu: 500m
    memory: 1Gi
  requests:
    cpu: 100m
    memory: 256Mi

# Service Account
serviceAccount:
  create: true
  name: "otel-collector"

# ClusterRole
clusterRole:
  create: true
  rules:
    - apiGroups: [""]
      resources: ["pods", "namespaces", "nodes", "nodes/proxy", "services", "endpoints", "events"]
      verbs: ["get", "list", "watch"]
    - apiGroups: ["apps"]
      resources: ["replicasets", "deployments", "daemonsets", "statefulsets"]
      verbs: ["get", "list", "watch"]
    - apiGroups: ["batch"]
      resources: ["jobs", "cronjobs"]
      verbs: ["get", "list", "watch"]
    - nonResourceURLs: ["/metrics", "/metrics/cadvisor"]
      verbs: ["get"]

# Security Context
podSecurityContext:
  runAsNonRoot: true
  runAsUser: 10001
  fsGroup: 10001

securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL
  readOnlyRootFilesystem: true

# Ports
ports:
  otlp:
    enabled: true
    containerPort: 4317
    servicePort: 4317
    hostPort: 4317
    protocol: TCP
    appProtocol: grpc
  otlp-http:
    enabled: true
    containerPort: 4318
    servicePort: 4318
    hostPort: 4318
    protocol: TCP
  metrics:
    enabled: true
    containerPort: 8888
    servicePort: 8888
    protocol: TCP

# Monitoring
serviceMonitor:
  enabled: true
  extraLabels:
    prometheus: kube-prometheus-stack
  metricsEndpoints:
    - port: metrics
      interval: 30s
      path: /metrics

podMonitor:
  enabled: true
  extraLabels:
    prometheus: kube-prometheus-stack

# Labels
podLabels:
  app.kubernetes.io/name: otel-collector
  app.kubernetes.io/component: telemetry
  app.kubernetes.io/part-of: observability-stack

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "8888"
  prometheus.io/path: "/metrics"

# Anti-Affinity
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values:
                  - otel-collector
          topologyKey: kubernetes.io/hostname

# Probes
livenessProbe:
  httpGet:
    port: 13133
    path: /

readinessProbe:
  httpGet:
    port: 13133
    path: /

Environment-Specific Overlays

Development

# Lower resources
resources:
  limits:
    cpu: 250m
    memory: 512Mi
  requests:
    cpu: 50m
    memory: 128Mi

# Spot instance tolerations (AKS)
tolerations:
  - key: kubernetes.azure.com/scalesetpriority
    operator: Equal
    value: "spot"
    effect: NoSchedule

# Debug enabled
config:
  service:
    telemetry:
      logs:
        level: debug
    pipelines:
      traces:
        exporters: [debug]

Production

# Higher resources
resources:
  limits:
    cpu: 1000m
    memory: 2Gi
  requests:
    cpu: 250m
    memory: 512Mi

# Autoscaling (deployment mode)
autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70
  targetMemoryUtilizationPercentage: 80

# PDB
podDisruptionBudget:
  enabled: true
  minAvailable: 1

# Sampling
config:
  processors:
    probabilistic_sampler:
      sampling_percentage: 10

ArgoCD ApplicationSet Example

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: otel
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - cluster: dev
            url: "https://dev-cluster:443"
          - cluster: prd
            url: "https://prd-cluster:443"
  template:
    metadata:
      name: "{{cluster}}-otel"
    spec:
      project: default
      sources:
        - chart: opentelemetry-collector
          repoURL: "https://open-telemetry.github.io/opentelemetry-helm-charts"
          targetRevision: "0.130.0"
          helm:
            valueFiles:
              - $values/otel/{{cluster}}/values.yaml
        - repoURL: "https://your-repo/helm-values"
          targetRevision: main
          ref: values
      destination:
        server: "{{url}}"
        namespace: monitoring
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true

RBAC Configuration

ClusterRole for Full Kubernetes Access

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-collector
rules:
  - apiGroups: [""]
    resources:
      - pods
      - namespaces
      - nodes
      - nodes/proxy
      - nodes/stats
      - services
      - endpoints
      - events
      - configmaps
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources:
      - replicasets
      - deployments
      - daemonsets
      - statefulsets
    verbs: ["get", "list", "watch"]
  - apiGroups: ["batch"]
    resources:
      - jobs
      - cronjobs
    verbs: ["get", "list", "watch"]
  - nonResourceURLs:
      - "/metrics"
      - "/metrics/cadvisor"
    verbs: ["get"]

Network Policies

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: otel-collector
  namespace: monitoring
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: otel-collector
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector: {}
      ports:
        - port: 4317  # OTLP gRPC
        - port: 4318  # OTLP HTTP
        - port: 8888  # Metrics
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              name: monitoring
      ports:
        - port: 9090  # Prometheus
        - port: 3100  # Loki
        - port: 4317  # Tempo

Verification Commands

# Check pods
kubectl get pods -n monitoring -l app.kubernetes.io/name=otel-collector

# Check DaemonSet rollout
kubectl rollout status daemonset/otel-collector -n monitoring

# Check logs
kubectl logs -n monitoring -l app.kubernetes.io/name=otel-collector --tail=100

# Check service endpoints
kubectl get endpoints -n monitoring otel-collector

# Port forward for testing
kubectl port-forward -n monitoring svc/otel-collector 4318:4318

# Check ServiceMonitor
kubectl get servicemonitor -n monitoring otel-collector -o yaml

# Describe pod for troubleshooting
kubectl describe pod -n monitoring -l app.kubernetes.io/name=otel-collector

OpenTelemetry Troubleshooting Guide

Diagnostic Tools

Enable Debug Logging

config:
  service:
    telemetry:
      logs:
        level: debug

Enable Debug Exporter

config:
  exporters:
    debug:
      verbosity: detailed
      sampling_initial: 5
      sampling_thereafter: 200

  service:
    pipelines:
      traces:
        exporters: [debug]
      metrics:
        exporters: [debug]
      logs:
        exporters: [debug]

Enable zPages Extension

extensions:
  zpages:
    endpoint: localhost:55679

service:
  extensions: [zpages]

Access at:

• /debug/tracez - Trace data inspection
• /debug/pipelinez - Pipeline status

Enable pprof for Profiling

extensions:
  pprof:
    endpoint: localhost:1777

service:
  extensions: [pprof]

Common Issues

1. Collector Not Starting

Check logs:

kubectl logs -n monitoring -l app.kubernetes.io/name=otel-collector --previous

Common causes:

• Invalid configuration syntax
• Missing required extensions
• Port conflicts

Validate config:

otelcol validate --config=config.yaml

2. Data Not Being Received

Symptoms: No data in exporters, empty pipelines

Diagnostic steps:

1. Check receiver is listening:

kubectl exec -n monitoring -it deploy/otel-collector -- netstat -tlnp

2. Test OTLP endpoint:

kubectl run test-otlp --image=curlimages/curl:latest --rm -it -- \
  curl -v http://otel-collector.monitoring:4318/v1/traces \
  -H "Content-Type: application/json" \
  -d '{"resourceSpans":[]}'

3. Check network policies:

kubectl get networkpolicies -n monitoring
kubectl describe networkpolicy -n monitoring

4. Verify service discovery:

kubectl get endpoints -n monitoring otel-collector

Common causes:

• Firewall/network policy blocking traffic
• Wrong endpoint configuration
• Service not ready

3. Data Not Being Exported

Symptoms: Data received but not appearing in backends

Diagnostic steps:

1. Check exporter logs:

kubectl logs -n monitoring -l app.kubernetes.io/name=otel-collector | grep -i export

2. Verify backend connectivity:

kubectl exec -n monitoring -it deploy/otel-collector -- \
  wget -O- http://prometheus:9090/-/healthy

3. Check exporter metrics:

kubectl exec -n monitoring -it deploy/otel-collector -- \
  curl localhost:8888/metrics | grep otelcol_exporter

Key metrics to check:

• otelcol_exporter_sent_metric_points - Successfully sent
• otelcol_exporter_send_failed_metric_points - Failed to send
• otelcol_exporter_queue_size - Queue backlog

Common causes:

• Backend unreachable
• TLS/authentication issues
• Rate limiting
• Backend capacity

4. High Memory Usage / OOM

Symptoms: Pods being OOMKilled, high memory consumption

Diagnostic steps:

1. Check memory usage:

kubectl top pods -n monitoring -l app.kubernetes.io/name=otel-collector

2. Check memory limiter:

kubectl logs -n monitoring -l app.kubernetes.io/name=otel-collector | grep -i "memory"

Solutions:

# Enable memory limiter processor
processors:
  memory_limiter:
    check_interval: 5s
    limit_percentage: 80
    spike_limit_percentage: 25

# Enable GOMEMLIMIT
useGOMEMLIMIT: true

# Add memory ballast
extensions:
  memory_ballast:
    size_in_percentage: 20

# Increase limits
resources:
  limits:
    memory: 2Gi

5. Collector Restarts / Crashes

Symptoms: Frequent pod restarts

Diagnostic steps:

1. Check restart count:

kubectl get pods -n monitoring -l app.kubernetes.io/name=otel-collector

2. Check previous logs:

kubectl logs -n monitoring -l app.kubernetes.io/name=otel-collector --previous

3. Check events:

kubectl get events -n monitoring --sort-by='.lastTimestamp' | grep otel

Common causes:

• OOM kills
• Liveness probe failures
• Configuration errors

6. High CPU Usage

Symptoms: CPU throttling, slow processing

Solutions:

# Increase batch size
processors:
  batch:
    timeout: 30s
    send_batch_size: 2048

# Enable sampling
processors:
  probabilistic_sampler:
    sampling_percentage: 10

# Increase CPU limits
resources:
  limits:
    cpu: 1000m

7. Pod Scheduling Issues

Symptoms: Pods pending, not scheduling

Diagnostic steps:

1. Check pod events:

kubectl describe pod -n monitoring -l app.kubernetes.io/name=otel-collector

2. Check node taints:

kubectl describe nodes | grep -A 5 Taints

Solution for spot instances (AKS):

tolerations:
  - key: kubernetes.azure.com/scalesetpriority
    operator: Equal
    value: "spot"
    effect: NoSchedule

Metrics to Monitor

Collector Health

# Uptime
up{job="otel-collector"}

# Process uptime
otelcol_process_uptime

# Memory usage
otelcol_process_runtime_heap_alloc_bytes

Pipeline Performance

# Received data points
rate(otelcol_receiver_accepted_metric_points[5m])
rate(otelcol_receiver_accepted_spans[5m])
rate(otelcol_receiver_accepted_log_records[5m])

# Refused data points
rate(otelcol_receiver_refused_metric_points[5m])

# Exported data points
rate(otelcol_exporter_sent_metric_points[5m])
rate(otelcol_exporter_sent_spans[5m])

# Export failures
rate(otelcol_exporter_send_failed_metric_points[5m])

# Queue size
otelcol_exporter_queue_size

Processor Performance

# Batch sizes
otelcol_processor_batch_batch_send_size

# Dropped data
rate(otelcol_processor_dropped_metric_points[5m])

Prometheus Alerts

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: otel-collector-alerts
  namespace: monitoring
spec:
  groups:
  - name: otel-collector
    rules:
    - alert: OTelCollectorDown
      expr: up{job="otel-collector"} == 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "OTel Collector is down"
        description: "OTel Collector {{ $labels.pod }} has been down for 5 minutes"

    - alert: OTelCollectorHighMemory
      expr: |
        container_memory_usage_bytes{pod=~"otel-collector.*"}
        / container_spec_memory_limit_bytes{pod=~"otel-collector.*"} > 0.9
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "OTel Collector memory usage above 90%"

    - alert: OTelCollectorExportFailures
      expr: rate(otelcol_exporter_send_failed_metric_points[5m]) > 0
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "OTel Collector failing to export metrics"

    - alert: OTelCollectorQueueFull
      expr: otelcol_exporter_queue_size > 1000
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "OTel Collector export queue is filling up"

    - alert: OTelCollectorReceiverRefused
      expr: rate(otelcol_receiver_refused_metric_points[5m]) > 100
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "OTel Collector refusing data points"

Log Analysis

Common Log Patterns

Successful startup:

Everything is ready. Begin running and processing data.

Memory pressure:

Memory usage is above soft limit
Data is being dropped due to memory pressure

Export failures:

Exporting failed. Will retry
rpc error: code = Unavailable
connection refused

Configuration errors:

Error decoding config
unknown component
failed to create

Useful grep commands

# Check for errors
kubectl logs -n monitoring -l app.kubernetes.io/name=otel-collector | grep -i error

# Check memory issues
kubectl logs -n monitoring -l app.kubernetes.io/name=otel-collector | grep -i memory

# Check export status
kubectl logs -n monitoring -l app.kubernetes.io/name=otel-collector | grep -i export

# Check receiver status
kubectl logs -n monitoring -l app.kubernetes.io/name=otel-collector | grep -i receiver

Quick Fixes

Restart Collector

kubectl rollout restart daemonset/otel-collector -n monitoring
# or
kubectl rollout restart deployment/otel-collector -n monitoring

Force Sync ArgoCD

argocd app sync <cluster>-otel --force

Scale Down/Up

kubectl scale deployment otel-collector -n monitoring --replicas=0
kubectl scale deployment otel-collector -n monitoring --replicas=3

Clear Checkpoints (Log Collection)

kubectl exec -n monitoring -it <pod> -- rm -rf /var/lib/otelcol/checkpoints