Spring Boot Observability

Spring Boot Observability

Production observability with Actuator endpoints, Micrometer metrics, and OpenTelemetry tracing.

Core Components

Core Workflow

1. Add starters → actuator, micrometer-registry-*, opentelemetry 2. Configure endpoint exposure → Secure sensitive endpoints 3. Define health groups → Separate liveness from readiness 4. Add custom metrics → Business-specific measurements 5. Configure tracing → Sampling, propagation, export

Quick Patterns

See EXAMPLES.md for complete working examples including:

• Production Actuator Configuration with health groups and Kubernetes probes
• Custom Health Indicator with latency monitoring (Java + Kotlin)
• Custom Micrometer Metrics with Counter, Timer, and Gauge patterns
• OpenTelemetry Span Customization with Observation API
• OpenTelemetry Configuration for OTLP export
• Actuator Endpoint Access Control (Boot 4)

Spring Boot 4 Specifics

• OpenTelemetry is the default tracer (replaces Brave)
• Health Indicator imports from org.springframework.boot.health.contributor.*
• Endpoint Access Control with access: none/unrestricted/read-only

Detailed References

• Examples: See EXAMPLES.md for complete working code examples
• Troubleshooting: See TROUBLESHOOTING.md for common issues and Boot 4 migration
• Actuator Endpoints: See references/ACTUATOR.md for endpoint configuration, security, custom endpoints
• Micrometer Metrics: See references/METRICS.md for Timer, Counter, Gauge, DistributionSummary patterns
• Distributed Tracing: See references/TRACING.md for OpenTelemetry, span customization, context propagation

Anti-Pattern Checklist

Related Skills

Critical Reminders

1. Separate liveness from readiness — Liveness: "is process alive?", Readiness: "can handle traffic?" 2. Low cardinality tags only — User IDs, request IDs = bad; status codes, regions = good 3. Secure Actuator endpoints — Use separate port or authentication 4. Sample traces in production — 100% sampling overwhelms collectors 5. Graceful shutdown — Allow in-flight requests to complete

Spring Boot Observability Examples

Complete working examples for Spring Boot 4 observability patterns.

Production Actuator Configuration

Comprehensive actuator setup with health groups and Kubernetes probes.

server:
  shutdown: graceful

spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s

management:
  server:
    port: 8081  # Separate management port
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus
      base-path: /manage
    access:
      default: none
  endpoint:
    health:
      show-details: when-authorized
      probes:
        enabled: true
      group:
        liveness:
          include: livenessState,ping
        readiness:
          include: readinessState,db,redis,diskSpace
  health:
    defaults:
      enabled: true

Key points:

• Use separate management port for production security
• Enable graceful shutdown to complete in-flight requests
• Separate liveness (is alive?) from readiness (can handle traffic?) probes

---

Custom Health Indicator

Health indicator with latency monitoring and threshold-based status.

Java

@Component
public class ExternalApiHealthIndicator implements HealthIndicator {

    private final ExternalApiClient apiClient;
    private static final Duration TIMEOUT = Duration.ofSeconds(5);

    @Override
    public Health health() {
        try {
            long start = System.currentTimeMillis();
            apiClient.ping();
            long latency = System.currentTimeMillis() - start;

            if (latency > 3000) {
                return Health.down()
                    .withDetail("latency_ms", latency)
                    .withDetail("reason", "Response time exceeded threshold")
                    .build();
            }
            return Health.up()
                .withDetail("latency_ms", latency)
                .build();
        } catch (Exception e) {
            return Health.down(e)
                .withDetail("error", e.getMessage())
                .build();
        }
    }
}

Kotlin

@Component
class ExternalApiHealthIndicator(private val apiClient: ExternalApiClient) : HealthIndicator {

    override fun health(): Health = runCatching {
        val start = System.currentTimeMillis()
        apiClient.ping()
        val latency = System.currentTimeMillis() - start

        if (latency > 3000) {
            Health.down()
                .withDetail("latency_ms", latency)
                .withDetail("reason", "Response time exceeded threshold")
                .build()
        } else {
            Health.up().withDetail("latency_ms", latency).build()
        }
    }.getOrElse { Health.down(it).build() }
}

Key points:

• Include latency details for debugging
• Define clear thresholds for degraded state
• Catch exceptions to prevent health check failures

---

Custom Micrometer Metrics

Complete metrics class with Counter, Timer, and Gauge patterns.

@Component
public class OrderMetrics {

    private final Counter ordersCreated;
    private final Timer orderProcessingTime;
    private final AtomicInteger activeOrders = new AtomicInteger(0);

    public OrderMetrics(MeterRegistry registry) {
        this.ordersCreated = Counter.builder("orders.created.total")
            .description("Total orders created")
            .tag("channel", "web")
            .register(registry);

        this.orderProcessingTime = Timer.builder("orders.processing.duration")
            .description("Order processing duration")
            .publishPercentiles(0.5, 0.95, 0.99)
            .publishPercentileHistogram()
            .serviceLevelObjectives(
                Duration.ofMillis(100),
                Duration.ofMillis(500),
                Duration.ofSeconds(1)
            )
            .register(registry);

        Gauge.builder("orders.active", activeOrders, AtomicInteger::get)
            .description("Currently active orders")
            .register(registry);
    }

    public void recordOrderCreated() {
        ordersCreated.increment();
        activeOrders.incrementAndGet();
    }

    public <T> T recordProcessing(Supplier<T> operation) {
        return orderProcessingTime.record(operation);
    }

    public void orderCompleted() {
        activeOrders.decrementAndGet();
    }
}

Key points:

• Use Counter for monotonically increasing values
• Use Timer with percentiles for duration measurements
• Use Gauge for current state values
• Always add descriptions for metric discovery

---

OpenTelemetry Span Customization

Custom spans with the Observation API.

@Component
public class PaymentProcessor {

    private final ObservationRegistry observationRegistry;

    public PaymentResult process(PaymentRequest request) {
        return Observation.createNotStarted("payment.processing", observationRegistry)
            .lowCardinalityKeyValue("payment.method", request.method().name())
            .lowCardinalityKeyValue("currency", request.currency())
            .highCardinalityKeyValue("merchant.id", request.merchantId())
            .observe(() -> executePayment(request));
    }
}

Key points:

• Use lowCardinalityKeyValue for tags used in aggregations (enum values, status codes)
• Use highCardinalityKeyValue for tracing only (IDs, user names)
• Observation API creates both spans and metrics automatically

---

OpenTelemetry Configuration

Complete OTLP exporter configuration.

management:
  tracing:
    sampling:
      probability: 0.1  # 10% in production
  opentelemetry:
    resource-attributes:
      service.name: my-service
      deployment.environment: production
    tracing:
      export:
        otlp:
          endpoint: http://otel-collector:4318/v1/traces

Key points:

• Set sampling probability based on traffic volume
• Include service.name for trace identification
• Use deployment.environment for filtering in observability tools

---

Actuator Endpoint Access Control (Boot 4)

Granular access control for endpoints.

management:
  endpoints:
    access:
      default: none  # Deny by default
    web:
      exposure:
        include: health,info,prometheus
  endpoint:
    health:
      access: unrestricted
    prometheus:
      access: read-only

Key points:

• Default to none access for defense in depth
• Use unrestricted only for health probes
• read-only prevents endpoint state changes

Actuator Endpoints

Endpoint configuration, security, and custom endpoints.

Table of Contents

• Endpoint Exposure Configuration
• Development Profile
• Production Profile
• Health Indicators
• Built-in Health Indicators
• Custom Health Indicator
• Kotlin Health Indicator
• Reactive Health Indicator
• Health Groups (Kubernetes Probes)
• Info Endpoint
• Custom Info Contributor
• Custom Actuator Endpoint
• Actuator Security
• Separate Security Filter Chain
• Users for Actuator
• Prometheus Integration
• Dependencies
• Configuration
• Prometheus Scrape Config
• Graceful Shutdown
• Environment and Loggers Endpoints

Endpoint Exposure Configuration

Development Profile

# application-dev.yml
management:
  endpoints:
    web:
      exposure:
        include: "*"
  endpoint:
    health:
      show-details: always

Production Profile

# application-prod.yml
management:
  server:
    port: 8081
    address: 127.0.0.1  # Only localhost
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus
      base-path: /actuator
    access:
      default: none
  endpoint:
    health:
      show-details: when-authorized
      access: unrestricted  # Health always accessible
    prometheus:
      access: read-only
    info:
      access: read-only

Health Indicators

Built-in Health Indicators

Custom Health Indicator

import org.springframework.boot.health.contributor.Health;
import org.springframework.boot.health.contributor.HealthIndicator;
import org.springframework.stereotype.Component;

@Component
public class PaymentGatewayHealthIndicator implements HealthIndicator {

    private final PaymentGatewayClient client;
    private final CircuitBreaker circuitBreaker;

    @Override
    public Health health() {
        if (circuitBreaker.getState() == CircuitBreaker.State.OPEN) {
            return Health.down()
                .withDetail("circuit_breaker", "OPEN")
                .withDetail("reason", "Too many failures")
                .build();
        }

        try {
            PaymentGatewayStatus status = client.getStatus();

            if (!status.isOperational()) {
                return Health.outOfService()
                    .withDetail("gateway_status", status.getMessage())
                    .build();
            }

            return Health.up()
                .withDetail("gateway_version", status.getVersion())
                .withDetail("response_time_ms", status.getLatency())
                .build();

        } catch (Exception e) {
            return Health.down()
                .withException(e)
                .build();
        }
    }
}

Kotlin Health Indicator

@Component
class PaymentGatewayHealthIndicator(
    private val client: PaymentGatewayClient,
    private val circuitBreaker: CircuitBreaker
) : HealthIndicator {

    override fun health(): Health {
        if (circuitBreaker.state == CircuitBreaker.State.OPEN) {
            return Health.down()
                .withDetail("circuit_breaker", "OPEN")
                .build()
        }

        return runCatching { client.getStatus() }
            .fold(
                onSuccess = { status ->
                    if (status.isOperational) {
                        Health.up()
                            .withDetail("gateway_version", status.version)
                            .withDetail("response_time_ms", status.latency)
                            .build()
                    } else {
                        Health.outOfService()
                            .withDetail("gateway_status", status.message)
                            .build()
                    }
                },
                onFailure = { Health.down().withException(it).build() }
            )
    }
}

Reactive Health Indicator

@Component
public class ReactiveExternalServiceHealthIndicator implements ReactiveHealthIndicator {

    private final WebClient webClient;

    @Override
    public Mono<Health> health() {
        return webClient.get()
            .uri("/health")
            .retrieve()
            .bodyToMono(HealthStatus.class)
            .map(status -> Health.up()
                .withDetail("service", status.name())
                .build())
            .timeout(Duration.ofSeconds(5))
            .onErrorResume(e -> Mono.just(
                Health.down()
                    .withException(e)
                    .build()
            ));
    }
}

Health Groups (Kubernetes Probes)

management:
  endpoint:
    health:
      probes:
        enabled: true
        add-additional-paths: true  # Exposes /livez and /readyz
      group:
        liveness:
          include: livenessState,ping
          show-details: never
        readiness:
          include: readinessState,db,redis,kafka,customService
          show-details: when-authorized
        startup:
          include: livenessState

Kubernetes deployment:

spec:
  containers:
    - name: app
      livenessProbe:
        httpGet:
          path: /actuator/health/liveness
          port: 8081
        initialDelaySeconds: 10
        periodSeconds: 10
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /actuator/health/readiness
          port: 8081
        initialDelaySeconds: 5
        periodSeconds: 5
        failureThreshold: 3
      startupProbe:
        httpGet:
          path: /actuator/health/liveness
          port: 8081
        initialDelaySeconds: 0
        periodSeconds: 5
        failureThreshold: 30  # 30 * 5 = 150s max startup time

Info Endpoint

management:
  info:
    env:
      enabled: true
    git:
      enabled: true
      mode: full
    build:
      enabled: true
    java:
      enabled: true
    os:
      enabled: true

info:
  app:
    name: "@project.name@"
    version: "@project.version@"
    description: "@project.description@"
  contact:
    team: platform-team
    slack: "#platform-support"

Custom Info Contributor

@Component
public class FeatureFlagsInfoContributor implements InfoContributor {

    private final FeatureFlagService featureFlags;

    @Override
    public void contribute(Info.Builder builder) {
        Map<String, Boolean> flags = featureFlags.getAllFlags();
        builder.withDetail("features", flags);
    }
}

Custom Actuator Endpoint

@Component
@Endpoint(id = "cache")
public class CacheEndpoint {

    private final CacheManager cacheManager;

    @ReadOperation
    public Map<String, CacheStats> caches() {
        return cacheManager.getCacheNames().stream()
            .collect(Collectors.toMap(
                name -> name,
                name -> getCacheStats(cacheManager.getCache(name))
            ));
    }

    @ReadOperation
    public CacheStats cache(@Selector String name) {
        Cache cache = cacheManager.getCache(name);
        if (cache == null) {
            throw new IllegalArgumentException("Cache not found: " + name);
        }
        return getCacheStats(cache);
    }

    @DeleteOperation
    public void clearCache(@Selector String name) {
        Cache cache = cacheManager.getCache(name);
        if (cache != null) {
            cache.clear();
        }
    }

    @WriteOperation
    public void clearAllCaches() {
        cacheManager.getCacheNames().forEach(name ->
            cacheManager.getCache(name).clear()
        );
    }

    private CacheStats getCacheStats(Cache cache) {
        // Implementation depends on cache provider
    }
}

Actuator Security

Separate Security Filter Chain

@Configuration
public class ActuatorSecurityConfig {

    @Bean
    @Order(1)
    public SecurityFilterChain actuatorSecurity(HttpSecurity http) throws Exception {
        http
            .securityMatcher(EndpointRequest.toAnyEndpoint())
            .authorizeHttpRequests(auth -> auth
                .requestMatchers(EndpointRequest.to("health", "info")).permitAll()
                .requestMatchers(EndpointRequest.to("prometheus")).hasRole("METRICS")
                .requestMatchers(EndpointRequest.to("loggers", "env")).hasRole("ADMIN")
                .anyRequest().hasRole("ACTUATOR")
            )
            .httpBasic(Customizer.withDefaults());
        return http.build();
    }
}

Users for Actuator

spring:
  security:
    user:
      name: actuator
      password: ${ACTUATOR_PASSWORD}
      roles: ACTUATOR,METRICS

Prometheus Integration

Dependencies

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

Configuration

management:
  endpoints:
    web:
      exposure:
        include: prometheus
  prometheus:
    metrics:
      export:
        enabled: true

Prometheus Scrape Config

# prometheus.yml
scrape_configs:
  - job_name: 'spring-boot-app'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 15s
    static_configs:
      - targets: ['app:8081']
    basic_auth:
      username: actuator
      password: secret

Graceful Shutdown

server:
  shutdown: graceful

spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s

management:
  endpoint:
    health:
      probes:
        enabled: true

Application responds to SIGTERM: 1. Health readiness → DOWN (stops receiving traffic) 2. Waits for in-flight requests (up to 30s) 3. Closes connections 4. Shuts down

Environment and Loggers Endpoints

management:
  endpoint:
    env:
      show-values: when-authorized  # or 'always', 'never'
    loggers:
      enabled: true

Change log level at runtime:

curl -X POST http://localhost:8081/actuator/loggers/com.example \
  -H 'Content-Type: application/json' \
  -d '{"configuredLevel": "DEBUG"}'

Micrometer Metrics

Timer, Counter, Gauge, and DistributionSummary patterns.

Table of Contents

• Meter Types
• Counter Patterns
• Basic Counter
• Counter with Tags
• Kotlin Counter
• Timer Patterns
• Basic Timer
• Timer with SLOs
• Timer with Tags
• @Timed Annotation
• Gauge Patterns
• Gauge from AtomicInteger
• Gauge from Collection
• Gauge from Method
• Kotlin Gauge
• DistributionSummary Patterns
• Request Size Tracking
• Batch Size Tracking
• MeterBinder Pattern
• Common Tags (Global)
• Tag Best Practices
• Registry Types
• Prometheus
• Datadog
• Multiple Registries

Meter Types

Counter Patterns

Basic Counter

@Component
public class OrderMetrics {

    private final Counter ordersCreated;
    private final Counter ordersFailed;

    public OrderMetrics(MeterRegistry registry) {
        this.ordersCreated = Counter.builder("orders.created")
            .description("Total orders created")
            .register(registry);

        this.ordersFailed = Counter.builder("orders.failed")
            .description("Total failed order attempts")
            .register(registry);
    }

    public void orderCreated() {
        ordersCreated.increment();
    }

    public void orderFailed() {
        ordersFailed.increment();
    }
}

Counter with Tags

@Component
public class PaymentMetrics {

    private final MeterRegistry registry;

    public void recordPayment(String method, String status, double amount) {
        Counter.builder("payments.total")
            .description("Total payments processed")
            .tag("method", method)      // credit_card, paypal, bank_transfer
            .tag("status", status)       // success, failed, pending
            .register(registry)
            .increment();

        // Also track amount
        DistributionSummary.builder("payments.amount")
            .description("Payment amounts")
            .tag("method", method)
            .tag("status", status)
            .baseUnit("dollars")
            .register(registry)
            .record(amount);
    }
}

Kotlin Counter

@Component
class OrderMetrics(registry: MeterRegistry) {

    private val ordersCreated = Counter.builder("orders.created")
        .description("Total orders created")
        .register(registry)

    private val ordersByChannel = mutableMapOf<String, Counter>()

    fun orderCreated(channel: String) {
        ordersCreated.increment()
        ordersByChannel.getOrPut(channel) {
            Counter.builder("orders.by_channel")
                .tag("channel", channel)
                .register(registry)
        }.increment()
    }
}

Timer Patterns

Basic Timer

@Component
public class ProcessingMetrics {

    private final Timer processingTimer;

    public ProcessingMetrics(MeterRegistry registry) {
        this.processingTimer = Timer.builder("processing.duration")
            .description("Processing duration")
            .publishPercentiles(0.5, 0.95, 0.99)
            .publishPercentileHistogram()
            .register(registry);
    }

    public <T> T recordProcessing(Supplier<T> operation) {
        return processingTimer.record(operation);
    }

    public void recordDuration(Duration duration) {
        processingTimer.record(duration);
    }
}

Timer with SLOs

Timer orderTimer = Timer.builder("order.processing.duration")
    .description("Order processing duration")
    .publishPercentiles(0.5, 0.75, 0.95, 0.99)
    .publishPercentileHistogram()
    .serviceLevelObjectives(
        Duration.ofMillis(100),   // Fast
        Duration.ofMillis(500),   // Normal
        Duration.ofSeconds(1),    // Slow
        Duration.ofSeconds(5)     // Very slow
    )
    .minimumExpectedValue(Duration.ofMillis(1))
    .maximumExpectedValue(Duration.ofSeconds(30))
    .register(registry);

Timer with Tags

public void recordApiCall(String endpoint, String method, int statusCode, Duration duration) {
    Timer.builder("http.client.requests")
        .description("HTTP client request duration")
        .tag("uri", endpoint)
        .tag("method", method)
        .tag("status", String.valueOf(statusCode))
        .tag("outcome", statusCode >= 400 ? "error" : "success")
        .register(registry)
        .record(duration);
}

@Timed Annotation

@Configuration
public class TimedConfig {
    @Bean
    public TimedAspect timedAspect(MeterRegistry registry) {
        return new TimedAspect(registry);
    }
}

@Service
public class OrderService {

    @Timed(value = "order.creation", percentiles = {0.5, 0.95, 0.99})
    public Order createOrder(CreateOrderRequest request) {
        // Method execution is automatically timed
    }

    @Timed(value = "order.processing", histogram = true)
    public void processOrder(Long orderId) {
        // With histogram for percentile approximation
    }
}

Gauge Patterns

Gauge from AtomicInteger

@Component
public class ConnectionMetrics {

    private final AtomicInteger activeConnections = new AtomicInteger(0);

    public ConnectionMetrics(MeterRegistry registry) {
        Gauge.builder("connections.active", activeConnections, AtomicInteger::get)
            .description("Number of active connections")
            .register(registry);
    }

    public void connectionOpened() {
        activeConnections.incrementAndGet();
    }

    public void connectionClosed() {
        activeConnections.decrementAndGet();
    }
}

Gauge from Collection

@Component
public class QueueMetrics {

    public QueueMetrics(MeterRegistry registry, BlockingQueue<?> workQueue) {
        Gauge.builder("queue.size", workQueue, BlockingQueue::size)
            .description("Current queue size")
            .register(registry);

        Gauge.builder("queue.remaining_capacity", workQueue, BlockingQueue::remainingCapacity)
            .description("Remaining queue capacity")
            .register(registry);
    }
}

Gauge from Method

@Component
public class CacheMetrics {

    private final CacheManager cacheManager;

    public CacheMetrics(MeterRegistry registry, CacheManager cacheManager) {
        this.cacheManager = cacheManager;

        Gauge.builder("cache.size", this, CacheMetrics::getTotalCacheSize)
            .description("Total items across all caches")
            .register(registry);
    }

    private double getTotalCacheSize() {
        return cacheManager.getCacheNames().stream()
            .mapToLong(name -> getCacheSize(cacheManager.getCache(name)))
            .sum();
    }
}

Kotlin Gauge

@Component
class QueueMetrics(registry: MeterRegistry, private val workQueue: BlockingQueue<*>) {

    init {
        Gauge.builder("queue.size", workQueue) { it.size.toDouble() }
            .description("Current queue size")
            .register(registry)
    }
}

DistributionSummary Patterns

Request Size Tracking

@Component
public class RequestMetrics {

    private final DistributionSummary requestSize;
    private final DistributionSummary responseSize;

    public RequestMetrics(MeterRegistry registry) {
        this.requestSize = DistributionSummary.builder("http.request.size")
            .description("Request body size")
            .baseUnit("bytes")
            .publishPercentiles(0.5, 0.95)
            .register(registry);

        this.responseSize = DistributionSummary.builder("http.response.size")
            .description("Response body size")
            .baseUnit("bytes")
            .publishPercentiles(0.5, 0.95)
            .register(registry);
    }

    public void recordRequest(long bytes) {
        requestSize.record(bytes);
    }

    public void recordResponse(long bytes) {
        responseSize.record(bytes);
    }
}

Batch Size Tracking

DistributionSummary batchSize = DistributionSummary.builder("batch.size")
    .description("Batch processing size")
    .publishPercentileHistogram()
    .serviceLevelObjectives(10, 50, 100, 500, 1000)
    .register(registry);

// Record batch sizes
batchSize.record(items.size());

MeterBinder Pattern

Auto-register metrics on startup:

@Component
public class DatabasePoolMetricsBinder implements MeterBinder {

    private final HikariDataSource dataSource;

    @Override
    public void bindTo(MeterRegistry registry) {
        Gauge.builder("db.pool.active", dataSource, ds -> ds.getHikariPoolMXBean().getActiveConnections())
            .description("Active database connections")
            .register(registry);

        Gauge.builder("db.pool.idle", dataSource, ds -> ds.getHikariPoolMXBean().getIdleConnections())
            .description("Idle database connections")
            .register(registry);

        Gauge.builder("db.pool.pending", dataSource, ds -> ds.getHikariPoolMXBean().getThreadsAwaitingConnection())
            .description("Threads waiting for connection")
            .register(registry);

        Gauge.builder("db.pool.total", dataSource, ds -> ds.getHikariPoolMXBean().getTotalConnections())
            .description("Total connections in pool")
            .register(registry);
    }
}

Common Tags (Global)

@Configuration
public class MetricsConfig {

    @Bean
    public MeterRegistryCustomizer<MeterRegistry> commonTags() {
        return registry -> registry.config()
            .commonTags(
                "application", "order-service",
                "environment", System.getenv("ENVIRONMENT"),
                "region", System.getenv("REGION")
            );
    }
}

Tag Best Practices

Registry Types

Prometheus

management:
  prometheus:
    metrics:
      export:
        enabled: true
        step: 1m

Datadog

management:
  datadog:
    metrics:
      export:
        api-key: ${DATADOG_API_KEY}
        enabled: true

Multiple Registries

@Configuration
public class MultiRegistryConfig {

    @Bean
    public CompositeMeterRegistry compositeMeterRegistry(
            PrometheusMeterRegistry prometheus,
            DatadogMeterRegistry datadog) {
        CompositeMeterRegistry composite = new CompositeMeterRegistry();
        composite.add(prometheus);
        composite.add(datadog);
        return composite;
    }
}

Distributed Tracing

OpenTelemetry integration, span customization, and context propagation.

Table of Contents

• OpenTelemetry Configuration
• Dependencies
• Configuration
• Production Configuration
• Custom Spans with Observation API
• Basic Span Creation
• Kotlin Span Creation
• Span with Error Handling
• Nested Spans
• @Observed Annotation
• Baggage Propagation
• Logging Correlation
• Automatic MDC Integration
• Custom Context in Logs
• HTTP Client Tracing
• RestClient (Spring Boot 4)
• WebClient
• Database Tracing
• Async Tracing
• Span Events and Attributes
• Sampling Strategies
• Probability Sampling
• Custom Sampler
• Exporters
• OTLP (OpenTelemetry Protocol)
• Zipkin
• Jaeger (via OTLP)
• Testing with Traces

OpenTelemetry Configuration

Dependencies

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

Configuration

management:
  tracing:
    enabled: true
    sampling:
      probability: 1.0  # 100% for dev, 0.1 for prod
    propagation:
      type: w3c  # or b3 for Zipkin compatibility
  opentelemetry:
    resource-attributes:
      service.name: order-service
      service.version: 1.0.0
      deployment.environment: ${ENVIRONMENT:local}
    tracing:
      export:
        otlp:
          endpoint: http://otel-collector:4318/v1/traces
          # For gRPC: http://otel-collector:4317

Production Configuration

# application-prod.yml
management:
  tracing:
    sampling:
      probability: 0.1  # 10% sampling
  opentelemetry:
    tracing:
      export:
        otlp:
          endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT}
          headers:
            Authorization: Bearer ${OTEL_AUTH_TOKEN}

Custom Spans with Observation API

Basic Span Creation

@Service
public class PaymentService {

    private final ObservationRegistry observationRegistry;

    public PaymentResult processPayment(PaymentRequest request) {
        return Observation.createNotStarted("payment.process", observationRegistry)
            .lowCardinalityKeyValue("payment.method", request.method().name())
            .lowCardinalityKeyValue("currency", request.currency())
            .observe(() -> doProcessPayment(request));
    }

    private PaymentResult doProcessPayment(PaymentRequest request) {
        // Business logic
    }
}

Kotlin Span Creation

@Service
class PaymentService(private val observationRegistry: ObservationRegistry) {

    fun processPayment(request: PaymentRequest): PaymentResult =
        Observation.createNotStarted("payment.process", observationRegistry)
            .lowCardinalityKeyValue("payment.method", request.method.name)
            .lowCardinalityKeyValue("currency", request.currency)
            .observe { doProcessPayment(request) }
}

Span with Error Handling

public Order createOrder(CreateOrderRequest request) {
    Observation observation = Observation.createNotStarted("order.create", observationRegistry)
        .lowCardinalityKeyValue("channel", request.channel());

    return observation.observe(() -> {
        try {
            Order order = orderRepository.save(mapToOrder(request));
            observation.lowCardinalityKeyValue("status", "success");
            return order;
        } catch (Exception e) {
            observation.lowCardinalityKeyValue("status", "error");
            observation.lowCardinalityKeyValue("error.type", e.getClass().getSimpleName());
            throw e;
        }
    });
}

Nested Spans

public OrderResult processOrder(Long orderId) {
    return Observation.createNotStarted("order.process", observationRegistry)
        .observe(() -> {
            Order order = fetchOrder(orderId);      // Creates child span
            validateOrder(order);                    // Creates child span
            PaymentResult payment = chargePayment(order);  // Creates child span
            return fulfillOrder(order, payment);     // Creates child span
        });
}

private Order fetchOrder(Long orderId) {
    return Observation.createNotStarted("order.fetch", observationRegistry)
        .highCardinalityKeyValue("order.id", orderId.toString())
        .observe(() -> orderRepository.findById(orderId).orElseThrow());
}

@Observed Annotation

@Configuration
public class ObservationConfig {
    @Bean
    public ObservedAspect observedAspect(ObservationRegistry registry) {
        return new ObservedAspect(registry);
    }
}

@Service
public class OrderService {

    @Observed(name = "order.creation",
              contextualName = "creating-order",
              lowCardinalityKeyValues = {"operation", "create"})
    public Order createOrder(CreateOrderRequest request) {
        // Automatically traced
    }
}

Baggage Propagation

Baggage propagates context across service boundaries:

@Component
public class TenantContextPropagator {

    private final Tracer tracer;

    public void setTenantContext(String tenantId) {
        try (BaggageInScope baggage = tracer.createBaggageInScope("tenant.id", tenantId)) {
            // Tenant ID propagates to all downstream calls
        }
    }

    public String getCurrentTenant() {
        Baggage baggage = tracer.getBaggage("tenant.id");
        return baggage != null ? baggage.get() : null;
    }
}

Configuration:

management:
  tracing:
    baggage:
      remote-fields:
        - tenant-id
        - correlation-id
      local-fields:
        - tenant-id
      correlation:
        fields:
          - tenant-id

Logging Correlation

Automatic MDC Integration

logging:
  pattern:
    console: "%d{yyyy-MM-dd HH:mm:ss} [%X{traceId:-},%X{spanId:-}] %-5level %logger{36} - %msg%n"

Logs automatically include trace and span IDs:

2025-01-15 10:30:45 [abc123def456,789xyz] INFO  c.e.OrderService - Processing order 12345

Custom Context in Logs

@Component
public class TracingFilter extends OncePerRequestFilter {

    private final Tracer tracer;

    @Override
    protected void doFilterInternal(HttpServletRequest request,
            HttpServletResponse response, FilterChain chain) {

        Span currentSpan = tracer.currentSpan();
        if (currentSpan != null) {
            MDC.put("traceId", currentSpan.context().traceId());
            MDC.put("spanId", currentSpan.context().spanId());
        }

        try {
            chain.doFilter(request, response);
        } finally {
            MDC.remove("traceId");
            MDC.remove("spanId");
        }
    }
}

HTTP Client Tracing

RestClient (Spring Boot 4)

@Configuration
public class RestClientConfig {

    @Bean
    public RestClient restClient(RestClient.Builder builder) {
        return builder
            .baseUrl("https://api.example.com")
            .build();
        // Tracing automatically instrumented
    }
}

WebClient

@Configuration
public class WebClientConfig {

    @Bean
    public WebClient webClient(WebClient.Builder builder) {
        return builder
            .baseUrl("https://api.example.com")
            .build();
        // Tracing automatically instrumented
    }
}

Database Tracing

JPA/JDBC operations are automatically traced. Add span tags:

@Repository
public class OrderRepositoryImpl {

    private final ObservationRegistry observationRegistry;
    private final JdbcTemplate jdbcTemplate;

    public List<Order> findLargeOrders(BigDecimal threshold) {
        return Observation.createNotStarted("db.query.large_orders", observationRegistry)
            .lowCardinalityKeyValue("db.operation", "SELECT")
            .lowCardinalityKeyValue("db.table", "orders")
            .observe(() ->
                jdbcTemplate.query(
                    "SELECT * FROM orders WHERE total > ?",
                    orderRowMapper,
                    threshold
                )
            );
    }
}

Async Tracing

Context propagates to async operations:

@Service
public class AsyncOrderProcessor {

    private final ObservationRegistry observationRegistry;

    @Async
    public CompletableFuture<ProcessingResult> processAsync(Order order) {
        // Observation context automatically propagated
        return Observation.createNotStarted("order.async_process", observationRegistry)
            .observe(() -> CompletableFuture.completedFuture(doProcess(order)));
    }
}

For manual propagation:

@Service
public class ManualAsyncService {

    private final Tracer tracer;
    private final ExecutorService executor;

    public void processInBackground(Runnable task) {
        Span currentSpan = tracer.currentSpan();

        executor.submit(() -> {
            try (Tracer.SpanInScope ws = tracer.withSpan(currentSpan)) {
                task.run();
            }
        });
    }
}

Span Events and Attributes

public void processOrder(Order order) {
    Observation observation = Observation.start("order.process", observationRegistry);

    try {
        observation.event(Observation.Event.of("validation.started"));
        validateOrder(order);
        observation.event(Observation.Event.of("validation.completed"));

        observation.event(Observation.Event.of("payment.started"));
        processPayment(order);
        observation.event(Observation.Event.of("payment.completed"));

        observation.lowCardinalityKeyValue("order.status", "completed");
    } catch (Exception e) {
        observation.error(e);
        throw e;
    } finally {
        observation.stop();
    }
}

Sampling Strategies

Probability Sampling

management:
  tracing:
    sampling:
      probability: 0.1  # 10%

Custom Sampler

@Bean
public Sampler customSampler() {
    return new Sampler() {
        @Override
        public SamplingResult shouldSample(
                Context parentContext,
                String traceId,
                String name,
                SpanKind spanKind,
                Attributes attributes,
                List<LinkData> parentLinks) {

            // Always sample errors
            if (name.contains("error")) {
                return SamplingResult.recordAndSample();
            }

            // Always sample health checks
            if (name.contains("health")) {
                return SamplingResult.drop();
            }

            // 10% for everything else
            return Math.random() < 0.1
                ? SamplingResult.recordAndSample()
                : SamplingResult.drop();
        }

        @Override
        public String getDescription() {
            return "CustomSampler";
        }
    };
}

Exporters

OTLP (OpenTelemetry Protocol)

management:
  opentelemetry:
    tracing:
      export:
        otlp:
          endpoint: http://otel-collector:4318/v1/traces

Zipkin

management:
  zipkin:
    tracing:
      endpoint: http://zipkin:9411/api/v2/spans

Jaeger (via OTLP)

management:
  opentelemetry:
    tracing:
      export:
        otlp:
          endpoint: http://jaeger:4317

Testing with Traces

@SpringBootTest
class TracingTest {

    @Autowired
    private TestObservationRegistry observationRegistry;

    @Autowired
    private OrderService orderService;

    @Test
    void shouldCreateSpanForOrderProcessing() {
        orderService.processOrder(testOrder);

        TestObservationRegistryAssert.assertThat(observationRegistry)
            .hasObservationWithNameEqualTo("order.process")
            .that()
            .hasLowCardinalityKeyValue("status", "success");
    }
}

Spring Boot Observability Troubleshooting

Common issues and solutions for Spring Boot 4 observability.

Common Issues

Issue: Actuator Endpoints Returning 404

Symptom: /actuator/health returns 404 Not Found

Cause: Endpoints not exposed or incorrect base path

Solution:

1. Check endpoint exposure:

management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus

2. Verify base path configuration:

management:
  endpoints:
    web:
      base-path: /actuator  # Default

3. Check if using separate management port:

management:
  server:
    port: 8081  # Access at localhost:8081/actuator/health

---

Issue: OpenTelemetry Traces Not Appearing

Symptom: No traces in Jaeger/Zipkin/OTLP collector

Cause: Missing dependencies, sampling at 0%, or wrong endpoint

Solution:

1. Verify dependencies:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

2. Check sampling configuration:

management:
  tracing:
    sampling:
      probability: 1.0  # 100% for debugging, reduce in production

3. Verify OTLP endpoint:

management:
  opentelemetry:
    tracing:
      export:
        otlp:
          endpoint: http://localhost:4318/v1/traces

---

Issue: Custom Metrics Not Registered

Symptom: Custom metrics don't appear in /actuator/metrics

Cause: Metrics created but not registered to MeterRegistry

Solution:

// Wrong - metric not registered
Timer timer = Timer.builder("my.timer").build();

// Correct - register to MeterRegistry
@Component
public class MyMetrics {
    private final Timer timer;

    public MyMetrics(MeterRegistry registry) {
        this.timer = Timer.builder("my.timer")
            .description("My custom timer")
            .register(registry);  // <-- Essential!
    }
}

---

Issue: Health Indicator Timeout

Symptom: Health check times out, returns UNKNOWN status

Cause: Slow external dependency blocking health check

Solution:

1. Add timeout to health checks:

management:
  endpoint:
    health:
      components:
        myIndicator:
          enabled: true
  health:
    livenessstate:
      enabled: true
    readinessstate:
      enabled: true

2. Implement async health indicator:

@Component
public class AsyncExternalApiHealthIndicator implements ReactiveHealthIndicator {

    private final WebClient webClient;

    @Override
    public Mono<Health> health() {
        return webClient.get()
            .uri("/health")
            .retrieve()
            .bodyToMono(String.class)
            .timeout(Duration.ofSeconds(5))
            .map(response -> Health.up().build())
            .onErrorResume(e -> Mono.just(Health.down(e).build()));
    }
}

3. Move slow checks to readiness only:

management:
  endpoint:
    health:
      group:
        liveness:
          include: livenessState,ping  # Fast checks only
        readiness:
          include: readinessState,db,externalApi  # Slower checks

---

Issue: Prometheus Endpoint Not Available

Symptom: /actuator/prometheus returns 404

Cause: Missing micrometer-registry-prometheus dependency

Solution:

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

And ensure endpoint is exposed:

management:
  endpoints:
    web:
      exposure:
        include: prometheus

---

Issue: High-Cardinality Tag Explosion

Symptom: OOM or slow metrics queries

Cause: Using user IDs, request IDs, or other high-cardinality values as tags

Solution:

// Wrong - high cardinality (millions of unique values)
Counter.builder("requests.total")
    .tag("user_id", userId)  // DON'T DO THIS
    .tag("request_id", requestId)  // DON'T DO THIS
    .register(registry);

// Correct - low cardinality (bounded set of values)
Counter.builder("requests.total")
    .tag("method", httpMethod)  // GET, POST, PUT, DELETE
    .tag("status", statusCode)  // 200, 201, 400, 404, 500
    .tag("endpoint", "/api/orders")  // Known endpoints
    .register(registry);

For high-cardinality data, use spans instead:

Observation.createNotStarted("request", observationRegistry)
    .highCardinalityKeyValue("user_id", userId)  // OK for tracing
    .observe(() -> processRequest());

---

Spring Boot 4 Migration Issues

Health Indicator Import Changes

// Before (Boot 3.x)
import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;

// After (Boot 4.x)
import org.springframework.boot.health.contributor.Health;
import org.springframework.boot.health.contributor.HealthIndicator;

Endpoint Access Control Migration

# Before (Boot 3.x)
management:
  endpoints:
    web:
      exposure:
        include: "*"
  endpoint:
    health:
      enabled: true

# After (Boot 4.x) - add explicit access control
management:
  endpoints:
    access:
      default: none
    web:
      exposure:
        include: health,info,prometheus
  endpoint:
    health:
      access: unrestricted

OpenTelemetry Default Tracer

Boot 4 uses OpenTelemetry by default instead of Brave:

<!-- Remove if present - no longer needed -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>

<!-- Add for Boot 4 -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>