Observability Sre

Observability & Site Reliability Engineering

Core Principles

• Three Pillars — Metrics, Logs, and Traces provide holistic visibility
• Observability-First — Build systems that explain their own behavior
• SLO-Driven — Define reliability targets that matter to users
• Proactive Detection — Find issues before customers do
• Blameless Culture — Learn from failures without blame
• Automate Toil — Reduce repetitive operational work
• Continuous Improvement — Each incident makes systems more resilient
• Full-Stack Visibility — Monitor from infrastructure to business metrics

---

Hard Rules (Must Follow)

These rules are mandatory. Violating them means the skill is not working correctly.

Symptom-Based Alerts Only

Alert on user-facing symptoms, not internal infrastructure metrics.

# ❌ FORBIDDEN: Alerting on internal metrics
- alert: CPUHigh
  expr: cpu_usage > 70%
  # Users don't care about CPU, they care about latency

- alert: MemoryHigh
  expr: memory_usage > 80%
  # Internal metric, may not affect users

# ✅ REQUIRED: Alert on user experience
- alert: APILatencyHigh
  expr: slo:api_latency:p95 > 0.200
  annotations:
    summary: "Users experiencing slow response times"

- alert: ErrorRateHigh
  expr: slo:api_errors:rate5m > 0.001
  annotations:
    summary: "Users encountering errors"

Low Cardinality Labels

Loki/Prometheus labels must have low cardinality (<10 unique labels).

# ❌ FORBIDDEN: High cardinality labels
labels:
  user_id: "usr_123"      # Millions of values!
  order_id: "ord_456"     # Millions of values!
  request_id: "req_789"   # Every request is unique!

# ✅ REQUIRED: Low cardinality only
labels:
  namespace: "production"  # Few values
  app: "api-server"        # Few values
  level: "error"           # 5-6 values
  method: "GET"            # ~10 values

# High cardinality data goes in log body:
logger.info({
  user_id: "usr_123",      # In JSON body, not label
  order_id: "ord_456",
}, "Order processed");

SLO-Based Error Budgets

Every service must have defined SLOs with error budget tracking.

# ❌ FORBIDDEN: No SLO definition
# Just monitoring without targets

# ✅ REQUIRED: Explicit SLO with budget
# SLO: 99.9% availability
# Error Budget: 0.1% = 43.2 minutes/month downtime

groups:
  - name: slo_tracking
    rules:
      - record: slo:api_availability:ratio
        expr: sum(rate(http_requests_total{status!~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

      - alert: ErrorBudgetBurnRate
        expr: slo:api_availability:ratio < 0.999
        for: 5m
        annotations:
          summary: "Burning error budget too fast"

Trace Context in Logs

All logs must include trace_id for correlation with distributed traces.

// ❌ FORBIDDEN: Logs without trace context
logger.info("Payment processed");

// ✅ REQUIRED: Include trace_id in every log
const span = trace.getActiveSpan();
logger.info({
  trace_id: span?.spanContext().traceId,
  span_id: span?.spanContext().spanId,
  order_id: "ord_123",
}, "Payment processed");

// Output includes correlation:
// {"trace_id":"abc123","span_id":"def456","order_id":"ord_123","msg":"Payment processed"}

---

Quick Reference

When to Use What

The Three Pillars

Fourth Pillar (Emerging): Continuous Profiling — Code-level performance data (CPU, memory usage at function level)

---

Observability Architecture

Layered Prometheus Setup

# 2025 Best Practice: Federated architecture
# Prevents metric chaos while enabling drill-down

# Layer 1: Application Prometheus
# - Detailed business logic metrics
# - High cardinality acceptable
# - Short retention (7 days)

# Layer 2: Cluster Prometheus
# - Per-environment/cluster metrics
# - Medium retention (30 days)
# - Aggregates from application level

# Layer 3: Global Prometheus
# - Cross-cluster critical metrics
# - Long retention (1 year)
# - Federation from cluster level

# Global Prometheus config
scrape_configs:
  - job_name: 'federate'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="kubernetes-nodes"}'
        - '{__name__=~"job:.*"}'  # Recording rules only
    static_configs:
      - targets:
        - 'cluster-prom-us-east.internal:9090'
        - 'cluster-prom-eu-west.internal:9090'

Recording Rules for Performance

# Precompute expensive queries
groups:
  - name: api_performance
    interval: 30s
    rules:
      # Request rate (requests per second)
      - record: job:api_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job, method, status)

      # Error rate
      - record: job:api_errors:rate5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
          /
          sum(rate(http_requests_total[5m])) by (job)

      # P95 latency
      - record: job:api_latency:p95
        expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le))

Resource Optimization

# Increase scrape interval for high-target deployments
scrape_interval: 30s  # Default: 15s reduces load by 50%

# Use relabeling to drop unnecessary metrics
metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'go_.*|process_.*'  # Drop Go runtime metrics
    action: drop

# Limit sample retention
storage:
  tsdb:
    retention.time: 15d  # Keep only 15 days locally
    retention.size: 50GB # Or max 50GB

---

Distributed Tracing with OpenTelemetry

Auto-Instrumentation Setup

// Node.js auto-instrumentation
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4318/v1/traces',
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      // Auto-instruments HTTP, Express, PostgreSQL, Redis, etc.
      '@opentelemetry/instrumentation-fs': { enabled: false }, // Too noisy
    }),
  ],
});

sdk.start();

Manual Instrumentation for Business Logic

import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('payment-service', '1.0.0');

async function processPayment(orderId: string, amount: number) {
  // Create custom span for business operation
  return tracer.startActiveSpan('processPayment', async (span) => {
    try {
      // Add business context
      span.setAttributes({
        'order.id': orderId,
        'payment.amount': amount,
        'payment.currency': 'USD',
      });

      // Child span for external API call
      const paymentResult = await tracer.startActiveSpan('stripe.charge', async (childSpan) => {
        const result = await stripe.charges.create({ amount, currency: 'usd' });
        childSpan.setAttribute('stripe.charge_id', result.id);
        childSpan.setStatus({ code: SpanStatusCode.OK });
        childSpan.end();
        return result;
      });

      span.setStatus({ code: SpanStatusCode.OK });
      return paymentResult;
    } catch (error) {
      span.recordException(error);
      span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
      throw error;
    } finally {
      span.end();
    }
  });
}

Sampling Strategies

# OpenTelemetry Collector config
processors:
  # Probabilistic sampling: Keep 10% of traces
  probabilistic_sampler:
    sampling_percentage: 10

  # Tail sampling: Make decisions after seeing full trace
  tail_sampling:
    policies:
      # Always sample errors
      - name: error-traces
        type: status_code
        status_code: {status_codes: [ERROR]}

      # Always sample slow requests
      - name: slow-traces
        type: latency
        latency: {threshold_ms: 1000}

      # Sample 5% of normal traffic
      - name: normal-traces
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

Context Propagation

// Ensure trace context flows across services
import { propagation, context } from '@opentelemetry/api';

// Outgoing HTTP request (automatic with auto-instrumentation)
fetch('https://api.example.com/data', {
  headers: {
    // W3C Trace Context headers injected automatically:
    // traceparent: 00-<trace-id>-<span-id>-01
    // tracestate: vendor=value
  },
});

// Manual propagation for non-HTTP (e.g., message queues)
const carrier = {};
propagation.inject(context.active(), carrier);
await publishMessage(queue, { data: payload, headers: carrier });

---

Structured Logging Best Practices

JSON Logging Format

// Use structured logging library
import pino from 'pino';

const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  formatters: {
    level: (label) => ({ level: label }),
  },
  timestamp: pino.stdTimeFunctions.isoTime,
  // Include trace context in logs
  mixin() {
    const span = trace.getActiveSpan();
    if (!span) return {};

    const { traceId, spanId } = span.spanContext();
    return {
      trace_id: traceId,
      span_id: spanId,
    };
  },
});

// Structured logging with context
logger.info(
  {
    user_id: '123',
    order_id: 'ord_456',
    amount: 99.99,
    payment_method: 'card',
  },
  'Payment processed successfully'
);

// Output:
// {"level":"info","time":"2025-01-15T10:30:00.000Z","trace_id":"abc123","span_id":"def456","user_id":"123","order_id":"ord_456","amount":99.99,"payment_method":"card","msg":"Payment processed successfully"}

Log Levels

// Follow standard severity levels
logger.trace({ details }, 'Low-level debugging');     // Very verbose
logger.debug({ state }, 'Debug information');          // Development
logger.info({ event }, 'Normal operation');            // Production default
logger.warn({ issue }, 'Warning condition');           // Potential issues
logger.error({ error, context }, 'Error occurred');    // Errors
logger.fatal({ critical }, 'Fatal error');             // Process crash

Grafana Loki Configuration

# Promtail config - ships logs to Loki
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: kubernetes
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Add pod labels as Loki labels (LOW cardinality only!)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
    pipeline_stages:
      # Parse JSON logs
      - json:
          expressions:
            level: level
            trace_id: trace_id
      # Extract fields as labels
      - labels:
          level:
          trace_id:

Loki Best Practices

• Low Cardinality Labels — Use only 5-10 labels (namespace, app, level)
• High Cardinality in Log Body — Put user_id, order_id in JSON, not labels
• LogQL for Filtering — Use {app="api"} | json | user_id="123"
• Retention Policy — Keep recent logs longer, compress old logs

# LogQL query examples
{namespace="production", app="api"} |= "error"  # Text search

{app="api"} | json | level="error" | line_format "{{.msg}}"  # JSON parsing

rate({app="api"}[5m])  # Log rate per second

sum by (level) (count_over_time({namespace="production"}[1h]))  # Count by level

---

Extended Reference

Detailed material starting at ## SLO/SLI/SLA Management has been moved to `reference/extended.md` to keep this skill concise. Load that reference when the task requires the moved examples, command catalogs, checklists, platform details, or implementation templates.

observability-sre Extended Reference

This file preserves detailed material moved out of SKILL.md for progressive disclosure. Load it only when the current task needs the specific examples, commands, templates, or checklists below.

Moved content starts at: ## SLO/SLI/SLA Management.

SLO/SLI/SLA Management

Definitions

• SLI (Service Level Indicator) — Quantifiable measurement of service behavior
• Examples: Request latency, error rate, availability, throughput

• SLO (Service Level Objective) — Target value/range for an SLI
• Examples: 99.9% availability, P95 latency < 200ms

• SLA (Service Level Agreement) — Formal commitment with consequences
• Examples: "99.9% uptime or 10% credit"

The Four Golden Signals

# Google SRE's key metrics for any service

1. Latency
   SLI: P95 request latency
   SLO: 95% of requests complete in < 200ms

2. Traffic
   SLI: Requests per second
   SLO: Handle 10,000 req/s peak load

3. Errors
   SLI: Error rate (5xx / total)
   SLO: < 0.1% error rate

4. Saturation
   SLI: Resource utilization (CPU, memory, disk)
   SLO: CPU < 70%, Memory < 80%

Error Budget

# Error budget = 1 - SLO
SLO = 99.9%  # "three nines"
Error_Budget = 100% - 99.9% = 0.1%

# Monthly calculation (30 days)
Total_Minutes = 30 * 24 * 60 = 43,200 minutes
Allowed_Downtime = 43,200 * 0.001 = 43.2 minutes

# If you've had 20 minutes downtime this month:
Budget_Remaining = 43.2 - 20 = 23.2 minutes
Budget_Consumed = 20 / 43.2 = 46.3%

# Policy: If budget > 90% consumed, freeze deployments

SLO Implementation with Prometheus

# Recording rules for SLI calculation
groups:
  - name: slo_availability
    interval: 30s
    rules:
      # Total requests
      - record: slo:api_requests:total
        expr: sum(rate(http_requests_total[5m]))

      # Successful requests (non-5xx)
      - record: slo:api_requests:success
        expr: sum(rate(http_requests_total{status!~"5.."}[5m]))

      # Availability SLI
      - record: slo:api_availability:ratio
        expr: slo:api_requests:success / slo:api_requests:total

      # 30-day availability
      - record: slo:api_availability:30d
        expr: avg_over_time(slo:api_availability:ratio[30d])

  - name: slo_latency
    interval: 30s
    rules:
      # P95 latency SLI
      - record: slo:api_latency:p95
        expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# Alerting on SLO burn rate
- alert: HighErrorBudgetBurnRate
  expr: |
    (
      slo:api_availability:ratio < 0.999  # Below 99.9% SLO
      and
      slo:api_availability:30d > 0.999    # But 30-day average still OK
    )
  for: 5m
  annotations:
    summary: "Burning error budget too fast"
    description: "Current availability {{ $value }} is below SLO. {{ $labels.service }}"

---

Incident Response

Incident Severity Levels

Incident Response Roles (IMAG Framework)

Incident Commander (IC):
  - Overall coordination and decision-making
  - Declares incident start/end
  - Decides on escalations
  - Owns communication to leadership

Operations Lead (OL):
  - Technical investigation and mitigation
  - Coordinates engineers
  - Implements fixes
  - Reports status to IC

Communications Lead (CL):
  - Internal/external status updates
  - Customer communication
  - Stakeholder notifications
  - Status page updates

Incident Workflow

1. Detection (Alert fires or user reports)
   ↓
2. Triage (Assess severity, assign IC)
   ↓
3. Response (Assemble team, create war room)
   ↓
4. Mitigation (Stop the bleeding, restore service)
   ↓
5. Resolution (Fix root cause)
   ↓
6. Postmortem (Blameless review, action items)
   ↓
7. Follow-up (Implement improvements)

On-Call Best Practices

• Rotation — 1-week shifts, balanced across timezones
• Escalation — Primary → Secondary → Manager (15 min each)
• Playbooks — Step-by-step debugging guides for common issues
• Runbooks — Automated remediation scripts
• Handoff — 15-min sync at rotation change
• Compensation — On-call pay or comp time
• Health — No more than 2 incidents/night target

Alert Fatigue Prevention

# Symptoms vs Causes alerting
# Alert on WHAT users experience, not WHY it's broken

# GOOD: Symptom-based alert
- alert: APILatencyHigh
  expr: slo:api_latency:p95 > 0.200  # User-facing metric
  annotations:
    summary: "API is slow for users"

# BAD: Cause-based alert
- alert: CPUHigh
  expr: cpu_usage > 70%  # Internal metric, might not impact users
  # Don't alert unless this affects SLOs

# Use SLO-based alerting
# Alert when error budget burn rate is too high

---

Blameless Postmortems

Core Principles

• Assume Good Intentions — Everyone did their best with available information
• Focus on Systems — Identify gaps in process/tooling, not people
• Psychological Safety — No punishment for honest mistakes
• Learning Culture — Incidents are opportunities to improve
• Separate from Performance Reviews — Postmortem participation never affects evaluations

Postmortem Template

# Incident Postmortem: [Title]

**Date:** 2025-01-15
**Duration:** 10:30 - 12:15 UTC (1h 45m)
**Severity:** SEV-2
**Incident Commander:** Jane Doe
**Responders:** John Smith, Alice Johnson

## Impact
- 15,000 users affected
- 12% error rate on payment processing
- $5,000 estimated revenue impact
- No data loss

## Timeline (UTC)
- 10:30 - Alert: Payment error rate > 5%
- 10:32 - IC assigned, war room created
- 10:45 - Identified: Database connection pool exhausted
- 11:00 - Mitigation: Increased pool size from 50 → 100
- 11:15 - Error rate back to normal
- 12:15 - Incident closed after monitoring

## Root Cause
Database connection pool configured for average load, not peak traffic.
Black Friday traffic spike (3x normal) exhausted connections.

## What Went Well
- Alert fired within 2 minutes of issue
- Clear escalation path, IC available immediately
- Mitigation applied quickly (30 minutes to fix)
- No data corruption or loss

## What Went Wrong
- No load testing at 3x scale
- No auto-scaling for connection pool
- No alert on connection pool saturation
- Insufficient monitoring of database metrics

## Action Items
- [ ] (@john) Add connection pool metrics to Grafana (Due: Jan 20)
- [ ] (@alice) Implement auto-scaling based on request rate (Due: Jan 25)
- [ ] (@jane) Add load testing to CI for 5x scale (Due: Feb 1)
- [ ] (@jane) Add alert: connection pool > 80% (Due: Jan 18)
- [ ] (@john) Document connection pool tuning runbook (Due: Jan 22)

## Lessons Learned
1. Black Friday load patterns need dedicated testing
2. Database metrics were missing from standard dashboards
3. Auto-scaling should cover ALL resources, not just pods

Follow-up

• Review postmortem in team meeting within 1 week
• Track action items to completion (not optional!)
• Share learnings across teams
• Update runbooks and playbooks
• Celebrate successful incident response

---

Chaos Engineering

Principles

1. Define Steady State — Normal system behavior (e.g., 99.9% success rate) 2. Hypothesize — Predict system will remain stable under failure 3. Inject Failures — Simulate real-world events 4. Disprove Hypothesis — Look for deviations from steady state 5. Learn and Improve — Fix weaknesses, increase resilience

Failure Types

Infrastructure:
  - Pod/node termination
  - Network latency/packet loss
  - DNS failures
  - Cloud region outage

Resources:
  - CPU stress
  - Memory exhaustion
  - Disk I/O saturation
  - File descriptor limits

Dependencies:
  - Database connection failures
  - API timeout/errors
  - Cache unavailability
  - Message queue backlog

Security:
  - DDoS simulation
  - Certificate expiration
  - Unauthorized access attempts

Chaos Mesh Example

# Network latency injection
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: network-delay
spec:
  action: delay
  mode: one
  selector:
    namespaces:
      - production
    labelSelectors:
      app: payment-service
  delay:
    latency: "100ms"
    correlation: "50"
    jitter: "50ms"
  duration: "5m"
  scheduler:
    cron: "@every 2h"  # Run every 2 hours

---
# Pod kill experiment
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: pod-kill
spec:
  action: pod-kill
  mode: fixed-percent
  value: "10"  # Kill 10% of pods
  selector:
    namespaces:
      - production
    labelSelectors:
      app: api-server
  duration: "30s"

Best Practices

• Start Small — Non-production first, then canary production
• Collect Baselines — Know normal metrics before experiments
• Define Success — Clear criteria for what "stable" means
• Monitor Everything — Watch metrics, logs, traces during tests
• Automate Rollback — Stop experiment if SLOs violated
• Game Days — Scheduled chaos exercises with full team
• Blameless Reviews — Treat chaos failures like production incidents

---

AIOps and AI in Observability

2025 Trends

• Anomaly Detection — AI spots unusual patterns in metrics/logs
• Root Cause Analysis — Correlate failures across services automatically
• Predictive Alerting — Predict failures before they happen
• Auto-Remediation — AI suggests or applies fixes autonomously
• Natural Language Queries — Ask "Why is checkout slow?" instead of writing PromQL
• AI Observability — Monitor AI model drift, hallucinations, token usage

AI-Driven Platforms (2025)

Dynatrace Davis AI:
  - Auto-detected 73% of incidents before customer impact
  - Reduced alert noise by 90%
  - Causal AI for root cause analysis

Datadog Watchdog:
  - Anomaly detection across metrics, logs, traces
  - Automated correlation of related issues
  - LLM-powered investigation assistant

Elastic AIOps:
  - Machine learning for log anomaly detection
  - Automated baseline learning
  - Predictive alerting

New Relic AI:
  - Natural language query interface
  - Automated incident summarization
  - Proactive capacity recommendations

Implementing AI Observability

# Monitor AI model performance
from opentelemetry import trace, metrics

tracer = trace.get_tracer(__name__)
meter = metrics.get_meter(__name__)

# Create metrics for AI model
model_latency = meter.create_histogram(
    "ai.model.latency",
    description="AI model inference latency",
    unit="ms"
)
model_tokens = meter.create_counter(
    "ai.model.tokens",
    description="Token usage"
)

async def run_ai_model(prompt: str):
    with tracer.start_as_current_span("ai.inference") as span:
        start = time.time()

        span.set_attribute("ai.model", "gpt-4")
        span.set_attribute("ai.prompt_length", len(prompt))

        response = await openai.chat.completions.create(
            model="gpt-4",
            messages=[{"role": "user", "content": prompt}]
        )

        latency = (time.time() - start) * 1000
        tokens = response.usage.total_tokens

        # Record metrics
        model_latency.record(latency, {"model": "gpt-4"})
        model_tokens.add(tokens, {"model": "gpt-4", "type": "total"})

        # Add to span
        span.set_attribute("ai.response_length", len(response.choices[0].message.content))
        span.set_attribute("ai.tokens_used", tokens)

        return response

---

Grafana Dashboards

3-3-3 Rule

• 3 rows of panels per dashboard
• 3 panels per row
• 3 key metrics per panel

Avoid "dashboard sprawl" — Each dashboard should answer ONE question.

Dashboard Categories

RED Dashboard (for services):
  - Rate: Requests per second
  - Errors: Error rate
  - Duration: Latency (P50, P95, P99)

USE Dashboard (for resources):
  - Utilization: % of capacity used
  - Saturation: Queue depth, wait time
  - Errors: Error count

Four Golden Signals Dashboard:
  - Latency
  - Traffic
  - Errors
  - Saturation

SLO Dashboard:
  - Current SLI value
  - Error budget remaining
  - Burn rate
  - Trend (30-day)

Panel Best Practices

{
  "title": "API Request Rate",
  "type": "graph",
  "targets": [
    {
      "expr": "sum(rate(http_requests_total[5m])) by (method)",
      "legendFormat": "{{ method }}"
    }
  ],
  "options": {
    "tooltip": { "mode": "multi" },
    "legend": { "displayMode": "table", "calcs": ["mean", "last"] }
  },
  "fieldConfig": {
    "defaults": {
      "unit": "reqps",  // Requests per second
      "color": { "mode": "palette-classic" },
      "custom": {
        "lineWidth": 2,
        "fillOpacity": 10
      }
    }
  }
}

---

Checklist

## Metrics (Prometheus + Grafana)
- [ ] Layered architecture (app/cluster/global)
- [ ] Recording rules for expensive queries
- [ ] Resource limits and retention configured
- [ ] Dashboards follow 3-3-3 rule
- [ ] Alerts based on SLOs, not internal metrics

## Tracing (OpenTelemetry)
- [ ] Auto-instrumentation enabled
- [ ] Custom spans for business operations
- [ ] Sampling strategy configured
- [ ] Trace context in logs (correlation)
- [ ] Backend connected (Tempo/Jaeger)

## Logging (Loki/ELK)
- [ ] Structured JSON logging
- [ ] Low cardinality labels (<10)
- [ ] Trace IDs in logs
- [ ] Appropriate log levels
- [ ] Retention policy defined

## SLOs
- [ ] SLIs defined for key user journeys
- [ ] SLOs documented and tracked
- [ ] Error budget calculated
- [ ] Burn rate alerting configured
- [ ] Monthly SLO review process

## Incident Response
- [ ] Severity levels defined
- [ ] On-call rotation scheduled
- [ ] Escalation policy documented
- [ ] Runbooks for common issues
- [ ] Postmortem template ready

## Culture
- [ ] Blameless postmortem process
- [ ] Action items tracked to completion
- [ ] Incident learnings shared
- [ ] On-call compensation policy
- [ ] Regular chaos engineering exercises

---

See Also

• reference/monitoring.md — Prometheus and Grafana deep dive
• reference/logging.md — Structured logging best practices
• reference/tracing.md — OpenTelemetry and distributed tracing
• reference/incident-response.md — Incident management and postmortems
• templates/slo-template.md — SLO definition template

Incident Response and Management

Incident Severity Levels

Classification Matrix

Severity Decision Tree

Is the service completely unavailable?
  YES → SEV-1

Is there a security breach or data loss?
  YES → SEV-1

Are core features unavailable or severely degraded?
  YES → SEV-2

Is there a workaround available?
  NO → SEV-2
  YES → Is customer impact moderate?
    YES → SEV-3
    NO → SEV-4

Is it only affecting internal systems?
  YES → Lower one level

Incident Response Framework (IMAG)

Roles and Responsibilities

Incident Commander (IC):
  Responsibilities:
    - Overall incident coordination
    - Declare incident start and end
    - Make final decisions on mitigation strategies
    - Manage escalations
    - Communicate to executive leadership
    - Own postmortem completion

  Should NOT:
    - Debug technical issues directly
    - Implement fixes themselves
    - Get pulled into tactical work

  Key Skills:
    - Stay calm under pressure
    - Clear communication
    - Decisive decision-making

Operations Lead (OL):
  Responsibilities:
    - Technical investigation and debugging
    - Coordinate engineers working on mitigation
    - Implement fixes and deploy changes
    - Report status to IC every 15-30 minutes
    - Maintain incident timeline
    - Validate mitigation effectiveness

  Should NOT:
    - Communicate with customers
    - Make unilateral decisions about incident severity

Communications Lead (CL):
  Responsibilities:
    - Update status page
    - Internal stakeholder notifications
    - Customer communication (if customer-facing)
    - Manage Slack incident channel
    - Document timeline in real-time
    - Notify when incident is resolved

  Should NOT:
    - Get involved in technical debugging
    - Make decisions about mitigation

Subject Matter Expert (SME):
  Responsibilities:
    - Provide domain expertise
    - Assist Operations Lead with debugging
    - Implement specific fixes in their area
    - Review and approve changes

  Should NOT:
    - Take over IC or OL roles without handoff

Incident Lifecycle

1. DETECTION
   ├─ Alert fires (automated)
   ├─ User report (manual)
   └─ Monitoring anomaly

2. TRIAGE (< 5 minutes)
   ├─ Assess severity
   ├─ Assign Incident Commander
   └─ Create incident channel

3. RESPONSE
   ├─ Assemble team (IC, OL, CL, SMEs)
   ├─ Create war room (Zoom/Slack)
   ├─ Begin investigation
   └─ Regular status updates (every 15-30 min)

4. MITIGATION
   ├─ Identify root cause
   ├─ Stop the bleeding (temporary fix)
   ├─ Monitor metrics for improvement
   └─ Validate customer impact reduced

5. RESOLUTION
   ├─ Permanent fix deployed
   ├─ Metrics back to normal
   ├─ Monitor for regression
   └─ Incident Commander declares resolved

6. POSTMORTEM
   ├─ Schedule within 48 hours
   ├─ Write blameless postmortem
   ├─ Identify action items
   └─ Share learnings

7. FOLLOW-UP
   ├─ Track action items to completion
   ├─ Update runbooks
   └─ Share with broader team

On-Call Best Practices

On-Call Rotation

Rotation Schedule:
  - Duration: 1 week per rotation
  - Coverage: 24/7
  - Handoff: 15-minute sync at rotation change
  - Backup: Secondary on-call for escalation

Rotation Requirements:
  - Minimum 2 people per rotation (primary + secondary)
  - Balanced across timezones for global teams
  - No more than 1 rotation per month per person
  - Automatic escalation after 15 minutes

Compensation:
  - On-call pay ($X/day)
  - Time off in lieu (TOIL) for incidents after hours
  - Incident bonuses for SEV-1/SEV-2

Escalation Policy

# PagerDuty/Opsgenie config
Escalation Levels:
  Level 1 (0 min):
    - Primary on-call engineer
    - Alert: SMS, Phone call, Push notification
    - Timeout: 15 minutes

  Level 2 (15 min):
    - Secondary on-call engineer
    - Alert: Same as Level 1
    - Timeout: 15 minutes

  Level 3 (30 min):
    - Team lead / Manager
    - Alert: Same as Level 1
    - Timeout: 15 minutes

  Level 4 (45 min):
    - Director / VP Engineering
    - Alert: SMS, Phone call
    - Continuous escalation until acknowledged

Alert Channels:
  - SEV-1: Phone call (loops until answered)
  - SEV-2: Push notification + SMS
  - SEV-3: Slack notification
  - SEV-4: Email

Runbooks and Playbooks

# Runbook Template: High API Error Rate

## Symptoms
- Alert: "HighErrorRate" firing
- Dashboard: Error rate > 5%
- Customer reports: 500 errors

## Severity
- Error rate 5-10% → SEV-3
- Error rate 10-20% → SEV-2
- Error rate > 20% → SEV-1

## Investigation Steps

### 1. Check Service Health

Check pod status

kubectl get pods -n production -l app=api

Check recent deployments

kubectl rollout history deployment/api -n production

Check pod logs for errors

kubectl logs -n production -l app=api --tail=100 | grep ERROR

### 2. Check Dependencies

Database connectivity

kubectl exec -n production <api-pod> -- nc -zv postgres 5432

Redis connectivity

kubectl exec -n production <api-pod> -- nc -zv redis 6379

External API health

curl https://partner-api.example.com/health

### 3. Check Resource Usage

CPU usage

100 (1 - avg(rate(container_cpu_usage_seconds_total{pod=~"api."}[5m])))

Memory usage

container_memory_usage_bytes{pod=~"api."} / container_spec_memory_limit_bytes{pod=~"api."}

Database connections

pg_stat_activity_count

## Common Causes

### 1. Recent Deployment
**Fix:** Rollback to previous version

kubectl rollout undo deployment/api -n production

### 2. Database Connection Pool Exhausted
**Fix:** Increase pool size temporarily

kubectl set env deployment/api -n production DB_POOL_SIZE=100

### 3. External API Down
**Fix:** Enable circuit breaker

kubectl set env deployment/api -n production CIRCUIT_BREAKER_ENABLED=true

### 4. High Traffic Spike
**Fix:** Scale up pods

kubectl scale deployment/api -n production --replicas=20

## Escalation
If error rate doesn't improve within 30 minutes:
- Escalate to @backend-team
- Contact @database-team if DB-related
- Page @on-call-manager for SEV-1

## Related Runbooks
- [Database Connection Issues](database-connection-issues.md)
- [High Latency Investigation](high-latency.md)
- [Kubernetes Pod Troubleshooting](k8s-pod-troubleshooting.md)

Incident Communication

Status Page Updates

# Statuspage.io / Atlassian Statuspage

Initial Update (< 15 minutes):
  Status: Investigating
  Template: |
    We are investigating reports of [issue description].
    We will provide an update within 30 minutes.

  Example: |
    We are investigating reports of elevated error rates on the API.
    Some users may experience failures when processing payments.
    We will provide an update within 30 minutes.

Ongoing Updates (every 30 minutes):
  Status: Identified / Monitoring
  Template: |
    We have identified the cause as [brief explanation].
    We are currently [mitigation action].
    Expected resolution: [timeframe or "unknown"].

  Example: |
    We have identified the cause as database connection pool exhaustion.
    We are currently increasing connection limits and scaling the database.
    Expected resolution: 45 minutes.

Resolution Update:
  Status: Resolved
  Template: |
    The issue has been resolved.
    [What was fixed].
    If you continue to experience issues, please contact support.

  Example: |
    The issue has been resolved at 14:30 UTC.
    We increased database connection limits and optimized query performance.
    If you continue to experience issues, please contact support at support@example.com.

Postmortem (within 72 hours):
  Status: Postmortem
  Template: |
    We have published a postmortem about the incident on [date].
    Read more: [link]

Internal Communication

Slack Incident Channel:
  Naming: incident-2025-01-15-api-errors
  Pin: Incident details, severity, IC, OL, CL
  Updates: Every 15-30 minutes from CL

  Template:
    ━━━━━━━━━━━━━━━━━━━━━━━━
    🚨 INCIDENT DECLARED
    ━━━━━━━━━━━━━━━━━━━━━━━━
    Severity: SEV-2
    Started: 2025-01-15 14:00 UTC

    Incident Commander: @jane
    Operations Lead: @john
    Communications Lead: @alice

    Summary: High API error rate (15%)
    Impact: Payment processing failures

    War Room: https://zoom.us/j/123456789
    Dashboard: https://grafana.example.com/incident-123
    ━━━━━━━━━━━━━━━━━━━━━━━━

  Status Updates (every 15-30 min):
    [14:30] Update from OL: Identified database connection issue.
            Increasing pool size from 50 → 100. Deploying now.

    [14:45] Update from OL: Deployment complete. Error rate dropped to 8%.
            Monitoring for further improvement.

    [15:00] Update from IC: Error rate back to normal (0.5%).
            Preparing to declare incident resolved.

Alert Fatigue Prevention

Symptom-Based Alerting

# GOOD: Alert on user-facing symptoms
- alert: UserFacingErrorRate
  expr: |
    sum(rate(http_requests_total{status=~"5.."}[5m]))
    /
    sum(rate(http_requests_total[5m])) > 0.05
  annotations:
    summary: "Users experiencing high error rate ({{ $value | humanizePercentage }})"

# BAD: Alert on internal causes that may not affect users
- alert: CPUHigh
  expr: cpu_usage > 70%
  # Might not impact users at all

# GOOD: SLO-based alerting
- alert: SLOBurnRateTooHigh
  expr: slo:api_availability:ratio < 0.999
  for: 5m
  annotations:
    summary: "Burning error budget too fast"

Alert Grouping and Deduplication

# Alertmanager config
route:
  group_by: ['alertname', 'cluster']
  group_wait: 10s        # Wait for similar alerts
  group_interval: 10s    # Batch additional alerts
  repeat_interval: 4h    # Don't re-alert for 4 hours

  routes:
    - match:
        severity: critical
      repeat_interval: 15m  # More frequent for critical

Alert Tuning

# Use 'for' to avoid flapping
- alert: HighErrorRate
  expr: error_rate > 0.05
  for: 5m  # Must be true for 5 minutes
  annotations:
    summary: "Sustained high error rate"

# Inhibit lower-priority alerts
inhibit_rules:
  - source_match:
      alertname: ServiceDown
    target_match:
      alertname: HighLatency
    equal: ['service']

Blameless Postmortems

Postmortem Template

# Postmortem: [Brief Title]

**Date:** 2025-01-15
**Authors:** Jane Doe (IC), John Smith (OL)
**Status:** Complete
**Severity:** SEV-2

## Executive Summary
One-paragraph summary of what happened, impact, and resolution.

Example: On January 15, 2025, our API experienced a 15% error rate for
1 hour and 45 minutes, affecting approximately 15,000 users attempting to
process payments. The incident was caused by database connection pool
exhaustion during a traffic spike. We resolved the issue by increasing
connection limits and implementing auto-scaling.

## Impact
- **Duration:** 1 hour 45 minutes (14:00 - 15:45 UTC)
- **Users Affected:** ~15,000 users
- **Error Rate:** 15% peak, 8% sustained
- **Revenue Impact:** Estimated $12,000 in failed transactions
- **SLO Impact:** Consumed 40% of monthly error budget
- **Data Loss:** None

## Timeline (All times UTC)
- **14:00** - Alert: "HighErrorRate" fires (error rate 12%)
- **14:02** - Jane assigned as IC, incident channel created
- **14:05** - John (OL) begins investigation
- **14:10** - Identified: Database connection pool exhausted (50/50 connections in use)
- **14:15** - First mitigation: Increased pool size to 75 connections
- **14:20** - Error rate decreased to 10%
- **14:25** - Second mitigation: Increased to 100 connections
- **14:30** - Error rate decreased to 8%
- **14:35** - Implemented auto-scaling based on connection usage
- **14:45** - Error rate back to normal (0.5%)
- **15:00** - Monitoring for regression
- **15:45** - Incident declared resolved

## Root Cause
The API's database connection pool was configured with a static size of 50
connections, which was adequate for average traffic but insufficient for
peak loads. On January 15, traffic spiked to 3x normal levels due to a
marketing campaign. Once all 50 connections were in use, new requests
queued and eventually timed out, resulting in 500 errors to users.

**Contributing Factors:**
1. No auto-scaling for database connection pool
2. No alert on connection pool saturation
3. Inadequate load testing (only tested at 2x normal load)
4. Marketing campaign not communicated to engineering

## What Went Well
- Alert fired within 2 minutes of elevated errors
- IC assigned immediately, clear command structure
- Root cause identified in 10 minutes
- Mitigation applied quickly (first fix in 15 minutes)
- No data loss or corruption
- Good communication with stakeholders

## What Went Wrong
- No proactive monitoring of connection pool usage
- Static connection pool configuration (should be dynamic)
- Load testing didn't cover 3x traffic scenarios
- Marketing campaign not coordinated with engineering
- No circuit breaker to fail fast instead of queueing

## Action Items

| Action | Owner | Due Date | Status |
|--------|-------|----------|--------|
| Add Prometheus metrics for DB connection pool usage | @john | Jan 18 | ✅ Done |
| Implement auto-scaling for connection pool | @alice | Jan 22 | 🔄 In Progress |
| Add alert: connection pool > 80% | @jane | Jan 17 | ✅ Done |
| Load test at 5x traffic | @bob | Jan 25 | ⏳ Not Started |
| Document connection pool tuning in runbook | @john | Jan 20 | ✅ Done |
| Create process for eng/marketing coordination | @jane | Jan 30 | ⏳ Not Started |
| Implement circuit breaker for DB calls | @alice | Feb 5 | ⏳ Not Started |

## Lessons Learned
1. **Static limits are failure points** - Any statically configured resource
   limit will eventually be exceeded under load. Use auto-scaling.

2. **Test at higher multiples** - If you expect 2x traffic, test at 5x.
   Real-world spikes can exceed expectations.

3. **Monitor saturation, not just errors** - By the time error rate spiked,
   the connection pool had been saturated for minutes. Earlier detection
   would have enabled proactive mitigation.

4. **Cross-functional communication matters** - Marketing campaigns that
   drive traffic need engineering input on infrastructure capacity.

## Related Incidents
- 2024-11-10: Similar issue with Redis connection pool (SEV-3)
- 2024-09-05: Database timeout during Black Friday (SEV-1)

## Appendix
- [Grafana dashboard during incident](https://grafana.example.com/incident-123)
- [Slack incident channel](https://slack.com/incident-2025-01-15-api-errors)
- [Runbook: Database connection issues](runbooks/database-connections.md)

Postmortem Best Practices

Timing:
  - Schedule within 48 hours of resolution
  - Complete within 1 week
  - Review with team within 2 weeks

Facilitation:
  - Incident Commander facilitates
  - All responders attend
  - No managers in initial meeting (psychological safety)
  - 60-90 minute session

Focus Areas:
  - Timeline accuracy (facts, not opinions)
  - Root cause (not root person)
  - System gaps (not individual mistakes)
  - Process improvements (not blame)

Language:
  - "The system failed to..." (not "Bob forgot to...")
  - "We lacked visibility into..." (not "We should have known...")
  - "The process allowed..." (not "They didn't follow...")

Action Items:
  - Specific and measurable
  - Assigned owner and due date
  - Tracked to completion
  - Reviewed in next postmortem

Sharing:
  - Share with entire engineering org
  - Post on company wiki
  - Include in monthly all-hands
  - Add to runbook library

Training and Drills

Wheel of Misfortune

Purpose:
  - Practice incident response in safe environment
  - Train new on-call engineers
  - Validate runbooks
  - Identify gaps in knowledge

Format:
  - 60-minute session
  - 1 facilitator, 3-5 participants
  - Use past incidents or hypothetical scenarios
  - Rotate roles (IC, OL, CL)

Process:
  1. Facilitator presents initial symptoms
  2. Participants ask questions (facilitator answers as if it's real)
  3. Participants investigate using real tools (read-only)
  4. Participants propose mitigations
  5. Facilitator reveals outcome
  6. Debrief: What went well, what to improve

Scenarios:
  - Database connection pool exhausted
  - Kubernetes node failure
  - DDoS attack
  - Certificate expiration
  - Cascading failure

Failure Friday / Chaos Engineering

Purpose:
  - Validate incident response readiness
  - Test systems under failure conditions
  - Train team on real-world scenarios

Schedule:
  - Weekly, Friday 2-4 PM (low-traffic window)
  - Announced 1 day in advance
  - IC on standby

Experiments:
  - Pod termination (10% of pods)
  - Network latency injection (100ms)
  - Database connection limit reduction
  - External API failures
  - Resource exhaustion (CPU/memory)

Success Criteria:
  - SLOs maintained during failure
  - Alerts fired appropriately
  - Team responded within SLA
  - Automatic recovery worked

Debrief:
  - Document findings
  - Update runbooks
  - Fix gaps discovered

This incident response guide provides production-ready processes for managing incidents effectively with blameless culture and continuous improvement.

Structured Logging Best Practices

Logging Principles

2025 Standards

• Structured Format — JSON for machine parsing
• Consistent Schema — Same fields across all services
• Correlation IDs — Link logs to traces
• Appropriate Levels — Don't log everything at INFO
• Low Cardinality Labels — For Loki indexing
• Sensitive Data — Never log PII, secrets, tokens
• Retention Policy — Define and enforce log lifecycle

Structured Logging Libraries

Node.js (Pino)

import pino from 'pino';
import { trace } from '@opentelemetry/api';

// Configure logger
const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  formatters: {
    level: (label) => ({ level: label }),
  },
  timestamp: pino.stdTimeFunctions.isoTime,

  // Inject trace context
  mixin() {
    const span = trace.getActiveSpan();
    if (!span) return {};

    const { traceId, spanId } = span.spanContext();
    return {
      trace_id: traceId,
      span_id: spanId,
    };
  },

  // Redact sensitive fields
  redact: {
    paths: [
      'password',
      'token',
      'api_key',
      'secret',
      'authorization',
      'credit_card',
      '*.password',
      '*.token',
    ],
    censor: '[REDACTED]',
  },
});

// Child logger with context
const requestLogger = logger.child({
  request_id: 'req_abc123',
  user_id: 'user_456',
});

// Log examples
logger.info('Application started');

logger.info(
  {
    user_id: '123',
    action: 'login',
    ip_address: '192.168.1.1',
  },
  'User logged in'
);

logger.error(
  {
    error: {
      message: err.message,
      stack: err.stack,
      code: err.code,
    },
    context: { order_id: 'ord_789' },
  },
  'Payment processing failed'
);

// Output:
// {"level":"info","time":"2025-01-15T10:30:00.000Z","msg":"Application started"}
// {"level":"info","time":"2025-01-15T10:30:01.000Z","trace_id":"abc123","span_id":"def456","user_id":"123","action":"login","ip_address":"192.168.1.1","msg":"User logged in"}

Python (structlog)

import structlog
from opentelemetry import trace

# Configure structlog
structlog.configure(
    processors=[
        structlog.stdlib.filter_by_level,
        structlog.stdlib.add_logger_name,
        structlog.stdlib.add_log_level,
        structlog.stdlib.PositionalArgumentsFormatter(),
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.StackInfoRenderer(),
        structlog.processors.format_exc_info,
        structlog.processors.UnicodeDecoder(),
        # Add trace context
        lambda _, __, event_dict: add_trace_context(event_dict),
        structlog.processors.JSONRenderer()
    ],
    wrapper_class=structlog.stdlib.BoundLogger,
    context_class=dict,
    logger_factory=structlog.stdlib.LoggerFactory(),
    cache_logger_on_first_use=True,
)

def add_trace_context(event_dict):
    span = trace.get_current_span()
    if span.is_recording():
        ctx = span.get_span_context()
        event_dict['trace_id'] = format(ctx.trace_id, '032x')
        event_dict['span_id'] = format(ctx.span_id, '016x')
    return event_dict

logger = structlog.get_logger()

# Log examples
logger.info("application_started")

logger.info(
    "user_logged_in",
    user_id="123",
    action="login",
    ip_address="192.168.1.1"
)

try:
    process_payment()
except Exception as e:
    logger.error(
        "payment_processing_failed",
        exc_info=True,
        order_id="ord_789",
        amount=99.99
    )

Go (zap)

package main

import (
    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
    "go.opentelemetry.io/otel/trace"
)

func newLogger() *zap.Logger {
    config := zap.Config{
        Level:       zap.NewAtomicLevelAt(zap.InfoLevel),
        Development: false,
        Encoding:    "json",
        EncoderConfig: zapcore.EncoderConfig{
            TimeKey:        "time",
            LevelKey:       "level",
            NameKey:        "logger",
            CallerKey:      "caller",
            MessageKey:     "msg",
            StacktraceKey:  "stacktrace",
            LineEnding:     zapcore.DefaultLineEnding,
            EncodeLevel:    zapcore.LowercaseLevelEncoder,
            EncodeTime:     zapcore.ISO8601TimeEncoder,
            EncodeDuration: zapcore.SecondsDurationEncoder,
            EncodeCaller:   zapcore.ShortCallerEncoder,
        },
        OutputPaths:      []string{"stdout"},
        ErrorOutputPaths: []string{"stderr"},
    }

    logger, _ := config.Build()
    return logger
}

func logWithTrace(logger *zap.Logger, span trace.Span) *zap.Logger {
    ctx := span.SpanContext()
    return logger.With(
        zap.String("trace_id", ctx.TraceID().String()),
        zap.String("span_id", ctx.SpanID().String()),
    )
}

func main() {
    logger := newLogger()
    defer logger.Sync()

    logger.Info("application started")

    logger.Info("user logged in",
        zap.String("user_id", "123"),
        zap.String("action", "login"),
        zap.String("ip_address", "192.168.1.1"),
    )

    logger.Error("payment processing failed",
        zap.Error(err),
        zap.String("order_id", "ord_789"),
        zap.Float64("amount", 99.99),
    )
}

Log Levels

Standard Levels and Usage

TRACE (10):
  Usage: Extremely detailed debugging
  Example: "Function entry/exit", "Variable values at each step"
  Production: Disabled
  Development: Use sparingly

DEBUG (20):
  Usage: Diagnostic information
  Example: "Query executed", "Cache miss", "Retry attempt"
  Production: Disabled or very limited
  Development: Common

INFO (30):
  Usage: Normal operational events
  Example: "Server started", "Request completed", "Job processed"
  Production: Default level
  Development: Common

WARN (40):
  Usage: Potentially harmful situations
  Example: "Deprecated API used", "Retry threshold approaching", "Slow query"
  Production: Always logged
  Development: Always logged

ERROR (50):
  Usage: Error events that allow continued execution
  Example: "API call failed", "Database timeout", "Validation error"
  Production: Always logged, may trigger alerts
  Development: Always logged

FATAL (60):
  Usage: Severe errors causing process termination
  Example: "Cannot connect to database", "Out of memory", "Critical failure"
  Production: Always logged, triggers critical alerts
  Development: Always logged

Level Selection Guidelines

// WRONG: Everything at INFO
logger.info('Function started');              // Too verbose
logger.info('Checking cache');                // Too verbose
logger.info({ result }, 'Cache hit');         // Too verbose
logger.info('Processing payment');            // OK, but could be DEBUG
logger.info('Payment successful');            // OK

// RIGHT: Appropriate levels
logger.debug('Entering processPayment function');
logger.debug({ cacheKey }, 'Checking cache');
logger.debug({ result }, 'Cache hit');
logger.info({ order_id, amount }, 'Payment processed');  // Business event
logger.warn({ retries: 2 }, 'Payment service slow');
logger.error({ error }, 'Payment failed');

// Business events → INFO
// System health → WARN
// Failures → ERROR
// Diagnostics → DEBUG

Grafana Loki Setup

Architecture

Applications → Promtail → Loki → Grafana
                ↓
            (Optional)
         OpenTelemetry Collector

Loki Configuration

# loki-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

# Retention configuration
limits_config:
  retention_period: 744h  # 31 days

  # Prevent high cardinality issues
  max_streams_per_user: 10000
  max_global_streams_per_user: 100000

  # Query limits
  max_query_length: 721h  # 30 days
  max_query_parallelism: 32

  # Ingestion limits
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  per_stream_rate_limit: 3MB

# Compactor for retention enforcement
compactor:
  working_directory: /loki/compactor
  shared_store: filesystem
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150

# Query frontend for caching
query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

Promtail Configuration

# promtail-config.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  # Docker container logs
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      # Container name as label
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
      # Container labels as Loki labels
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: 'service'
    pipeline_stages:
      # Parse JSON logs
      - json:
          expressions:
            level: level
            msg: msg
            trace_id: trace_id
            user_id: user_id
      # Extract level as label (LOW cardinality)
      - labels:
          level:
      # Only log trace_id, not as label (HIGH cardinality)
      - output:
          source: msg

  # Kubernetes pods
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Only pods with logging=enabled annotation
      - source_labels:
          - __meta_kubernetes_pod_annotation_logging_enabled
        action: keep
        regex: true
      # Add namespace
      - source_labels:
          - __meta_kubernetes_namespace
        target_label: namespace
      # Add pod name
      - source_labels:
          - __meta_kubernetes_pod_name
        target_label: pod
      # Add app label
      - source_labels:
          - __meta_kubernetes_pod_label_app
        target_label: app
      # Path to pod logs
      - source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
        target_label: __path__
        separator: /
        replacement: /var/log/pods/*$1/*.log
    pipeline_stages:
      # Parse JSON
      - json:
          expressions:
            level: level
            trace_id: trace_id
            msg: msg
      # Extract log level
      - labels:
          level:
      # Drop noisy logs
      - match:
          selector: '{level="debug"}'
          action: drop

  # System logs
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*.log

LogQL Queries

# Basic text search
{namespace="production", app="api"} |= "error"

# JSON parsing and filtering
{app="api"}
  | json
  | level="error"
  | user_id="123"

# Regex filtering
{app="api"} |~ "error|failed|timeout"

# Line format (extract specific fields)
{app="api"}
  | json
  | line_format "{{.time}} {{.level}} {{.msg}}"

# Log rate (logs per second)
rate({app="api"}[5m])

# Count logs by level
sum by (level) (
  count_over_time({namespace="production"}[1h])
)

# Error percentage
sum(rate({app="api", level="error"}[5m]))
/
sum(rate({app="api"}[5m]))

# Top 10 error messages
topk(10,
  sum by (msg) (
    count_over_time({level="error"} | json [1h])
  )
)

# Logs with specific trace ID (correlation)
{namespace="production"}
  | json
  | trace_id="abc123def456"

# Slow requests (>1s latency)
{app="api"}
  | json
  | duration > 1000
  | line_format "{{.method}} {{.path}} took {{.duration}}ms"

# Pattern matching
{app="api"}
  | pattern `<_> level=<level> msg="<msg>"`
  | level="error"

ELK Stack (Alternative to Loki)

When to Use ELK vs Loki

Use Loki when:
  - Cost-sensitive (10x cheaper than ELK)
  - Already using Prometheus/Grafana
  - Simple log queries (grep-like)
  - Cloud-native/Kubernetes environment

Use ELK when:
  - Advanced full-text search needed
  - Complex log analytics and aggregations
  - Security/compliance (detailed audit logs)
  - Multiple data sources beyond logs

Elasticsearch Configuration

# elasticsearch.yml
cluster.name: logging-cluster
node.name: node-1
network.host: 0.0.0.0

# Index lifecycle management
xpack.ilm.enabled: true

# Security
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

Logstash Pipeline

# logstash.conf
input {
  beats {
    port => 5044
  }
}

filter {
  # Parse JSON logs
  json {
    source => "message"
  }

  # Add timestamp
  date {
    match => ["time", "ISO8601"]
    target => "@timestamp"
  }

  # Grok for non-JSON logs
  grok {
    match => {
      "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:msg}"
    }
  }

  # GeoIP enrichment
  geoip {
    source => "ip_address"
  }

  # Remove sensitive fields
  mutate {
    remove_field => ["password", "token", "api_key"]
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "logs-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "${ELASTIC_PASSWORD}"
  }
}

Filebeat Configuration

# filebeat.yml
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/app/*.log
    json.keys_under_root: true
    json.add_error_key: true
    fields:
      app: api
      environment: production

  - type: container
    paths:
      - '/var/lib/docker/containers/*/*.log'

processors:
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

output.logstash:
  hosts: ["logstash:5044"]

# Or directly to Elasticsearch
# output.elasticsearch:
#   hosts: ["http://elasticsearch:9200"]
#   index: "logs-%{+yyyy.MM.dd}"

Best Practices

Cardinality Management

# LOW cardinality labels (good for Loki)
Good labels:
  - namespace (5-20 values)
  - app/service (10-50 values)
  - level (5 values: trace, debug, info, warn, error)
  - environment (3-5 values: dev, staging, prod)

# HIGH cardinality data (keep in log body)
Bad labels:
  - user_id (millions of values)
  - request_id (unique per request)
  - trace_id (unique per trace)
  - session_id (millions of values)
  - ip_address (thousands of values)

# Rule: Keep total label combinations < 10,000
# Example: 5 namespaces × 20 apps × 5 levels = 500 combinations ✓

Sensitive Data Handling

// Redact sensitive data
const logger = pino({
  redact: {
    paths: [
      'req.headers.authorization',
      'req.headers.cookie',
      'password',
      'creditCard',
      'ssn',
      '*.password',
      '*.token',
      '*.secret',
    ],
    censor: '[REDACTED]',
  },
});

// Custom serializers
const logger = pino({
  serializers: {
    user: (user) => ({
      id: user.id,
      email: user.email.replace(/(.{2}).*(@.*)/, '$1***$2'), // ma***@example.com
      // Omit sensitive fields
    }),
  },
});

Log Rotation

# logrotate config
/var/log/app/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0644 app app
    postrotate
        # Signal app to reopen log file
        killall -SIGUSR1 app
    endscript
}

Sampling for High-Volume Logs

// Sample debug logs (keep 10%)
const shouldLog = (level: string) => {
  if (level !== 'debug') return true;
  return Math.random() < 0.1; // 10% sample rate
};

if (shouldLog('debug')) {
  logger.debug({ details }, 'Debug information');
}

// Or use structured sampling
const logger = pino({
  level: 'debug',
  hooks: {
    logMethod(inputArgs, method, level) {
      // Sample debug logs
      if (level === 20 && Math.random() > 0.1) {
        return; // Skip this log
      }
      return method.apply(this, inputArgs);
    },
  },
});

This logging guide provides production-ready structured logging configurations for comprehensive log management with Loki and ELK.

Monitoring with Prometheus and Grafana

Prometheus Architecture

Core Components

Prometheus Server:
  - Scrapes and stores time-series data
  - Executes PromQL queries
  - Evaluates alerting rules
  - Forwards alerts to Alertmanager

Exporters:
  - node_exporter: Hardware and OS metrics
  - kube-state-metrics: Kubernetes object states
  - blackbox_exporter: Endpoint probes
  - Custom exporters: Application metrics

Alertmanager:
  - Alert deduplication
  - Grouping and routing
  - Silences and inhibitions
  - Notification channels

Pushgateway:
  - For short-lived jobs
  - Batch jobs that can't be scraped
  - Use sparingly (anti-pattern for most cases)

Scrape Configuration

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'production-us-east-1'
    environment: 'prod'

# Alertmanager connection
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - alertmanager:9093

# Recording and alerting rules
rule_files:
  - 'rules/recording/*.yml'
  - 'rules/alerting/*.yml'

scrape_configs:
  # Prometheus self-monitoring
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Node exporter (system metrics)
  - job_name: 'node'
    static_configs:
      - targets:
        - 'node1.internal:9100'
        - 'node2.internal:9100'

  # Kubernetes service discovery
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Only scrape pods with prometheus.io/scrape annotation
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      # Use custom port if specified
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)
        replacement: $1:${1}
      # Add pod labels
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      # Add namespace
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      # Add pod name
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

Application Metrics Instrumentation

// Node.js with prom-client
import express from 'express';
import client from 'prom-client';

const app = express();

// Create a Registry
const register = new client.Registry();

// Add default metrics (CPU, memory, event loop lag)
client.collectDefaultMetrics({ register });

// Custom metrics
const httpRequestDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status'],
  buckets: [0.001, 0.01, 0.1, 0.5, 1, 2, 5], // 1ms to 5s
  registers: [register],
});

const httpRequestsTotal = new client.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'route', 'status'],
  registers: [register],
});

const activeConnections = new client.Gauge({
  name: 'http_active_connections',
  help: 'Number of active HTTP connections',
  registers: [register],
});

// Middleware to track metrics
app.use((req, res, next) => {
  const start = Date.now();
  activeConnections.inc();

  res.on('finish', () => {
    const duration = (Date.now() - start) / 1000;
    httpRequestDuration.labels(req.method, req.route?.path || req.path, res.statusCode).observe(duration);
    httpRequestsTotal.labels(req.method, req.route?.path || req.path, res.statusCode).inc();
    activeConnections.dec();
  });

  next();
});

// Metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

app.listen(3000);

PromQL Query Language

Basic Queries

# Instant vector - current value
http_requests_total

# With label filtering
http_requests_total{status="200", method="GET"}

# Regex matching
http_requests_total{status=~"2..", method!="OPTIONS"}

# Range vector - time series over duration
http_requests_total[5m]

# Rate of requests per second
rate(http_requests_total[5m])

# Sum across all instances
sum(rate(http_requests_total[5m]))

# Group by label
sum(rate(http_requests_total[5m])) by (method, status)

Advanced Queries

# P95 latency from histogram
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)

# Error rate (5xx / total)
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

# Requests per second by method
sum(rate(http_requests_total[5m])) by (method)

# CPU usage percentage
100 * (1 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])))

# Memory usage percentage
100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes))

# Disk space remaining
node_filesystem_avail_bytes{mountpoint="/"}

# Predict when disk will be full (linear regression)
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[1h], 4*3600) < 0

Subqueries

# Max request rate in the last hour, sampled every 5 minutes
max_over_time(
  rate(http_requests_total[5m])[1h:5m]
)

# 99th percentile latency over the last 24 hours
quantile_over_time(0.99,
  http_request_duration_seconds[24h]
)

Recording Rules

# rules/recording/api_performance.yml
groups:
  - name: api_performance
    interval: 30s
    rules:
      # Request rate by service
      - record: job:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job, method)

      # Error rate by service
      - record: job:http_errors:rate5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
          /
          sum(rate(http_requests_total[5m])) by (job)

      # P50, P95, P99 latency
      - record: job:http_latency:p50
        expr: histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le))

      - record: job:http_latency:p95
        expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le))

      - record: job:http_latency:p99
        expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (job, le))

      # Aggregation for federation
      - record: instance:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (instance)

Alerting Rules

# rules/alerting/api_alerts.yml
groups:
  - name: api_alerts
    interval: 30s
    rules:
      # High error rate
      - alert: HighErrorRate
        expr: job:http_errors:rate5m > 0.05  # 5% error rate
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High error rate on {{ $labels.job }}"
          description: "Error rate is {{ $value | humanizePercentage }} (threshold: 5%)"

      # Critical error rate
      - alert: CriticalErrorRate
        expr: job:http_errors:rate5m > 0.10  # 10% error rate
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "CRITICAL: Error rate on {{ $labels.job }}"
          description: "Error rate is {{ $value | humanizePercentage }} (threshold: 10%)"
          runbook_url: "https://wiki.example.com/runbooks/high-error-rate"

      # High latency
      - alert: HighLatency
        expr: job:http_latency:p95 > 0.5  # 500ms P95
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High latency on {{ $labels.job }}"
          description: "P95 latency is {{ $value }}s (threshold: 0.5s)"

      # Service down
      - alert: ServiceDown
        expr: up{job="api"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Service {{ $labels.job }} is down"
          description: "{{ $labels.instance }} has been down for more than 1 minute"

      # High memory usage
      - alert: HighMemoryUsage
        expr: |
          100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is {{ $value | humanizePercentage }}"

      # Disk space low
      - alert: DiskSpaceLow
        expr: |
          (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Only {{ $value | humanizePercentage }} disk space remaining"

      # Disk will fill in 4 hours
      - alert: DiskWillFillSoon
        expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[1h], 4*3600) < 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Disk {{ $labels.mountpoint }} will fill in 4 hours"
          description: "Based on current usage trend"

Alertmanager Configuration

# alertmanager.yml
global:
  resolve_timeout: 5m
  slack_api_url: 'https://hooks.slack.com/services/XXX'
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'

# Templates for notifications
templates:
  - '/etc/alertmanager/templates/*.tmpl'

# Route alerts based on labels
route:
  receiver: 'default'
  group_by: ['alertname', 'cluster', 'job']
  group_wait: 10s        # Wait before sending first notification
  group_interval: 10s    # Wait before sending batch of new alerts
  repeat_interval: 12h   # Wait before re-sending notification

  routes:
    # Critical alerts to PagerDuty
    - match:
        severity: critical
      receiver: 'pagerduty'
      group_wait: 0s
      repeat_interval: 5m

    # Database alerts to database team
    - match_re:
        job: '.*database.*'
      receiver: 'database-team'

    # Non-critical to Slack
    - match:
        severity: warning
      receiver: 'slack'

receivers:
  - name: 'default'
    email_configs:
      - to: 'ops@example.com'

  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: '<PAGERDUTY_SERVICE_KEY>'
        description: '{{ .GroupLabels.alertname }}'
        severity: '{{ .CommonLabels.severity }}'

  - name: 'slack'
    slack_configs:
      - channel: '#alerts'
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}'
        send_resolved: true

  - name: 'database-team'
    email_configs:
      - to: 'db-team@example.com'
    slack_configs:
      - channel: '#database-alerts'

# Inhibition rules - suppress alerts when others fire
inhibit_rules:
  # If service is down, don't alert on high latency
  - source_match:
      alertname: ServiceDown
    target_match:
      alertname: HighLatency
    equal: ['job', 'instance']

Grafana Dashboards

RED Dashboard for Services

{
  "dashboard": {
    "title": "API Service - RED Metrics",
    "rows": [
      {
        "title": "Request Rate",
        "panels": [
          {
            "title": "Requests per Second",
            "targets": [
              {
                "expr": "sum(rate(http_requests_total{job=\"api\"}[5m])) by (method)"
              }
            ],
            "type": "graph"
          },
          {
            "title": "Request Rate by Route",
            "targets": [
              {
                "expr": "topk(10, sum(rate(http_requests_total{job=\"api\"}[5m])) by (route))"
              }
            ],
            "type": "graph"
          }
        ]
      },
      {
        "title": "Error Rate",
        "panels": [
          {
            "title": "Error Rate %",
            "targets": [
              {
                "expr": "100 * (sum(rate(http_requests_total{job=\"api\",status=~\"5..\"}[5m])) / sum(rate(http_requests_total{job=\"api\"}[5m])))"
              }
            ],
            "type": "graph",
            "fieldConfig": {
              "defaults": {
                "unit": "percent"
              }
            }
          },
          {
            "title": "Errors by Status Code",
            "targets": [
              {
                "expr": "sum(rate(http_requests_total{job=\"api\",status=~\"5..\"}[5m])) by (status)"
              }
            ],
            "type": "graph"
          }
        ]
      },
      {
        "title": "Duration (Latency)",
        "panels": [
          {
            "title": "Request Latency Percentiles",
            "targets": [
              {
                "expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{job=\"api\"}[5m])) by (le))",
                "legendFormat": "P50"
              },
              {
                "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job=\"api\"}[5m])) by (le))",
                "legendFormat": "P95"
              },
              {
                "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job=\"api\"}[5m])) by (le))",
                "legendFormat": "P99"
              }
            ],
            "type": "graph",
            "fieldConfig": {
              "defaults": {
                "unit": "s"
              }
            }
          }
        ]
      }
    ]
  }
}

USE Dashboard for Resources

{
  "dashboard": {
    "title": "Node Resources - USE Metrics",
    "templating": {
      "list": [
        {
          "name": "instance",
          "type": "query",
          "query": "label_values(node_cpu_seconds_total, instance)"
        }
      ]
    },
    "rows": [
      {
        "title": "Utilization",
        "panels": [
          {
            "title": "CPU Utilization %",
            "targets": [
              {
                "expr": "100 * (1 - avg(rate(node_cpu_seconds_total{mode=\"idle\",instance=\"$instance\"}[5m])))"
              }
            ],
            "type": "graph"
          },
          {
            "title": "Memory Utilization %",
            "targets": [
              {
                "expr": "100 * (1 - (node_memory_MemAvailable_bytes{instance=\"$instance\"} / node_memory_MemTotal_bytes{instance=\"$instance\"}))"
              }
            ],
            "type": "graph"
          },
          {
            "title": "Disk Utilization %",
            "targets": [
              {
                "expr": "100 * (1 - (node_filesystem_avail_bytes{instance=\"$instance\",mountpoint=\"/\"} / node_filesystem_size_bytes{instance=\"$instance\",mountpoint=\"/\"}))"
              }
            ],
            "type": "graph"
          }
        ]
      },
      {
        "title": "Saturation",
        "panels": [
          {
            "title": "Load Average (1m, 5m, 15m)",
            "targets": [
              {
                "expr": "node_load1{instance=\"$instance\"}",
                "legendFormat": "1m"
              },
              {
                "expr": "node_load5{instance=\"$instance\"}",
                "legendFormat": "5m"
              },
              {
                "expr": "node_load15{instance=\"$instance\"}",
                "legendFormat": "15m"
              }
            ],
            "type": "graph"
          },
          {
            "title": "Disk I/O Utilization %",
            "targets": [
              {
                "expr": "rate(node_disk_io_time_seconds_total{instance=\"$instance\"}[5m]) * 100"
              }
            ],
            "type": "graph"
          }
        ]
      },
      {
        "title": "Errors",
        "panels": [
          {
            "title": "Network Errors",
            "targets": [
              {
                "expr": "rate(node_network_receive_errs_total{instance=\"$instance\"}[5m])",
                "legendFormat": "RX {{ device }}"
              },
              {
                "expr": "rate(node_network_transmit_errs_total{instance=\"$instance\"}[5m])",
                "legendFormat": "TX {{ device }}"
              }
            ],
            "type": "graph"
          }
        ]
      }
    ]
  }
}

Performance Optimization

Metric Cardinality

# GOOD: Low cardinality labels
http_requests_total{method="GET", status="200", service="api"}
# Cardinality = methods (5) × statuses (10) × services (20) = 1,000

# BAD: High cardinality labels (DON'T DO THIS!)
http_requests_total{user_id="12345", session_id="abc..."}
# Cardinality = users (1M) × sessions (10M) = EXPLODES

# Rule: Limit label cardinality to < 10,000 combinations
# Use high-cardinality data in logs/traces, not metrics

Query Optimization

# SLOW: Calculates rate for each series, then sums
sum(rate(http_requests_total[5m]))

# FAST: Sums first, then calculates rate (fewer series)
rate(sum(http_requests_total)[5m])

# SLOW: Large range for high-resolution data
rate(http_requests_total[1h])

# FAST: Smaller range with recording rules
rate(http_requests_total[5m])

# Use recording rules for dashboard queries
# Dashboards should query recording rules, not raw metrics

Retention and Storage

# Prometheus config
storage:
  tsdb:
    path: /prometheus/data
    retention.time: 15d  # Keep 15 days locally
    retention.size: 50GB # Or max 50GB

# Use remote write for long-term storage
remote_write:
  - url: "https://thanos.example.com/api/v1/receive"
    queue_config:
      capacity: 10000
      max_shards: 50
      min_shards: 1
      max_samples_per_send: 5000
      batch_send_deadline: 5s

# Thanos for long-term storage (years)
# VictoriaMetrics for cost-effective storage
# Cortex for multi-tenant setups

Common Patterns

Blackbox Monitoring

# Probe endpoints from outside
scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://api.example.com/health
        - https://www.example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

# Check SSL certificate expiry
- alert: SSLCertExpiringSoon
  expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30  # 30 days
  annotations:
    summary: "SSL certificate expires in {{ $value | humanizeDuration }}"

Service Discovery

# EC2 instances
- job_name: 'ec2'
  ec2_sd_configs:
    - region: us-east-1
      port: 9100
  relabel_configs:
    - source_labels: [__meta_ec2_tag_Name]
      target_label: instance
    - source_labels: [__meta_ec2_tag_Environment]
      target_label: environment

# Consul service discovery
- job_name: 'consul'
  consul_sd_configs:
    - server: 'consul.example.com:8500'
      services: ['web', 'api', 'database']

This monitoring guide provides production-ready Prometheus and Grafana configurations for comprehensive system observability.

Distributed Tracing with OpenTelemetry

OpenTelemetry Overview

What is OpenTelemetry?

OpenTelemetry (OTel) is a CNCF standard for collecting telemetry data:

• Vendor-neutral — Works with any backend (Jaeger, Tempo, Datadog, etc.)
• Auto-instrumentation — Automatic tracing for common frameworks
• Unified API — Single SDK for metrics, logs, and traces
• Production-ready — 79% of organizations use or are considering OTel

Core Concepts

Trace:
  - Entire journey of a request through the system
  - Unique trace ID shared by all related spans
  - Example: User checkout flow across 5 microservices

Span:
  - Single unit of work within a trace
  - Has start time, end time, attributes, events
  - Parent-child relationships form trace tree
  - Example: Database query within API handler

Context:
  - Metadata that flows across service boundaries
  - Contains trace ID, span ID, trace flags
  - Propagated via HTTP headers, message queues, etc.

Attributes:
  - Key-value pairs attached to spans
  - Describe the operation (http.method, db.statement)
  - Enable filtering and grouping in backends

Auto-Instrumentation

Node.js

// instrumentation.ts - Load BEFORE any other imports
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'payment-service',
    [SemanticResourceAttributes.SERVICE_VERSION]: '1.2.3',
    [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: 'production',
  }),

  traceExporter: new OTLPTraceExporter({
    // Send to OpenTelemetry Collector
    url: 'http://otel-collector:4318/v1/traces',
  }),

  instrumentations: [
    getNodeAutoInstrumentations({
      // Configure auto-instrumentation
      '@opentelemetry/instrumentation-http': {
        enabled: true,
        ignoreIncomingPaths: ['/health', '/metrics'],
      },
      '@opentelemetry/instrumentation-express': { enabled: true },
      '@opentelemetry/instrumentation-pg': { enabled: true }, // PostgreSQL
      '@opentelemetry/instrumentation-redis': { enabled: true },
      '@opentelemetry/instrumentation-mongodb': { enabled: true },
      '@opentelemetry/instrumentation-aws-sdk': { enabled: true },
      '@opentelemetry/instrumentation-fs': { enabled: false }, // Too noisy
    }),
  ],
});

sdk.start();

// Graceful shutdown
process.on('SIGTERM', async () => {
  await sdk.shutdown();
  process.exit(0);
});

// Now import your app
import './app';

Python

# Auto-instrumentation via CLI
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor

# Configure resource
resource = Resource(attributes={
    "service.name": "payment-service",
    "service.version": "1.2.3",
    "deployment.environment": "production",
})

# Setup tracer provider
provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(OTLPSpanExporter(
    endpoint="http://otel-collector:4317",
))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

# Auto-instrument libraries
FlaskInstrumentor().instrument()
RequestsInstrumentor().instrument()
SQLAlchemyInstrumentor().instrument()

# Or use CLI: opentelemetry-instrument python app.py

Go

package main

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.17.0"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
)

func initTracer() func() {
    ctx := context.Background()

    // Resource describes the service
    res, _ := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceName("payment-service"),
            semconv.ServiceVersion("1.2.3"),
            semconv.DeploymentEnvironment("production"),
        ),
    )

    // OTLP exporter
    exporter, _ := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("otel-collector:4317"),
        otlptracegrpc.WithInsecure(),
    )

    // Tracer provider
    provider := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(res),
        sdktrace.WithSampler(sdktrace.ParentBased(
            sdktrace.TraceIDRatioBased(0.1), // 10% sampling
        )),
    )
    otel.SetTracerProvider(provider)

    return func() { provider.Shutdown(ctx) }
}

func main() {
    cleanup := initTracer()
    defer cleanup()

    // Auto-instrumented HTTP client
    client := &http.Client{
        Transport: otelhttp.NewTransport(http.DefaultTransport),
    }

    // Auto-instrumented HTTP server
    handler := otelhttp.NewHandler(http.HandlerFunc(handleRequest), "handleRequest")
    http.Handle("/", handler)

    http.ListenAndServe(":8080", nil)
}

Manual Instrumentation

Creating Custom Spans

import { trace, SpanStatusCode, SpanKind } from '@opentelemetry/api';

const tracer = trace.getTracer('payment-service', '1.0.0');

async function processPayment(orderId: string, amount: number) {
  // Start a new span
  return await tracer.startActiveSpan(
    'processPayment',
    {
      kind: SpanKind.INTERNAL,
      attributes: {
        'order.id': orderId,
        'payment.amount': amount,
        'payment.currency': 'USD',
      },
    },
    async (span) => {
      try {
        // Add events (logs within span)
        span.addEvent('validating payment');

        // Validate
        await validatePayment(orderId, amount);

        // Child span for external API call
        const result = await chargeCustomer(orderId, amount);

        // Update span attributes
        span.setAttribute('payment.transaction_id', result.transactionId);
        span.setAttribute('payment.status', 'completed');

        // Mark span as successful
        span.setStatus({ code: SpanStatusCode.OK });

        return result;
      } catch (error) {
        // Record exception
        span.recordException(error);
        span.setStatus({
          code: SpanStatusCode.ERROR,
          message: error.message,
        });

        throw error;
      } finally {
        // Always end the span
        span.end();
      }
    }
  );
}

async function chargeCustomer(orderId: string, amount: number) {
  return await tracer.startActiveSpan(
    'stripe.charge',
    {
      kind: SpanKind.CLIENT,
      attributes: {
        'stripe.method': 'charges.create',
        'http.url': 'https://api.stripe.com/v1/charges',
      },
    },
    async (span) => {
      const result = await stripe.charges.create({
        amount: amount * 100,
        currency: 'usd',
      });

      span.setAttribute('stripe.charge_id', result.id);
      span.end();

      return result;
    }
  );
}

Span Types

import { SpanKind } from '@opentelemetry/api';

// INTERNAL: Internal operations (default)
tracer.startActiveSpan('calculateTotal', { kind: SpanKind.INTERNAL });

// SERVER: Incoming request handler
tracer.startActiveSpan('handleRequest', { kind: SpanKind.SERVER });

// CLIENT: Outgoing request
tracer.startActiveSpan('fetchUser', { kind: SpanKind.CLIENT });

// PRODUCER: Message queue producer
tracer.startActiveSpan('publishMessage', { kind: SpanKind.PRODUCER });

// CONSUMER: Message queue consumer
tracer.startActiveSpan('processMessage', { kind: SpanKind.CONSUMER });

Context Propagation

HTTP Headers (W3C Trace Context)

// Automatic propagation with auto-instrumentation
fetch('https://api.example.com/data', {
  // These headers are injected automatically:
  // traceparent: 00-<trace-id>-<span-id>-01
  // tracestate: vendor1=value1,vendor2=value2
});

// Manual propagation
import { propagation, context } from '@opentelemetry/api';

const carrier = {};
propagation.inject(context.active(), carrier);

// carrier now contains:
// {
//   traceparent: "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"
// }

fetch('https://api.example.com/data', {
  headers: carrier,
});

Message Queues

// Producer: Inject trace context into message
import { propagation, context } from '@opentelemetry/api';

async function publishMessage(queue: string, data: any) {
  return await tracer.startActiveSpan('publishMessage', async (span) => {
    const carrier = {};
    propagation.inject(context.active(), carrier);

    await messageQueue.publish(queue, {
      data,
      headers: carrier, // Propagate trace context
    });

    span.end();
  });
}

// Consumer: Extract trace context from message
async function processMessage(message: Message) {
  const carrier = message.headers;
  const extractedContext = propagation.extract(context.active(), carrier);

  return await context.with(extractedContext, async () => {
    return await tracer.startActiveSpan('processMessage', async (span) => {
      // This span is now part of the original trace
      await handleMessage(message.data);
      span.end();
    });
  });
}

Sampling Strategies

Head Sampling (at span creation)

import { TraceIdRatioBasedSampler, ParentBasedSampler } from '@opentelemetry/sdk-trace-base';

// Sample 10% of traces
const sampler = new TraceIdRatioBasedSampler(0.1);

// Parent-based sampling (respect parent decision)
const parentBasedSampler = new ParentBasedSampler({
  root: new TraceIdRatioBasedSampler(0.1),
});

const provider = new NodeTracerProvider({
  sampler: parentBasedSampler,
});

Tail Sampling (OpenTelemetry Collector)

# otel-collector-config.yaml
processors:
  tail_sampling:
    decision_wait: 10s  # Wait to see full trace
    num_traces: 100000
    expected_new_traces_per_sec: 100

    policies:
      # Always sample errors
      - name: error-traces
        type: status_code
        status_code:
          status_codes: [ERROR]

      # Always sample slow requests (>1s)
      - name: slow-traces
        type: latency
        latency:
          threshold_ms: 1000

      # Sample traces with specific attributes
      - name: vip-users
        type: string_attribute
        string_attribute:
          key: user.tier
          values: [vip, premium]

      # Sample 5% of normal traffic
      - name: probabilistic
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [otlp/jaeger]

OpenTelemetry Collector

Configuration

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  # Add resource attributes
  resource:
    attributes:
      - key: cluster.name
        value: production
        action: insert

  # Batch spans for efficiency
  batch:
    timeout: 10s
    send_batch_size: 1024

  # Memory limiter to prevent OOM
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

  # Sampling
  probabilistic_sampler:
    sampling_percentage: 10

  # Attribute manipulation
  attributes:
    actions:
      - key: http.url
        action: delete  # Remove sensitive URLs
      - key: db.statement
        action: delete  # Remove SQL queries

exporters:
  # Jaeger
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

  # Tempo
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

  # Datadog
  datadog:
    api:
      key: ${DD_API_KEY}

  # Logging (for debugging)
  logging:
    loglevel: debug

extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  pprof:
    endpoint: 0.0.0.0:1777

service:
  extensions: [health_check, pprof]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, probabilistic_sampler]
      exporters: [otlp/tempo, logging]

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
spec:
  replicas: 3
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: otel-collector
        image: otel/opentelemetry-collector-contrib:0.91.0
        args:
          - "--config=/conf/otel-collector-config.yaml"
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1000m"
        volumeMounts:
        - name: config
          mountPath: /conf
        ports:
        - containerPort: 4317  # OTLP gRPC
        - containerPort: 4318  # OTLP HTTP
        - containerPort: 13133 # Health check
      volumes:
      - name: config
        configMap:
          name: otel-collector-config

---
apiVersion: v1
kind: Service
metadata:
  name: otel-collector
spec:
  selector:
    app: otel-collector
  ports:
  - name: otlp-grpc
    port: 4317
    targetPort: 4317
  - name: otlp-http
    port: 4318
    targetPort: 4318

Trace Backends

Grafana Tempo

# tempo.yaml
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317

ingester:
  trace_idle_period: 10s
  max_block_bytes: 1_000_000
  max_block_duration: 5m

compactor:
  compaction:
    block_retention: 720h  # 30 days

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
      endpoint: s3.amazonaws.com
    wal:
      path: /var/tempo/wal
    pool:
      max_workers: 100
      queue_depth: 10000

Jaeger

# Docker Compose
version: '3'
services:
  jaeger:
    image: jaegertracing/all-in-one:1.52
    environment:
      - COLLECTOR_OTLP_ENABLED=true
    ports:
      - "16686:16686"  # UI
      - "4317:4317"    # OTLP gRPC
      - "4318:4318"    # OTLP HTTP

Best Practices

Span Naming

// GOOD: Operation names, not URLs
tracer.startActiveSpan('GET /users/:id')
tracer.startActiveSpan('database.query')
tracer.startActiveSpan('stripe.charge')

// BAD: High cardinality, not useful
tracer.startActiveSpan('GET /users/12345')  // User ID in name
tracer.startActiveSpan('SELECT * FROM users WHERE id=12345')  // Full query

Attribute Selection

// Standard semantic conventions
span.setAttribute('http.method', 'GET');
span.setAttribute('http.status_code', 200);
span.setAttribute('http.url', 'https://api.example.com/users');
span.setAttribute('db.system', 'postgresql');
span.setAttribute('db.statement', 'SELECT * FROM users WHERE id = $1');
span.setAttribute('db.name', 'myapp');

// Business attributes
span.setAttribute('order.id', 'ord_123');
span.setAttribute('user.id', 'user_456');
span.setAttribute('payment.amount', 99.99);
span.setAttribute('payment.currency', 'USD');

// Avoid high-cardinality values in span names
// Put them in attributes instead

Error Handling

try {
  await doSomething();
  span.setStatus({ code: SpanStatusCode.OK });
} catch (error) {
  // Record full exception with stack trace
  span.recordException(error);

  // Set error status
  span.setStatus({
    code: SpanStatusCode.ERROR,
    message: error.message,
  });

  // Add error attributes
  span.setAttribute('error.type', error.constructor.name);
  span.setAttribute('error.handled', true);

  throw error;
}

Performance Optimization

// Limit span attributes
const limitConfig = {
  attributeValueLengthLimit: 1024,  // Max 1KB per attribute
  attributeCountLimit: 128,         // Max 128 attributes
  eventCountLimit: 128,             // Max 128 events
  linkCountLimit: 128,              // Max 128 links
};

// Batch export for efficiency
const exporter = new OTLPTraceExporter();
const processor = new BatchSpanProcessor(exporter, {
  maxQueueSize: 2048,
  maxExportBatchSize: 512,
  scheduledDelayMillis: 5000,  // Export every 5 seconds
});

Trace Analysis Queries

Grafana Tempo TraceQL

# Find slow traces
{ duration > 1s }

# Find errors
{ status = error }

# Find specific service
{ service.name = "payment-service" }

# Combine conditions
{
  service.name = "api" &&
  http.status_code >= 500 &&
  duration > 500ms
}

# Resource attributes
{ cluster.name = "production" && namespace = "default" }

# Span attributes
{ db.system = "postgresql" && db.statement =~ "SELECT.*users.*" }

# Count spans
{ service.name = "api" } | count() > 100

Common Analysis Patterns

# Top 10 slowest endpoints
topk(10, {service.name="api"} | sort by duration desc)

# Error rate by service
rate({status=error}[5m]) by service.name

# P95 latency by endpoint
histogram_quantile(0.95, {service.name="api"}) by http.route

# Traces with database calls
{service.name="api"} | select(span.db.system)

# Find specific user's traces
{user.id="12345"}

This tracing guide provides production-ready OpenTelemetry configurations for comprehensive distributed tracing.

SLO Definition Template

Service: [Service Name]

Owner: [Team Name] Last Updated: [Date] Review Frequency: Quarterly

---

Service Overview

Description: [Brief description of what this service does]

User Journey: [Describe the critical user paths this service supports]

Dependencies:

• [Upstream service 1]
• [Upstream service 2]
• [Database/cache/queue]

---

SLIs (Service Level Indicators)

1. Availability

Definition: Percentage of successful requests over total requests

Measurement:

# Recording rule
- record: slo:service_availability:ratio
  expr: |
    sum(rate(http_requests_total{service="my-service",status!~"5.."}[5m]))
    /
    sum(rate(http_requests_total{service="my-service"}[5m]))

Data Source: Prometheus metrics from application instrumentation

Valid Requests:

• Include: All HTTP requests to API endpoints
• Exclude: Health checks (/health, /ready)
• Exclude: Internal monitoring requests

Success Criteria:

• HTTP status codes 200-499 (4xx are user errors, not service errors)
• Response received within timeout (30 seconds)

---

2. Latency

Definition: Percentage of requests completed within target latency

Measurement:

# P95 latency
- record: slo:service_latency:p95
  expr: |
    histogram_quantile(0.95,
      sum(rate(http_request_duration_seconds_bucket{service="my-service"}[5m])) by (le)
    )

# Percentage of requests under 200ms
- record: slo:service_latency:success_ratio
  expr: |
    sum(rate(http_request_duration_seconds_bucket{service="my-service",le="0.2"}[5m]))
    /
    sum(rate(http_request_duration_seconds_count{service="my-service"}[5m]))

Target: 95% of requests complete in < 200ms (P95 latency)

Measurement Window: 5-minute rolling window

Breakdown by Endpoint:

• GET /users - 100ms
• POST /orders - 200ms
• GET /search - 500ms (complex query)

---

3. Error Rate

Definition: Percentage of requests that result in server errors

Measurement:

# Error rate
- record: slo:service_errors:rate
  expr: |
    sum(rate(http_requests_total{service="my-service",status=~"5.."}[5m]))
    /
    sum(rate(http_requests_total{service="my-service"}[5m]))

Error Types Counted:

• 5xx HTTP status codes
• Request timeouts
• Circuit breaker open
• Database connection failures

Error Types Excluded:

• 4xx client errors (user mistakes)
• Validation errors
• Authentication failures (user input)

---

SLOs (Service Level Objectives)

Production SLOs

Per-Endpoint SLOs

---

Error Budget

Calculation

SLO = 99.9%
Error_Budget = 100% - 99.9% = 0.1%

# Monthly (30 days)
Total_Minutes = 30 * 24 * 60 = 43,200 minutes
Allowed_Downtime = 43,200 * 0.001 = 43.2 minutes

# Weekly
Weekly_Minutes = 7 * 24 * 60 = 10,080 minutes
Weekly_Allowed_Downtime = 10,080 * 0.001 = 10.08 minutes

Error Budget Policy

Burn Rate Alerts

# Multi-window burn rate alerting
# Based on Google SRE Workbook

- alert: SLOBurnRateCritical
  # Burning 14.4x (will exhaust budget in 2 days)
  expr: |
    (
      slo:service_availability:ratio < 0.856  # 99.9% - (14.4 * 0.1%) = 98.56%
      and
      slo:service_availability:ratio_1h < 0.856
    )
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Burning error budget at 14.4x rate"
    description: "At current rate, will exhaust monthly budget in 2 days"

- alert: SLOBurnRateHigh
  # Burning 6x (will exhaust budget in 5 days)
  expr: |
    (
      slo:service_availability:ratio < 0.94  # 99.9% - (6 * 0.1%) = 99.4%
      and
      slo:service_availability:ratio_6h < 0.94
    )
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "Burning error budget at 6x rate"

- alert: SLOBurnRateModerate
  # Burning 3x (will exhaust budget in 10 days)
  expr: |
    (
      slo:service_availability:ratio < 0.97  # 99.9% - (3 * 0.1%) = 99.7%
      and
      slo:service_availability:ratio_1d < 0.97
    )
  for: 1h
  labels:
    severity: warning

---

SLA (Service Level Agreement)

Customer Commitment: 99.9% monthly availability

Measurement: Based on SLI availability metric

Credits:

• 99.9% - 99.0% availability: 10% service credit
• 99.0% - 95.0% availability: 25% service credit
• < 95.0% availability: 50% service credit

Exclusions:

• Planned maintenance (announced 7 days in advance)
• Customer-caused issues (invalid requests, quota exceeded)
• Force majeure (natural disasters, DDoS attacks)

Claim Process: 1. Customer submits ticket within 30 days 2. SRE team validates metrics 3. Credit applied to next invoice

---

Dashboard

Grafana Dashboard: [Link to dashboard]

Panels: 1. Current availability (gauge) 2. 30-day availability trend (graph) 3. Error budget remaining (gauge + graph) 4. P95 latency by endpoint (graph) 5. Error rate (graph) 6. Request rate (graph)

Example PromQL Queries:

# Current availability (last 5 minutes)
avg_over_time(slo:service_availability:ratio[5m])

# 30-day availability
avg_over_time(slo:service_availability:ratio[30d])

# Error budget remaining
1 - (
  (1 - avg_over_time(slo:service_availability:ratio[30d]))
  /
  (1 - 0.999)  # SLO target
)

# Burn rate
(1 - slo:service_availability:ratio)
/
(1 - 0.999)

---

Alerting Strategy

Alert Hierarchy

1. Symptom-based (SLO violations) — Primary alerts

• Alert when SLO is at risk (burn rate too high)
• Page on-call for critical burn rate

2. Cause-based (system health) — Secondary alerts

• Database connection pool saturation
• High CPU/memory usage
• External dependency failures
• Send to Slack, don't page

Alert Routing

# Alertmanager config
routes:
  - match:
      alertname: SLOBurnRateCritical
    receiver: pagerduty-critical
    group_wait: 0s
    repeat_interval: 5m

  - match:
      alertname: SLOBurnRateHigh
    receiver: pagerduty-warning
    repeat_interval: 30m

  - match:
      alertname: SLOBurnRateModerate
    receiver: slack-alerts
    repeat_interval: 4h

---

Review and Iteration

Quarterly Review

Review Date: [First week of each quarter]

Review Checklist:

• [ ] Are SLOs still aligned with user expectations?
• [ ] Have there been repeated SLO violations?
• [ ] Is error budget consistently under/over-utilized?
• [ ] Do SLOs match actual system capabilities?
• [ ] Are there new critical user journeys to track?

Adjustment Process: 1. Review SLO violations from past quarter 2. Analyze error budget consumption trend 3. Gather user feedback on performance 4. Propose SLO adjustments (stricter or looser) 5. Get approval from product and engineering leads 6. Update SLO definitions and alerts 7. Communicate changes to stakeholders

Historical SLO Performance

---

Runbooks

Related Runbooks:

• High Error Rate Investigation
• Latency Debugging
• Database Connection Issues

---

Stakeholders

Service Owner: [Name, @slack] Product Manager: [Name, @slack] On-Call Team: [Team name] Escalation Contact: [Manager name, @slack]

---

Example: Filled-In SLO

Service: Payment Processing API

Owner: Payment Team Last Updated: 2025-01-15 Review Frequency: Quarterly

---

SLOs (Service Level Objectives)

Status: ✅ All SLOs met Error Budget: 80% remaining (safe to deploy)

---

Recent SLO Violations

---

Notes

• This template should be customized for each service
• SLOs should be based on user expectations, not arbitrary numbers
• Start with looser SLOs and tighten over time
• Review and adjust quarterly based on actual performance
• Error budget is a tool for decision-making, not a punishment