Implementing Observability
- 53 installs
- 426 repo stars
- Updated December 11, 2025
- ancoleman/ai-design-components
implementing-observability is a Claude Code skill that implements production monitoring, logging, and tracing using OpenTelemetry and the LGTM stack across Python, Rust, Go, and TypeScript.
About
This skill implements production observability using OpenTelemetry for metrics, logs, and traces. Developers use it when building production systems that need visibility into performance and errors, debugging distributed systems, or setting up monitoring and alerting. It covers the LGTM stack (Loki, Grafana, Tempo, Mimir), structured logging with trace correlation, and language SDKs for Python, Rust, Go, and TypeScript.
- Production observability using OpenTelemetry as the unified standard
- Covers the three pillars: metrics, logs, and traces with log-trace correlation
- Includes the LGTM stack (Loki, Grafana, Tempo, Mimir) and structured logging
Implementing Observability by the numbers
- 53 all-time installs (skills.sh)
- Ranked #710 of 1,435 DevOps & CI/CD skills by installs in the Skillselion catalog
- Data as of Aug 5, 2026 (Skillselion catalog sync)
implementing-observability capabilities & compatibility
- Capabilities
- metrics instrumentation · distributed tracing · structured logging · alerting
- Works with
- grafana · datadog
- Use cases
- devops · debugging
- Pricing
- Free
What implementing-observability says it does
Monitoring, logging, and tracing implementation using OpenTelemetry as the unified standard.
npx skills add https://github.com/ancoleman/ai-design-components --skill implementing-observabilityAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 53 |
|---|---|
| repo stars | ★ 426 |
| Last updated | December 11, 2025 |
| Repository | ancoleman/ai-design-components ↗ |
What it does
Instrumenting production systems with OpenTelemetry metrics, logs, and traces and deploying the LGTM stack.
Who is it for?
Production and distributed systems needing OpenTelemetry-based metrics, logs, and traces.
Skip if: Proof-of-concept apps or systems under 100 requests/day where console logging suffices.
When should I use this skill?
You are building production systems requiring visibility into performance, errors, and behavior.
What you get
Instrumented services exporting correlated telemetry to an LGTM or SaaS backend with alerting.
- OpenTelemetry instrumentation
- LGTM stack deployment
- Structured logging with trace correlation
By the numbers
- 3 pillars of observability (metrics, logs, traces)
- OpenTelemetry Context7 score 85.9 with 5,888 snippets
Files
Production Observability with OpenTelemetry
Purpose
Implement production-grade observability using OpenTelemetry as the 2025 industry standard. Covers the three pillars (metrics, logs, traces), LGTM stack deployment, and critical log-trace correlation patterns.
When to Use
Use when:
- Building production systems requiring visibility into performance and errors
- Debugging distributed systems with multiple services
- Setting up monitoring, logging, or tracing infrastructure
- Implementing structured logging with trace correlation
- Configuring alerting rules for production systems
Skip if:
- Building proof-of-concept without production deployment
- System has < 100 requests/day (console logging may suffice)
The OpenTelemetry Standard (2025)
OpenTelemetry is the CNCF graduated project unifying observability:
┌────────────────────────────────────────────────────────┐
│ OpenTelemetry: The Unified Standard │
├────────────────────────────────────────────────────────┤
│ │
│ ONE SDK for ALL signals: │
│ ├── Metrics (Prometheus-compatible) │
│ ├── Logs (structured, correlated) │
│ ├── Traces (distributed, standardized) │
│ └── Context (propagates across services) │
│ │
│ Language SDKs: │
│ ├── Python: opentelemetry-api, opentelemetry-sdk │
│ ├── Rust: opentelemetry, tracing-opentelemetry │
│ ├── Go: go.opentelemetry.io/otel │
│ └── TypeScript: @opentelemetry/api │
│ │
│ Export to ANY backend: │
│ ├── LGTM Stack (Loki, Grafana, Tempo, Mimir) │
│ ├── Prometheus + Jaeger │
│ ├── Datadog, New Relic, Honeycomb (SaaS) │
│ └── Custom backends via OTLP protocol │
│ │
└────────────────────────────────────────────────────────┘Context7 Reference: /websites/opentelemetry_io (Trust: High, Snippets: 5,888, Score: 85.9)
The Three Pillars of Observability
1. Metrics (What is happening?)
Track system health and performance over time.
Metric Types: Counters (always increase), Gauges (up/down), Histograms (distributions), Summaries (percentiles).
Brief Example (Python):
from opentelemetry import metrics
meter = metrics.get_meter(__name__)
http_requests = meter.create_counter("http.server.requests")
http_requests.add(1, {"method": "GET", "status": 200})2. Logs (What happened?)
Record discrete events with context.
CRITICAL: Always inject trace_id/span_id for log-trace correlation.
Brief Example (Python + structlog):
import structlog
from opentelemetry import trace
logger = structlog.get_logger()
span = trace.get_current_span()
ctx = span.get_span_context()
logger.info(
"processing_request",
trace_id=format(ctx.trace_id, '032x'),
span_id=format(ctx.span_id, '016x'),
user_id=user_id
)See: references/structured-logging.md for complete configuration.
3. Traces (Where did time go?)
Track request flow across distributed services.
Key Concepts: Trace (end-to-end journey), Span (individual operation), Parent-Child (nested operations).
Brief Example (Python + FastAPI):
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
app = FastAPI()
FastAPIInstrumentor.instrument_app(app) # Auto-traces all HTTP requestsSee: references/opentelemetry-setup.md for SDK installation by language.
The LGTM Stack (Self-Hosted Observability)
LGTM = Loki (Logs) + Grafana (Visualization) + Tempo (Traces) + Mimir (Metrics)
┌────────────────────────────────────────────────────────┐
│ LGTM Architecture │
├────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────────────────────────────────┐ │
│ │ Grafana Dashboard (Port 3000) │ │
│ │ Unified UI for Logs, Metrics, Traces │ │
│ └──────┬──────────────┬─────────────┬─────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Loki │ │ Tempo │ │ Mimir │ │
│ │ (Logs) │ │ (Traces) │ │(Metrics) │ │
│ │Port 3100 │ │Port 3200 │ │Port 9009 │ │
│ └────▲─────┘ └────▲─────┘ └────▲─────┘ │
│ │ │ │ │
│ └──────────────┴─────────────┘ │
│ │ │
│ ┌───────▼────────┐ │
│ │ Grafana Alloy │ │
│ │ (Collector) │ │
│ │ Port 4317/8 │ ← OTLP gRPC/HTTP │
│ └───────▲────────┘ │
│ │ │
│ OpenTelemetry Instrumented Apps │
│ │
└────────────────────────────────────────────────────────┘Quick Start: Run examples/lgtm-docker-compose/docker-compose.yml for a complete LGTM stack.
See: references/lgtm-stack.md for production deployment guide.
Critical Pattern: Log-Trace Correlation
The Problem: Logs and traces live in separate systems. You see an error log but can't find the related trace.
The Solution: Inject trace_id and span_id into every log record.
Python (structlog)
import structlog
from opentelemetry import trace
logger = structlog.get_logger()
span = trace.get_current_span()
ctx = span.get_span_context()
logger.info(
"request_processed",
trace_id=format(ctx.trace_id, '032x'), # 32-char hex
span_id=format(ctx.span_id, '016x'), # 16-char hex
user_id=user_id
)Rust (tracing)
use tracing::{info, instrument};
#[instrument(fields(user_id = %user_id))]
async fn process_request(user_id: u64) -> Result<Response> {
// trace_id/span_id automatically included
info!(user_id = user_id, "processing request");
Ok(result)
}See: references/trace-context.md for Go and TypeScript patterns.
Query in Grafana
{job="api-service"} |= "trace_id=4bf92f3577b34da6a3ce929d0e0e4736"Quick Setup Guide
1. Choose Your Stack
Decision Tree:
- Greenfield: OpenTelemetry SDK + LGTM Stack (self-hosted) or Grafana Cloud (managed)
- Existing Prometheus: Add Loki (logs) + Tempo (traces)
- Kubernetes: LGTM via Helm, Alloy DaemonSet
- Zero-ops: Managed SaaS (Grafana Cloud, Datadog, New Relic)
2. Install OpenTelemetry SDK
Bootstrap Script:
python scripts/setup_otel.py --language python --framework fastapiManual (Python):
pip install opentelemetry-api opentelemetry-sdk \
opentelemetry-instrumentation-fastapi \
opentelemetry-exporter-otlpSee: references/opentelemetry-setup.md for Rust, Go, TypeScript installation.
3. Deploy LGTM Stack
Docker Compose (development):
cd examples/lgtm-docker-compose
docker-compose up -d
# Grafana: http://localhost:3000 (admin/admin)
# OTLP: localhost:4317 (gRPC), localhost:4318 (HTTP)See: references/lgtm-stack.md for production Kubernetes deployment.
4. Configure Structured Logging
See: references/structured-logging.md for complete setup (Python, Rust, Go, TypeScript).
5. Set Up Alerting
See: references/alerting-rules.md for Prometheus and Loki alert patterns.
Auto-Instrumentation
OpenTelemetry auto-instruments popular frameworks:
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
app = FastAPI()
FastAPIInstrumentor.instrument_app(app) # Auto-trace all HTTP requestsSupported: FastAPI, Flask, Django, Express, Gin, Echo, Nest.js
See: references/opentelemetry-setup.md for framework-specific setup.
Common Patterns
Custom Spans
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("fetch_user_details") as span:
span.set_attribute("user_id", user_id)
user = await db.fetch_user(user_id)
span.set_attribute("user_found", user is not None)Error Tracking
from opentelemetry.trace import Status, StatusCode
with tracer.start_as_current_span("process_payment") as span:
try:
result = process_payment(amount, card_token)
span.set_status(Status(StatusCode.OK))
except PaymentError as e:
span.set_status(Status(StatusCode.ERROR, str(e)))
span.record_exception(e)
raiseSee: references/trace-context.md for background job tracing and context propagation.
Validation and Testing
# Test log-trace correlation
# 1. Make request to your app
# 2. Copy trace_id from logs
# 3. Query in Grafana: {job="myapp"} |= "trace_id=<TRACE_ID>"
# Validate metrics
python scripts/validate_metrics.pyIntegration with Other Skills
- Dashboards: Embed Grafana panels, query Prometheus metrics
- Feedback: Alert routing (Slack, PagerDuty), notification UI
- Data-Viz: Time-series charts, trace waterfall, latency heatmaps
See: examples/fastapi-otel/ for complete integration.
Progressive Disclosure
Setup Guides:
references/opentelemetry-setup.md- SDK installation (Python, Rust, Go, TypeScript)references/structured-logging.md- structlog, tracing, slog, pino configurationreferences/lgtm-stack.md- LGTM deployment (Docker, Kubernetes)references/trace-context.md- Log-trace correlation patternsreferences/alerting-rules.md- Prometheus and Loki alert templates
Examples:
examples/fastapi-otel/- FastAPI + OpenTelemetry + LGTMexamples/axum-tracing/- Rust Axum + tracing + LGTMexamples/lgtm-docker-compose/- Production-ready LGTM stack
Scripts:
scripts/setup_otel.py- Bootstrap OpenTelemetry SDKscripts/generate_dashboards.py- Generate Grafana dashboardsscripts/validate_metrics.py- Validate metric naming
Key Principles
1. OpenTelemetry is THE standard - Use OTel SDK, not vendor-specific SDKs 2. Auto-instrumentation first - Prefer auto over manual spans 3. Always correlate logs and traces - Inject trace_id/span_id into every log 4. Use structured logging - JSON format, consistent field names 5. LGTM stack for self-hosting - Production-ready open-source stack
Common Pitfalls
Don't:
- Use vendor-specific SDKs (use OpenTelemetry)
- Log without trace_id/span_id context
- Manually instrument what auto-instrumentation covers
- Mix logging libraries (pick one: structlog, tracing, slog, pino)
Do:
- Start with auto-instrumentation
- Add manual spans only for business-critical operations
- Use semantic conventions for span attributes
- Export to OTLP (gRPC preferred over HTTP)
- Test locally with LGTM docker-compose before production
Success Metrics
1. 100% of logs include trace_id when in request context 2. Mean time to resolution (MTTR) decreases by >50% 3. Developers use Grafana as first debugging tool 4. 80%+ of telemetry from auto-instrumentation 5. Alert noise < 5% false positives
Rust Axum + OpenTelemetry Tracing Example
Production-ready Rust application with OpenTelemetry instrumentation using tracing crate.
Features
- Axum web framework
- tracing crate (structured logging + spans)
- tracing-opentelemetry bridge
- OpenTelemetry export to OTLP
- Log-trace correlation
- Prometheus metrics
Files
axum-tracing/
├── src/
│ ├── main.rs # Server with tracing setup
│ ├── routes.rs # API routes
│ └── telemetry.rs # OpenTelemetry configuration
├── Cargo.toml
└── .env.exampleQuick Start
# Start LGTM stack (in separate directory)
cd ../lgtm-docker-compose
docker-compose up -d
# Run Axum app
cargo runAccess:
- API: http://localhost:3000
- Grafana: http://localhost:3000 (admin/admin)
Implementation
Dependencies (Cargo.toml)
[dependencies]
axum = "0.7"
tokio = { version = "1", features = ["full"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
tracing-opentelemetry = "0.22"
opentelemetry = { version = "0.21", features = ["trace"] }
opentelemetry-otlp = { version = "0.14", features = ["grpc-tonic"] }
opentelemetry_sdk = { version = "0.21", features = ["rt-tokio"] }Tracing Setup
// src/telemetry.rs
use opentelemetry::global;
use opentelemetry_otlp::WithExportConfig;
use opentelemetry_sdk::{runtime, trace as sdktrace};
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};
pub fn init_telemetry() {
// OpenTelemetry tracer
let tracer = opentelemetry_otlp::new_pipeline()
.tracing()
.with_exporter(
opentelemetry_otlp::new_exporter()
.tonic()
.with_endpoint("http://localhost:4317")
)
.with_trace_config(
sdktrace::config()
.with_resource(opentelemetry_sdk::Resource::new(vec![
opentelemetry::KeyValue::new("service.name", "axum-api"),
]))
)
.install_batch(runtime::Tokio)
.unwrap();
// Tracing subscriber with OpenTelemetry layer
tracing_subscriber::registry()
.with(tracing_subscriber::EnvFilter::new("info"))
.with(tracing_subscriber::fmt::layer().json())
.with(tracing_opentelemetry::layer().with_tracer(tracer))
.init();
}Instrumented Handler
// src/routes.rs
use axum::{extract::Path, Json};
use tracing::{info, instrument};
use serde::Serialize;
#[derive(Serialize)]
struct User {
id: u64,
name: String,
}
#[instrument(fields(user_id = %user_id))]
async fn get_user(Path(user_id): Path<u64>) -> Json<User> {
// Log with trace context automatically included
info!(user_id = user_id, "Fetching user from database");
// Simulate DB query
let user = User {
id: user_id,
name: "John Doe".to_string(),
};
info!(user_id = user_id, "User found");
Json(user)
}Main Server
// src/main.rs
mod routes;
mod telemetry;
use axum::{routing::get, Router};
#[tokio::main]
async fn main() {
// Initialize telemetry
telemetry::init_telemetry();
let app = Router::new()
.route("/users/:id", get(routes::get_user));
println!("Server running on http://localhost:3000");
axum::Server::bind(&"0.0.0.0:3000".parse().unwrap())
.serve(app.into_make_service())
.await
.unwrap();
// Shutdown telemetry
opentelemetry::global::shutdown_tracer_provider();
}Log Output
{
"timestamp": "2025-12-03T10:30:00.123Z",
"level": "INFO",
"message": "Fetching user from database",
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
"span_id": "00f067aa0ba902b7",
"user_id": 42
}Query in Grafana
{service_name="axum-api"} |= "trace_id=4bf92f3577b34da6a3ce929d0e0e4736"Benefits
- Automatic trace context propagation
- Zero-cost abstractions (compile-time only)
- Structured logging built-in
- OpenTelemetry standard compliance
"""
FastAPI + OpenTelemetry Complete Example
Demonstrates:
- Auto-instrumentation with FastAPI
- Manual span creation
- Log-trace correlation
- HTTP client instrumentation
- Database instrumentation
"""
from fastapi import FastAPI, HTTPException
from fastapi.responses import JSONResponse
import httpx
import structlog
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.trace import Status, StatusCode
import signal
import sys
# Configure structured logging with trace context
def otel_processor(logger, log_method, event_dict):
"""Inject OpenTelemetry trace context into logs."""
span = trace.get_current_span()
if span.is_recording():
ctx = span.get_span_context()
event_dict["trace_id"] = format(ctx.trace_id, '032x')
event_dict["span_id"] = format(ctx.span_id, '016x')
return event_dict
structlog.configure(
processors=[
structlog.stdlib.add_log_level,
structlog.processors.TimeStamper(fmt="iso"),
otel_processor, # Add trace context
structlog.processors.JSONRenderer()
],
logger_factory=structlog.stdlib.LoggerFactory(),
)
logger = structlog.get_logger()
# Initialize OpenTelemetry
resource = Resource.create({"service.name": "fastapi-example"})
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(
BatchSpanProcessor(
OTLPSpanExporter(
endpoint="localhost:4317",
insecure=True,
)
)
)
trace.set_tracer_provider(tracer_provider)
# Get tracer
tracer = trace.get_tracer(__name__)
# Create FastAPI app
app = FastAPI(
title="FastAPI OpenTelemetry Example",
description="Complete example with tracing, logging, and metrics"
)
# Auto-instrument FastAPI
FastAPIInstrumentor.instrument_app(app)
# Auto-instrument HTTP client
HTTPXClientInstrumentor().instrument()
# Graceful shutdown
def shutdown_handler(signum, frame):
logger.info("shutting_down", signal=signum)
tracer_provider.shutdown()
sys.exit(0)
signal.signal(signal.SIGTERM, shutdown_handler)
signal.signal(signal.SIGINT, shutdown_handler)
@app.on_event("startup")
async def startup_event():
logger.info("application_started", service="fastapi-example")
@app.get("/")
async def root():
"""Root endpoint - automatically traced by FastAPI instrumentation."""
logger.info("root_endpoint_called")
return {"message": "Hello from FastAPI with OpenTelemetry", "status": "ok"}
@app.get("/api/users/{user_id}")
async def get_user(user_id: int):
"""
Fetch user data with manual span creation.
Demonstrates:
- Manual span creation
- Span attributes
- Log-trace correlation
"""
# Automatic HTTP span created by FastAPI instrumentation
# Create manual span for business logic
with tracer.start_as_current_span("fetch_user_data") as span:
span.set_attribute("user_id", user_id)
span.set_attribute("operation", "fetch_user")
logger.info(
"fetching_user",
user_id=user_id,
# trace_id and span_id automatically added by processor
)
# Simulate database query
if user_id == 0:
logger.error("invalid_user_id", user_id=user_id)
span.set_status(Status(StatusCode.ERROR, "Invalid user ID"))
raise HTTPException(status_code=400, detail="Invalid user ID")
# Simulate user data
user = {
"user_id": user_id,
"name": f"User {user_id}",
"email": f"user{user_id}@example.com"
}
span.set_attribute("user_found", True)
logger.info(
"user_fetched",
user_id=user_id,
user_name=user["name"]
)
return user
@app.get("/api/external")
async def call_external_service():
"""
Call external service with trace propagation.
Demonstrates:
- HTTP client auto-instrumentation
- Trace context propagation
- Error handling with spans
"""
logger.info("calling_external_service", url="https://jsonplaceholder.typicode.com/posts/1")
with tracer.start_as_current_span("call_external_api") as span:
span.set_attribute("http.url", "https://jsonplaceholder.typicode.com/posts/1")
try:
async with httpx.AsyncClient() as client:
# httpx automatically propagates trace context via headers
response = await client.get("https://jsonplaceholder.typicode.com/posts/1")
response.raise_for_status()
data = response.json()
span.set_attribute("http.status_code", response.status_code)
span.set_status(Status(StatusCode.OK))
logger.info(
"external_call_success",
status_code=response.status_code,
response_size=len(response.content)
)
return {"external_data": data, "status": "success"}
except httpx.HTTPError as e:
logger.error(
"external_call_failed",
error=str(e),
exc_info=True
)
span.record_exception(e)
span.set_status(Status(StatusCode.ERROR, str(e)))
raise HTTPException(status_code=502, detail="External service unavailable")
@app.get("/api/error")
async def simulate_error():
"""
Simulate an error to test error tracking.
Demonstrates:
- Exception recording in spans
- Error logging with trace context
"""
logger.warning("simulating_error")
with tracer.start_as_current_span("error_operation") as span:
try:
# Simulate error
raise ValueError("Simulated error for testing")
except ValueError as e:
logger.error(
"operation_failed",
error=str(e),
error_type=type(e).__name__,
exc_info=True
)
span.record_exception(e)
span.set_status(Status(StatusCode.ERROR, str(e)))
# Return error response with trace_id for debugging
trace_id = format(span.get_span_context().trace_id, '032x')
return JSONResponse(
status_code=500,
content={
"error": "Internal server error",
"trace_id": trace_id, # Client can reference this for support
"message": "An error occurred. Please reference this trace_id when contacting support."
}
)
@app.get("/health")
async def health():
"""Health check endpoint."""
return {"status": "healthy", "service": "fastapi-example"}
if __name__ == "__main__":
import uvicorn
logger.info("starting_server", host="0.0.0.0", port=8000)
uvicorn.run(
app,
host="0.0.0.0",
port=8000,
log_config=None # Use structlog instead of uvicorn's default logger
)
FastAPI + OpenTelemetry Complete Example
Complete working example of FastAPI with OpenTelemetry tracing and structured logging.
Features
- ✅ Auto-instrumentation with FastAPI
- ✅ Manual span creation for business logic
- ✅ Log-trace correlation (trace_id/span_id in logs)
- ✅ HTTP client auto-instrumentation
- ✅ Error tracking and exception recording
- ✅ Structured JSON logging with structlog
- ✅ OTLP export to LGTM stack
Quick Start
1. Install Dependencies
pip install -r requirements.txt2. Start LGTM Stack
# From the lgtm-docker-compose directory
cd ../lgtm-docker-compose
docker-compose up -d
# Wait for services to start (30 seconds)3. Run Application
python main.py4. Generate Traces
# Root endpoint
curl http://localhost:8000/
# User endpoint with manual span
curl http://localhost:8000/api/users/123
# External API call with trace propagation
curl http://localhost:8000/api/external
# Error handling example
curl http://localhost:8000/api/error5. View in Grafana
1. Navigate to http://localhost:3000 (login: admin/admin) 2. Go to Explore → Tempo 3. Search for service.name="fastapi-example" 4. Click on a trace to see spans and timing 5. Click "Logs for this span" to see correlated logs
Project Structure
fastapi-otel/
├── main.py # FastAPI app with OTel instrumentation
├── requirements.txt # Python dependencies
├── README.md # This file
└── docker-compose.yml # Optional: Run with DockerCode Walkthrough
OpenTelemetry Setup
# Initialize tracer provider
tracer_provider = TracerProvider(
resource=Resource.create({"service.name": "fastapi-example"})
)
# Add OTLP exporter
tracer_provider.add_span_processor(
BatchSpanProcessor(
OTLPSpanExporter(endpoint="localhost:4317")
)
)
# Set as global provider
trace.set_tracer_provider(tracer_provider)
# Auto-instrument FastAPI
FastAPIInstrumentor.instrument_app(app)Manual Span Creation
@app.get("/api/users/{user_id}")
async def get_user(user_id: int):
# Create manual span
with tracer.start_as_current_span("fetch_user_data") as span:
span.set_attribute("user_id", user_id)
# Business logic here
user = fetch_from_db(user_id)
span.set_attribute("user_found", True)
return userLog-Trace Correlation
# Custom processor to inject trace context
def otel_processor(logger, log_method, event_dict):
span = trace.get_current_span()
if span.is_recording():
ctx = span.get_span_context()
event_dict["trace_id"] = format(ctx.trace_id, '032x')
event_dict["span_id"] = format(ctx.span_id, '016x')
return event_dict
# Configure structlog
structlog.configure(
processors=[
structlog.stdlib.add_log_level,
otel_processor, # Add trace context
structlog.processors.JSONRenderer()
]
)Error Handling
@app.get("/api/error")
async def simulate_error():
with tracer.start_as_current_span("error_operation") as span:
try:
raise ValueError("Simulated error")
except ValueError as e:
# Record exception in span
span.record_exception(e)
span.set_status(Status(StatusCode.ERROR, str(e)))
# Log with trace context
logger.error("operation_failed", error=str(e))
# Return trace_id to client for support
trace_id = format(span.get_span_context().trace_id, '032x')
return {"error": "Internal error", "trace_id": trace_id}Observability Queries
Grafana Tempo (Traces)
# Find all traces for this service
{service.name="fastapi-example"}
# Find traces with errors
{service.name="fastapi-example" && status=error}
# Find slow requests (>1s)
{service.name="fastapi-example" && duration>1s}Grafana Loki (Logs)
# All logs for this service
{service="fastapi-example"}
# Error logs with trace_id
{service="fastapi-example"} | json | level="error"
# Find logs for specific trace
{service="fastapi-example"} | json | trace_id="4bf92f3577b34da6a3ce929d0e0e4736"Testing
Manual Testing
# Test normal request
curl http://localhost:8000/api/users/123
# Test error handling
curl http://localhost:8000/api/users/0
# Test external call
curl http://localhost:8000/api/externalLoad Testing
# Install hey
go install github.com/rakyll/hey@latest
# Generate load
hey -n 1000 -c 10 http://localhost:8000/api/users/123
# View traces in Grafana to see performance distributionConfiguration
Environment Variables
# OTLP endpoint
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
# Service name
export OTEL_SERVICE_NAME=fastapi-example
# Sampling (1.0 = 100%, 0.1 = 10%)
export OTEL_TRACES_SAMPLER_ARG=1.0
# Log level
export LOG_LEVEL=INFOCustom OTLP Endpoint
Edit main.py:
OTLPSpanExporter(
endpoint="your-collector:4317",
insecure=False, # Use TLS
headers=(("x-api-key", "your-key"),) # Authentication
)Troubleshooting
Traces not appearing in Tempo:
1. Check OTLP endpoint is reachable:
telnet localhost 43172. Verify Grafana Alloy is running:
docker ps | grep alloy3. Enable debug logging:
export OTEL_LOG_LEVEL=debug
python main.pyLogs missing trace_id:
- Verify logs are emitted inside a span context
- Check structlog processor is configured correctly
- Ensure OpenTelemetry is initialized before logging
High memory usage:
- Reduce batch size in exporter
- Enable sampling for high-traffic endpoints
- Use async exporter for better performance
Next Steps
1. Add database instrumentation (see references/opentelemetry-setup.md) 2. Configure alerting rules (see references/alerting-rules.md) 3. Create custom Grafana dashboards 4. Add metrics collection 5. Deploy to production with proper sampling and batching
Resources
- OpenTelemetry Python Docs: https://opentelemetry-python.readthedocs.io
- FastAPI Docs: https://fastapi.tiangolo.com
- Grafana Tempo Docs: https://grafana.com/docs/tempo
# FastAPI + OpenTelemetry Example Dependencies
# Web framework
fastapi==0.109.0
uvicorn[standard]==0.27.0
httpx==0.26.0
# OpenTelemetry core
opentelemetry-api==1.22.0
opentelemetry-sdk==1.22.0
opentelemetry-exporter-otlp==1.22.0
# OpenTelemetry auto-instrumentation
opentelemetry-instrumentation-fastapi==0.43b0
opentelemetry-instrumentation-httpx==0.43b0
# Structured logging
structlog==24.1.0
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"liveNow": false,
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"tooltip": false,
"viz": false,
"legend": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "reqps"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
},
"id": 1,
"options": {
"legend": {
"calcs": [
"mean",
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "sum(rate(http_server_requests_total{job=\"$job\"}[5m])) by (method, status)",
"legendFormat": "{{method}} - {{status}}",
"range": true,
"refId": "A"
}
],
"title": "Request Rate (by Method & Status)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 1000
},
{
"color": "red",
"value": 5000
}
]
},
"unit": "reqps"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 6,
"x": 12,
"y": 0
},
"id": 2,
"options": {
"orientation": "auto",
"reduceOptions": {
"values": false,
"calcs": [
"lastNotNull"
],
"fields": ""
},
"showThresholdLabels": false,
"showThresholdMarkers": true
},
"pluginVersion": "10.0.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "sum(rate(http_server_requests_total{job=\"$job\"}[5m]))",
"legendFormat": "Total RPS",
"range": true,
"refId": "A"
}
],
"title": "Total Request Rate",
"type": "gauge"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 1
},
{
"color": "red",
"value": 5
}
]
},
"unit": "percentunit"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 6,
"x": 18,
"y": 0
},
"id": 3,
"options": {
"orientation": "auto",
"reduceOptions": {
"values": false,
"calcs": [
"lastNotNull"
],
"fields": ""
},
"showThresholdLabels": false,
"showThresholdMarkers": true
},
"pluginVersion": "10.0.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "sum(rate(http_server_requests_total{job=\"$job\",status=~\"5..\"}[5m])) / sum(rate(http_server_requests_total{job=\"$job\"}[5m]))",
"legendFormat": "Error Rate",
"range": true,
"refId": "A"
}
],
"title": "Error Rate (5xx)",
"type": "gauge"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"tooltip": false,
"viz": false,
"legend": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "ms"
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "p50"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "green",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "p95"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "yellow",
"mode": "fixed"
}
}
]
},
{
"matcher": {
"id": "byName",
"options": "p99"
},
"properties": [
{
"id": "color",
"value": {
"fixedColor": "red",
"mode": "fixed"
}
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 4
},
"id": 4,
"options": {
"legend": {
"calcs": [
"mean",
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "histogram_quantile(0.50, sum(rate(http_server_duration_milliseconds_bucket{job=\"$job\"}[5m])) by (le))",
"legendFormat": "p50",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "histogram_quantile(0.95, sum(rate(http_server_duration_milliseconds_bucket{job=\"$job\"}[5m])) by (le))",
"legendFormat": "p95",
"range": true,
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "histogram_quantile(0.99, sum(rate(http_server_duration_milliseconds_bucket{job=\"$job\"}[5m])) by (le))",
"legendFormat": "p99",
"range": true,
"refId": "C"
}
],
"title": "Response Latency (p50, p95, p99)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"tooltip": false,
"viz": false,
"legend": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "percentunit"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 8
},
"id": 5,
"options": {
"legend": {
"calcs": [
"mean",
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "sum(rate(http_server_requests_total{job=\"$job\",status=~\"2..\"}[5m])) / sum(rate(http_server_requests_total{job=\"$job\"}[5m]))",
"legendFormat": "2xx (Success)",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "sum(rate(http_server_requests_total{job=\"$job\",status=~\"4..\"}[5m])) / sum(rate(http_server_requests_total{job=\"$job\"}[5m]))",
"legendFormat": "4xx (Client Error)",
"range": true,
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "sum(rate(http_server_requests_total{job=\"$job\",status=~\"5..\"}[5m])) / sum(rate(http_server_requests_total{job=\"$job\"}[5m]))",
"legendFormat": "5xx (Server Error)",
"range": true,
"refId": "C"
}
],
"title": "Error Rate (by Status Class)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"tooltip": false,
"viz": false,
"legend": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "percent"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 12
},
"id": 6,
"options": {
"legend": {
"calcs": [
"mean",
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "100 * (1 - avg(rate(process_cpu_seconds_total{job=\"$job\"}[5m])))",
"legendFormat": "CPU",
"range": true,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "100 * (1 - (process_resident_memory_bytes{job=\"$job\"} / process_virtual_memory_max_bytes{job=\"$job\"}))",
"legendFormat": "Memory",
"range": true,
"refId": "B"
}
],
"title": "Saturation (Resource Utilization)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"tooltip": false,
"viz": false,
"legend": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "never",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "reqps"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 16
},
"id": 7,
"options": {
"legend": {
"calcs": [
"mean",
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "sum(rate(http_server_requests_total{job=\"$job\"}[5m])) by (route)",
"legendFormat": "{{route}}",
"range": true,
"refId": "A"
}
],
"title": "Request Rate (by Endpoint)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": false
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": [
{
"matcher": {
"id": "byName",
"options": "RPS"
},
"properties": [
{
"id": "unit",
"value": "reqps"
},
{
"id": "decimals",
"value": 2
}
]
},
{
"matcher": {
"id": "byName",
"options": "p50 Latency"
},
"properties": [
{
"id": "unit",
"value": "ms"
},
{
"id": "decimals",
"value": 2
}
]
},
{
"matcher": {
"id": "byName",
"options": "p95 Latency"
},
"properties": [
{
"id": "unit",
"value": "ms"
},
{
"id": "decimals",
"value": 2
}
]
},
{
"matcher": {
"id": "byName",
"options": "Error Rate"
},
"properties": [
{
"id": "unit",
"value": "percentunit"
},
{
"id": "decimals",
"value": 2
},
{
"id": "custom.cellOptions",
"value": {
"type": "color-background"
}
},
{
"id": "thresholds",
"value": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 0.01
},
{
"color": "red",
"value": 0.05
}
]
}
}
]
}
]
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 20
},
"id": 8,
"options": {
"cellHeight": "sm",
"footer": {
"countRows": false,
"fields": "",
"reducer": [
"sum"
],
"show": false
},
"showHeader": true,
"sortBy": [
{
"desc": true,
"displayName": "RPS"
}
]
},
"pluginVersion": "10.0.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"exemplar": false,
"expr": "sum(rate(http_server_requests_total{job=\"$job\"}[5m])) by (route)",
"format": "table",
"instant": true,
"legendFormat": "__auto",
"range": false,
"refId": "A"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"exemplar": false,
"expr": "histogram_quantile(0.50, sum(rate(http_server_duration_milliseconds_bucket{job=\"$job\"}[5m])) by (le, route))",
"format": "table",
"hide": false,
"instant": true,
"legendFormat": "__auto",
"range": false,
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"exemplar": false,
"expr": "histogram_quantile(0.95, sum(rate(http_server_duration_milliseconds_bucket{job=\"$job\"}[5m])) by (le, route))",
"format": "table",
"hide": false,
"instant": true,
"legendFormat": "__auto",
"range": false,
"refId": "C"
},
{
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"editorMode": "code",
"exemplar": false,
"expr": "sum(rate(http_server_requests_total{job=\"$job\",status=~\"5..\"}[5m])) by (route) / sum(rate(http_server_requests_total{job=\"$job\"}[5m])) by (route)",
"format": "table",
"hide": false,
"instant": true,
"legendFormat": "__auto",
"range": false,
"refId": "D"
}
],
"title": "Endpoint Summary",
"transformations": [
{
"id": "merge",
"options": {}
},
{
"id": "organize",
"options": {
"excludeByName": {
"Time": true,
"le": true
},
"indexByName": {
"Time": 0,
"Value #A": 2,
"Value #B": 3,
"Value #C": 4,
"Value #D": 5,
"route": 1
},
"renameByName": {
"Value #A": "RPS",
"Value #B": "p50 Latency",
"Value #C": "p95 Latency",
"Value #D": "Error Rate",
"route": "Endpoint"
}
}
}
],
"type": "table"
}
],
"refresh": "30s",
"schemaVersion": 38,
"style": "dark",
"tags": [
"api",
"opentelemetry",
"monitoring"
],
"templating": {
"list": [
{
"current": {
"selected": false,
"text": "Prometheus",
"value": "prometheus"
},
"hide": 0,
"includeAll": false,
"label": "Datasource",
"multi": false,
"name": "datasource",
"options": [],
"query": "prometheus",
"queryValue": "",
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"type": "datasource"
},
{
"current": {
"selected": false,
"text": "api-service",
"value": "api-service"
},
"datasource": {
"type": "prometheus",
"uid": "${datasource}"
},
"definition": "label_values(http_server_requests_total, job)",
"hide": 0,
"includeAll": false,
"label": "Job",
"multi": false,
"name": "job",
"options": [],
"query": {
"query": "label_values(http_server_requests_total, job)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"type": "query"
}
]
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "API Overview (OpenTelemetry)",
"uid": "api-overview-otel",
"version": 1,
"weekStart": ""
}
# LGTM Stack Docker Compose
# Production-ready observability platform with unified telemetry collection
#
# Components:
# - Grafana Alloy: OpenTelemetry collector (receives OTLP, forwards to backends)
# - Loki: Log aggregation and querying
# - Tempo: Distributed tracing backend
# - Mimir: Prometheus-compatible metrics with long-term storage
# - Grafana: Unified visualization and dashboard UI
#
# Quick Start:
# docker-compose up -d
# Access Grafana: http://localhost:3000 (admin/admin)
# Send telemetry to: localhost:4317 (OTLP gRPC) or localhost:4318 (OTLP HTTP)
version: '3.8'
# Shared network for all LGTM services
networks:
lgtm:
driver: bridge
# Persistent volumes for data storage
volumes:
loki-data:
driver: local
tempo-data:
driver: local
mimir-data:
driver: local
grafana-data:
driver: local
services:
# ============================================================================
# Grafana Alloy - OpenTelemetry Collector
# ============================================================================
# Receives telemetry via OTLP protocol and forwards to Loki, Tempo, and Mimir
# Acts as the single entry point for all application telemetry
alloy:
image: grafana/alloy:latest
container_name: lgtm-alloy
restart: unless-stopped
ports:
- "4317:4317" # OTLP gRPC (recommended for production)
- "4318:4318" # OTLP HTTP (easier for debugging)
- "12345:12345" # Alloy UI for debugging config
volumes:
- ./alloy/config.alloy:/etc/alloy/config.alloy:ro
command:
- run
- --server.http.listen-addr=0.0.0.0:12345
- --storage.path=/var/lib/alloy/data
- /etc/alloy/config.alloy
networks:
- lgtm
depends_on:
loki:
condition: service_healthy
tempo:
condition: service_healthy
mimir:
condition: service_healthy
healthcheck:
test: ["CMD", "wget", "--spider", "-q", "http://localhost:12345/ready"]
interval: 10s
timeout: 5s
retries: 5
start_period: 10s
# ============================================================================
# Loki - Log Aggregation System
# ============================================================================
# Stores and indexes logs with label-based querying (LogQL)
# Optimized for Kubernetes-style structured logs
loki:
image: grafana/loki:3.0.0
container_name: lgtm-loki
restart: unless-stopped
ports:
- "3100:3100" # HTTP API for queries and ingestion
volumes:
- ./loki/loki-config.yaml:/etc/loki/local-config.yaml:ro
- loki-data:/loki # Persistent storage for log chunks and index
command: -config.file=/etc/loki/local-config.yaml
networks:
- lgtm
healthcheck:
test: ["CMD", "wget", "--spider", "-q", "http://localhost:3100/ready"]
interval: 10s
timeout: 5s
retries: 5
start_period: 15s
# Resource limits to prevent memory issues with high log volume
deploy:
resources:
limits:
memory: 1G
reservations:
memory: 512M
# ============================================================================
# Tempo - Distributed Tracing Backend
# ============================================================================
# Stores and queries traces with TraceQL
# Supports Jaeger, Zipkin, and OpenTelemetry trace formats
tempo:
image: grafana/tempo:2.4.0
container_name: lgtm-tempo
restart: unless-stopped
ports:
- "3200:3200" # HTTP API for queries
- "4317" # OTLP gRPC (internal, routed through Alloy)
- "4318" # OTLP HTTP (internal, routed through Alloy)
- "9411:9411" # Zipkin compatible endpoint
- "14268:14268" # Jaeger ingest endpoint
volumes:
- ./tempo/tempo-config.yaml:/etc/tempo.yaml:ro
- tempo-data:/var/tempo # Persistent storage for trace data
command: -config.file=/etc/tempo.yaml
networks:
- lgtm
healthcheck:
test: ["CMD", "wget", "--spider", "-q", "http://localhost:3200/ready"]
interval: 10s
timeout: 5s
retries: 5
start_period: 15s
# Tempo can be memory-intensive with high trace volume
deploy:
resources:
limits:
memory: 2G
reservations:
memory: 1G
# ============================================================================
# Mimir - Prometheus-Compatible Metrics Storage
# ============================================================================
# Horizontally scalable, multi-tenant metrics backend
# Drop-in replacement for Prometheus with long-term storage
mimir:
image: grafana/mimir:2.11.0
container_name: lgtm-mimir
restart: unless-stopped
ports:
- "9009:9009" # HTTP API for queries and ingestion
volumes:
- ./mimir/mimir-config.yaml:/etc/mimir.yaml:ro
- mimir-data:/data # Persistent storage for metrics blocks
command:
- -config.file=/etc/mimir.yaml
- -target=all
networks:
- lgtm
healthcheck:
test: ["CMD", "wget", "--spider", "-q", "http://localhost:9009/ready"]
interval: 10s
timeout: 5s
retries: 5
start_period: 20s
# Mimir requires significant memory for compaction and queries
deploy:
resources:
limits:
memory: 2G
reservations:
memory: 1G
# ============================================================================
# Grafana - Unified Observability UI
# ============================================================================
# Pre-configured with Loki, Tempo, and Mimir datasources
# Provides dashboards, explore views, and alerting
grafana:
image: grafana/grafana:10.4.0
container_name: lgtm-grafana
restart: unless-stopped
ports:
- "3000:3000" # Web UI
environment:
# Authentication settings
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=admin
- GF_AUTH_ANONYMOUS_ENABLED=false
# Enable explore view for ad-hoc queries
- GF_EXPLORE_ENABLED=true
# Auto-provision datasources and dashboards
- GF_PATHS_PROVISIONING=/etc/grafana/provisioning
# Feature toggles
- GF_FEATURE_TOGGLES_ENABLE=traceqlEditor,correlations,traceToMetrics,traceToLogs
# Performance settings
- GF_LOG_LEVEL=info
- GF_ANALYTICS_REPORTING_ENABLED=false
volumes:
# Auto-provision datasources (Loki, Tempo, Mimir)
- ./grafana/datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml:ro
# Auto-provision dashboards (if any)
- ./grafana/dashboards:/etc/grafana/provisioning/dashboards:ro
# Persistent storage for user settings, dashboards, and plugins
- grafana-data:/var/lib/grafana
networks:
- lgtm
depends_on:
loki:
condition: service_healthy
tempo:
condition: service_healthy
mimir:
condition: service_healthy
healthcheck:
test: ["CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health"]
interval: 10s
timeout: 5s
retries: 5
start_period: 30s
# ============================================================================
# Prometheus (Optional) - Alternative to Mimir for simpler setups
# ============================================================================
# Uncomment if you prefer traditional Prometheus over Mimir
# Note: Prometheus has limited long-term storage compared to Mimir
#
# prometheus:
# image: prom/prometheus:v2.50.1
# container_name: lgtm-prometheus
# restart: unless-stopped
#
# ports:
# - "9090:9090"
#
# volumes:
# - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
# - prometheus-data:/prometheus
#
# command:
# - '--config.file=/etc/prometheus/prometheus.yml'
# - '--storage.tsdb.path=/prometheus'
# - '--storage.tsdb.retention.time=15d'
# - '--web.console.libraries=/usr/share/prometheus/console_libraries'
# - '--web.console.templates=/usr/share/prometheus/consoles'
#
# networks:
# - lgtm
#
# healthcheck:
# test: ["CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy"]
# interval: 10s
# timeout: 5s
# retries: 5
# ============================================================================
# Usage Instructions
# ============================================================================
#
# 1. Start the stack:
# docker-compose up -d
#
# 2. Check service health:
# docker-compose ps
# docker-compose logs -f # Follow logs from all services
#
# 3. Access Grafana:
# URL: http://localhost:3000
# Username: admin
# Password: admin
#
# 4. Configure your application to send telemetry:
# OTLP gRPC endpoint: localhost:4317 (production recommended)
# OTLP HTTP endpoint: localhost:4318 (easier for testing)
#
# Example (Python):
# from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
# exporter = OTLPSpanExporter(endpoint="localhost:4317", insecure=True)
#
# Example (TypeScript):
# import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
# const exporter = new OTLPTraceExporter({ url: 'http://localhost:4317' });
#
# 5. Verify telemetry in Grafana:
# - Navigate to Explore
# - Select datasource: Loki (logs), Tempo (traces), or Mimir (metrics)
# - Run queries using LogQL, TraceQL, or PromQL
#
# 6. Stop the stack:
# docker-compose down
#
# To remove volumes (WARNING: deletes all data):
# docker-compose down -v
#
# ============================================================================
# Production Considerations
# ============================================================================
#
# 1. Resource Limits:
# - Adjust memory limits based on telemetry volume
# - Monitor with: docker stats
#
# 2. Data Retention:
# - Configure retention in respective config files
# - Default: Loki (31d), Tempo (7d), Mimir (15d)
#
# 3. Security:
# - Change default Grafana password: GF_SECURITY_ADMIN_PASSWORD
# - Enable TLS for OTLP endpoints in production
# - Use authentication for datasource endpoints
#
# 4. Scaling:
# - For high volume, run multiple Alloy collectors
# - Consider Kubernetes deployment for auto-scaling
# - Use object storage (S3, GCS) for long-term data
#
# 5. Networking:
# - In production, expose only necessary ports (3000, 4317, 4318)
# - Use reverse proxy (nginx, Traefik) for TLS termination
#
# 6. Monitoring the Monitors:
# - Check health endpoints: /ready, /health
# - Set up alerts for stack component failures
# - Monitor disk usage for volumes
#
# ============================================================================
LGTM Stack Docker Compose
Production-ready observability stack: Loki (Logs), Grafana (Visualization), Tempo (Traces), Mimir (Metrics).
What is LGTM?
Open-source observability platform with unified UI for logs, metrics, and traces.
Components:
- Loki - Log aggregation (like Prometheus for logs)
- Grafana - Dashboards and visualization
- Tempo - Distributed tracing
- Mimir - Prometheus-compatible metrics (long-term storage)
- Grafana Alloy - OpenTelemetry collector
Files
lgtm-docker-compose/
├── docker-compose.yml # All services
├── grafana/
│ ├── datasources.yml # Pre-configured datasources
│ └── dashboards/
├── loki/
│ └── loki-config.yaml
├── tempo/
│ └── tempo-config.yaml
├── mimir/
│ └── mimir-config.yaml
└── alloy/
└── config.alloy # OpenTelemetry collector configQuick Start
# Start all services
docker-compose up -d
# Check status
docker-compose ps
# View logs
docker-compose logs -f grafana
# Stop
docker-compose downAccess Points
- Grafana UI: http://localhost:3000 (admin/admin)
- Loki API: http://localhost:3100
- Tempo API: http://localhost:3200
- Mimir API: http://localhost:9009
- OTLP gRPC: localhost:4317
- OTLP HTTP: localhost:4318
docker-compose.yml
version: '3.8'
services:
# OpenTelemetry Collector
alloy:
image: grafana/alloy:latest
ports:
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
volumes:
- ./alloy/config.alloy:/etc/alloy/config.alloy
command: run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy
# Loki (Logs)
loki:
image: grafana/loki:latest
ports:
- "3100:3100"
volumes:
- ./loki/loki-config.yaml:/etc/loki/local-config.yaml
- loki-data:/loki
command: -config.file=/etc/loki/local-config.yaml
# Tempo (Traces)
tempo:
image: grafana/tempo:latest
ports:
- "3200:3200" # Tempo API
- "4317" # OTLP gRPC
volumes:
- ./tempo/tempo-config.yaml:/etc/tempo.yaml
- tempo-data:/var/tempo
command: -config.file=/etc/tempo.yaml
# Mimir (Metrics)
mimir:
image: grafana/mimir:latest
ports:
- "9009:9009"
volumes:
- ./mimir/mimir-config.yaml:/etc/mimir.yaml
- mimir-data:/data
command: -config.file=/etc/mimir.yaml
# Grafana (UI)
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
- GF_SECURITY_ADMIN_PASSWORD=admin
volumes:
- ./grafana/datasources.yml:/etc/grafana/provisioning/datasources/datasources.yml
- grafana-data:/var/lib/grafana
volumes:
loki-data:
tempo-data:
mimir-data:
grafana-data:Pre-configured Datasources
# grafana/datasources.yml
apiVersion: 1
datasources:
- name: Loki
type: loki
access: proxy
url: http://loki:3100
isDefault: false
editable: true
- name: Tempo
type: tempo
access: proxy
url: http://tempo:3200
isDefault: false
editable: true
- name: Mimir
type: prometheus
access: proxy
url: http://mimir:9009/prometheus
isDefault: true
editable: trueApplication Integration
Python (FastAPI)
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
# Configure exporter
tracer_provider = TracerProvider()
otlp_exporter = OTLPSpanExporter(endpoint="localhost:4317", insecure=True)
tracer_provider.add_span_processor(BatchSpanProcessor(otlp_exporter))
# Set global tracer
from opentelemetry import trace
trace.set_tracer_provider(tracer_provider)
# Auto-instrument FastAPI
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
FastAPIInstrumentor.instrument_app(app)TypeScript
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
const sdk = new NodeSDK({
traceExporter: new OTLPTraceExporter({
url: 'http://localhost:4317',
}),
serviceName: 'my-app',
});
sdk.start();Querying in Grafana
LogQL (Loki)
# All logs from service
{service_name="my-app"}
# Error logs only
{service_name="my-app"} |= "error"
# By trace ID
{service_name="my-app"} |= "trace_id=abc123"TraceQL (Tempo)
# Find slow requests
{ duration > 1s }
# By service and endpoint
{ service.name = "api" && http.route = "/users" }PromQL (Mimir)
# Request rate
rate(http_requests_total[5m])
# Error rate
rate(http_requests_total{status=~"5.."}[5m])Production Deployment
For Kubernetes deployment, see the observability skill's Kubernetes manifests.
Resources
- Grafana LGTM: https://grafana.com/oss/
- OpenTelemetry: https://opentelemetry.io/docs/
skill: "implementing-observability"
version: "1.0"
domain: "backend"
base_outputs:
# OpenTelemetry configuration
- path: "observability/otel-collector.yaml"
must_contain: ["receivers:", "processors:", "exporters:", "service:"]
# Prometheus configuration
- path: "observability/prometheus.yml"
must_contain: ["global:", "scrape_configs:", "job_name:"]
# Alerting rules
- path: "observability/alerts/"
must_contain: ["groups:", "alert:", "expr:"]
# Structured logging configuration
- path: "observability/logging/"
must_contain: [] # Language-specific files
conditional_outputs:
maturity:
starter:
- path: "observability/docker-compose.yml"
must_contain: ["prometheus:", "grafana:", "services:"]
- path: "observability/grafana/dashboards/overview.json"
must_contain: ["dashboard", "panels"]
- path: "observability/prometheus.yml"
must_contain: ["scrape_configs:"]
intermediate:
- path: "observability/lgtm-stack/"
must_contain: ["loki", "grafana", "tempo", "mimir"]
- path: "observability/alloy/config.alloy"
must_contain: ["otelcol.receiver", "otelcol.exporter"]
- path: "observability/grafana/dashboards/"
must_contain: [] # Multiple dashboard files
- path: "observability/alerts/slos.yml"
must_contain: ["record:", "expr:", "labels:"]
- path: "observability/loki/config.yaml"
must_contain: ["auth_enabled:", "server:", "ingester:"]
- path: "observability/tempo/config.yaml"
must_contain: ["server:", "distributor:", "ingester:"]
advanced:
- path: "observability/kubernetes/alloy-daemonset.yaml"
must_contain: ["DaemonSet", "containers:", "opentelemetry"]
- path: "observability/kubernetes/prometheus-operator.yaml"
must_contain: ["Prometheus", "ServiceMonitor"]
- path: "observability/kubernetes/loki-distributed.yaml"
must_contain: ["StatefulSet", "loki"]
- path: "observability/alerts/advanced-slos.yml"
must_contain: ["record:", "expr:", "multiburn"]
- path: "observability/grafana/dashboards/advanced/"
must_contain: [] # Advanced dashboards directory
- path: "observability/tempo/tempo-distributed.yaml"
must_contain: ["StatefulSet", "distributor", "ingester", "querier"]
observability:
prometheus_grafana:
- path: "observability/prometheus.yml"
must_contain: ["scrape_configs:", "job_name:"]
- path: "observability/grafana/datasources.yaml"
must_contain: ["apiVersion:", "datasources:", "prometheus"]
- path: "observability/grafana/dashboards/"
must_contain: []
- path: "observability/alerts/prometheus-rules.yml"
must_contain: ["groups:", "alert:"]
lgtm_stack:
- path: "observability/lgtm/docker-compose.yml"
must_contain: ["loki:", "grafana:", "tempo:", "mimir:", "alloy:"]
- path: "observability/lgtm/alloy/config.alloy"
must_contain: ["otelcol.receiver.otlp", "loki.write", "prometheus.remote_write"]
- path: "observability/lgtm/loki/config.yaml"
must_contain: ["auth_enabled:", "ingester:", "schema_config:"]
- path: "observability/lgtm/tempo/config.yaml"
must_contain: ["server:", "distributor:", "storage:"]
- path: "observability/lgtm/mimir/config.yaml"
must_contain: ["target:", "server:", "ingester:"]
- path: "observability/grafana/datasources.yaml"
must_contain: ["prometheus", "loki", "tempo"]
datadog:
- path: "observability/datadog/datadog.yaml"
must_contain: ["api_key:", "site:", "logs_enabled:", "apm_config:"]
- path: "observability/datadog/agent-daemonset.yaml"
must_contain: ["DaemonSet", "datadog/agent"]
- path: "observability/datadog/monitors/"
must_contain: [] # Monitor definitions
newrelic:
- path: "observability/newrelic/newrelic.yml"
must_contain: ["license_key:", "app_name:", "log_level:"]
- path: "observability/newrelic/otel-collector.yaml"
must_contain: ["exporters:", "otlp:", "endpoint: https://otlp.nr-data.net"]
cloudwatch:
- path: "observability/cloudwatch/otel-collector.yaml"
must_contain: ["exporters:", "awsemf:", "awsxray:"]
- path: "observability/cloudwatch/alarms.tf"
must_contain: ["aws_cloudwatch_metric_alarm", "comparison_operator"]
- path: "observability/cloudwatch/log-groups.tf"
must_contain: ["aws_cloudwatch_log_group", "retention_in_days"]
infrastructure:
docker:
- path: "observability/docker-compose.yml"
must_contain: ["services:", "prometheus:", "grafana:"]
- path: "observability/Dockerfile.alloy"
must_contain: ["FROM grafana/alloy"]
kubernetes:
- path: "observability/kubernetes/namespace.yaml"
must_contain: ["kind: Namespace", "name: observability"]
- path: "observability/kubernetes/alloy-daemonset.yaml"
must_contain: ["kind: DaemonSet", "opentelemetry"]
- path: "observability/kubernetes/prometheus/"
must_contain: [] # Prometheus Operator manifests
- path: "observability/kubernetes/grafana/"
must_contain: [] # Grafana Deployment
- path: "observability/kubernetes/loki/"
must_contain: [] # Loki StatefulSet
- path: "observability/kubernetes/tempo/"
must_contain: [] # Tempo StatefulSet
helm:
- path: "observability/helm/values-lgtm.yaml"
must_contain: ["loki:", "grafana:", "tempo:", "mimir:"]
- path: "observability/helm/Chart.yaml"
must_contain: ["apiVersion:", "name:", "dependencies:"]
scaffolding:
# Core OpenTelemetry SDK setup (language-specific)
- path: "src/observability/otel.py"
template: "examples/fastapi-otel/main.py"
condition: "language == 'python'"
description: "OpenTelemetry SDK initialization for Python"
- path: "src/observability/otel.rs"
template: "examples/axum-tracing/src/main.rs"
condition: "language == 'rust'"
description: "OpenTelemetry SDK initialization for Rust with tracing"
# Structured logging configuration
- path: "src/observability/logging.py"
template: "references/structured-logging.md"
condition: "language == 'python'"
description: "structlog configuration with trace correlation"
- path: "src/observability/tracing.rs"
template: "references/structured-logging.md"
condition: "language == 'rust'"
description: "tracing-subscriber configuration"
# LGTM Stack docker-compose
- path: "observability/lgtm/docker-compose.yml"
template: "examples/lgtm-docker-compose/docker-compose.yml"
condition: "maturity == 'starter' OR maturity == 'intermediate'"
description: "Complete LGTM stack for local development"
# Grafana Alloy collector configuration
- path: "observability/alloy/config.alloy"
template: "examples/lgtm-docker-compose/alloy-config.alloy"
condition: "observability == 'lgtm_stack'"
description: "Grafana Alloy OpenTelemetry collector configuration"
# Prometheus configuration
- path: "observability/prometheus.yml"
template: "examples/lgtm-docker-compose/prometheus.yml"
description: "Prometheus scrape configuration"
# Base alert rules
- path: "observability/alerts/base-alerts.yml"
template: "references/alerting-rules.md"
description: "Essential Prometheus alert rules (SLOs, error rates, latency)"
# Loki alert rules
- path: "observability/alerts/loki-alerts.yml"
template: "references/alerting-rules.md"
description: "Log-based alerting with Loki"
# Grafana dashboards
- path: "observability/grafana/dashboards/api-overview.json"
template: "examples/grafana-dashboards/api-overview.json"
description: "API service overview dashboard (RED metrics)"
- path: "observability/grafana/dashboards/traces.json"
template: "examples/grafana-dashboards/traces.json"
description: "Distributed tracing visualization dashboard"
# Kubernetes manifests (advanced)
- path: "observability/kubernetes/alloy-daemonset.yaml"
template: "references/lgtm-stack.md"
condition: "infrastructure == 'kubernetes'"
description: "Grafana Alloy DaemonSet for Kubernetes"
- path: "observability/kubernetes/prometheus-operator.yaml"
template: "references/lgtm-stack.md"
condition: "infrastructure == 'kubernetes' AND observability == 'prometheus_grafana'"
description: "Prometheus Operator with ServiceMonitors"
# Helper scripts
- path: "scripts/setup_otel.py"
template: "scripts/setup_otel.py"
description: "Bootstrap OpenTelemetry SDK installation"
- path: "scripts/generate_dashboards.py"
template: "scripts/generate_dashboards.py"
description: "Generate Grafana dashboards from templates"
- path: "scripts/validate_metrics.py"
template: "scripts/validate_metrics.py"
description: "Validate metric naming and cardinality"
metadata:
primary_blueprints:
- "observability"
contributes_to:
- "Monitoring and observability"
- "Distributed tracing"
- "Structured logging"
- "Metrics collection"
- "Alerting and incident response"
related_skills:
- "implementing-kubernetes"
- "implementing-cicd"
- "handling-errors"
- "implementing-databases"
key_technologies:
- "OpenTelemetry"
- "Prometheus"
- "Grafana"
- "Loki"
- "Tempo"
- "Mimir"
- "Grafana Alloy"
- "Jaeger"
- "structlog"
- "tracing (Rust)"
- "slog (Go)"
- "pino (Node.js)"
output_patterns:
logs:
- "observability/loki/"
- "observability/logging/"
- "src/observability/logging.*"
metrics:
- "observability/prometheus.yml"
- "observability/mimir/"
- "observability/alerts/"
traces:
- "observability/tempo/"
- "observability/jaeger/"
- "src/observability/otel.*"
visualization:
- "observability/grafana/dashboards/"
- "observability/grafana/datasources.yaml"
deployment:
- "observability/docker-compose.yml"
- "observability/kubernetes/"
- "observability/helm/"
Alerting Rules and Notification Patterns
Complete guide for creating alerting rules in Prometheus, Loki, and Grafana with notification routing.
Table of Contents
- Alerting Architecture
- Prometheus Alert Rules
- Loki Alert Rules
- Grafana Alerting
- Notification Channels
- Best Practices
---
Alerting Architecture
┌────────────────────────────────────────────────────────┐
│ Alerting Data Flow │
├────────────────────────────────────────────────────────┤
│ │
│ Metrics/Logs Sources │
│ ├── Prometheus (metrics) │
│ ├── Loki (logs) │
│ └── Tempo (traces - no native alerting) │
│ │ │
│ ▼ │
│ ┌─────────────────────┐ │
│ │ Alert Rules │ │
│ │ (PromQL/LogQL) │ │
│ └──────────┬──────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────┐ │
│ │ Alertmanager / │ │
│ │ Grafana Alerting │ │
│ └──────────┬──────────┘ │
│ │ │
│ ├──────────┬────────────┬──────────┐ │
│ ▼ ▼ ▼ ▼ │
│ Slack PagerDuty Email Webhook │
│ │
└────────────────────────────────────────────────────────┘---
Prometheus Alert Rules
Rule File Format
alerts.yaml:
groups:
- name: http_alerts
interval: 30s # Evaluation interval
rules:
- alert: HighHTTPErrorRate
expr: |
(
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)
) > 0.05
for: 5m # Alert fires after condition is true for 5 minutes
labels:
severity: critical
team: backend
annotations:
summary: "High HTTP error rate on {{ $labels.service }}"
description: |
Service {{ $labels.service }} has {{ $value | humanizePercentage }} error rate.
Current error rate: {{ $value | humanize }}
runbook_url: "https://wiki.example.com/runbooks/high-error-rate"Common Metric Alerts
1. High Error Rate
- alert: HighHTTPErrorRate
expr: |
(
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)
) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "{{ $labels.service }} error rate above 5%"2. High Latency (P95)
- alert: HighP95Latency
expr: |
histogram_quantile(0.95,
rate(http_request_duration_seconds_bucket[5m])
) > 0.5
for: 10m
labels:
severity: warning
annotations:
summary: "P95 latency above 500ms on {{ $labels.service }}"
description: "Current P95: {{ $value }}s"3. High Memory Usage
- alert: HighMemoryUsage
expr: |
(
container_memory_usage_bytes{container!=""}
/
container_spec_memory_limit_bytes{container!=""}
) > 0.9
for: 5m
labels:
severity: warning
annotations:
summary: "Container {{ $labels.container }} memory usage above 90%"4. High CPU Usage
- alert: HighCPUUsage
expr: |
rate(container_cpu_usage_seconds_total{container!=""}[5m]) > 0.8
for: 10m
labels:
severity: warning
annotations:
summary: "Container {{ $labels.container }} CPU usage above 80%"5. Service Down (No Metrics)
- alert: ServiceDown
expr: |
up{job="my-service"} == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Service {{ $labels.job }} is down"
description: "No metrics received from {{ $labels.instance }}"6. High Request Rate
- alert: HighRequestRate
expr: |
sum(rate(http_requests_total[1m])) by (service) > 1000
for: 5m
labels:
severity: warning
annotations:
summary: "High request rate on {{ $labels.service }}"
description: "Current rate: {{ $value }} req/s"7. Disk Usage
- alert: HighDiskUsage
expr: |
(
node_filesystem_avail_bytes{mountpoint="/"}
/
node_filesystem_size_bytes{mountpoint="/"}
) < 0.1
for: 5m
labels:
severity: critical
annotations:
summary: "Disk usage above 90% on {{ $labels.instance }}"---
Loki Alert Rules
Rule File Format
loki-alerts.yaml:
groups:
- name: log_alerts
interval: 1m
rules:
- alert: HighLogErrorRate
expr: |
sum(rate({job="api-service"} | json | level="error" [5m])) by (service)
/
sum(rate({job="api-service"}[5m])) by (service)
> 0.1
for: 5m
labels:
severity: warning
annotations:
summary: "High error log rate for {{ $labels.service }}"
description: "{{ $value | humanizePercentage }} of logs are errors"Common Log Alerts
1. Error Log Spike
- alert: ErrorLogSpike
expr: |
sum(rate({job=~".+"} | json | level="error" [5m])) by (service)
> 10
for: 2m
labels:
severity: warning
annotations:
summary: "Error log spike on {{ $labels.service }}"
description: "{{ $value }} errors/second"2. Application Crash Detected
- alert: ApplicationCrashDetected
expr: |
sum(count_over_time({job=~".+"} |~ "(?i)(panic|fatal|crashed|exception)" [1m])) by (service)
> 0
for: 0m # Immediate alert
labels:
severity: critical
annotations:
summary: "Application crash detected on {{ $labels.service }}"3. Authentication Failures
- alert: HighAuthFailureRate
expr: |
sum(rate({job="auth-service"} | json | message="authentication_failed" [5m]))
> 5
for: 3m
labels:
severity: warning
annotations:
summary: "High authentication failure rate"
description: "{{ $value }} failed auth attempts per second"4. Slow Query Logs
- alert: SlowQueriesDetected
expr: |
sum(count_over_time({job="api-service"} | json | duration_ms > 1000 [5m]))
> 10
for: 5m
labels:
severity: warning
annotations:
summary: "Slow queries detected on {{ $labels.service }}"5. Database Connection Errors
- alert: DatabaseConnectionErrors
expr: |
sum(rate({job=~".+"} |~ "(?i)(connection refused|connection timeout|database unavailable)" [5m]))
> 1
for: 2m
labels:
severity: critical
annotations:
summary: "Database connection errors on {{ $labels.service }}"---
Grafana Alerting
Contact Points
Slack Integration:
apiVersion: 1
contactPoints:
- orgId: 1
name: slack-critical
receivers:
- uid: slack-critical
type: slack
settings:
url: https://hooks.slack.com/services/YOUR/WEBHOOK/URL
recipient: '#alerts-critical'
username: Grafana
text: |
{{ range .Alerts }}
*Alert:* {{ .Labels.alertname }}
*Severity:* {{ .Labels.severity }}
*Summary:* {{ .Annotations.summary }}
*Description:* {{ .Annotations.description }}
{{ end }}Email Integration:
- name: email-team
receivers:
- uid: email-team
type: email
settings:
addresses: team@example.com
subject: "[{{ .Status }}] {{ .Labels.alertname }}"PagerDuty Integration:
- name: pagerduty-oncall
receivers:
- uid: pagerduty
type: pagerduty
settings:
integrationKey: YOUR_INTEGRATION_KEY
severity: criticalWebhook Integration:
- name: custom-webhook
receivers:
- uid: webhook
type: webhook
settings:
url: https://api.example.com/alerts
httpMethod: POSTNotification Policies
apiVersion: 1
policies:
- orgId: 1
receiver: default-email
group_by: ['alertname', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- receiver: slack-critical
matchers:
- severity = critical
continue: true
group_wait: 10s
repeat_interval: 1h
- receiver: pagerduty-oncall
matchers:
- severity = critical
- team = backend
continue: false
- receiver: email-team
matchers:
- severity = warning
repeat_interval: 12hAlert Rule (Grafana UI Format)
apiVersion: 1
groups:
- orgId: 1
name: performance_alerts
folder: Production
interval: 1m
rules:
- uid: high_p99_latency
title: High P99 Latency
condition: C
data:
- refId: A
datasourceUid: mimir
model:
expr: |
histogram_quantile(0.99,
rate(http_request_duration_seconds_bucket[5m])
)
range: true
intervalMs: 1000
- refId: B
datasourceUid: __expr__
model:
type: reduce
expression: A
reducer: last
- refId: C
datasourceUid: __expr__
model:
type: threshold
expression: B
conditions:
- evaluator:
params: [1.0]
type: gt
noDataState: NoData
execErrState: Error
for: 5m
annotations:
summary: P99 latency above 1 second
labels:
severity: warning---
Notification Channels
Slack Message Template
{
"channel": "#alerts",
"username": "Grafana",
"icon_emoji": ":warning:",
"attachments": [
{
"color": "{{ if eq .Status \"firing\" }}danger{{ else }}good{{ end }}",
"title": "{{ .Labels.alertname }}",
"text": "{{ .Annotations.summary }}",
"fields": [
{
"title": "Service",
"value": "{{ .Labels.service }}",
"short": true
},
{
"title": "Severity",
"value": "{{ .Labels.severity }}",
"short": true
},
{
"title": "Description",
"value": "{{ .Annotations.description }}",
"short": false
}
],
"footer": "Grafana",
"footer_icon": "https://grafana.com/static/img/about/grafana_icon.svg",
"ts": {{ .StartsAt.Unix }}
}
]
}Email Template
<!DOCTYPE html>
<html>
<head>
<style>
.alert { padding: 20px; border-left: 5px solid #f44336; }
.warning { border-left-color: #ff9800; }
.resolved { border-left-color: #4caf50; }
</style>
</head>
<body>
<div class="alert {{ .Status }}">
<h2>{{ .Labels.alertname }}</h2>
<p><strong>Status:</strong> {{ .Status }}</p>
<p><strong>Severity:</strong> {{ .Labels.severity }}</p>
<p><strong>Summary:</strong> {{ .Annotations.summary }}</p>
<p><strong>Description:</strong> {{ .Annotations.description }}</p>
<p><strong>Started:</strong> {{ .StartsAt }}</p>
{{ if .EndsAt }}
<p><strong>Ended:</strong> {{ .EndsAt }}</p>
{{ end }}
</div>
</body>
</html>PagerDuty Payload
{
"routing_key": "YOUR_INTEGRATION_KEY",
"event_action": "{{ if eq .Status \"firing\" }}trigger{{ else }}resolve{{ end }}",
"dedup_key": "{{ .GroupLabels.alertname }}-{{ .GroupLabels.service }}",
"payload": {
"summary": "{{ .Annotations.summary }}",
"source": "{{ .Labels.instance }}",
"severity": "{{ .Labels.severity }}",
"custom_details": {
"alert_name": "{{ .Labels.alertname }}",
"service": "{{ .Labels.service }}",
"description": "{{ .Annotations.description }}",
"runbook_url": "{{ .Annotations.runbook_url }}"
}
}
}---
Best Practices
1. Alert Severity Levels
CRITICAL: Service down, data loss, security breach
- Page on-call engineer immediately
- Example: 100% error rate, database unavailable
WARNING: Degraded performance, high resource usage
- Notify during business hours
- Example: High latency, 80% CPU
INFO: Informational events
- Log only, no notification
- Example: Deployment completed, scaling event2. Alert Naming Convention
[Component][Metric][Condition]
Good:
- APIHighErrorRate
- DatabaseHighLatency
- CacheMemoryExhausted
Bad:
- Alert1
- Problem
- IssueDetected3. Alert Grouping
Group related alerts:
group_by: ['alertname', 'service', 'region']
group_wait: 30s # Wait to batch initial alerts
group_interval: 5m # Wait before sending additional grouped alerts
repeat_interval: 4h # Resend if still firing4. Runbook Links
Always include runbook URL:
annotations:
runbook_url: "https://wiki.example.com/runbooks/{{ $labels.alertname }}"5. Alert Inhibition
Suppress dependent alerts:
inhibit_rules:
- source_match:
severity: critical
alertname: ServiceDown
target_match:
severity: warning
equal: ['service', 'instance']
# If service is down, don't alert on high latency/errors6. Alert Thresholds
Use appropriate thresholds:
# Error rate: > 5% for 5 minutes
# Latency: P95 > 500ms for 10 minutes
# Memory: > 90% for 5 minutes
# CPU: > 80% for 10 minutesAvoid flapping:
- Use
for: <duration>to require sustained condition - Set reasonable repeat intervals
- Use hysteresis (different thresholds for firing vs resolving)
7. Alert Fatigue Prevention
Reduce noise:
- Start with higher thresholds, tune down based on false positives
- Use alert inhibition for cascading failures
- Group related alerts
- Set appropriate repeat intervals
- Auto-resolve when condition clears
---
Testing Alerts
Test Prometheus Alert
Create test metric:
# Send high error rate
curl -X POST http://localhost:9090/api/v1/admin/tsdb/delete_series \
-d 'match[]={__name__="http_requests_total"}'
# Push test metrics
cat <<EOF | curl --data-binary @- http://localhost:9091/metrics/job/test
http_requests_total{status="500"} 100
http_requests_total{status="200"} 10
EOFVerify alert fires:
# Check pending/firing alerts
curl http://localhost:9090/api/v1/alerts | jq .Test Loki Alert
Generate error logs:
# Send error logs to Loki
curl -X POST http://localhost:3100/loki/api/v1/push \
-H "Content-Type: application/json" \
-d '{
"streams": [{
"stream": {"job": "test-service", "level": "error"},
"values": [
["'$(date +%s%N)'", "error: test alert"]
]
}]
}'Test Notification
Test Slack webhook:
curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
-H "Content-Type: application/json" \
-d '{
"text": "Test alert from Grafana",
"attachments": [{
"color": "danger",
"title": "Test Alert",
"text": "This is a test notification"
}]
}'---
Auto-Generated Alerts
Use script to generate alerts:
python scripts/generate_dashboards.py --alerts \
--metrics http_request_duration_seconds \
--thresholds p95:0.5,p99:1.0 \
--output alerts.yamlGenerated output:
groups:
- name: http_request_duration_seconds_alerts
rules:
- alert: HighP95Latency
expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 0.5
for: 5m
labels:
severity: warning
annotations:
summary: "P95 latency above 500ms"
- alert: HighP99Latency
expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 1.0
for: 5m
labels:
severity: critical
annotations:
summary: "P99 latency above 1s"---
Troubleshooting
Alerts not firing:
- Check PromQL/LogQL query returns data
- Verify
for:duration hasn't been met yet - Check alert rule evaluation interval
- Verify data source is connected
Notifications not sent:
- Test webhook URL manually
- Check notification policy routing
- Verify contact point configuration
- Check Alertmanager/Grafana logs
Too many alerts (alert fatigue):
- Increase thresholds
- Add inhibit rules
- Increase repeat interval
- Group related alerts
- Use more specific matchers
LGTM Stack Deployment Guide
Complete guide for deploying the LGTM stack (Loki, Grafana, Tempo, Mimir) in Docker Compose and Kubernetes.
Table of Contents
- Architecture Overview
- Docker Compose (Development)
- Kubernetes (Production)
- Grafana Configuration
- Scaling Considerations
---
Architecture Overview
┌────────────────────────────────────────────────────────────────┐
│ LGTM Stack Flow │
├────────────────────────────────────────────────────────────────┤
│ │
│ Application │
│ ├── OpenTelemetry SDK │
│ └── Exports OTLP (gRPC port 4317 or HTTP port 4318) │
│ │ │
│ ▼ │
│ ┌─────────────────────┐ │
│ │ Grafana Alloy │ (Unified Collector) │
│ │ Receives: OTLP │ │
│ │ Exports to: │ │
│ │ ├─ Loki (logs) │ │
│ │ ├─ Tempo (traces) │ │
│ │ └─ Mimir (metrics) │ │
│ └─────────────────────┘ │
│ │ │
│ ├────────────────┬──────────────────┐ │
│ ▼ ▼ ▼ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Loki │ │ Tempo │ │ Mimir │ │
│ │ (Logs) │ │ (Traces) │ │ (Metrics) │ │
│ │ Port 3100 │ │ Port 3200 │ │ Port 9009 │ │
│ │ Storage: │ │ Storage: │ │ Storage: │ │
│ │ Object/FS │ │ Object/FS │ │ Object/FS │ │
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
│ │ │ │ │
│ └─────────────────┴──────────────────┘ │
│ │ │
│ ┌────────▼────────┐ │
│ │ Grafana │ │
│ │ Port 3000 │ │
│ │ Datasources: │ │
│ │ ├─ Loki │ │
│ │ ├─ Tempo │ │
│ │ └─ Mimir │ │
│ └─────────────────┘ │
│ │
└────────────────────────────────────────────────────────────────┘---
Docker Compose (Development)
Complete Stack (see examples/lgtm-docker-compose/)
docker-compose.yml:
version: '3.8'
services:
# Grafana - Visualization
grafana:
image: grafana/grafana:10.2.3
container_name: grafana
ports:
- "3000:3000"
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
- GF_SECURITY_ADMIN_PASSWORD=admin
volumes:
- ./grafana/datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
- grafana-data:/var/lib/grafana
networks:
- lgtm
# Loki - Logs
loki:
image: grafana/loki:2.9.3
container_name: loki
ports:
- "3100:3100"
command: -config.file=/etc/loki/local-config.yaml
volumes:
- ./loki/loki-config.yaml:/etc/loki/local-config.yaml
- loki-data:/loki
networks:
- lgtm
# Tempo - Traces
tempo:
image: grafana/tempo:2.3.1
container_name: tempo
ports:
- "3200:3200" # Tempo
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
command: -config.file=/etc/tempo/tempo.yaml
volumes:
- ./tempo/tempo-config.yaml:/etc/tempo/tempo.yaml
- tempo-data:/var/tempo
networks:
- lgtm
# Mimir - Metrics
mimir:
image: grafana/mimir:2.11.0
container_name: mimir
ports:
- "9009:9009"
command:
- -config.file=/etc/mimir/mimir.yaml
volumes:
- ./mimir/mimir-config.yaml:/etc/mimir/mimir.yaml
- mimir-data:/data
networks:
- lgtm
# Grafana Alloy - Collector
alloy:
image: grafana/alloy:v1.0.0
container_name: alloy
ports:
- "12345:12345" # Alloy UI
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
command:
- run
- /etc/alloy/config.alloy
- --server.http.listen-addr=0.0.0.0:12345
volumes:
- ./alloy/config.alloy:/etc/alloy/config.alloy
networks:
- lgtm
depends_on:
- loki
- tempo
- mimir
volumes:
grafana-data:
loki-data:
tempo-data:
mimir-data:
networks:
lgtm:
driver: bridgeConfiguration Files
grafana/datasources.yaml:
apiVersion: 1
datasources:
- name: Loki
type: loki
uid: loki
access: proxy
url: http://loki:3100
jsonData:
derivedFields:
- datasourceUid: tempo
matcherRegex: "trace_id=(\\w+)"
name: TraceID
url: "$${__value.raw}"
- name: Tempo
type: tempo
uid: tempo
access: proxy
url: http://tempo:3200
jsonData:
tracesToLogsV2:
datasourceUid: loki
filterByTraceID: true
filterBySpanID: true
tracesToMetrics:
datasourceUid: mimir
serviceMap:
datasourceUid: tempo
- name: Mimir
type: prometheus
uid: mimir
access: proxy
url: http://mimir:9009/prometheusloki/loki-config.yaml:
auth_enabled: false
server:
http_listen_port: 3100
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
limits_config:
retention_period: 168h # 7 daystempo/tempo-config.yaml:
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
storage:
trace:
backend: local
local:
path: /var/tempo/traces
wal:
path: /var/tempo/wal
pool:
max_workers: 100
queue_depth: 10000mimir/mimir-config.yaml:
target: all
server:
http_listen_port: 9009
grpc_listen_port: 9095
common:
storage:
backend: filesystem
filesystem:
dir: /data
blocks_storage:
backend: filesystem
filesystem:
dir: /data/blocks
compactor:
data_dir: /data/compactor
ingester:
ring:
kvstore:
store: inmemory
replication_factor: 1
limits:
ingestion_rate: 50000
ingestion_burst_size: 100000alloy/config.alloy:
otelcol.receiver.otlp "default" {
grpc {
endpoint = "0.0.0.0:4317"
}
http {
endpoint = "0.0.0.0:4318"
}
output {
metrics = [otelcol.processor.batch.default.input]
logs = [otelcol.processor.batch.default.input]
traces = [otelcol.processor.batch.default.input]
}
}
otelcol.processor.batch "default" {
output {
metrics = [otelcol.exporter.prometheus.mimir.input]
logs = [otelcol.exporter.loki.default.input]
traces = [otelcol.exporter.otlp.tempo.input]
}
}
otelcol.exporter.loki "default" {
forward_to = [loki.write.default.receiver]
}
loki.write "default" {
endpoint {
url = "http://loki:3100/loki/api/v1/push"
}
}
otelcol.exporter.otlp "tempo" {
client {
endpoint = "tempo:4317"
tls {
insecure = true
}
}
}
otelcol.exporter.prometheus "mimir" {
forward_to = [prometheus.remote_write.mimir.receiver]
}
prometheus.remote_write "mimir" {
endpoint {
url = "http://mimir:9009/api/v1/push"
}
}Start Stack
cd examples/lgtm-docker-compose
docker-compose up -d
# Verify services
docker-compose ps
# Check logs
docker-compose logs -f
# Access Grafana: http://localhost:3000 (admin/admin)
# OTLP endpoint: localhost:4317 (gRPC) or localhost:4318 (HTTP)---
Kubernetes (Production)
Namespace
# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: implementing-observabilityLoki (StatefulSet)
# loki-statefulset.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: loki-config
namespace: observability
data:
loki.yaml: |
auth_enabled: false
server:
http_listen_port: 3100
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: loki
namespace: observability
spec:
serviceName: loki
replicas: 1
selector:
matchLabels:
app: loki
template:
metadata:
labels:
app: loki
spec:
containers:
- name: loki
image: grafana/loki:2.9.3
args:
- -config.file=/etc/loki/loki.yaml
ports:
- containerPort: 3100
name: http
volumeMounts:
- name: config
mountPath: /etc/loki
- name: storage
mountPath: /loki
volumes:
- name: config
configMap:
name: loki-config
volumeClaimTemplates:
- metadata:
name: storage
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
name: loki
namespace: observability
spec:
ports:
- port: 3100
targetPort: 3100
name: http
selector:
app: lokiTempo (StatefulSet)
# tempo-statefulset.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: tempo-config
namespace: observability
data:
tempo.yaml: |
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
storage:
trace:
backend: local
local:
path: /var/tempo/traces
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: tempo
namespace: observability
spec:
serviceName: tempo
replicas: 1
selector:
matchLabels:
app: tempo
template:
metadata:
labels:
app: tempo
spec:
containers:
- name: tempo
image: grafana/tempo:2.3.1
args:
- -config.file=/etc/tempo/tempo.yaml
ports:
- containerPort: 3200
name: http
- containerPort: 4317
name: otlp-grpc
volumeMounts:
- name: config
mountPath: /etc/tempo
- name: storage
mountPath: /var/tempo
volumes:
- name: config
configMap:
name: tempo-config
volumeClaimTemplates:
- metadata:
name: storage
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 50Gi
---
apiVersion: v1
kind: Service
metadata:
name: tempo
namespace: observability
spec:
ports:
- port: 3200
targetPort: 3200
name: http
- port: 4317
targetPort: 4317
name: otlp-grpc
selector:
app: tempoGrafana (Deployment)
# grafana-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana
namespace: observability
spec:
replicas: 1
selector:
matchLabels:
app: grafana
template:
metadata:
labels:
app: grafana
spec:
containers:
- name: grafana
image: grafana/grafana:10.2.3
env:
- name: GF_SECURITY_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: grafana-admin
key: password
ports:
- containerPort: 3000
name: http
volumeMounts:
- name: datasources
mountPath: /etc/grafana/provisioning/datasources
- name: storage
mountPath: /var/lib/grafana
volumes:
- name: datasources
configMap:
name: grafana-datasources
- name: storage
persistentVolumeClaim:
claimName: grafana-pvc
---
apiVersion: v1
kind: Service
metadata:
name: grafana
namespace: observability
spec:
type: LoadBalancer
ports:
- port: 3000
targetPort: 3000
selector:
app: grafanaGrafana Alloy (DaemonSet)
# alloy-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: alloy
namespace: observability
spec:
selector:
matchLabels:
app: alloy
template:
metadata:
labels:
app: alloy
spec:
containers:
- name: alloy
image: grafana/alloy:v1.0.0
args:
- run
- /etc/alloy/config.alloy
ports:
- containerPort: 4317
name: otlp-grpc
- containerPort: 4318
name: otlp-http
volumeMounts:
- name: config
mountPath: /etc/alloy
volumes:
- name: config
configMap:
name: alloy-config
---
apiVersion: v1
kind: Service
metadata:
name: alloy
namespace: observability
spec:
type: ClusterIP
ports:
- port: 4317
targetPort: 4317
name: otlp-grpc
- port: 4318
targetPort: 4318
name: otlp-http
selector:
app: alloyDeploy to Kubernetes
# Create namespace
kubectl apply -f namespace.yaml
# Deploy LGTM stack
kubectl apply -f loki-statefulset.yaml
kubectl apply -f tempo-statefulset.yaml
kubectl apply -f mimir-statefulset.yaml
kubectl apply -f grafana-deployment.yaml
kubectl apply -f alloy-daemonset.yaml
# Check status
kubectl get pods -n observability
# Get Grafana URL (if LoadBalancer)
kubectl get svc grafana -n observability---
Grafana Configuration
Pre-configured Dashboards
Create ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-dashboards
namespace: observability
data:
dashboard-provider.yaml: |
apiVersion: 1
providers:
- name: 'default'
folder: 'General'
type: file
options:
path: /var/lib/grafana/dashboards
http-overview.json: |
{
"title": "HTTP Overview",
"panels": [
{
"title": "Request Rate",
"targets": [
{
"expr": "sum(rate(http_requests_total[5m]))"
}
]
}
]
}Alerting
Create Alert Rules:
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-alerts
namespace: observability
data:
alerts.yaml: |
groups:
- name: http_alerts
interval: 1m
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "High HTTP error rate"
description: "Error rate is {{ $value | humanizePercentage }}"---
Scaling Considerations
Horizontal Scaling
Loki:
- Read path: Scale queriers
- Write path: Scale ingesters
- Use object storage (S3, GCS) for long-term storage
Tempo:
- Scale distributors for ingestion
- Scale queriers for query performance
- Object storage is mandatory for production
Mimir:
- Distribute ingesters across AZs
- Scale compactors based on series count
- Use object storage for blocks
Resource Requests (Production)
# Loki
resources:
requests:
memory: "2Gi"
cpu: "1"
limits:
memory: "4Gi"
cpu: "2"
# Tempo
resources:
requests:
memory: "4Gi"
cpu: "2"
limits:
memory: "8Gi"
cpu: "4"
# Mimir
resources:
requests:
memory: "8Gi"
cpu: "4"
limits:
memory: "16Gi"
cpu: "8"High Availability
3-replica setup:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: loki
spec:
replicas: 3
template:
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: loki
topologyKey: kubernetes.io/hostname---
Helm Charts (Alternative)
Using official Grafana Helm charts:
# Add Grafana Helm repo
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
# Install Loki
helm install loki grafana/loki -n observability
# Install Tempo
helm install tempo grafana/tempo -n observability
# Install Mimir
helm install mimir grafana/mimir-distributed -n observability
# Install Grafana
helm install grafana grafana/grafana -n observability---
Validation
Test OTLP endpoint:
# Test gRPC endpoint
grpcurl -plaintext localhost:4317 list
# Send test trace
curl -X POST http://localhost:4318/v1/traces \
-H "Content-Type: application/json" \
-d '{
"resourceSpans": [{
"resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "test"}}]},
"scopeSpans": [{
"spans": [{
"traceId": "5B8EFFF798038103D269B633813FC60C",
"spanId": "EEE19B7EC3C1B174",
"name": "test-span",
"startTimeUnixNano": "1544712660000000000",
"endTimeUnixNano": "1544712661000000000"
}]
}]
}]
}'Check Grafana: 1. Navigate to http://localhost:3000 2. Explore → Tempo → Search for trace 3. Explore → Loki → Query logs 4. Explore → Mimir → Query metrics
---
Troubleshooting
Services not communicating:
- Check network connectivity:
kubectl exec -it <pod> -- nc -zv loki 3100 - Verify ConfigMaps are mounted correctly
- Check service DNS:
kubectl exec -it <pod> -- nslookup loki.observability.svc.cluster.local
High memory usage:
- Reduce retention period in Loki config
- Enable compaction in Mimir
- Tune batch sizes in Alloy
Missing data:
- Verify OTLP endpoint is accessible from apps
- Check Alloy logs:
kubectl logs -n observability -l app=alloy - Validate data is reaching backends:
kubectl logs -n observability -l app=loki
#!/usr/bin/env python3
"""
Generate Grafana Dashboards from Templates
Creates Grafana dashboard JSON from templates for common observability patterns.
Usage:
python scripts/generate_dashboards.py --type api --service my-api --output dashboard.json
python scripts/generate_dashboards.py --type database --service postgres --output db-dashboard.json
"""
import argparse
import json
import sys
from typing import Dict, Any
DASHBOARD_TEMPLATES = {
"api": {
"title": "{service} API Metrics",
"panels": [
{
"title": "Request Rate",
"targets": [
{
"expr": 'rate(http_requests_total{{service="{service}"}}[5m])',
"legendFormat": "{{method}} {{route}}",
}
],
"type": "graph",
},
{
"title": "Error Rate",
"targets": [
{
"expr": 'rate(http_requests_total{{service="{service}",status=~"5.."}}[5m])',
"legendFormat": "{{status}} {{route}}",
}
],
"type": "graph",
},
{
"title": "Latency (p50, p95, p99)",
"targets": [
{
"expr": 'histogram_quantile(0.50, rate(http_request_duration_seconds_bucket{{service="{service}"}}[5m]))',
"legendFormat": "p50",
},
{
"expr": 'histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{{service="{service}"}}[5m]))',
"legendFormat": "p95",
},
{
"expr": 'histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{{service="{service}"}}[5m]))',
"legendFormat": "p99",
},
],
"type": "graph",
},
],
},
"database": {
"title": "{service} Database Metrics",
"panels": [
{
"title": "Active Connections",
"targets": [
{
"expr": 'pg_stat_activity_count{{service="{service}"}}',
"legendFormat": "{{state}}",
}
],
"type": "graph",
},
{
"title": "Query Duration",
"targets": [
{
"expr": 'rate(pg_stat_statements_mean_exec_time{{service="{service}"}}[5m])',
"legendFormat": "{{query}}",
}
],
"type": "graph",
},
],
},
"llm": {
"title": "{service} LLM Serving Metrics",
"panels": [
{
"title": "Tokens per Second",
"targets": [
{
"expr": 'rate(llm_tokens_generated_total{{service="{service}"}}[5m])',
"legendFormat": "{{model}}",
}
],
"type": "graph",
},
{
"title": "Time to First Token (TTFT)",
"targets": [
{
"expr": 'histogram_quantile(0.95, rate(llm_time_to_first_token_seconds_bucket{{service="{service}"}}[5m]))',
"legendFormat": "p95 TTFT",
}
],
"type": "graph",
},
{
"title": "GPU Utilization",
"targets": [
{
"expr": 'nvidia_gpu_duty_cycle{{service="{service}"}}',
"legendFormat": "GPU {{gpu}}",
}
],
"type": "graph",
},
],
},
}
def generate_dashboard(dashboard_type: str, service: str) -> Dict[str, Any]:
"""Generate Grafana dashboard JSON"""
if dashboard_type not in DASHBOARD_TEMPLATES:
raise ValueError(f"Unknown dashboard type: {dashboard_type}. Valid: {list(DASHBOARD_TEMPLATES.keys())}")
template = DASHBOARD_TEMPLATES[dashboard_type]
# Build panels with service name
panels = []
for i, panel_template in enumerate(template["panels"]):
panel = {
"id": i + 1,
"title": panel_template["title"],
"type": panel_template["type"],
"gridPos": {"x": 0, "y": i * 8, "w": 24, "h": 8},
"targets": [],
}
# Substitute service name in queries
for target in panel_template["targets"]:
panel["targets"].append({
"expr": target["expr"].format(service=service),
"legendFormat": target["legendFormat"],
"refId": f"A{i}",
})
panels.append(panel)
# Build complete dashboard
dashboard = {
"dashboard": {
"title": template["title"].format(service=service),
"panels": panels,
"editable": True,
"timezone": "browser",
"schemaVersion": 36,
"version": 1,
"refresh": "30s",
},
"overwrite": True,
}
return dashboard
def main():
parser = argparse.ArgumentParser(description="Generate Grafana dashboards")
parser.add_argument(
"--type",
required=True,
choices=["api", "database", "llm"],
help="Dashboard type"
)
parser.add_argument(
"--service",
required=True,
help="Service name (used in metric labels)"
)
parser.add_argument(
"--output",
default="dashboard.json",
help="Output file path"
)
args = parser.parse_args()
try:
dashboard = generate_dashboard(args.type, args.service)
with open(args.output, 'w') as f:
json.dump(dashboard, f, indent=2)
print(f"✓ Dashboard generated: {args.output}")
print(f" Type: {args.type}")
print(f" Service: {args.service}")
print(f" Panels: {len(dashboard['dashboard']['panels'])}")
print(f"\nImport to Grafana:")
print(f" curl -X POST http://localhost:3000/api/dashboards/db \\")
print(f" -H 'Content-Type: application/json' \\")
print(f" -d @{args.output}")
except Exception as e:
print(f"Error: {e}", file=sys.stderr)
sys.exit(1)
if __name__ == "__main__":
main()
Related skills
Azure DeploySafely execute production deployments of already-prepared applications to Microsoft Azure.485k1.3kAzure ValidateRun deep pre-deployment checks on Azure configuration, infrastructure definitions, RBAC roles, and managed identities before pushing to production.485k1.3k
Github Actions DocsGet precise, docs-grounded answers about GitHub Actions workflows, syntax, security, and migration instead of relying on stale knowledge.275k72
Setup Pre CommitAutomatically run Prettier, type checking, and tests on every commit via Husky and lint-staged.170k199k
Deploy To VercelSafely turn any local project into a live Vercel preview with one instruction.99.3k29.7kVercel Cli With TokensDeploy projects to Vercel from agents and scripts using token authentication instead of interactive browser login.74.6k29.7kFAQ
What is the LGTM stack?
LGTM is Loki (logs), Grafana (visualization), Tempo (traces), and Mimir (metrics), a self-hosted observability stack fed by OpenTelemetry via Grafana Alloy.
Why inject trace_id into logs?
The skill flags log-trace correlation as critical: injecting trace_id and span_id lets you jump from a log line to the full distributed trace.