Loki

Grafana Loki Skill

Comprehensive guide for Grafana Loki - the cost-effective, horizontally-scalable log aggregation system inspired by Prometheus.

What is Loki?

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system that:

• Indexes only metadata (labels) - Not full log content like traditional systems
• Stores compressed chunks in affordable object storage (S3, GCS, Azure Blob)
• Uses Prometheus-style labels for organizing log streams
• Multi-tenant by default with built-in tenant isolation
• Cost-efficient - Dramatically smaller index and lower operational costs

Architecture Overview

Core Components

Data Flow

Write Path:

Log Source → Distributor → Ingester → Object Storage
                                    ↓
                              Chunks + Indexes

Read Path:

Query → Query Frontend → Query Scheduler → Querier
                                             ↓
                                    Ingesters + Storage

Deployment Modes

1. Monolithic Mode (-target=all)

• All components in single process
• Best for: Initial experimentation, small-scale (~20GB logs/day)
• Simplest approach

2. Simple Scalable Deployment (SSD) - Recommended Default

deploymentMode: SimpleScalable

write:
  replicas: 3   # Distributor + Ingester

read:
  replicas: 2   # Query Frontend + Querier

backend:
  replicas: 2   # Compactor + Index Gateway + Query Scheduler + Ruler

3. Microservices Mode (Distributed)

deploymentMode: Distributed

ingester:
  replicas: 3
  zoneAwareReplication:
    enabled: true

distributor:
  replicas: 3

querier:
  replicas: 3

queryFrontend:
  replicas: 2

queryScheduler:
  replicas: 2

compactor:
  replicas: 1

indexGateway:
  replicas: 2

Schema Configuration

Recommended: TSDB with Schema v13

loki:
  schemaConfig:
    configs:
      - from: "2024-04-01"
        store: tsdb
        object_store: azure  # or s3, gcs
        schema: v13
        index:
          prefix: loki_index_
          period: 24h

Storage Configuration

Azure Blob Storage (Recommended for Azure)

loki:
  storage:
    type: azure
    bucketNames:
      chunks: loki-chunks
      ruler: loki-ruler
      admin: loki-admin
    azure:
      accountName: <storage-account-name>
      # Option 1: User-Assigned Managed Identity (Recommended)
      useManagedIdentity: true
      useFederatedToken: false
      userAssignedId: <identity-client-id>
      # Option 2: Account Key (Dev only)
      # accountKey: <account-key>
      requestTimeout: 30s

AWS S3

loki:
  storage:
    type: s3
    bucketNames:
      chunks: my-loki-chunks-2024
      ruler: my-loki-ruler-2024
      admin: my-loki-admin-2024
    s3:
      endpoint: s3.us-east-1.amazonaws.com
      region: us-east-1
      # Use IAM roles or access keys
      accessKeyId: <access-key>
      secretAccessKey: <secret-key>
      s3ForcePathStyle: false

Google Cloud Storage

loki:
  storage:
    type: gcs
    bucketNames:
      chunks: my-loki-gcs-bucket
    gcs:
      bucketName: my-loki-gcs-bucket
      # Uses Workload Identity or service account

Chunk Configuration Best Practices

loki:
  ingester:
    chunk_encoding: snappy        # Recommended (fast + efficient)
    chunk_target_size: 1572864    # ~1.5MB compressed
    max_chunk_age: 2h             # Max time before flush
    chunk_idle_period: 30m        # Flush idle chunks
    flush_check_period: 30s
    flush_op_timeout: 10m

Limits Configuration

loki:
  limits_config:
    # Retention
    retention_period: 744h              # 31 days

    # Ingestion limits
    ingestion_rate_mb: 50
    ingestion_burst_size_mb: 100
    per_stream_rate_limit: 3MB
    per_stream_rate_limit_burst: 15MB

    # Query limits
    max_query_series: 10000
    max_query_lookback: 720h
    max_entries_limit_per_query: 10000

    # Required for OTLP
    allow_structured_metadata: true
    volume_enabled: true

    # Sample rejection
    reject_old_samples: true
    reject_old_samples_max_age: 168h    # 7 days
    max_label_names_per_series: 25

Compactor Configuration

loki:
  compactor:
    retention_enabled: true
    retention_delete_delay: 2h
    retention_delete_worker_count: 50
    compaction_interval: 10m
    delete_request_store: azure         # Match your storage type

Caching Configuration

Recommended: Separate Memcached instances

# Helm values for Loki caching
memcached:
  # Results cache
  frontend:
    replicas: 3
    memcached:
      maxItemMemory: 1024               # 1GB
      maxItemSize: 5m
      connectionLimit: 1024

  # Chunks cache
  chunks:
    replicas: 3
    memcached:
      maxItemMemory: 4096               # 4GB
      maxItemSize: 2m
      connectionLimit: 1024

# Enable caching in Loki config
loki:
  chunk_store_config:
    chunk_cache_config:
      memcached_client:
        host: loki-memcached-chunks.monitoring.svc
        service: memcached-client

LogQL Query Language

Basic Queries

# Stream selector
{job="api-server"}

# Multiple labels
{job="api-server", env="prod"}

# Label matchers
{namespace=~".*-prod"}           # Regex match
{level!="debug"}                  # Not equal

# Filter expressions
{job="api-server"} |= "error"     # Contains
{job="api-server"} != "debug"     # Not contains
{job="api-server"} |~ "err.*"     # Regex match
{job="api-server"} !~ "debug.*"   # Regex not match

Pipeline Stages

# JSON parsing
{job="api-server"} | json

# Extract specific fields
{job="api-server"} | json | line_format "{{.message}}"

# Label extraction
{job="api-server"} | logfmt | level="error"

# Pattern matching
{job="api-server"} | pattern "<ip> - - [<_>] \"<method> <path>\"" | method="POST"

Metric Queries

# Count logs per minute
count_over_time({job="api-server"}[1m])

# Rate of errors
rate({job="api-server"} |= "error" [5m])

# Bytes rate
bytes_rate({job="api-server"}[5m])

# Sum by label
sum by (namespace) (rate({job="api-server"}[5m]))

# Top 10 by volume
topk(10, sum by (namespace) (bytes_rate({}[5m])))

OpenTelemetry Integration

Native OTLP (Recommended - Loki 3.0+)

OpenTelemetry Collector Config:

exporters:
  otlphttp:
    endpoint: http://loki-gateway:3100/otlp
    headers:
      X-Scope-OrgID: "my-tenant"

service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [otlphttp]

Loki Config:

loki:
  limits_config:
    allow_structured_metadata: true    # Required for OTLP

Key Benefits:

• Log body stored as plain text (not JSON encoded)
• 17 default resource attributes auto-indexed
• Simpler queries without JSON parsing
• Better storage efficiency

Resource Attribute Mapping

Kubernetes Helm Deployment

Add Repository

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Install with Values

helm install loki grafana/loki \
  --namespace monitoring \
  --values values.yaml

Production Values Example

deploymentMode: Distributed

loki:
  auth_enabled: true

  schemaConfig:
    configs:
      - from: "2024-04-01"
        store: tsdb
        object_store: azure
        schema: v13
        index:
          prefix: loki_index_
          period: 24h

  storage:
    type: azure
    azure:
      accountName: mystorageaccount
      useManagedIdentity: true
      userAssignedId: <client-id>
    bucketNames:
      chunks: loki-chunks
      ruler: loki-ruler
      admin: loki-admin

  limits_config:
    retention_period: 2160h             # 90 days
    allow_structured_metadata: true

ingester:
  replicas: 3
  zoneAwareReplication:
    enabled: true
  resources:
    requests:
      cpu: 2
      memory: 8Gi
    limits:
      cpu: 4
      memory: 16Gi

querier:
  replicas: 3
  maxUnavailable: 2

queryFrontend:
  replicas: 2

distributor:
  replicas: 3

compactor:
  replicas: 1

indexGateway:
  replicas: 2
  maxUnavailable: 1

# Gateway for external access
gateway:
  service:
    type: LoadBalancer

# Monitoring
monitoring:
  serviceMonitor:
    enabled: true

Azure Identity Configuration

User-Assigned Managed Identity (Recommended)

1. Create Identity:

az identity create \
  --name loki-identity \
  --resource-group <rg>

IDENTITY_CLIENT_ID=$(az identity show --name loki-identity --resource-group <rg> --query clientId -o tsv)
IDENTITY_PRINCIPAL_ID=$(az identity show --name loki-identity --resource-group <rg> --query principalId -o tsv)

2. Assign to Node Pool:

az vmss identity assign \
  --resource-group <aks-node-rg> \
  --name <vmss-name> \
  --identities /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/loki-identity

3. Grant Storage Permission:

az role assignment create \
  --role "Storage Blob Data Contributor" \
  --assignee-object-id $IDENTITY_PRINCIPAL_ID \
  --scope /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<storage>

4. Configure Loki:

loki:
  storage:
    azure:
      useManagedIdentity: true
      userAssignedId: <IDENTITY_CLIENT_ID>

Multi-Tenancy

loki:
  auth_enabled: true

# Query with tenant header
curl -H "X-Scope-OrgID: tenant-a" \
  "http://loki:3100/loki/api/v1/query?query={job=\"app\"}"

# Multi-tenant queries (if enabled)
# X-Scope-OrgID: tenant-a|tenant-b

Troubleshooting

Common Issues

1. Container Not Found (Azure)

# Create required containers
az storage container create --name loki-chunks --account-name <storage>
az storage container create --name loki-ruler --account-name <storage>
az storage container create --name loki-admin --account-name <storage>

2. Authorization Failure (Azure)

# Verify RBAC assignment
az role assignment list --scope /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<storage>

# Assign if missing
az role assignment create \
  --role "Storage Blob Data Contributor" \
  --assignee-object-id <principal-id> \
  --scope <storage-scope>

# Restart pod to refresh token
kubectl delete pod -n monitoring <ingester-pod>

3. Ingester OOM

# Increase memory limits
ingester:
  resources:
    limits:
      memory: 16Gi

4. Query Timeout

loki:
  querier:
    query_timeout: 5m
    max_concurrent: 8
  query_scheduler:
    max_outstanding_requests_per_tenant: 2048

Diagnostic Commands

# Check pod status
kubectl get pods -n monitoring -l app.kubernetes.io/name=loki

# Check ingester logs
kubectl logs -n monitoring -l app.kubernetes.io/component=ingester --tail=100

# Check compactor logs
kubectl logs -n monitoring -l app.kubernetes.io/component=compactor --tail=100

# Verify readiness
kubectl exec -it <loki-pod> -n monitoring -- wget -qO- http://localhost:3100/ready

# Check configuration
kubectl exec -it <loki-pod> -n monitoring -- cat /etc/loki/config/config.yaml

API Reference

Ingestion

# Push logs
POST /loki/api/v1/push

# OTLP logs
POST /otlp/v1/logs

Query

# Instant query
GET /loki/api/v1/query?query={job="app"}&time=<timestamp>

# Range query
GET /loki/api/v1/query_range?query={job="app"}&start=<start>&end=<end>

# Labels
GET /loki/api/v1/labels
GET /loki/api/v1/label/<name>/values

# Series
GET /loki/api/v1/series

# Tail (WebSocket)
GET /loki/api/v1/tail?query={job="app"}

Health

GET /ready
GET /metrics

Reference Documentation

For detailed configuration by topic:

• [Storage Configuration](references/storage.md): Object stores, retention, WAL
• [LogQL Reference](references/logql.md): Query syntax and examples
• [OpenTelemetry Integration](references/opentelemetry.md): OTLP configuration

External Resources

• Official Loki Documentation
• Loki Helm Chart
• LogQL Documentation
• Loki GitHub Repository

---

Gotchas

• Ingester rejects labels that change cardinality mid-stream — a label switched from low to high cardinality silently splits the stream into a ghost gap.
• LogQL `|=` (line filter) is faster than label filter — query plan optimizes only line filters; label filters fire after parsing.
• Tenant separation via `X-Scope-OrgID`: missing header writes to tenant "fake" silently — your logs aren't lost, they're in the wrong tenant.
• Retention is per-tenant; global retention env var is fallback only — a misconfigured tenant silently overrides global.
• Compactor not running = orphan chunks pile up — storage grows without bounds; the compactor's failure is in a separate component's log.
• Promtail vs Alloy migration: label normalization differs in subtle ways (e.g., __path__ semantics); migrating during high traffic loses logs.

LogQL Query Language Reference

LogQL is Loki's Prometheus-inspired query language for logs.

Query Types

Log Queries

Filter and return log lines.

Metric Queries

Extract numeric values and aggregate.

Stream Selectors

Label Matchers

Examples

# Single label
{job="api-server"}

# Multiple labels (AND)
{job="api-server", env="prod"}

# Regex match
{namespace=~"(prod|staging)-.*"}

# All logs from namespace
{namespace="monitoring"}

# Exclude specific job
{namespace="monitoring", job!="loki"}

Line Filters

Applied after stream selector to filter log content.

Examples

# Contains "error"
{job="api"} |= "error"

# Case-insensitive contains
{job="api"} |~ "(?i)error"

# Multiple filters (AND)
{job="api"} |= "error" != "timeout"

# Regex filter
{job="api"} |~ "status=[45][0-9]{2}"

Parser Expressions

JSON Parser

# Parse entire line as JSON
{job="api"} | json

# Extract specific keys
{job="api"} | json level, message, user_id

# Filter on parsed field
{job="api"} | json | level="error"

# Access nested fields
{job="api"} | json | request_body_user="admin"

Logfmt Parser

# Parse key=value format
{job="api"} | logfmt

# Filter on parsed field
{job="api"} | logfmt | level="error"

Pattern Parser

# Extract from structured patterns
{job="nginx"} | pattern "<ip> - - [<_>] \"<method> <path> <_>\" <status> <size>"

# Filter on extracted field
{job="nginx"} | pattern "<ip> - - [<_>] \"<method> <path> <_>\" <status> <size>" | status >= 400

Regexp Parser

# Extract with named groups
{job="api"} | regexp "(?P<ip>\\d+\\.\\d+\\.\\d+\\.\\d+)"

# Multiple extractions
{job="api"} | regexp "user=(?P<user>\\w+).*status=(?P<status>\\d+)"

Unpack Parser

# Unpack Loki's structured metadata
{job="api"} | unpack

Label Filter Expressions

After parsing, filter on extracted labels:

# String comparison
{job="api"} | json | level="error"

# Numeric comparison
{job="api"} | json | status >= 400

# Regex on label
{job="api"} | json | path=~"/api/v[12]/.*"

# Multiple conditions
{job="api"} | json | level="error" and status >= 500

Line Format Expression

Transform output format:

# Simple template
{job="api"} | json | line_format "{{.message}}"

# Multiple fields
{job="api"} | json | line_format "{{.timestamp}} [{{.level}}] {{.message}}"

# Conditional formatting
{job="api"} | json | line_format "{{if eq .level \"error\"}}ERROR: {{end}}{{.message}}"

# With functions
{job="api"} | json | line_format "{{.message | upper}}"

Template Functions

Label Format Expression

Rename or modify labels:

# Rename label
{job="api"} | json | label_format app=job

# Transform label value
{job="api"} | json | label_format level=`{{.level | upper}}`

# Create new label
{job="api"} | json | label_format severity=`{{if eq .level "error"}}high{{else}}low{{end}}`

Drop Labels

Remove labels from output:

# Drop specific label
{job="api"} | json | drop __error__

# Drop multiple labels
{job="api"} | json | drop __error__, __error_details__

Keep Labels

Keep only specified labels:

# Keep only these labels
{job="api"} | json | keep level, message

Decolorize

Remove ANSI color codes:

{job="api"} | decolorize

Metric Queries

Range Aggregations

# Count errors in 5 minutes
count_over_time({job="api"} |= "error" [5m])

# Rate of logs per second
rate({job="api"} [5m])

# Bytes ingested
bytes_over_time({job="api"} [1h])

# Bytes per second
bytes_rate({job="api"} [5m])

Unwrap Expressions

Extract numeric values from logs:

# Extract duration from logs
{job="api"} | json | unwrap duration

# Apply range function
sum_over_time({job="api"} | json | unwrap response_time [5m])

# Average response time
avg_over_time({job="api"} | json | unwrap latency_ms [5m])

# Percentile
quantile_over_time(0.99, {job="api"} | json | unwrap duration [5m])

Unwrap Aggregation Functions

Vector Aggregations

Aggregate across streams:

# Sum by namespace
sum by (namespace) (rate({job="api"} [5m]))

# Average excluding job
avg without (job) (rate({namespace="prod"} [5m]))

# Top 10 by volume
topk(10, sum by (namespace) (bytes_rate({} [5m])))

# Count unique streams
count(rate({namespace="prod"} [5m]))

Binary Operations

# Multiply rate
rate({job="api"} [5m]) * 60

# Divide
bytes_rate({job="api"} [5m]) / 1024

# Compare
rate({job="api"} |= "error" [5m]) > 0.1

# Boolean
count_over_time({job="api"} |= "error" [5m]) > 100 and count_over_time({job="api"} |= "error" [5m]) < 1000

Structured Metadata Queries

For OTLP ingested logs:

# Filter by structured metadata
{job="api"} | severity_text="ERROR"

# Access trace context
{job="api"} | trace_id="abc123"

# Combine with parsers
{job="api"} | json | severity_text="ERROR" | line_format "{{.message}}"

Common Query Patterns

Error Analysis

# Error rate
sum(rate({namespace="prod"} |= "error" [5m])) by (job)

# Error percentage
sum(rate({namespace="prod"} |= "error" [5m])) by (job)
/
sum(rate({namespace="prod"} [5m])) by (job) * 100

# Top error messages
topk(10, sum by (message) (count_over_time({job="api"} | json | level="error" [1h])))

Latency Analysis

# Average response time
avg_over_time({job="api"} | json | unwrap duration_ms [5m])

# P99 latency
quantile_over_time(0.99, {job="api"} | json | unwrap duration_ms [5m])

# Slow requests
{job="api"} | json | duration_ms > 1000

Traffic Analysis

# Requests per second by endpoint
sum by (path) (rate({job="nginx"} | pattern "<_> <method> <path> <_>" [5m]))

# Traffic by status code
sum by (status) (rate({job="nginx"} | json [5m]))

# Top talkers
topk(10, sum by (client_ip) (bytes_rate({job="nginx"} [5m])))

Kubernetes Analysis

# Logs from specific pod
{namespace="prod", pod=~"api-.*"}

# Logs from deployment
{namespace="prod", deployment="api-server"}

# Container restarts
{namespace="prod"} |= "Starting container"

# OOMKilled events
{namespace="kube-system"} |= "OOMKilled"

Query Optimization Tips

1. Use stream selectors first - Narrow down streams before filtering 2. Avoid `{}` - Always specify at least one label 3. Use line filters before parsers - Filter raw logs before parsing 4. Limit time ranges - Smaller ranges = faster queries 5. Use `limit` - Add | limit 1000 for exploratory queries 6. Avoid high-cardinality labels - Use structured metadata instead

API Query Parameters

# Range query
GET /loki/api/v1/query_range?query={job="api"}&start=<timestamp>&end=<timestamp>&limit=1000&step=60s

# Instant query
GET /loki/api/v1/query?query={job="api"}&time=<timestamp>&limit=100

# With direction
GET /loki/api/v1/query_range?query={job="api"}&direction=backward&limit=100

OpenTelemetry Integration Reference

Overview

Grafana Loki supports two methods for OpenTelemetry log ingestion:

1. Native OTLP Endpoint (Recommended - Loki 3.0+) 2. LokiExporter (Deprecated)

Native OTLP Integration (Recommended)

Key Benefits

• Log body stored as plain text (not JSON encoded)
• 17 default resource attributes auto-indexed as labels
• Structured metadata for non-indexed attributes
• Simpler queries without JSON parsing
• Better storage efficiency
• All future enhancements focus on this method

Loki Configuration

loki:
  limits_config:
    # Required for OTLP (default in Loki 3.0+)
    allow_structured_metadata: true

    # Optional: Customize OTLP attribute mapping
    otlp_config:
      resource_attributes:
        attributes_config:
          # Promote additional attributes to index labels
          - action: index_label
            attributes:
              - custom.attribute
          # Drop sensitive attributes
          - action: drop
            attributes:
              - sensitive.field

OpenTelemetry Collector Configuration

Basic Configuration:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlphttp:
    endpoint: http://loki-gateway:3100/otlp

service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [otlphttp]

With Authentication:

extensions:
  basicauth:
    client_auth:
      username: ${LOKI_USERNAME}
      password: ${LOKI_PASSWORD}

exporters:
  otlphttp:
    endpoint: http://loki-gateway:3100/otlp
    auth:
      authenticator: basicauth

service:
  extensions: [basicauth]
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [otlphttp]

With Multi-Tenancy:

exporters:
  otlphttp:
    endpoint: http://loki-gateway:3100/otlp
    headers:
      X-Scope-OrgID: "my-tenant"

Resource Attribute Mapping

Loki automatically indexes these default resource attributes as labels:

Transformation Rules:

• Dots converted to underscores: service.name → service_name
• Nested attributes flattened: http.request.body → http_request_body
• Non-string values stringified automatically

OTLP Data Model

LogRecord Structure:

LogRecord:
  Timestamp            # Event occurrence time
  ObservedTimestamp    # System detection time
  TraceContext:
    TraceId            # Links to distributed trace
    SpanId             # Links to operation span
    TraceFlags         # Sampling info
  SeverityNumber       # 1-24 scale
  SeverityText         # TRACE, DEBUG, INFO, WARN, ERROR, FATAL
  Body                 # Log message
  Resource             # Source metadata
  InstrumentationScope # Emitting library info
  Attributes           # Custom key-value pairs

Severity Level Mapping:

Querying OTLP Logs

Direct Attribute Access:

# Filter by severity
{service_name="api"} | severity_text="ERROR"

# Filter by trace context
{service_name="api"} | trace_id="abc123"

# Access structured metadata
{service_name="api"} | user_id="12345"

Compare with LokiExporter (legacy):

# OTLP Native (simple)
{service_name="api"} | severity_text="ERROR"

# LokiExporter (complex - requires parsing)
{job="my-namespace/api"} | json | severity="ERROR"

LokiExporter (Deprecated)

Status: No longer recommended. No new feature development.

Why Deprecated

• All data encoded into JSON blobs
• Requires query-time JSON parsing
• Fixed index labels only: job, instance, exporter, level
• Higher query overhead
• Inefficient storage

Migration to Native OTLP

Step 1: Update Loki Configuration

loki:
  limits_config:
    allow_structured_metadata: true

Step 2: Update Collector Configuration

# Old (LokiExporter)
exporters:
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
    labels:
      attributes:
        severity: ""

# New (Native OTLP)
exporters:
  otlphttp:
    endpoint: http://loki:3100/otlp

Step 3: Update LogQL Queries

# Old (LokiExporter)
{job="namespace/service"} | json | level="error"

# New (Native OTLP)
{service_name="service", service_namespace="namespace"} | severity_text="ERROR"

Grafana Alloy Configuration

Grafana Alloy is the recommended collector for Loki.

Basic OTLP to Loki:

otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
  }
  http {
    endpoint = "0.0.0.0:4318"
  }

  output {
    logs = [otelcol.exporter.otlphttp.loki.input]
  }
}

otelcol.exporter.otlphttp "loki" {
  client {
    endpoint = "http://loki-gateway:3100/otlp"
    headers = {
      "X-Scope-OrgID" = "default",
    }
  }
}

With Kubernetes Attributes:

otelcol.processor.k8sattributes "default" {
  extract {
    metadata = [
      "k8s.namespace.name",
      "k8s.pod.name",
      "k8s.deployment.name",
      "k8s.node.name",
    ]
  }

  output {
    logs = [otelcol.exporter.otlphttp.loki.input]
  }
}

Application SDK Configuration

Java (Log4j2)

<!-- log4j2.xml -->
<Configuration>
  <Appenders>
    <OpenTelemetry name="OpenTelemetryAppender"/>
  </Appenders>
  <Loggers>
    <Root level="INFO">
      <AppenderRef ref="OpenTelemetryAppender"/>
    </Root>
  </Loggers>
</Configuration>

# application.properties
otel.exporter.otlp.endpoint=http://collector:4317
otel.service.name=my-java-app

Python

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter

# Setup logging
logger_provider = LoggerProvider()
logger_provider.add_log_record_processor(
    BatchLogRecordProcessor(OTLPLogExporter(endpoint="http://collector:4317"))
)

Go

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc"
    "go.opentelemetry.io/otel/sdk/log"
)

func initLogger() {
    exporter, _ := otlploggrpc.New(ctx,
        otlploggrpc.WithEndpoint("collector:4317"),
        otlploggrpc.WithInsecure(),
    )

    provider := log.NewLoggerProvider(
        log.WithProcessor(log.NewBatchProcessor(exporter)),
    )
}

Node.js

const { OTLPLogExporter } = require('@opentelemetry/exporter-logs-otlp-grpc');
const { LoggerProvider } = require('@opentelemetry/sdk-logs');

const loggerProvider = new LoggerProvider();
loggerProvider.addLogRecordProcessor(
  new BatchLogRecordProcessor(
    new OTLPLogExporter({
      url: 'http://collector:4317',
    })
  )
);

Best Practices

Resource Attribute Strategy

1. Use Semantic Conventions

• Use standard OpenTelemetry semantic conventions
• Consistent naming across services

2. Avoid High Cardinality

• Don't use request IDs, user IDs as resource attributes
• Store high-cardinality data in log attributes (structured metadata)

3. Kubernetes-Native Attributes

• Let collector auto-detect k8s attributes
• Use k8sattributes processor in collector

Query Optimization

1. Index Labels First

   # Good - uses indexed labels
   {service_name="api", k8s_namespace_name="prod"} | severity_text="ERROR"

   # Bad - no index filter
   {} | severity_text="ERROR" | service_name="api"

2. Use Structured Metadata for Secondary Filters

   {service_name="api"} | user_id="12345"

Trace Correlation

# Find logs for a specific trace
{service_name="api"} | trace_id="abc123def456"

# Link to Grafana Tempo
# Use trace_id to navigate from logs to traces

Troubleshooting

OTLP Payloads Rejected

Error: malformed request or structured metadata errors

Solution:

loki:
  limits_config:
    allow_structured_metadata: true

Missing Attributes in Labels

Issue: Resource attributes not appearing as index labels

Check:

1. Verify attribute is in default list or custom config 2. Check attribute naming follows conventions 3. Verify collector is sending attributes

High Cardinality Warnings

Issue: Too many unique label values

Solution:

1. Move high-cardinality attributes to structured metadata 2. Use otlp_config to drop or not index certain attributes

Connection Issues

# Test collector to Loki connectivity
curl -v http://loki-gateway:3100/ready

# Check collector logs
kubectl logs -l app=otel-collector -c collector

# Verify endpoint format
# Correct: http://loki:3100/otlp
# Wrong: http://loki:3100/loki/api/v1/push (that's the push API)

Loki Storage Configuration Reference

Storage Architecture

Loki manages three primary data types:

• Chunks: Compressed log entries stored in object stores
• Indexes: Stream metadata and references linking to chunks
• Bloom Blocks: Optional advanced indexes for accelerated search

Index Engines

TSDB (Recommended - Loki 2.8+)

The recommended index store for all new deployments.

Benefits:

• Stores index files directly in object storage
• More efficient, faster, and more scalable than BoltDB
• Feature parity with all previous approaches
• Dynamic query sharding (targets 300-600 MBs per shard)
• Index caching not required

Configuration:

loki:
  schemaConfig:
    configs:
      - from: "2024-04-01"
        store: tsdb
        object_store: azure
        schema: v13
        index:
          prefix: loki_index_
          period: 24h

  storage_config:
    tsdb_shipper:
      active_index_directory: /loki/tsdb-index
      cache_location: /loki/tsdb-cache
      cache_ttl: 24h

BoltDB Shipper (Legacy)

Suitable for Loki 2.0-2.7.x deployments only.

Characteristics:

• Index period MUST be 24 hours
• Requires compactor for deduplication
• Write deduplication disabled when replication factor > 1

Object Store Backends

AWS S3

Required IAM Permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::my-loki-bucket",
        "arn:aws:s3:::my-loki-bucket/*"
      ]
    }
  ]
}

Configuration:

loki:
  storage:
    type: s3
    s3:
      endpoint: s3.us-east-1.amazonaws.com
      region: us-east-1
      bucketnames: my-loki-bucket
      # Option 1: IAM Role (Recommended)
      # Use service account with IAM role annotation
      # Option 2: Access Keys
      accessKeyId: ${AWS_ACCESS_KEY_ID}
      secretAccessKey: ${AWS_SECRET_ACCESS_KEY}
      s3ForcePathStyle: false
      insecure: false
    bucketNames:
      chunks: my-loki-chunks
      ruler: my-loki-ruler
      admin: my-loki-admin

SSE-KMS Encryption:

loki:
  storage:
    s3:
      sse:
        type: SSE-KMS
        kms_key_id: <kms-key-arn>

Azure Blob Storage

Authentication Methods:

1. User-Assigned Managed Identity (Recommended)

loki:
  storage:
    type: azure
    azure:
      accountName: mystorageaccount
      useManagedIdentity: true
      useFederatedToken: false
      userAssignedId: <identity-client-id>
      requestTimeout: 30s

2. Workload Identity Federation

loki:
  podLabels:
    azure.workload.identity/use: "true"

serviceAccount:
  annotations:
    azure.workload.identity/client-id: <identity-client-id>

loki:
  storage:
    azure:
      accountName: mystorageaccount
      useManagedIdentity: false
      useFederatedToken: true

3. Account Key (Dev only)

loki:
  storage:
    azure:
      accountName: mystorageaccount
      accountKey: ${AZURE_STORAGE_KEY}

4. SAS Token

loki:
  storage:
    azure:
      accountName: mystorageaccount
      sasToken: ${AZURE_SAS_TOKEN}

Required RBAC Role:

• Storage Blob Data Contributor on the storage account

Google Cloud Storage

Configuration:

loki:
  storage:
    type: gcs
    gcs:
      bucketName: my-loki-bucket
      # Uses Workload Identity or service account JSON
      service_account: |
        ${GCS_SERVICE_ACCOUNT_JSON}
    bucketNames:
      chunks: chunks
      ruler: ruler
      admin: admin

MinIO (On-Premises)

loki:
  storage:
    type: s3
    s3:
      endpoint: minio.minio.svc:9000
      accessKeyId: ${MINIO_ACCESS_KEY}
      secretAccessKey: ${MINIO_SECRET_KEY}
      s3ForcePathStyle: true
      insecure: true   # Set false for TLS
    bucketNames:
      chunks: loki-chunks
      ruler: loki-ruler
      admin: loki-admin

Filesystem (Development Only)

loki:
  storage:
    type: filesystem
    filesystem:
      directory: /loki/chunks

  storage_config:
    filesystem:
      directory: /tmp/loki/

Limitations:

• NOT production-supported
• Directory limits at ~5.5M+ files
• Requires shared filesystem (NFS) for HA
• Durability depends on filesystem reliability

Retention Configuration

Enable Retention:

loki:
  compactor:
    retention_enabled: true
    retention_delete_delay: 2h
    retention_delete_worker_count: 50
    compaction_interval: 10m
    delete_request_store: azure

  limits_config:
    retention_period: 744h    # 31 days (minimum: 24h)

Stream-Level Retention:

loki:
  limits_config:
    retention_period: 744h    # Default

    retention_stream:
      - selector: '{environment="dev"}'
        priority: 1
        period: 168h          # 7 days

      - selector: '{environment="prod"}'
        priority: 1
        period: 2160h         # 90 days

      - selector: '{namespace="audit"}'
        priority: 2
        period: 8760h         # 1 year

Per-Tenant Overrides:

# runtime-config.yaml
overrides:
  tenant-a:
    retention_period: 2160h
  tenant-b:
    retention_period: 720h

Write Ahead Log (WAL)

Purpose: Records incoming data for crash recovery.

Configuration:

loki:
  ingester:
    wal:
      enabled: true
      dir: /loki/wal
      checkpoint_duration: 5m
      replay_memory_ceiling: 4GB   # ~75% of available memory

Requirements:

• Use StatefulSets with persistent volumes
• Each ingester must have unique WAL directory
• Expect ~10-15GB disk usage per ingester

Monitoring Metrics:

• loki_ingester_wal_records_logged
• loki_ingester_wal_logged_bytes_total
• loki_ingester_wal_corruptions_total
• loki_ingester_wal_disk_full_failures_total

Caching

Results Cache (Frontend)

loki:
  query_frontend:
    results_cache:
      cache:
        memcached_client:
          host: loki-memcached-frontend.monitoring.svc
          service: memcached-client
          timeout: 500ms
          max_idle_conns: 16
          update_interval: 1m

Chunks Cache

loki:
  chunk_store_config:
    chunk_cache_config:
      memcached_client:
        host: loki-memcached-chunks.monitoring.svc
        service: memcached-client
        timeout: 500ms
        max_idle_conns: 16
        batch_size: 256
        parallelism: 10

Memcached Deployment

# Chunks cache (larger)
memcached-chunks:
  replicas: 3
  args:
    - --memory-limit=4096
    - --max-item-size=2m
    - --conn-limit=1024

# Results cache (smaller)
memcached-frontend:
  replicas: 3
  args:
    - --memory-limit=1024
    - --max-item-size=5m
    - --conn-limit=1024

Compaction

Configuration:

loki:
  compactor:
    working_directory: /loki/compactor
    compaction_interval: 10m
    retention_enabled: true
    retention_delete_delay: 2h
    retention_delete_worker_count: 50
    delete_request_store: azure

Component Requirements:

• Must run as singleton instance
• Requires delete permissions on object storage
• Handles index deduplication and merging

Schema Migration

Adding New Schema:

loki:
  schemaConfig:
    configs:
      # Old schema (keep for historical data)
      - from: "2023-01-01"
        store: boltdb-shipper
        object_store: azure
        schema: v12
        index:
          prefix: loki_index_
          period: 24h

      # New schema (future date, UTC 00:00:00)
      - from: "2024-04-01"
        store: tsdb
        object_store: azure
        schema: v13
        index:
          prefix: loki_index_
          period: 24h

Rules:

• from date must be in future (UTC 00:00:00)
• Schema changes are irreversible
• Multiple schemas can coexist
• Queries span schemas transparently

Storage Troubleshooting

Azure Container Not Found

az storage container create --name loki-chunks --account-name <storage>
az storage container create --name loki-ruler --account-name <storage>
az storage container create --name loki-admin --account-name <storage>

Azure Authorization Failure

# Check role assignments
az role assignment list --scope <storage-scope> --query "[?principalId=='<principal-id>']"

# Assign role if missing
az role assignment create \
  --role "Storage Blob Data Contributor" \
  --assignee-object-id <principal-id> \
  --scope <storage-scope>

# Restart ingester to refresh token
kubectl delete pod -n monitoring <ingester-pod>

S3 Access Denied

# Verify IAM policy
aws iam get-policy --policy-arn <policy-arn>

# Test bucket access
aws s3 ls s3://my-loki-bucket/

Compactor Issues

# Check compactor logs
kubectl logs -n monitoring -l app.kubernetes.io/component=compactor --tail=200

# Verify compactor is running as singleton
kubectl get pods -n monitoring -l app.kubernetes.io/component=compactor