Pyroscope

Grafana Pyroscope Skill

Comprehensive guide for Grafana Pyroscope - the open-source continuous profiling platform for analyzing application performance at the code level.

What is Pyroscope?

Pyroscope is a horizontally-scalable, highly-available, multi-tenant continuous profiling system that:

• Collects profiling data continuously with minimal overhead (~2-5% CPU)
• Provides code-level visibility with source-line granularity
• Stores compressed profiles in object storage (S3, GCS, Azure Blob)
• Integrates with Grafana for correlating profiles with metrics, logs, and traces
• Supports multiple languages - Go, Java, Python, .NET, Ruby, Node.js, Rust

Architecture Overview

Core Components

Data Flow

Write Path:

SDK/Alloy → Distributor → Ingester → Object Storage
                                   ↓
                             Blocks + Indexes

Read Path:

Query → Query Frontend → Query Scheduler → Querier
                                             ↓
                                    Ingesters + Store Gateway

Deployment Modes

1. Monolithic Mode (-target=all)

• All components in single process
• Best for: Development, small-scale deployments
• Query URL: http://pyroscope:4040/

2. Microservices Mode (Production)

• Each component runs independently
• Horizontally scalable
• Query URL: http://pyroscope-querier:4040/

# Microservices deployment
architecture:
  microservices:
    enabled: true

querier:
  replicas: 3
distributor:
  replicas: 2
ingester:
  replicas: 3
compactor:
  replicas: 3
storeGateway:
  replicas: 3

Quick Start - Kubernetes Helm

Add Repository

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Install Single Binary

kubectl create namespace pyroscope
helm install pyroscope grafana/pyroscope -n pyroscope

Install Microservices Mode

curl -Lo values-micro-services.yaml \
  https://raw.githubusercontent.com/grafana/pyroscope/main/operations/pyroscope/helm/pyroscope/values-micro-services.yaml

helm install pyroscope grafana/pyroscope \
  -n pyroscope \
  --values values-micro-services.yaml

Profile Types

Client Configuration Methods

Method 1: SDK Instrumentation (Push Mode)

Go SDK:

import "github.com/grafana/pyroscope-go"

pyroscope.Start(pyroscope.Config{
    ApplicationName: "my-app",
    ServerAddress:   "http://pyroscope:4040",
    ProfileTypes: []pyroscope.ProfileType{
        pyroscope.ProfileCPU,
        pyroscope.ProfileAllocObjects,
        pyroscope.ProfileAllocSpace,
        pyroscope.ProfileInuseObjects,
        pyroscope.ProfileInuseSpace,
        pyroscope.ProfileGoroutines,
        pyroscope.ProfileMutexCount,
        pyroscope.ProfileMutexDuration,
        pyroscope.ProfileBlockCount,
        pyroscope.ProfileBlockDuration,
    },
    Tags: map[string]string{
        "env": "production",
    },
})

Java SDK:

PyroscopeAgent.start(
    new Config.Builder()
        .setApplicationName("my-app")
        .setServerAddress("http://pyroscope:4040")
        .setProfilingEvent(EventType.ITIMER)
        .setFormat(Format.JFR)
        .build()
);

Python SDK:

import pyroscope

pyroscope.configure(
    application_name="my-app",
    server_address="http://pyroscope:4040",
    tags={"env": "production"},
)

Method 2: Grafana Alloy (Pull Mode)

Auto-instrumentation via Annotations:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    metadata:
      annotations:
        profiles.grafana.com/cpu.scrape: "true"
        profiles.grafana.com/cpu.port: "8080"
        profiles.grafana.com/memory.scrape: "true"
        profiles.grafana.com/memory.port: "8080"
        profiles.grafana.com/goroutine.scrape: "true"
        profiles.grafana.com/goroutine.port: "8080"

Alloy Configuration:

pyroscope.scrape "default" {
  targets = discovery.kubernetes.pods.targets
  forward_to = [pyroscope.write.default.receiver]

  profiling_config {
    profile.process_cpu { enabled = true }
    profile.memory { enabled = true }
    profile.goroutine { enabled = true }
  }
}

pyroscope.write "default" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

Method 3: eBPF Profiling (Linux)

For compiled languages (C/C++, Go, Rust):

pyroscope.ebpf "default" {
  forward_to = [pyroscope.write.default.receiver]
  targets = discovery.kubernetes.pods.targets
}

Storage Configuration

Azure Blob Storage

pyroscope:
  config:
    storage:
      backend: azure
      azure:
        container_name: pyroscope-data
        account_name: mystorageaccount
        account_key: ${AZURE_ACCOUNT_KEY}

AWS S3

pyroscope:
  config:
    storage:
      backend: s3
      s3:
        bucket_name: pyroscope-data
        region: us-east-1
        endpoint: s3.us-east-1.amazonaws.com
        access_key_id: ${AWS_ACCESS_KEY_ID}
        secret_access_key: ${AWS_SECRET_ACCESS_KEY}

Google Cloud Storage

pyroscope:
  config:
    storage:
      backend: gcs
      gcs:
        bucket_name: pyroscope-data
        # Uses GOOGLE_APPLICATION_CREDENTIALS

Grafana Integration

Data Source Configuration

apiVersion: 1
datasources:
  - name: Pyroscope
    type: grafana-pyroscope-datasource
    access: proxy
    url: http://pyroscope-querier:4040
    isDefault: false
    editable: true

Trace-to-Profile Linking

Enable span profiles to correlate traces with profiles:

Go with OpenTelemetry:

import (
    "github.com/grafana/pyroscope-go"
    otelpyroscope "github.com/grafana/otel-profiling-go"
)

tp := trace.NewTracerProvider(
    trace.WithSpanProcessor(otelpyroscope.NewSpanProcessor()),
)

Requirements:

• Minimum span duration: 20ms
• Supported: Go, Java, .NET, Python, Ruby

Resource Requirements

Single Binary (Development)

resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 1
    memory: 2Gi

Microservices (Production)

Common Helm Values

# Production values
architecture:
  microservices:
    enabled: true

pyroscope:
  persistence:
    enabled: true
    size: 50Gi

  config:
    storage:
      backend: s3
      s3:
        bucket_name: pyroscope-prod
        region: us-east-1

# High availability
ingester:
  replicas: 3
  terminationGracePeriodSeconds: 600

querier:
  replicas: 3

distributor:
  replicas: 2

compactor:
  replicas: 3
  terminationGracePeriodSeconds: 1200

storeGateway:
  replicas: 3

# Pod disruption budget
podDisruptionBudget:
  enabled: true
  maxUnavailable: 1

# Topology spread
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule

# Monitoring
serviceMonitor:
  enabled: true

# Alloy for profile collection
alloy:
  enabled: true

API Endpoints

Ingestion

# Push profiles (Connect API)
POST /push.v1.PusherService/Push

# Legacy HTTP (pprof, JFR formats)
POST /ingest

Query

# Merged profile
POST /querier.v1.QuerierService/SelectMergeProfile

# Flame graph data
POST /querier.v1.QuerierService/SelectMergeStacktraces

# Available labels
POST /querier.v1.QuerierService/LabelNames

# Profile types
POST /querier.v1.QuerierService/ProfileTypes

# Legacy render
GET /pyroscope/render?query={}&from=now-1h&until=now

System

# Readiness
GET /ready

# Configuration
GET /config

# Metrics
GET /metrics

Troubleshooting

Diagnostic Commands

# Check pod status
kubectl get pods -n pyroscope -l app.kubernetes.io/name=pyroscope

# View ingester logs
kubectl logs -n pyroscope -l app.kubernetes.io/component=ingester --tail=100

# Check ring status
kubectl exec -it pyroscope-0 -n pyroscope -- \
  curl http://localhost:4040/ingester/ring

# Verify readiness
kubectl exec -it pyroscope-0 -n pyroscope -- \
  curl http://localhost:4040/ready

# Check configuration
kubectl exec -it pyroscope-0 -n pyroscope -- \
  curl http://localhost:4040/config

Common Issues

1. Ingester OOM:

ingester:
  resources:
    limits:
      memory: 16Gi

2. Storage Authentication Failed:

# Azure - verify RBAC
az role assignment create \
  --role "Storage Blob Data Contributor" \
  --assignee-object-id <principal-id> \
  --scope <storage-scope>

3. High Cardinality Labels:

# Limit label cardinality
pyroscope:
  config:
    validation:
      max_label_names_per_series: 25

4. Query Timeout:

pyroscope:
  config:
    querier:
      query_timeout: 5m
      max_concurrent: 8

Reference Documentation

For detailed configuration by topic:

• [Helm Deployment](references/helm-deployment.md): Complete Helm values reference
• [Architecture](references/architecture.md): Component details and scaling
• [SDK Instrumentation](references/sdk-instrumentation.md): Language SDK guides
• [Troubleshooting](references/troubleshooting.md): Common issues and diagnostics

External Resources

• Official Pyroscope Documentation
• Pyroscope Helm Chart
• Pyroscope GitHub Repository
• Grafana Profiles Drilldown

---

Gotchas

• Profile sampling rate per-application: 100Hz on a busy service adds non-trivial overhead — profile in canary first.
• Storage compaction blocks new writes during merge — large compactions cause write stalls; chunk size and compaction window need tuning.
• Tag cardinality explosion: per-request tags balloon storage; cap tag values explicitly or storage doubles weekly.
• Trace integration: span → profile linking requires matching traceparent header propagation end-to-end; one un-instrumented hop loses linkage silently.
• eBPF profiler vs SDK profiler: different attribution, sometimes conflicting. Don't run both for the same service.
• gRPC vs HTTP push: same data, different rate limits at the ingester; HTTP push is throttled harder under load.

Pyroscope Architecture Reference

Complete reference for Grafana Pyroscope architecture, deployment modes, and scaling.

Deployment Modes

Monolithic Mode

Flag: -target=all (default)

All components run in a single process. Simplest configuration for getting started.

┌─────────────────────────────────────────────┐
│              Pyroscope (Single Process)     │
│                                             │
│  ┌──────────┐ ┌──────────┐ ┌─────────────┐ │
│  │Distributor│ │ Ingester │ │   Querier   │ │
│  └──────────┘ └──────────┘ └─────────────┘ │
│  ┌──────────┐ ┌──────────┐ ┌─────────────┐ │
│  │Compactor │ │Store-GW  │ │Query-Frontend│ │
│  └──────────┘ └──────────┘ └─────────────┘ │
└─────────────────────────────────────────────┘

Use Cases:

• Development and testing
• Small-scale deployments (~20GB profiles/day)
• Quick experimentation

Limitations:

• No horizontal scaling
• Multiple instances don't share data
• Not suitable for production at scale

Query URL: http://pyroscope:4040/

Microservices Mode

Flag: -target=[component_name]

Each component runs as a separate process, enabling horizontal scaling and fault isolation.

                    ┌───────────────┐
                    │  SDK / Alloy  │
                    └───────┬───────┘
                            │
                    ┌───────▼───────┐
                    │  Distributor  │  (Stateless)
                    └───────┬───────┘
                            │
                    ┌───────▼───────┐
                    │   Ingesters   │  (Stateful - 3+ replicas)
                    └───────┬───────┘
                            │
              ┌─────────────┼─────────────┐
              │             │             │
      ┌───────▼───────┐     │     ┌───────▼───────┐
      │   Compactor   │     │     │ Store-Gateway │
      └───────────────┘     │     └───────────────┘
                            │
                    ┌───────▼───────┐
                    │Object Storage │
                    │(S3/GCS/Azure) │
                    └───────────────┘

Query Path:
┌─────────────┐   ┌────────────────┐   ┌─────────────┐
│Query-Frontend│──▶│Query-Scheduler │──▶│   Querier   │
└─────────────┘   └────────────────┘   └─────────────┘

Use Cases:

• Production environments
• High-scale deployments
• Multi-tenant systems
• Independent component scaling

Query URL: http://pyroscope-querier:4040/

Core Components

Distributor (Stateless)

Purpose: Entry point for profile ingestion. Routes profiles to ingesters.

Responsibilities:

1. Data Validation - Validates incoming profiles 2. Series Sharding - Distributes series across ingesters using consistent hashing 3. Replication - Replicates each series to multiple ingesters

Replication Model:

• Dynamo-style quorum consistency
• Default replication factor: 3
• Write success requires n/2 + 1 confirmations (2 of 3)

Configuration:

distributor:
  push_timeout: 5s
  ingestion_tenant_shard_size: 4  # Shuffle sharding
  ring:
    kvstore: memberlist
    heartbeat_timeout: 5m

Scaling: Horizontally scalable, stateless

Ingester (Stateful)

Purpose: Receives and temporarily stores profiles before writing to long-term storage.

Responsibilities:

1. Profile Reception - Receives profiles from distributors 2. In-Memory Buffering - Stores recent data in memory 3. Batch Compression - Compresses and batches samples 4. Storage Upload - Writes blocks to object storage

Ring States:

Configuration:

ingester:
  num_tokens: 128
  heartbeat_interval: 5s
  heartbeat_timeout: 10s
  lifecycle:
    num_flush_instances: 3
    claim_on_rollout: true
    ring_timeout: 5m

Data Safety:

• Each profile replicates to 3 ingesters by default
• Single ingester failure causes no data loss
• Write-ahead log for crash recovery

Scaling: Stateful, minimum 3 replicas for production

Querier (Stateless)

Purpose: Handles query execution on the read path.

Responsibilities:

1. Query Evaluation - Executes profile queries 2. Data Fetching - Retrieves from ingesters (recent) and store-gateways (historical) 3. Result Aggregation - Merges results from multiple sources

Data Sources:

• Ingesters - Recent profile data (not yet flushed)
• Store-Gateways - Historical data from object storage

Configuration:

querier:
  query_store_after: 4h
  query_ingesters_within: 1h
  shuffle_sharding_ingesters_enabled: true
  max_query_lookback: 30d

Scaling: Horizontally scalable, stateless

Query Frontend (Stateless)

Purpose: API gateway for the read path. Optimizes query execution.

Responsibilities:

1. Query Reception - Receives incoming queries 2. Queue Management - Enqueues queries via query-scheduler 3. Result Aggregation - Collects and returns results

Benefits:

• Fair scheduling across tenants
• Query splitting and caching
• Horizontal scaling of query capacity

Configuration:

query_frontend:
  max_outstanding_per_tenant: 200
  querier_forget_delay: 15m

Scaling: Horizontally scalable, minimum 2 replicas for HA

Query Scheduler (Stateless)

Purpose: Manages query queue and distributes work to queriers.

Flow:

1. Query-frontend receives and processes queries 2. Query-frontend enqueues to query-scheduler 3. Query-scheduler maintains in-memory queue 4. Queriers pull and execute queries 5. Results route back through query-frontend

Configuration:

query_scheduler:
  max_outstanding_requests_per_tenant: 2048

Scaling: Minimum 2 replicas for HA

Compactor (Stateless)

Purpose: Optimizes storage by merging blocks and managing retention.

Responsibilities:

1. Block Compaction - Merges small blocks into larger ones 2. De-duplication - Removes duplicate data from replication 3. Retention Enforcement - Deletes expired data 4. Bucket Index - Maintains per-tenant block index

Compaction Process:

Stage 1 - Vertical Compaction:

• Merges blocks from same time range
• Eliminates duplicates from replication

Stage 2 - Horizontal Compaction:

• Combines adjacent time ranges
• Reduces index size significantly

Configuration:

compactor:
  data_dir: ./data
  block_ranges:
    - 1h
    - 2h
    - 8h
  compaction_interval: 30m
  compaction_concurrency: 4
  deletion_delay: 12h

Disk Requirements:

Minimum disk = compaction_concurrency × max_block_size × 2

Scaling: Stateless, shuffle sharding for horizontal scaling

Store Gateway (Distributed)

Purpose: Provides access to long-term storage for historical queries.

Responsibilities:

1. Block Management - Each instance manages subset of blocks 2. Query Serving - Retrieves profile data from object storage 3. Caching - Caches frequently accessed blocks

Configuration:

store_gateway:
  sharding_ring:
    enabled: true
    kvstore: memberlist
  sync_interval: 15m
  ignore_blocks_within: 3h

Scaling: Distributed block assignment enables horizontal scaling

Data Structures

Block Format

Each block contains profiling data for a single tenant:

<block-ulid>/
├── meta.json      # Block metadata (time range, etc.)
├── index.tsdb     # TSDB index (labels → profiles)
├── profiles.parquet  # Profile data (Parquet format)
└── symbols.symdb  # Symbol information for profiles

Characteristics:

• Unique ULID identifier
• Single-tenant isolation
• Google pprof protocol compatible
• Label-based organization

Bucket Index

Per-tenant file listing all blocks:

{
  "blocks": [...],
  "block_deletion_marks": [...],
  "updated_at": 1234567890
}

Benefits:

• Eliminates costly "list objects" API calls
• Store-gateways consult index directly
• Updated by compactor periodically

Hash Rings

Used for consistent data distribution:

Memberlist Configuration:

memberlist:
  bind_addr: 0.0.0.0
  bind_port: 7946
  join:
    - dnssrv+pyroscope-memberlist._tcp.pyroscope.svc.cluster.local
  gossip_interval: 1s
  gossip_nodes: 3
  retransmit_factor: 4

Ring Types:

• Ingester Ring - Distributes profiles across ingesters
• Store-Gateway Ring - Assigns blocks to store-gateways
• Compactor Ring - Coordinates compaction work

Shuffle Sharding

Limits which components handle specific tenants:

# Ingesters per tenant
distributor:
  ingestion_tenant_shard_size: 4

# Queriers per tenant
query_frontend:
  max_queriers_per_tenant: 2

# Store-gateways per tenant
store_gateway:
  tenant_shard_size: 3

# Compactors per tenant
compactor:
  compactor_tenant_shard_size: 1

Benefits:

• Failure isolation between tenants
• Resource fairness
• Reduced blast radius

Scaling Guidelines

Horizontal Scaling

Vertical Scaling

Production Replica Counts

Data Flow Details

Write Path

1. SDK/Alloy pushes profile to Distributor
2. Distributor validates and hashes profile labels
3. Distributor identifies target ingesters via hash ring
4. Profile replicated to N ingesters (default: 3)
5. Ingester buffers profile in memory
6. After block duration, ingester flushes to object storage
7. Compactor merges and optimizes blocks

Read Path

1. Query arrives at Query Frontend
2. Query Frontend enqueues to Query Scheduler
3. Querier pulls query from scheduler
4. Querier determines time range:
   - Recent: Query ingesters
   - Historical: Query store-gateways
5. Store-gateways fetch blocks from object storage
6. Results aggregated and returned via Query Frontend

Consistency Model

Write Consistency:

• Quorum-based (n/2 + 1 replicas)
• Default: 2 of 3 ingesters must confirm

Read Consistency:

• Eventual consistency for recent data
• Strong consistency for flushed data

Data Durability:

• 3-way replication prevents data loss
• WAL recovery for crash scenarios

Deployment Decision Matrix

Pyroscope Helm Deployment Reference

Complete reference for deploying Grafana Pyroscope on Kubernetes via Helm.

Chart Information

Installation

Add Repository

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Single Binary Mode

kubectl create namespace pyroscope
helm install pyroscope grafana/pyroscope -n pyroscope

Microservices Mode

curl -Lo values-micro-services.yaml \
  https://raw.githubusercontent.com/grafana/pyroscope/main/operations/pyroscope/helm/pyroscope/values-micro-services.yaml

helm install pyroscope grafana/pyroscope \
  -n pyroscope \
  --values values-micro-services.yaml

Complete Values Reference

Root Level Values

# Replica count for single binary mode
replicaCount: 1

# Image configuration
image:
  repository: grafana/pyroscope
  pullPolicy: IfNotPresent
  tag: ""  # Defaults to chart.appVersion

# Image pull secrets
imagePullSecrets: []

# Pod service account
serviceAccount:
  create: true
  annotations: {}
  name: ""

# Pod annotations (profile scraping)
podAnnotations:
  profiles.grafana.com/cpu.scrape: "true"
  profiles.grafana.com/cpu.port: "4040"
  profiles.grafana.com/memory.scrape: "true"
  profiles.grafana.com/memory.port: "4040"
  profiles.grafana.com/goroutine.scrape: "true"
  profiles.grafana.com/goroutine.port: "4040"

# Pod security context
podSecurityContext:
  fsGroup: 10001
  runAsUser: 10001
  runAsNonRoot: true

# Container security context
securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL
  readOnlyRootFilesystem: true

Architecture Configuration

architecture:
  # Enable microservices mode
  microservices:
    enabled: false    # Set true for production

  # Deploy unified services (compatible endpoints)
  deployUnifiedServices: false

  # Storage architecture
  storage:
    v1: true          # Legacy storage layer
    v2: false         # Modern segment-writer storage
    migration:
      ingesterWeight: 0        # Traffic split (0-1)
      segmentWriterWeight: 0   # V2 write path

Service Configuration

service:
  type: ClusterIP
  port: 4040
  portName: http2
  annotations: {}

  # Additional ports
  grpc:
    port: 9095
  memberlist:
    port: 7946
  metastore:
    port: 9099

Ingress Configuration

ingress:
  enabled: false
  className: ""
  annotations: {}
  hosts:
    - host: pyroscope.local
      paths:
        - path: /
          pathType: Prefix
  tls: []

Persistence Configuration

pyroscope:
  persistence:
    enabled: false
    size: 10Gi
    accessModes:
      - ReadWriteOnce
    storageClassName: ""
    annotations: {}
    subPath: ""

  # Disable self-profiling
  disableSelfProfile: true

  # Cluster domain
  cluster_domain: .cluster.local

  # Custom config (merged with defaults)
  config: {}

  # Extra arguments
  extraArgs: []

  # Environment variables
  env: []

  # Volume mounts
  volumeMounts: []

Resource Configuration

resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 1
    memory: 2Gi

Pod Disruption Budget

podDisruptionBudget:
  enabled: true
  maxUnavailable: 1

Autoscaling

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 10

Node Scheduling

nodeSelector: {}
affinity: {}
tolerations: []
topologySpreadConstraints: []

Monitoring

serviceMonitor:
  enabled: false
  interval: 15s
  scrapeTimeout: 10s

podMonitor:
  enabled: false

Microservices Mode Values

Querier

querier:
  replicas: 3
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: 1
      memory: 1Gi

Query Frontend

queryFrontend:
  replicas: 2
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: 100m
      memory: 1Gi

Query Scheduler

queryScheduler:
  replicas: 2
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: 100m
      memory: 1Gi

Distributor

distributor:
  replicas: 2
  resources:
    requests:
      cpu: 500m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 1Gi

Ingester (StatefulSet)

ingester:
  replicas: 3
  resources:
    requests:
      cpu: 1
      memory: 8Gi
    limits:
      cpu: 1
      memory: 16Gi
  terminationGracePeriodSeconds: 600  # 10 minutes
  persistence:
    enabled: true
    size: 50Gi

Compactor (StatefulSet)

compactor:
  replicas: 3
  resources:
    requests:
      cpu: 1
      memory: 8Gi
    limits:
      cpu: 1
      memory: 16Gi
  terminationGracePeriodSeconds: 1200  # 20 minutes

Store Gateway (StatefulSet)

storeGateway:
  replicas: 3
  shardSize: 3  # Replication factor
  resources:
    requests:
      cpu: 1
      memory: 8Gi
    limits:
      cpu: 1
      memory: 16Gi
  readinessProbe:
    initialDelaySeconds: 60

Storage Backend Configuration

AWS S3

pyroscope:
  config:
    storage:
      backend: s3
      s3:
        bucket_name: pyroscope-data
        region: us-east-1
        endpoint: s3.us-east-1.amazonaws.com
        access_key_id: ${AWS_ACCESS_KEY_ID}
        secret_access_key: ${AWS_SECRET_ACCESS_KEY}
        # Optional
        s3ForcePathStyle: false
        insecure: false

Azure Blob Storage

pyroscope:
  config:
    storage:
      backend: azure
      azure:
        container_name: pyroscope-data
        account_name: mystorageaccount
        # Option 1: Account Key
        account_key: ${AZURE_ACCOUNT_KEY}
        # Option 2: Managed Identity
        # useManagedIdentity: true
        # userAssignedId: <client-id>
        request_timeout: 30s

Google Cloud Storage

pyroscope:
  config:
    storage:
      backend: gcs
      gcs:
        bucket_name: pyroscope-data
        # Uses GOOGLE_APPLICATION_CREDENTIALS

MinIO (Development)

minio:
  enabled: true
  rootUser: minioadmin
  rootPassword: minioadmin
  buckets:
    - name: grafana-pyroscope-data
  persistence:
    enabled: true
    size: 50Gi

pyroscope:
  config:
    storage:
      backend: s3
      s3:
        bucket_name: grafana-pyroscope-data
        endpoint: pyroscope-minio:9000
        access_key_id: minioadmin
        secret_access_key: minioadmin
        insecure: true

Dependency Charts

Grafana Alloy (Profile Collection)

alloy:
  enabled: true
  controller:
    type: statefulset
    replicas: 1
  alloy:
    configMap:
      content: |
        pyroscope.scrape "default" {
          targets = discovery.kubernetes.pods.targets
          forward_to = [pyroscope.write.default.receiver]
        }
        pyroscope.write "default" {
          endpoint {
            url = "http://pyroscope:4040"
          }
        }

Grafana Agent (Alternative)

grafana-agent:
  enabled: false

Production Values Example

# values-production.yaml
architecture:
  microservices:
    enabled: true

# Object storage
pyroscope:
  config:
    storage:
      backend: s3
      s3:
        bucket_name: pyroscope-prod
        region: us-east-1

# High availability replicas
querier:
  replicas: 5
  resources:
    limits:
      cpu: 2
      memory: 2Gi

distributor:
  replicas: 3
  resources:
    limits:
      cpu: 1
      memory: 1Gi

ingester:
  replicas: 5
  resources:
    limits:
      cpu: 2
      memory: 16Gi
  persistence:
    enabled: true
    size: 100Gi
  terminationGracePeriodSeconds: 600

compactor:
  replicas: 3
  resources:
    limits:
      cpu: 2
      memory: 16Gi
  terminationGracePeriodSeconds: 1200

storeGateway:
  replicas: 3
  resources:
    limits:
      cpu: 2
      memory: 16Gi

queryFrontend:
  replicas: 2

queryScheduler:
  replicas: 2

# Pod disruption
podDisruptionBudget:
  enabled: true
  maxUnavailable: 1

# Topology spread
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: pyroscope

# Monitoring
serviceMonitor:
  enabled: true
  interval: 30s

# Alloy for collection
alloy:
  enabled: true
  controller:
    type: daemonset

Development Values Example

# values-dev.yaml
replicaCount: 1

image:
  tag: "1.16.0"

resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 1
    memory: 2Gi

pyroscope:
  persistence:
    enabled: true
    size: 5Gi

architecture:
  microservices:
    enabled: false

minio:
  enabled: true
  persistence:
    size: 10Gi

alloy:
  enabled: true

Grafana Data Source Configuration

# ConfigMap for Grafana provisioning
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  labels:
    grafana_datasource: "1"
data:
  pyroscope-ds.yaml: |
    apiVersion: 1
    datasources:
      - name: Pyroscope
        type: grafana-pyroscope-datasource
        access: proxy
        url: http://pyroscope-querier.pyroscope.svc.cluster.local:4040
        isDefault: false
        editable: true

Upgrade Commands

# Update repository
helm repo update grafana

# Upgrade release
helm upgrade pyroscope grafana/pyroscope \
  -n pyroscope \
  --values values.yaml

# Rollback if needed
helm rollback pyroscope 1 -n pyroscope

# View release history
helm history pyroscope -n pyroscope

Validation Commands

# Verify pods
kubectl get pods -n pyroscope

# Check services
kubectl get svc -n pyroscope

# View logs
kubectl logs -n pyroscope -l app.kubernetes.io/name=pyroscope --tail=100

# Test endpoint
kubectl run test-pyroscope --image=curlimages/curl:latest --rm -it -- \
  curl -v http://pyroscope.pyroscope:4040/ready

Pyroscope SDK Instrumentation Reference

Complete guide for instrumenting applications with Pyroscope SDKs and Grafana Alloy.

Client Configuration Methods

Method 1: SDK Instrumentation (Push Mode)

Applications directly push profiles to Pyroscope server.

Application with SDK → Pyroscope Server (port 4040)

Method 2: Grafana Alloy (Pull Mode)

Alloy scrapes pprof endpoints from applications.

Application ← Grafana Alloy (collector) → Pyroscope Server

Method 3: Hybrid Mode

SDKs send to local Alloy, which forwards to Pyroscope.

Application with SDK → Alloy → Pyroscope Server

Profile Types

Language SDKs

Go SDK

Installation:

go get github.com/grafana/pyroscope-go

Basic Usage:

package main

import (
    "github.com/grafana/pyroscope-go"
)

func main() {
    // Start profiling
    pyroscope.Start(pyroscope.Config{
        ApplicationName: "my-app",
        ServerAddress:   "http://pyroscope:4040",
        Logger:          pyroscope.StandardLogger,

        // Tags for filtering
        Tags: map[string]string{
            "env":     "production",
            "version": "1.0.0",
        },

        // Profile types to collect
        ProfileTypes: []pyroscope.ProfileType{
            pyroscope.ProfileCPU,
            pyroscope.ProfileAllocObjects,
            pyroscope.ProfileAllocSpace,
            pyroscope.ProfileInuseObjects,
            pyroscope.ProfileInuseSpace,
            pyroscope.ProfileGoroutines,
            pyroscope.ProfileMutexCount,
            pyroscope.ProfileMutexDuration,
            pyroscope.ProfileBlockCount,
            pyroscope.ProfileBlockDuration,
        },
    })
    defer pyroscope.Stop()

    // Application code
}

With Basic Authentication:

pyroscope.Start(pyroscope.Config{
    ApplicationName: "my-app",
    ServerAddress:   "http://pyroscope:4040",
    BasicAuthUser:   "user",
    BasicAuthPassword: "password",
})

With Tenant ID (Multi-tenant):

pyroscope.Start(pyroscope.Config{
    ApplicationName: "my-app",
    ServerAddress:   "http://pyroscope:4040",
    TenantID:        "my-tenant",
})

Dynamic Tags:

// Add dynamic tags for specific code sections
pyroscope.TagWrapper(context.Background(), pyroscope.Labels(
    "controller", "OrderController",
    "method", "CreateOrder",
), func(ctx context.Context) {
    // Profiled code with these tags
    processOrder(ctx)
})

OpenTelemetry Integration (Span Profiles):

import (
    "github.com/grafana/pyroscope-go"
    otelpyroscope "github.com/grafana/otel-profiling-go"
    "go.opentelemetry.io/otel/sdk/trace"
)

// Configure tracer with span profiling
tp := trace.NewTracerProvider(
    trace.WithSpanProcessor(otelpyroscope.NewSpanProcessor()),
    // ... other options
)

Java SDK

Maven Dependency:

<dependency>
    <groupId>io.pyroscope</groupId>
    <artifactId>agent</artifactId>
    <version>0.12.0</version>
</dependency>

Gradle:

implementation 'io.pyroscope:agent:0.12.0'

Programmatic Configuration:

import io.pyroscope.javaagent.PyroscopeAgent;
import io.pyroscope.javaagent.config.Config;
import io.pyroscope.javaagent.EventType;
import io.pyroscope.http.Format;

public class Application {
    public static void main(String[] args) {
        PyroscopeAgent.start(
            new Config.Builder()
                .setApplicationName("my-java-app")
                .setServerAddress("http://pyroscope:4040")
                .setProfilingEvent(EventType.ITIMER)  // CPU profiling
                .setProfilingAlloc("512k")            // Memory profiling
                .setProfilingLock("10ms")             // Lock profiling
                .setFormat(Format.JFR)
                .setLabels(Map.of(
                    "env", "production",
                    "version", "1.0.0"
                ))
                .build()
        );

        // Application code
    }
}

Java Agent (JVM Argument):

java -javaagent:pyroscope.jar \
  -Dpyroscope.application.name=my-app \
  -Dpyroscope.server.address=http://pyroscope:4040 \
  -Dpyroscope.profiling.event=itimer \
  -Dpyroscope.profiling.alloc=512k \
  -Dpyroscope.profiling.lock=10ms \
  -jar myapp.jar

Spring Boot Integration:

# application.yml
pyroscope:
  application-name: my-spring-app
  server-address: http://pyroscope:4040
  profiling-event: itimer
  labels:
    env: ${ENVIRONMENT:development}

Dynamic Labels:

import io.pyroscope.javaagent.api.Pyroscope;

public void processRequest(String userId) {
    Pyroscope.LabelsWrapper.run(
        new LabelsSet("user_id", userId),
        () -> {
            // Code profiled with user_id label
            doWork();
        }
    );
}

Python SDK

Installation:

pip install pyroscope-io

Basic Usage:

import pyroscope

pyroscope.configure(
    application_name="my-python-app",
    server_address="http://pyroscope:4040",
    tags={
        "env": "production",
        "version": "1.0.0",
    },
)

# Application code

With Authentication:

pyroscope.configure(
    application_name="my-python-app",
    server_address="http://pyroscope:4040",
    basic_auth_username="user",
    basic_auth_password="password",
)

Multi-tenant:

pyroscope.configure(
    application_name="my-python-app",
    server_address="http://pyroscope:4040",
    tenant_id="my-tenant",
)

Django Integration:

# settings.py
import pyroscope

pyroscope.configure(
    application_name="my-django-app",
    server_address="http://pyroscope:4040",
    tags={
        "env": os.environ.get("ENVIRONMENT", "development"),
    },
)

Flask Integration:

from flask import Flask
import pyroscope

app = Flask(__name__)

pyroscope.configure(
    application_name="my-flask-app",
    server_address="http://pyroscope:4040",
)

@app.route("/")
def index():
    return "Hello World"

Dynamic Tags:

with pyroscope.tag_wrapper({"endpoint": "/api/users", "method": "GET"}):
    # Code profiled with these tags
    process_request()

.NET SDK

NuGet Package:

dotnet add package Pyroscope

Basic Usage:

using Pyroscope;

public class Program
{
    public static void Main(string[] args)
    {
        Pyroscope.Profiler.Instance.Configure(new Configuration
        {
            ApplicationName = "my-dotnet-app",
            ServerAddress = "http://pyroscope:4040",
            Labels = new Dictionary<string, string>
            {
                {"env", "production"},
                {"version", "1.0.0"}
            }
        });

        Pyroscope.Profiler.Instance.Start();

        // Application code

        Pyroscope.Profiler.Instance.Stop();
    }
}

ASP.NET Core Integration:

// Program.cs
var builder = WebApplication.CreateBuilder(args);

// Configure Pyroscope
Pyroscope.Profiler.Instance.Configure(new Configuration
{
    ApplicationName = "my-aspnet-app",
    ServerAddress = "http://pyroscope:4040",
});
Pyroscope.Profiler.Instance.Start();

var app = builder.Build();
// ... rest of app

Ruby SDK

Gemfile:

gem 'pyroscope'

Basic Usage:

require 'pyroscope'

Pyroscope.configure do |config|
  config.application_name = 'my-ruby-app'
  config.server_address = 'http://pyroscope:4040'
  config.tags = {
    env: 'production',
    version: '1.0.0'
  }
end

# Application code

Rails Integration:

# config/initializers/pyroscope.rb
Pyroscope.configure do |config|
  config.application_name = 'my-rails-app'
  config.server_address = ENV.fetch('PYROSCOPE_SERVER', 'http://pyroscope:4040')
  config.tags = {
    env: Rails.env,
    version: ENV.fetch('APP_VERSION', 'unknown')
  }
end

Node.js SDK

Installation:

npm install @pyroscope/nodejs

Basic Usage:

const Pyroscope = require('@pyroscope/nodejs');

Pyroscope.init({
  serverAddress: 'http://pyroscope:4040',
  appName: 'my-nodejs-app',
  tags: {
    env: 'production',
    version: '1.0.0'
  }
});

Pyroscope.start();

// Application code

Express Integration:

const express = require('express');
const Pyroscope = require('@pyroscope/nodejs');

Pyroscope.init({
  serverAddress: 'http://pyroscope:4040',
  appName: 'my-express-app',
});
Pyroscope.start();

const app = express();

app.get('/', (req, res) => {
  res.send('Hello World');
});

app.listen(3000);

Rust SDK

Cargo.toml:

[dependencies]
pyroscope = "0.5"
pyroscope_pprofrs = "0.2"

Basic Usage:

use pyroscope::PyroscopeAgent;
use pyroscope_pprofrs::{pprof_backend, PprofConfig};

fn main() {
    let agent = PyroscopeAgent::builder(
        "http://pyroscope:4040",
        "my-rust-app"
    )
    .backend(pprof_backend(PprofConfig::new().sample_rate(100)))
    .tags([("env", "production")].to_vec())
    .build()
    .unwrap();

    agent.start();

    // Application code

    agent.stop();
}

Grafana Alloy Configuration

eBPF Profiling (Linux)

For compiled languages without code modification:

// discovery.kubernetes.pods for target discovery
discovery.kubernetes "pods" {
  role = "pod"
}

// eBPF profiler
pyroscope.ebpf "default" {
  forward_to = [pyroscope.write.default.receiver]

  targets = discovery.kubernetes.pods.targets

  demangle = "none"
}

// Write to Pyroscope
pyroscope.write "default" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

Supported Languages (eBPF):

• Go (with frame pointers)
• C/C++ (with frame pointers)
• Rust (with frame pointers)
• Python (with python_enabled=true)

Pull Mode (pprof Scraping)

For Go applications with pprof endpoints:

// Scrape pprof endpoints
pyroscope.scrape "default" {
  targets = [
    {"__address__" = "my-app:6060", "service_name" = "my-app"},
  ]

  forward_to = [pyroscope.write.default.receiver]

  profiling_config {
    profile.process_cpu {
      enabled = true
    }
    profile.memory {
      enabled = true
      path = "/debug/pprof/allocs"
    }
    profile.goroutine {
      enabled = true
    }
    profile.mutex {
      enabled = true
    }
    profile.block {
      enabled = true
    }
  }
}

pyroscope.write "default" {
  endpoint {
    url = "http://pyroscope:4040"
  }
}

Java Pull Mode

pyroscope.java "default" {
  forward_to = [pyroscope.write.default.receiver]

  targets = discovery.kubernetes.pods.targets
}

Kubernetes Discovery with Annotations

// Discover pods with profile annotations
discovery.kubernetes "pods" {
  role = "pod"
}

// Relabel for annotation-based scraping
discovery.relabel "pods" {
  targets = discovery.kubernetes.pods.targets

  rule {
    source_labels = ["__meta_kubernetes_pod_annotation_profiles_grafana_com_cpu_scrape"]
    action = "keep"
    regex = "true"
  }

  rule {
    source_labels = ["__meta_kubernetes_pod_annotation_profiles_grafana_com_cpu_port"]
    target_label = "__address__"
    regex = "(\\d+)"
    replacement = "${1}"
  }
}

pyroscope.scrape "kubernetes" {
  targets = discovery.relabel.pods.output
  forward_to = [pyroscope.write.default.receiver]
}

Kubernetes Pod Annotations

Enable profile scraping with annotations:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    metadata:
      annotations:
        # CPU profiling
        profiles.grafana.com/cpu.scrape: "true"
        profiles.grafana.com/cpu.port: "8080"
        profiles.grafana.com/cpu.path: "/debug/pprof/profile"

        # Memory profiling
        profiles.grafana.com/memory.scrape: "true"
        profiles.grafana.com/memory.port: "8080"
        profiles.grafana.com/memory.path: "/debug/pprof/allocs"

        # Goroutine profiling
        profiles.grafana.com/goroutine.scrape: "true"
        profiles.grafana.com/goroutine.port: "8080"

        # Block profiling
        profiles.grafana.com/block.scrape: "true"
        profiles.grafana.com/block.port: "8080"

        # Mutex profiling
        profiles.grafana.com/mutex.scrape: "true"
        profiles.grafana.com/mutex.port: "8080"

Annotation Parameters:

Trace-to-Profile Linking

Requirements

• Minimum span duration: 20ms
• CPU profile type only
• Supported: Go, Java, .NET, Python, Ruby

Go + OpenTelemetry

import (
    "github.com/grafana/pyroscope-go"
    otelpyroscope "github.com/grafana/otel-profiling-go"
    "go.opentelemetry.io/otel/sdk/trace"
)

func main() {
    // Start Pyroscope
    pyroscope.Start(pyroscope.Config{
        ApplicationName: "my-app",
        ServerAddress:   "http://pyroscope:4040",
    })

    // Configure tracer with span profiling
    tp := trace.NewTracerProvider(
        trace.WithSpanProcessor(otelpyroscope.NewSpanProcessor()),
    )
    otel.SetTracerProvider(tp)
}

Java + OpenTelemetry

// Use async-profiler event type for span profiles
PyroscopeAgent.start(
    new Config.Builder()
        .setApplicationName("my-java-app")
        .setServerAddress("http://pyroscope:4040")
        .setProfilingEvent(EventType.ITIMER)
        .build()
);

// OpenTelemetry SDK will automatically correlate

Python + OpenTelemetry

import pyroscope
from opentelemetry import trace

pyroscope.configure(
    application_name="my-python-app",
    server_address="http://pyroscope:4040",
)

# Traces will be automatically correlated with profiles

AWS Lambda Extension

Configuration

# Environment variables
PYROSCOPE_SERVER_ADDRESS: "http://pyroscope:4040"
PYROSCOPE_APPLICATION_NAME: "my-lambda"
PYROSCOPE_TAGS: "env=production,region=us-east-1"

Lambda Layer

# serverless.yml
functions:
  myFunction:
    handler: handler.main
    layers:
      - arn:aws:lambda:us-east-1:123456789012:layer:pyroscope:1
    environment:
      PYROSCOPE_SERVER_ADDRESS: http://pyroscope:4040
      PYROSCOPE_APPLICATION_NAME: my-lambda

Best Practices

Naming Conventions

# Application name format
{service-name}.{component}

# Examples
api-gateway.http
order-service.worker
payment-service.processor

Tag Strategy

// Recommended tags
Tags: map[string]string{
    "env":        "production",      // Environment
    "version":    "1.2.3",           // App version
    "region":     "us-east-1",       // Deployment region
    "instance":   os.Getenv("POD_NAME"), // Instance ID
}

Resource Overhead

• CPU overhead: ~2-5%
• Memory overhead: ~50MB per pod
• Network: Profiles sent every 15 seconds
• Crash safety: Never crashes app if backend unavailable

Sampling Configuration

// Reduce overhead with sampling
pyroscope.Start(pyroscope.Config{
    ApplicationName: "my-app",
    ServerAddress:   "http://pyroscope:4040",
    // Default: 100Hz sampling rate
})

Language Support Matrix

Pyroscope Troubleshooting Reference

Complete guide for diagnosing and resolving common Pyroscope issues.

Diagnostic Commands

Pod Status

# Check all Pyroscope pods
kubectl get pods -n pyroscope -l app.kubernetes.io/name=pyroscope

# Describe specific pod
kubectl describe pod pyroscope-ingester-0 -n pyroscope

# Check pod events
kubectl get events -n pyroscope --sort-by='.lastTimestamp'

Logs

# All components
kubectl logs -n pyroscope -l app.kubernetes.io/name=pyroscope --tail=100

# Specific component
kubectl logs -n pyroscope -l app.kubernetes.io/component=ingester --tail=200
kubectl logs -n pyroscope -l app.kubernetes.io/component=distributor --tail=200
kubectl logs -n pyroscope -l app.kubernetes.io/component=querier --tail=200
kubectl logs -n pyroscope -l app.kubernetes.io/component=compactor --tail=200

# Follow logs
kubectl logs -f -n pyroscope -l app.kubernetes.io/component=ingester

# Previous container logs (after restart)
kubectl logs -n pyroscope pyroscope-ingester-0 --previous

Health Checks

# Readiness
kubectl exec -it pyroscope-0 -n pyroscope -- curl http://localhost:4040/ready

# Ring status (ingesters)
kubectl exec -it pyroscope-0 -n pyroscope -- curl http://localhost:4040/ingester/ring

# Configuration
kubectl exec -it pyroscope-0 -n pyroscope -- curl http://localhost:4040/config

# Metrics
kubectl exec -it pyroscope-0 -n pyroscope -- curl http://localhost:4040/metrics

Connectivity Tests

# Test from within cluster
kubectl run test-pyroscope --image=curlimages/curl:latest --rm -it -- \
  curl -v http://pyroscope.pyroscope:4040/ready

# Test push endpoint
kubectl run test-push --image=curlimages/curl:latest --rm -it -- \
  curl -X POST http://pyroscope.pyroscope:4040/ingest \
  -H "Content-Type: application/json" \
  -d '{"name":"test"}'

# Test from SDK perspective
kubectl run test-sdk --image=curlimages/curl:latest --rm -it -- \
  curl -v http://pyroscope-distributor.pyroscope:4040/ready

Common Issues

1. Ingester Out of Memory (OOM)

Symptoms:

• Pod restarts with OOMKilled status
• Memory usage spikes before crash
• Ingester logs show memory pressure

Diagnosis:

# Check OOM status
kubectl describe pod pyroscope-ingester-0 -n pyroscope | grep -A5 "State:"

# Check memory usage
kubectl top pods -n pyroscope -l app.kubernetes.io/component=ingester

Solutions:

# Increase memory limits
ingester:
  resources:
    limits:
      memory: 16Gi
    requests:
      memory: 8Gi

# Reduce batch size
pyroscope:
  config:
    ingester:
      max_block_duration: 30m  # Flush more frequently

2. Storage Authentication Failed

Symptoms:

• Ingester/Compactor failing to start
• "access denied" or "unauthorized" in logs
• Storage-related error messages

AWS S3:

# Verify IAM role
aws sts get-caller-identity

# Check bucket policy
aws s3 ls s3://pyroscope-bucket/

# Verify environment variables
kubectl exec -it pyroscope-ingester-0 -n pyroscope -- env | grep AWS

Azure Blob:

# Verify managed identity
az identity show --name pyroscope-identity --resource-group <rg>

# Check RBAC assignment
az role assignment list --scope /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<storage>

# Assign if missing
az role assignment create \
  --role "Storage Blob Data Contributor" \
  --assignee-object-id <principal-id> \
  --scope <storage-scope>

# Create containers if missing
az storage container create --name pyroscope-data --account-name <storage>

GCS:

# Verify service account
kubectl get secret pyroscope-gcs-sa -n pyroscope -o yaml

# Check IAM binding
gcloud projects get-iam-policy <project> --filter="bindings.members:serviceAccount:*"

3. Profiles Not Appearing

Symptoms:

• No data in Grafana Pyroscope datasource
• SDKs appear to send successfully
• No errors in SDK logs

Diagnosis:

# Check distributor is receiving data
kubectl logs -n pyroscope -l app.kubernetes.io/component=distributor --tail=100 | grep -i "push"

# Check ingester is receiving from distributor
kubectl logs -n pyroscope -l app.kubernetes.io/component=ingester --tail=100 | grep -i "append"

# Verify ring membership
kubectl exec -it pyroscope-0 -n pyroscope -- curl http://localhost:4040/ingester/ring | jq '.shards'

Solutions:

# Verify SDK configuration
pyroscope.Start(pyroscope.Config{
    ApplicationName: "my-app",       # Must be set
    ServerAddress:   "http://pyroscope:4040", # Verify URL
    Logger:          pyroscope.StandardLogger, # Enable logging
})

# Check network policy allows traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-pyroscope-ingestion
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: pyroscope
  ingress:
    - from: []
      ports:
        - port: 4040

4. Query Timeouts

Symptoms:

• Queries hang or timeout
• "context deadline exceeded" errors
• Slow flame graph rendering

Diagnosis:

# Check querier logs
kubectl logs -n pyroscope -l app.kubernetes.io/component=querier --tail=100 | grep -i "timeout\|slow"

# Check query-frontend logs
kubectl logs -n pyroscope -l app.kubernetes.io/component=query-frontend --tail=100

# Check store-gateway for historical queries
kubectl logs -n pyroscope -l app.kubernetes.io/component=store-gateway --tail=100

Solutions:

# Increase query timeout
pyroscope:
  config:
    querier:
      query_timeout: 5m
      max_concurrent: 8

    query_scheduler:
      max_outstanding_requests_per_tenant: 2048

# Scale queriers
querier:
  replicas: 5

5. High Cardinality Labels

Symptoms:

• Memory usage increases rapidly
• Slow queries
• "too many labels" errors

Diagnosis:

# Check unique label combinations
kubectl exec -it pyroscope-0 -n pyroscope -- \
  curl "http://localhost:4040/querier.v1.QuerierService/Series" \
  -H "Content-Type: application/json" \
  -d '{"matchers":[]}'

Solutions:

# Limit label cardinality
pyroscope:
  config:
    validation:
      max_label_names_per_series: 25
      max_label_name_length: 1024
      max_label_value_length: 2048

6. Ring Unhealthy

Symptoms:

• Ingesters showing as UNHEALTHY
• "ring not ready" errors
• Failed writes

Diagnosis:

# Check ring status
kubectl exec -it pyroscope-0 -n pyroscope -- curl http://localhost:4040/ingester/ring

# Check memberlist connectivity
kubectl exec -it pyroscope-0 -n pyroscope -- curl http://localhost:4040/memberlist

# Verify DNS resolution
kubectl exec -it pyroscope-0 -n pyroscope -- nslookup pyroscope-memberlist

Solutions:

# Verify memberlist configuration
pyroscope:
  config:
    memberlist:
      bind_port: 7946
      join:
        - dnssrv+pyroscope-memberlist._tcp.pyroscope.svc.cluster.local

# Check headless service exists
apiVersion: v1
kind: Service
metadata:
  name: pyroscope-memberlist
spec:
  clusterIP: None
  selector:
    app.kubernetes.io/name: pyroscope
  ports:
    - name: memberlist
      port: 7946
      protocol: UDP

7. Compactor Not Running

Symptoms:

• Blocks not being compacted
• Storage usage growing rapidly
• Old data not being deleted

Diagnosis:

# Check compactor logs
kubectl logs -n pyroscope -l app.kubernetes.io/component=compactor --tail=200

# Check compactor ring
kubectl exec -it pyroscope-compactor-0 -n pyroscope -- \
  curl http://localhost:4040/compactor/ring

Solutions:

# Verify compactor configuration
pyroscope:
  config:
    compactor:
      compaction_interval: 30m
      retention_enabled: true
      retention_delete_delay: 2h
      compaction_concurrency: 4

8. SDK Connection Refused

Symptoms:

• "connection refused" in application logs
• Profiles not being sent
• SDK timeout errors

Diagnosis:

# Test connectivity from application pod
kubectl exec -it <app-pod> -n <app-ns> -- \
  curl -v http://pyroscope.pyroscope:4040/ready

# Check service exists
kubectl get svc -n pyroscope pyroscope

Solutions:

# Verify service DNS
kubectl run test-dns --image=busybox --rm -it -- nslookup pyroscope.pyroscope.svc.cluster.local

# Check endpoint
kubectl get endpoints -n pyroscope pyroscope

9. Alloy Not Scraping

Symptoms:

• No profiles from auto-instrumented apps
• Alloy running but not collecting data
• Annotations present but ignored

Diagnosis:

# Check Alloy logs
kubectl logs -n pyroscope -l app.kubernetes.io/name=alloy --tail=200

# Verify targets
kubectl exec -it <alloy-pod> -n pyroscope -- \
  wget -qO- http://localhost:12345/agent/api/v1/targets

Solutions:

# Verify pod annotations
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    metadata:
      annotations:
        profiles.grafana.com/cpu.scrape: "true"
        profiles.grafana.com/cpu.port: "8080"

# Check Alloy configuration
alloy:
  alloy:
    configMap:
      content: |
        discovery.kubernetes "pods" {
          role = "pod"
        }
        pyroscope.scrape "default" {
          targets = discovery.kubernetes.pods.targets
          forward_to = [pyroscope.write.default.receiver]
        }

10. Store-Gateway Slow Sync

Symptoms:

• Historical queries slow
• Store-gateway high CPU during sync
• "bucket sync" messages in logs

Diagnosis:

# Check sync status
kubectl logs -n pyroscope -l app.kubernetes.io/component=store-gateway --tail=200 | grep -i "sync"

# Check block count
kubectl exec -it pyroscope-store-gateway-0 -n pyroscope -- \
  curl http://localhost:4040/store-gateway/blocks

Solutions:

# Tune sync interval
pyroscope:
  config:
    store_gateway:
      sync_interval: 30m  # Increase if too frequent
      ignore_blocks_within: 6h  # Skip recent blocks

Performance Tuning

Ingestion Optimization

# High throughput configuration
pyroscope:
  config:
    distributor:
      push_timeout: 10s
      ingestion_tenant_shard_size: 8

    ingester:
      max_block_duration: 1h
      flush_check_period: 30s

Query Optimization

# Faster queries
pyroscope:
  config:
    querier:
      max_concurrent: 16
      query_store_after: 2h

    query_frontend:
      max_outstanding_per_tenant: 400

Storage Optimization

# Efficient storage
pyroscope:
  config:
    compactor:
      compaction_concurrency: 8
      block_ranges:
        - 1h
        - 4h
        - 12h

Metrics to Monitor

Key Metrics

Prometheus Rules

groups:
  - name: pyroscope
    rules:
      - alert: PyroscopeIngesterOOM
        expr: |
          container_memory_working_set_bytes{container="ingester"}
          / container_spec_memory_limit_bytes{container="ingester"} > 0.9
        for: 5m
        labels:
          severity: warning

      - alert: PyroscopeCompactorNotRunning
        expr: |
          increase(pyroscope_compactor_runs_completed_total[2h]) == 0
        for: 10m
        labels:
          severity: warning

      - alert: PyroscopeRingUnhealthy
        expr: |
          pyroscope_ring_members{state="UNHEALTHY"} > 0
        for: 5m
        labels:
          severity: critical

Log Analysis

Common Log Patterns

Successful ingestion:

level=info component=distributor msg="pushed profiles" tenant=anonymous count=100

Storage errors:

level=error component=ingester msg="failed to upload block" err="access denied"

Ring issues:

level=warn component=ring msg="instance not found in ring" instance=pyroscope-ingester-0

Query issues:

level=error component=querier msg="query failed" err="context deadline exceeded"

Grep Commands

# Find errors
kubectl logs -n pyroscope -l app.kubernetes.io/name=pyroscope | grep -i "error\|fail\|panic"

# Find storage issues
kubectl logs -n pyroscope -l app.kubernetes.io/component=ingester | grep -i "storage\|upload\|s3\|azure\|gcs"

# Find ring issues
kubectl logs -n pyroscope -l app.kubernetes.io/name=pyroscope | grep -i "ring\|memberlist"

# Find query issues
kubectl logs -n pyroscope -l app.kubernetes.io/component=querier | grep -i "query\|timeout"

Recovery Procedures

Restart Components

# Restart specific component
kubectl rollout restart deployment/pyroscope-distributor -n pyroscope
kubectl rollout restart statefulset/pyroscope-ingester -n pyroscope

# Restart all
kubectl rollout restart deployment -n pyroscope
kubectl rollout restart statefulset -n pyroscope

Clear Stuck State

# Delete PVC if corrupted (data loss!)
kubectl delete pvc pyroscope-ingester-data-pyroscope-ingester-0 -n pyroscope

# Force delete stuck pod
kubectl delete pod pyroscope-ingester-0 -n pyroscope --force --grace-period=0

Scale Down/Up

# Scale down
kubectl scale statefulset pyroscope-ingester --replicas=0 -n pyroscope

# Scale up
kubectl scale statefulset pyroscope-ingester --replicas=3 -n pyroscope