Promql Generator

PromQL Query Generator

Overview

This skill provides a comprehensive, interactive workflow for generating production-ready PromQL queries with best practices built-in. Generate queries for monitoring dashboards, alerting rules, and ad-hoc analysis with an emphasis on user collaboration and planning before code generation.

When to Use This Skill

Invoke this skill when:

• Creating new PromQL queries from scratch
• Building monitoring dashboards (Grafana, Prometheus UI, etc.)
• Implementing alerting rules for Prometheus Alertmanager
• Analyzing metrics for troubleshooting or capacity planning
• Converting monitoring requirements into PromQL expressions
• Learning PromQL or teaching others
• The user asks to "create", "generate", "build", or "write" PromQL queries
• Working with Prometheus metrics (counters, gauges, histograms, summaries)
• Implementing RED (Rate, Errors, Duration) or USE (Utilization, Saturation, Errors) metrics

Interactive Query Planning Workflow

CRITICAL: This skill emphasizes interactive planning before query generation. Always engage the user in a collaborative planning process to ensure the generated query matches their exact intentions.

Follow this workflow when generating PromQL queries:

Stage 1: Understand the Monitoring Goal

Start by understanding what the user wants to monitor or measure. Ask clarifying questions to gather requirements:

1. Primary Goal: What are you trying to monitor or measure?

• Request rate (requests per second)
• Error rate (percentage of failed requests)
• Latency/duration (response times, percentiles)
• Resource usage (CPU, memory, disk, network)
• Availability/uptime
• Queue depth, saturation, throughput
• Custom business metrics

2. Use Case: What will this query be used for?

• Dashboard visualization (Grafana, Prometheus UI)
• Alerting rule (firing when threshold exceeded)
• Ad-hoc troubleshooting/analysis
• Recording rule (pre-computed aggregation)
• Capacity planning or SLO tracking

3. Context: Any additional context?

• Service/application name
• Team or project
• Priority level
• Existing metrics or naming conventions

Use the AskUserQuestion tool to gather this information if not provided.

When to Ask vs. Infer: If the user's initial request already clearly specifies the goal, use case, and context (e.g., "Create an alert for P95 latency > 500ms for payment-service"), you may acknowledge these details in your response instead of re-asking. Only ask clarifying questions for information that is missing or ambiguous.

Stage 2: Identify Available Metrics

Determine which metrics are available and relevant:

1. Metric Discovery: What metrics are available?

• Ask the user for metric names
• If uncertain, suggest common naming patterns
• Check for metric type indicators in the name:
• _total suffix → Counter
• _bucket, _sum, _count suffix → Histogram
• No suffix → Likely Gauge
• _created suffix → Counter creation timestamp

2. Metric Type Identification: Confirm the metric type(s)

• Counter: Cumulative metric that only increases (or resets to zero)
• Examples: http_requests_total, errors_total, bytes_sent_total
• Use with: rate(), irate(), increase()
• Gauge: Point-in-time value that can go up or down
• Examples: memory_usage_bytes, cpu_temperature_celsius, queue_length
• Use with: avg_over_time(), min_over_time(), max_over_time(), or directly
• Histogram: Buckets of observations with cumulative counts
• Examples: http_request_duration_seconds_bucket, response_size_bytes_bucket
• Use with: histogram_quantile(), rate()
• Summary: Pre-calculated quantiles with count and sum
• Examples: rpc_duration_seconds{quantile="0.95"}
• Use _sum and _count for averages; don't average quantiles

3. Label Discovery: What labels are available on these metrics?

• Common labels: job, instance, environment, service, endpoint, status_code, method
• Ask which labels are important for filtering or grouping

Use the AskUserQuestion tool to confirm metric names, types, and available labels.

Stage 3: Determine Query Parameters

Gather specific requirements for the query.

Pre-confirmation for User-Provided Parameters

IMPORTANT: When the user has already specified parameters in their initial request (e.g., "5-minute window", "500ms threshold", "> 5% error rate"), you MUST:

1. Acknowledge the provided values explicitly in your response 2. Present them as pre-filled defaults in AskUserQuestion with the first option being "Use specified values" 3. Allow quick confirmation rather than re-asking for information already given

Example: If user says "alert when P95 latency exceeds 500ms", use:

AskUserQuestion:
- Question: "Confirm the alert threshold?"
- Options:
  1. "500ms (as specified)" - Use the threshold from your request
  2. "Different threshold" - Let me specify a different value

This respects the user's input and speeds up the workflow while still allowing modifications.

1. Time Range: What time window should the query cover?

• Instant value (current)
• Rate over time ([5m], [1h], [1d])
• For rate calculations: typically [1m] to [5m] for real-time, [1h] to [1d] for trends
• Rule of thumb: Rate range should be at least 4x the scrape interval

2. Label Filtering: Which labels should filter the data?

• Exact matches: job="api-server", status_code="200"
• Negative matches: status_code!="200"
• Regex matches: instance=~"prod-.*"
• Multiple conditions: {job="api", environment="production"}

3. Aggregation: Should the data be aggregated?

• No aggregation: Return all time series as-is
• Aggregate by labels: sum by (job, endpoint), avg by (instance)
• Aggregate without labels: sum without (instance, pod), avg without (job)
• Common aggregations: sum, avg, max, min, count, topk, bottomk

4. Thresholds or Conditions: Are there specific conditions?

• For alerting: threshold values (e.g., error rate > 5%)
• For filtering: only show series above/below a value
• For comparison: compare against historical data (offset)

Use the AskUserQuestion tool to gather or confirm these parameters. When the user has already provided values (e.g., "5-minute window", "> 5%"), present them as the default option for confirmation.

Stage 4: Present the Query Plan

BEFORE GENERATING ANY CODE, present a plain-English query plan and ask for user confirmation:

## PromQL Query Plan

Based on your requirements, here's what the query will do:

**Goal**: [Describe the monitoring goal in plain English]

**Query Structure**:
1. Start with metric: `[metric_name]`
2. Filter by labels: `{label1="value1", label2="value2"}`
3. Apply function: `[function_name]([metric][time_range])`
4. Aggregate: `[aggregation] by ([label_list])`
5. Additional operations: [any calculations, ratios, or transformations]

**Expected Output**:
- Data type: [instant vector/scalar]
- Labels in result: [list of labels]
- Value represents: [what the number means]
- Typical range: [expected value range]

**Example Interpretation**:
If the query returns `0.05`, it means: [plain English explanation]

**Does this match your intentions?**
- If yes, I'll generate the query and validate it
- If no, let me know what needs to change

Use the AskUserQuestion tool to confirm the plan with options:

• "Yes, generate this query"
• "Modify [specific aspect]"
• "Show me alternative approaches"

When the user chooses:

• "Modify [specific aspect]": ask one focused follow-up question about what to change (metric, labels, function, time range, threshold, or output shape), then present an updated plan before generating.
• "Show me alternative approaches": provide at least two valid query plans with trade-offs (accuracy, cost, cardinality, readability), then ask the user to choose one before generating.

Stage 5: Generate the PromQL Query

Once the user confirms the plan, generate the actual PromQL query following best practices.

IMPORTANT: Consult Reference Files Before Generating

Before writing any query code, you MUST:

1. Identify the query category first (histogram, RED, USE, function-specific, optimization, etc.).

2. Read only the relevant reference section(s) using the Read tool:

• For histogram queries → Read references/metric_types.md (Histogram section)
• For error/latency patterns → Read references/promql_patterns.md (RED method section)
• For resource monitoring → Read references/promql_patterns.md (USE method section)
• For optimization questions → Read references/best_practices.md
• For specific functions → Read references/promql_functions.md
• Re-read a section only if requirements changed or you have not consulted it yet in the current thread.

3. If a needed reference cannot be read, state the issue and continue with best-effort generation using the most applicable documented pattern you already have.

4. Cite the applicable pattern or best practice in your response:

   As documented in references/promql_patterns.md (Pattern 3: Latency Percentile):
   # 95th percentile latency
   histogram_quantile(0.95, sum by (le) (rate(...)))

5. Reference example files when generating similar queries:

   Based on examples/red_method.promql (lines 64-82):
   # P95 latency with proper histogram_quantile usage

This keeps generated queries aligned with documented patterns while avoiding unnecessary full-file rereads on iterative follow-ups.

Best Practices for Query Generation

1. Always Use Label Filters

   # Good: Specific filtering reduces cardinality
   rate(http_requests_total{job="api-server", environment="prod"}[5m])

   # Bad: Matches all time series, high cardinality
   rate(http_requests_total[5m])

2. Use Appropriate Functions for Metric Types

   # Counter: Use rate() or increase()
   rate(http_requests_total[5m])

   # Gauge: Use directly or with *_over_time()
   memory_usage_bytes
   avg_over_time(memory_usage_bytes[5m])

   # Histogram: Use histogram_quantile()
   histogram_quantile(0.95,
     sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
   )

3. Apply Aggregations with by() or without()

   # Aggregate by specific labels (keeps only these labels)
   sum by (job, endpoint) (rate(http_requests_total[5m]))

   # Aggregate without specific labels (removes these labels)
   sum without (instance, pod) (rate(http_requests_total[5m]))

4. Use Exact Matches Over Regex When Possible

   # Good: Faster exact match
   http_requests_total{status_code="200"}

   # Bad: Slower regex match when not needed
   http_requests_total{status_code=~"200"}

5. Calculate Ratios Properly

   # Error rate: errors / total requests
   sum(rate(http_requests_total{status_code=~"5.."}[5m]))
   /
   sum(rate(http_requests_total[5m]))

6. Use Recording Rules for Complex Queries

• If a query is used frequently or is computationally expensive
• Pre-aggregate data to reduce query load
• Follow naming convention: level:metric:operations

7. Format for Readability

   # Good: Multi-line for complex queries
   histogram_quantile(0.95,
     sum by (le, job) (
       rate(http_request_duration_seconds_bucket{job="api-server"}[5m])
     )
   )

Common Query Patterns

Pattern 1: Request Rate

# Requests per second
rate(http_requests_total{job="api-server"}[5m])

# Total requests per second across all instances
sum(rate(http_requests_total{job="api-server"}[5m]))

Pattern 2: Error Rate

# Error ratio (0 to 1)
sum(rate(http_requests_total{job="api-server", status_code=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api-server"}[5m]))

# Error percentage (0 to 100)
(
  sum(rate(http_requests_total{job="api-server", status_code=~"5.."}[5m]))
  /
  sum(rate(http_requests_total{job="api-server"}[5m]))
) * 100

Pattern 3: Latency Percentile (Histogram)

# 95th percentile latency
histogram_quantile(0.95,
  sum by (le) (
    rate(http_request_duration_seconds_bucket{job="api-server"}[5m])
  )
)

Pattern 4: Resource Usage

# Current memory usage
process_resident_memory_bytes{job="api-server"}

# Average CPU usage over 5 minutes
avg_over_time(process_cpu_seconds_total{job="api-server"}[5m])

Pattern 5: Availability

# Percentage of up instances
(
  count(up{job="api-server"} == 1)
  /
  count(up{job="api-server"})
) * 100

Pattern 6: Saturation/Queue Depth

# Average queue length
avg_over_time(queue_depth{job="worker"}[5m])

# Maximum queue depth in the last hour
max_over_time(queue_depth{job="worker"}[1h])

Stage 6: Validate the Generated Query

ALWAYS attempt to validate the generated query first using the devops-skills:promql-validator skill:

After generating the query, automatically invoke:
Skill(devops-skills:promql-validator)

The devops-skills:promql-validator skill will:
1. Check syntax correctness
2. Validate semantic logic (correct functions for metric types)
3. Identify anti-patterns and inefficiencies
4. Suggest optimizations
5. Explain what the query does
6. Verify it matches user intent

Validation checklist:

• Syntax is correct (balanced brackets, valid operators)
• Metric type matches function usage
• Label filters are specific enough
• Aggregation is appropriate
• Time ranges are reasonable
• No known anti-patterns
• Query is optimized for performance

If validation fails, fix issues and re-validate until all checks pass.

If the validator skill is unavailable, fails to run, or cannot complete after two fix/re-validate cycles:

• Report the validator failure briefly (tool unavailable, timeout, parsing error, etc.).
• Run a manual fallback check (syntax shape, metric/function compatibility, label filtering, aggregation, time range sanity).
• Mark any unchecked areas as UNVERIFIED and ask the user whether to proceed with best-effort output or provide more context for another validation attempt.

IMPORTANT: Display Validation Results to User

After running validation, you MUST display the structured results to the user in this format:

## PromQL Validation Results

### Syntax Check
- Status: ✅ VALID / ⚠️ WARNING / ❌ ERROR / ⚠️ UNVERIFIED
- Issues: [list any syntax errors]

### Best Practices Check
- Status: ✅ OPTIMIZED / ⚠️ CAN BE IMPROVED / ❌ HAS ISSUES / ⚠️ UNVERIFIED
- Issues: [list any problems found]
- Suggestions: [list optimization opportunities]

### Validation Coverage
- Validator tool run: [successful / failed / unavailable]
- Checks completed: [syntax, semantics, anti-patterns, performance, intent-match]
- Checks skipped: [list any skipped checks, or "None"]

### Query Explanation
- **What it measures**: [plain English description]
- **Output labels**: [list labels in result, or "None (scalar)"]
- **Expected result structure**: [instant vector / scalar / etc.]

This transparency helps users understand the validation process and any recommendations.

Stage 7: Provide Usage Instructions

After generation and validation (or manual fallback validation), provide the user with:

1. The Final Query:

   [Generated and validated PromQL query]

2. Query Explanation:

• What the query measures
• How to interpret the results
• Expected value range
• Labels in the output

3. How to Use It:

• For Dashboards: Copy into Grafana/Prometheus UI panel query
• For Alerts: Integrate into Alertmanager rule with threshold
• For Recording Rules: Add to Prometheus recording rule config
• For Ad-hoc: Run directly in Prometheus expression browser

4. Customization Notes:

• Time ranges that might need adjustment
• Labels to modify for different environments
• Threshold values to tune
• Alternative functions if requirements change

5. Related Queries:

• Suggest complementary queries
• Mention recording rule opportunities
• Recommend dashboard panels

Native Histograms (Prometheus 3.x+)

Native histograms are now stable in Prometheus 3.0+ (released November 2024). They offer significant advantages over classic histograms:

• Sparse bucket representation with near-zero cost for empty buckets
• No configuration of bucket boundaries during instrumentation
• Coverage of the full float64 range
• Efficient mergeability across histograms
• Simpler query syntax

Important: Starting with Prometheus v3.8.0, native histograms are fully stable. However, scraping native histograms still requires explicit activation via the scrape_native_histograms configuration setting. Starting with v3.9, no feature flag is needed but scrape_native_histograms must be set explicitly.

Native vs Classic Histogram Syntax

# Classic histogram (requires _bucket suffix and le label)
histogram_quantile(0.95,
  sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))
)

# Native histogram (simpler - no _bucket suffix, no le label needed)
histogram_quantile(0.95,
  sum by (job) (rate(http_request_duration_seconds[5m]))
)

Native Histogram Functions

# Get observation count rate from native histogram
histogram_count(rate(http_request_duration_seconds[5m]))

# Get sum of observations from native histogram
histogram_sum(rate(http_request_duration_seconds[5m]))

# Calculate fraction of observations between two values
histogram_fraction(0, 0.1, rate(http_request_duration_seconds[5m]))

# Average request duration from native histogram
histogram_sum(rate(http_request_duration_seconds[5m]))
/
histogram_count(rate(http_request_duration_seconds[5m]))

Detecting Native vs Classic Histograms

Native histograms are identified by:

• No `_bucket` suffix on the metric name
• No `le` label in the time series
• The metric stores histogram data directly (not separate bucket counters)

When querying, check if your Prometheus instance has native histograms enabled:

# prometheus.yml - Enable native histogram scraping
scrape_configs:
  - job_name: 'my-app'
    scrape_native_histogram: true  # Prometheus 3.x+

Custom Bucket Native Histograms (NHCB)

Prometheus 3.4+ supports custom bucket native histograms (schema -53), allowing classic histogram to native histogram conversion. This is a key migration path for users with existing classic histograms.

Benefits of NHCB:

• Keep existing instrumentation (no code changes needed)
• Store classic histograms as native histograms for lower costs
• Query with native histogram syntax
• Improved reliability and compression

Configuration (Prometheus 3.4+):

# prometheus.yml - Convert classic histograms to NHCB on scrape
global:
  scrape_configs:
    - job_name: 'my-app'
      convert_classic_histograms_to_nhcb: true  # Prometheus 3.4+

Querying NHCB:

# Query NHCB metrics the same way as native histograms
histogram_quantile(0.95, sum by (job) (rate(http_request_duration_seconds[5m])))

# histogram_fraction also works with NHCB (Prometheus 3.4+)
histogram_fraction(0, 0.2, rate(http_request_duration_seconds[5m]))

Note: Schema -53 indicates custom bucket boundaries. These histograms with different custom bucket boundaries are generally not mergeable with each other.

---

SLO, Error Budget, and Burn Rate Patterns

Service Level Objectives (SLOs) are critical for modern SRE practices. These patterns help implement SLO-based monitoring and alerting.

Error Budget Calculation

# Error budget remaining (for 99.9% SLO over 30 days)
# Returns value between 0 and 1 (1 = full budget, 0 = exhausted)
1 - (
  sum(rate(http_requests_total{job="api", status_code=~"5.."}[30d]))
  /
  sum(rate(http_requests_total{job="api"}[30d]))
) / 0.001  # 0.001 = 1 - 0.999 (allowed error rate)

# Simplified: Availability over 30 days
sum(rate(http_requests_total{job="api", status_code!~"5.."}[30d]))
/
sum(rate(http_requests_total{job="api"}[30d]))

Burn Rate Calculation

Burn rate measures how fast you're consuming error budget. A burn rate of 1 means you'll exhaust the budget exactly at the end of the SLO window.

# Current burn rate (1 hour window, 99.9% SLO)
# Burn rate = (current error rate) / (allowed error rate)
(
  sum(rate(http_requests_total{job="api", status_code=~"5.."}[1h]))
  /
  sum(rate(http_requests_total{job="api"}[1h]))
) / 0.001  # 0.001 = allowed error rate for 99.9% SLO

# Burn rate > 1 means consuming budget faster than allowed
# Burn rate of 14.4 consumes 2% of monthly budget in 1 hour

Multi-Window, Multi-Burn-Rate Alerts (Google SRE Standard)

The recommended approach for SLO alerting uses multiple windows to balance detection speed and precision:

# Page-level alert: 2% budget in 1 hour (burn rate 14.4)
# Long window (1h) AND short window (5m) must both exceed threshold
(
  (
    sum(rate(http_requests_total{job="api", status_code=~"5.."}[1h]))
    /
    sum(rate(http_requests_total{job="api"}[1h]))
  ) > 14.4 * 0.001
)
and
(
  (
    sum(rate(http_requests_total{job="api", status_code=~"5.."}[5m]))
    /
    sum(rate(http_requests_total{job="api"}[5m]))
  ) > 14.4 * 0.001
)

# Ticket-level alert: 5% budget in 6 hours (burn rate 6)
(
  (
    sum(rate(http_requests_total{job="api", status_code=~"5.."}[6h]))
    /
    sum(rate(http_requests_total{job="api"}[6h]))
  ) > 6 * 0.001
)
and
(
  (
    sum(rate(http_requests_total{job="api", status_code=~"5.."}[30m]))
    /
    sum(rate(http_requests_total{job="api"}[30m]))
  ) > 6 * 0.001
)

SLO Recording Rules

Pre-compute SLO metrics for efficient alerting:

# Recording rules for SLO calculations
groups:
  - name: slo_recording_rules
    interval: 30s
    rules:
      # Error ratio over different windows
      - record: job:slo_errors_per_request:ratio_rate1h
        expr: |
          sum by (job) (rate(http_requests_total{status_code=~"5.."}[1h]))
          /
          sum by (job) (rate(http_requests_total[1h]))

      - record: job:slo_errors_per_request:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{status_code=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))

      # Availability (success ratio)
      - record: job:slo_availability:ratio_rate1h
        expr: |
          1 - job:slo_errors_per_request:ratio_rate1h

Latency SLO Queries

# Percentage of requests faster than SLO target (200ms)
(
  sum(rate(http_request_duration_seconds_bucket{le="0.2", job="api"}[5m]))
  /
  sum(rate(http_request_duration_seconds_count{job="api"}[5m]))
) * 100

# Requests violating latency SLO (slower than 500ms)
(
  sum(rate(http_request_duration_seconds_count{job="api"}[5m]))
  -
  sum(rate(http_request_duration_seconds_bucket{le="0.5", job="api"}[5m]))
)
/
sum(rate(http_request_duration_seconds_count{job="api"}[5m]))

Burn Rate Reference Table

---

Advanced Query Techniques

Using Subqueries

Subqueries enable complex time-based calculations:

# Maximum 5-minute rate over the past 30 minutes
max_over_time(
  rate(http_requests_total[5m])[30m:1m]
)

Syntax: <query>[<range>:<resolution>]

• <range>: Time window to evaluate over
• <resolution>: Step size between evaluations

Using Offset Modifier

Compare current data with historical data:

# Compare current rate with rate from 1 week ago
rate(http_requests_total[5m])
-
rate(http_requests_total[5m] offset 1w)

Using @ Modifier

Query metrics at specific timestamps:

# Rate at the end of the range query
rate(http_requests_total[5m] @ end())

# Rate at specific Unix timestamp
rate(http_requests_total[5m] @ 1609459200)

Binary Operators and Vector Matching

Combine metrics with operators and control label matching:

# One-to-one matching (default)
metric_a + metric_b

# Many-to-one with group_left
rate(http_requests_total[5m])
* on (job, instance) group_left (version)
  app_version_info

# Ignoring specific labels
metric_a + ignoring(instance) metric_b

Logical Operators

Filter time series based on conditions:

# Return series only where value > 100
http_requests_total > 100

# Return series present in both
metric_a and metric_b

# Return series in A but not in B
metric_a unless metric_b

Documentation Lookup

If the user asks about specific Prometheus features, operators, or custom metrics:

1. Try context7 MCP first (preferred):

   Use mcp__context7__resolve-library-id with "prometheus"
   Then use mcp__context7__get-library-docs with:
   - context7CompatibleLibraryID: /prometheus/docs
   - topic: [specific feature, function, or operator]
   - page: 1 (fetch additional pages if needed)

2. Fallback to WebSearch:

   Search query pattern:
   "Prometheus PromQL [function/operator/feature] documentation [version] examples"

   Examples:
   "Prometheus PromQL rate function documentation examples"
   "Prometheus PromQL histogram_quantile documentation best practices"
   "Prometheus PromQL aggregation operators documentation"

Common Monitoring Scenarios

RED Method (for Request-Driven Services)

1. Rate: Request throughput

   sum(rate(http_requests_total{job="api"}[5m])) by (endpoint)

2. Errors: Error rate

   sum(rate(http_requests_total{job="api", status_code=~"5.."}[5m]))
   /
   sum(rate(http_requests_total{job="api"}[5m]))

3. Duration: Latency percentiles

   histogram_quantile(0.95,
     sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[5m]))
   )

USE Method (for Resources)

1. Utilization: Resource usage percentage

   (
     avg(rate(node_cpu_seconds_total{mode!="idle"}[5m]))
     /
     count(node_cpu_seconds_total{mode="idle"})
   ) * 100

2. Saturation: Queue depth or resource contention

   avg_over_time(node_load1[5m])

3. Errors: Error counters

   rate(node_network_receive_errs_total[5m])

Alerting Rules

When generating queries for alerting:

1. Include the Threshold: Make the condition explicit

   # Alert when error rate exceeds 5%
   (
     sum(rate(http_requests_total{status_code=~"5.."}[5m]))
     /
     sum(rate(http_requests_total[5m]))
   ) > 0.05

2. Use Boolean Operators: Return 1 (fire) or 0 (no alert)

   # Returns 1 when memory usage > 90%
   (process_resident_memory_bytes / node_memory_MemTotal_bytes) > 0.9

3. Consider for Duration: Alerts typically use for clause

   alert: HighErrorRate
   expr: |
     (
       sum(rate(http_requests_total{status_code=~"5.."}[5m]))
       /
       sum(rate(http_requests_total[5m]))
     ) > 0.05
   for: 10m  # Only fire after 10 minutes of continuous violation

Recording Rules

When generating queries for recording rules:

1. Follow Naming Convention: level:metric:operations

   # level: aggregation level (job, instance, etc.)
   # metric: base metric name
   # operations: functions applied

   - record: job:http_requests:rate5m
     expr: sum by (job) (rate(http_requests_total[5m]))

2. Pre-aggregate Expensive Queries:

   # Recording rule for frequently-used latency query
   - record: job_endpoint:http_request_duration_seconds:p95
     expr: |
       histogram_quantile(0.95,
         sum by (job, endpoint, le) (
           rate(http_request_duration_seconds_bucket[5m])
         )
       )

3. Use Recorded Metrics in Dashboards:

   # Instead of expensive query, use pre-recorded metric
   job_endpoint:http_request_duration_seconds:p95{job="api-server"}

Error Handling

Common Issues and Solutions

1. Empty Results:

• Check if metrics exist: up{job="your-job"}
• Verify label filters are correct
• Check time range is appropriate
• Confirm metric is being scraped

2. Too Many Series (High Cardinality):

• Add more specific label filters
• Use aggregation to reduce series count
• Consider using recording rules
• Check for label explosion (dynamic labels)

3. Incorrect Values:

• Verify metric type (counter vs gauge)
• Check function usage (rate on counters, not gauges)
• Verify time range is appropriate
• Check for counter resets

4. Performance Issues:

• Reduce time range for range vectors
• Add label filters to reduce cardinality
• Use recording rules for complex queries
• Avoid expensive regex patterns
• Consider query timeout settings

Communication Guidelines

When generating queries:

1. Explain the Plan: Always present a plain-English plan before generating 2. Ask Questions: Use AskUserQuestion tool to gather requirements 3. Confirm Intent: Verify the query matches user goals before finalizing 4. Educate: Explain why certain functions or patterns are used 5. Provide Context: Show how to interpret results 6. Suggest Improvements: Offer optimizations or alternative approaches 7. Validate Proactively: Always validate and fix issues 8. Follow Up: Ask if adjustments are needed

Fallback When AskUserQuestion Is Unavailable

If a structured question tool is unavailable, continue with an explicit inline questionnaire in plain text: 1. Ask for goal, metric names/types, labels, time range, aggregation, and use case in one compact prompt. 2. If the user provides partial answers, proceed with conservative defaults and clearly mark assumptions. 3. If core inputs are still ambiguous, offer 2-3 concrete query-plan options and ask the user to pick one. 4. Do not block generation indefinitely waiting for perfect context; generate a best-effort query with assumption notes.

Relevant Reference Criteria and Trivial-Case Skip Rules

Use references deterministically, but avoid unnecessary reads for trivial requests.

Read references when ANY of the following is true:

• Histogram or summary quantiles are requested
• Query uses joins/vector matching, subqueries, offsets, or recording/alerting rules
• Query is for SLO/burn-rate/error-budget workflows
• Query includes optimization or cardinality concerns
• Metric type is unknown or contested

Skip reference reads only when ALL of the following are true:

• Single-metric, single-function query (rate, increase, sum, avg, max, min)
• No joins, no recording/alert rules, no advanced functions
• Metric type and labels are clearly provided by the user

When skipping, explicitly state: Reference read skipped (trivial case) and keep validation mandatory.

Integration with devops-skills:promql-validator

After generating any PromQL query, automatically invoke the devops-skills:promql-validator skill to ensure quality:

Steps:
1. Generate the PromQL query based on user requirements
2. Invoke devops-skills:promql-validator skill with the generated query
3. Review validation results (syntax, semantics, performance)
4. Fix any issues identified by the validator
5. Re-validate until all checks pass
6. Provide the final validated query with usage instructions
7. Ask user if further refinements are needed

This ensures all generated queries follow best practices and are production-ready.

Resources

IMPORTANT: Explicit Reference Consultation

>

When generating queries, you SHOULD explicitly read the relevant reference files using the Read tool and cite applicable best practices. This ensures generated queries follow documented patterns and helps users understand why certain approaches are recommended.

references/

promql_functions.md

• Comprehensive reference of all PromQL functions
• Grouped by category (aggregation, math, time, histogram, etc.)
• Usage examples for each function
• Read this file when: implementing specific function requirements or when user asks about function behavior

promql_patterns.md

• Common query patterns for typical monitoring scenarios
• RED method patterns (Rate, Errors, Duration)
• USE method patterns (Utilization, Saturation, Errors)
• Alerting and recording rule patterns
• Read this file when: implementing standard monitoring patterns like error rates, latency, or resource usage

best_practices.md

• PromQL best practices and anti-patterns
• Performance optimization guidelines
• Cardinality management
• Query structure recommendations
• Read this file when: optimizing queries, reviewing for anti-patterns, or when cardinality concerns arise

metric_types.md

• Detailed guide to Prometheus metric types
• Counter, Gauge, Histogram, Summary
• When to use each type
• Appropriate functions for each type
• Read this file when: clarifying metric type questions or determining appropriate functions for a metric

examples/

common_queries.promql

• Collection of commonly-used PromQL queries
• Request rate, error rate, latency queries
• Resource usage queries
• Availability and uptime queries
• Can be copied and customized

red_method.promql

• Complete RED method implementation
• Request rate queries
• Error rate queries
• Duration/latency queries

use_method.promql

• Complete USE method implementation
• Utilization queries
• Saturation queries
• Error queries

alerting_rules.yaml

• Example Prometheus alerting rules
• Various threshold-based alerts
• Best practices for alert expressions

recording_rules.yaml

• Example Prometheus recording rules
• Pre-aggregated metrics
• Naming conventions

slo_patterns.promql

• SLO, error budget, and burn rate queries
• Multi-window, multi-burn-rate alerting patterns
• Latency SLO compliance queries

kubernetes_patterns.promql

• Kubernetes monitoring patterns
• kube-state-metrics queries (pods, deployments, nodes)
• cAdvisor container metrics (CPU, memory)
• Vector matching and joins for Kubernetes

Important Notes

1. Always Plan Interactively: Never generate a query without confirming the plan with the user 2. Use AskUserQuestion: Leverage the tool to gather requirements and confirm plans 3. Validate Everything: Always invoke devops-skills:promql-validator after generation 4. Educate Users: Explain what the query does and why it's structured that way 5. Consider Use Case: Tailor the query based on whether it's for dashboards, alerts, or analysis 6. Think About Performance: Always include label filters and consider cardinality 7. Follow Metric Types: Use appropriate functions for counters, gauges, and histograms 8. Format for Readability: Use multi-line formatting for complex queries

Success Criteria

A successful query generation session should meet these measurable checkpoints: 1. Requirement capture completed: goal/use-case/metric/time-range/aggregation recorded. 2. Plan confirmation completed: user approved plan OR explicit assumption set documented. 3. Reference decision recorded: consulted with file names OR skipped (trivial case) with reason. 4. Query validity completed: syntax passes validator or manual fallback check. 5. Semantic sanity completed: function choice matches metric type (counter/gauge/histogram/summary). 6. Cardinality guard completed: query includes explicit filters or aggregation rationale. 7. Delivery completed: final query + interpretation + next-step customization guidance provided.

Remember

The goal is to collaboratively plan and generate PromQL queries that exactly match user intentions. Always prioritize clarity, correctness, and performance. The interactive planning phase is the most important part of this skill—never skip it!

PromQL Best Practices

Comprehensive guide to writing efficient, maintainable, and correct PromQL queries.

Table of Contents

1. Label Selection and Filtering 2. Metric Type Usage 3. Aggregation Best Practices 4. Performance Optimization 5. Time Range Selection 6. Recording Rules 7. Alerting Best Practices 8. Query Readability 9. Common Anti-Patterns 10. Testing and Validation

---

Label Selection and Filtering

Always Use Label Filters

Problem: Querying metrics without label filters can match thousands or millions of time series, causing performance issues and timeouts.

# ❌ Bad: No filtering, matches all time series
rate(http_requests_total[5m])

# ✅ Good: Specific filtering
rate(http_requests_total{job="api-server", environment="production"}[5m])

Best practices:

• Always include at least job label filter
• Add environment or cluster for multi-environment setups
• Use instance for single-instance queries
• Add functional labels like endpoint, method, status_code as needed

Use Exact Matches Over Regex

Problem: Regex matching (=~) is significantly slower than exact matching (=).

# ❌ Bad: Unnecessary regex for exact match
http_requests_total{status_code=~"200"}

# ✅ Good: Exact match is faster
http_requests_total{status_code="200"}

# ✅ Good: Regex when truly needed
http_requests_total{status_code=~"2.."}  # All 2xx codes
http_requests_total{instance=~"prod-.*"}  # Pattern matching

When regex is appropriate:

• Matching patterns: instance=~"prod-.*"
• Multiple values: status_code=~"200|201|202"
• Character classes: status_code=~"5.."

Optimization tips:

• Anchor regex patterns when possible: =~"^prod-.*"
• Keep patterns simple and specific
• Use multiple exact matchers instead of single regex when possible

Avoid High-Cardinality Labels

Problem: Labels with many unique values create massive number of time series.

# ❌ Bad: user_id creates one series per user (high cardinality)
sum by (user_id) (rate(requests_total[5m]))

# ✅ Good: Aggregate without high-cardinality labels
sum(rate(requests_total[5m]))

# ✅ Good: Use low-cardinality labels
sum by (service, environment) (rate(requests_total[5m]))

High-cardinality labels to avoid in aggregations:

• User IDs, session IDs, request IDs
• IP addresses (unless specifically needed)
• Timestamps
• Full URLs or paths (use path patterns instead)
• UUIDs

Solutions:

• Aggregate out high-cardinality labels with without()
• Use lower-cardinality alternatives (e.g., path_pattern instead of full_url)
• Implement recording rules to pre-aggregate

---

Metric Type Usage

Use rate() with Counters

Problem: Counter metrics always increase; raw values are not useful for analysis.

# ❌ Bad: Raw counter value is not meaningful
http_requests_total

# ✅ Good: Calculate rate (requests per second)
rate(http_requests_total[5m])

# ✅ Good: Calculate total increase over period
increase(http_requests_total[1h])

Counter identification:

• Metrics ending in _total (e.g., requests_total, errors_total)
• Metrics ending in _count (e.g., http_requests_count)
• Metrics ending in _sum (e.g., request_duration_seconds_sum)
• Metrics ending in _bucket (e.g., request_duration_seconds_bucket)

Don't Use rate() with Gauges

Problem: Gauge metrics represent current state, not cumulative values.

# ❌ Bad: rate() on gauge doesn't make sense
rate(memory_usage_bytes[5m])

# ✅ Good: Use gauge value directly
memory_usage_bytes

# ✅ Good: Use *_over_time functions for analysis
avg_over_time(memory_usage_bytes[5m])
max_over_time(memory_usage_bytes[1h])

Gauge examples:

• memory_usage_bytes
• cpu_temperature_celsius
• queue_length
• active_connections

Histogram Quantiles Require Aggregation

Problem: histogram_quantile() requires proper aggregation and the le label.

# ❌ Bad: Missing aggregation
histogram_quantile(0.95, rate(request_duration_seconds_bucket[5m]))

# ❌ Bad: Missing le label in aggregation
histogram_quantile(0.95, sum(rate(request_duration_seconds_bucket[5m])))

# ❌ Bad: Missing rate() on buckets
histogram_quantile(0.95, sum by (le) (request_duration_seconds_bucket))

# ✅ Good: Correct usage
histogram_quantile(0.95,
  sum by (le) (rate(request_duration_seconds_bucket[5m]))
)

# ✅ Good: Preserving additional labels
histogram_quantile(0.95,
  sum by (service, le) (rate(request_duration_seconds_bucket[5m]))
)

Requirements for histogram_quantile(): 1. Must apply rate() or irate() to bucket counters 2. Must aggregate with sum 3. Must include le label in aggregation 4. Can include other labels for grouping

Never Average Pre-Calculated Quantiles

Problem: Averaging quantiles is mathematically invalid and produces incorrect results.

# ❌ Bad: Averaging quantiles is wrong
avg(request_duration_seconds{quantile="0.95"})

# ✅ Good: Use _sum and _count to calculate average
sum(rate(request_duration_seconds_sum[5m]))
/
sum(rate(request_duration_seconds_count[5m]))

# ✅ Good: If you need quantiles, use histogram
histogram_quantile(0.95,
  sum by (le) (rate(request_duration_seconds_bucket[5m]))
)

---

Aggregation Best Practices

Choose Between by() and without()

by(): Keeps only specified labels, removes all others without(): Removes specified labels, keeps all others

# Use by() when you know exactly what labels you want to keep
sum by (service, environment) (rate(requests_total[5m]))

# Use without() when you want to remove specific labels
sum without (instance, pod) (rate(requests_total[5m]))

When to use each:

• by(): When aggregating to specific dimensions (service-level metrics)
• without(): When removing noise (instance-level details)

Aggregate Before histogram_quantile()

Always aggregate before calling histogram_quantile():

# ❌ Bad: Trying to aggregate after quantile calculation
sum(
  histogram_quantile(0.95, rate(request_duration_seconds_bucket[5m]))
)

# ✅ Good: Aggregate first, then calculate quantile
histogram_quantile(0.95,
  sum by (le) (rate(request_duration_seconds_bucket[5m]))
)

# ✅ Good: Aggregate with grouping
histogram_quantile(0.95,
  sum by (service, le) (rate(request_duration_seconds_bucket[5m]))
)

Use Appropriate Aggregation Operators

Choose the right aggregation for your use case:

# sum: For counting, totaling
sum(up{job="api"})  # Total number of instances

# avg: For average values
avg(cpu_usage_percent)  # Average CPU across instances

# max/min: For identifying extremes
max(memory_usage_bytes)  # Instance with highest memory use

# count: For counting series
count(up{job="api"} == 1)  # Number of healthy instances

# topk/bottomk: For top/bottom N
topk(10, rate(requests_total[5m]))  # Top 10 by request rate

# quantile: For percentiles across simple metrics
quantile(0.95, response_time_seconds)  # 95th percentile

---

Performance Optimization

Limit Cardinality

The number of time series matters most for query performance.

# Check cardinality of a metric
count(metric_name)

# Check cardinality by label
count by (label_name) (metric_name)

# Identify high-cardinality metrics
topk(10, count by (__name__) ({__name__=~".+"}))

Strategies to reduce cardinality: 1. Add more specific label filters 2. Use aggregation to reduce dimensions 3. Remove high-cardinality labels from queries 4. Use recording rules for frequently-queried aggregations

Optimize Time Ranges

Larger time ranges process more data and run slower.

# ❌ Slow: Very large range for rate
rate(requests_total[1h])

# ✅ Fast: Appropriate range for rate
rate(requests_total[5m])

# For recording rules: Pre-compute common ranges
# Then use the recorded metric instead
job:requests:rate5m  # Recorded metric

Time range guidelines:

• Rate functions: [1m] to [5m] for real-time monitoring
• Trend analysis: [1h] to [1d] when needed
• Rule of thumb: Range should be 4× scrape interval minimum
• Recording rules: Use for ranges longer than [5m] if queried frequently

Avoid Expensive Subqueries

Subqueries can exponentially increase query cost.

# ❌ Expensive: Subquery over long range
max_over_time(rate(metric[5m])[7d:1h])

# ✅ Better: Use recording rule
max_over_time(job:metric:rate5m[7d])

# ✅ Better: Reduce range if possible
max_over_time(rate(metric[5m])[1d:1h])

Subquery cost = range_duration / resolution × base_query_cost

Use Recording Rules for Complex Queries

Recording rules pre-compute expensive queries.

# Recording rule configuration
groups:
  - name: request_rates
    interval: 30s
    rules:
      # Pre-compute expensive aggregation
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      # Pre-compute complex quantile
      - record: job:http_latency:p95
        expr: |
          histogram_quantile(0.95,
            sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))
          )

Use recording rules when:

• Query is used in multiple dashboards
• Query is computationally expensive
• Query is accessed frequently (every dashboard refresh)
• You need faster dashboard/alert evaluation

---

Time Range Selection

Choose Appropriate Ranges for rate()

Too short: Noisy, sensitive to scraping jitter Too long: Hides important spikes, slow to react

# Real-time monitoring: 1-5 minutes
rate(requests_total[2m])
rate(requests_total[5m])

# Trend analysis: 15 minutes to 1 hour
rate(requests_total[15m])
rate(requests_total[1h])

# Historical analysis: Hours to days
rate(requests_total[6h])
rate(requests_total[1d])

Guidelines:

• Minimum range: 4× scrape interval
• For 15s scrape interval: minimum [1m]
• For 30s scrape interval: minimum [2m]
• Default choice: [5m] works well for most cases

Use irate() for Volatile Metrics

# rate(): Average over time range, smooth
rate(requests_total[5m])

# irate(): Instant based on last 2 points, volatile
irate(requests_total[5m])

When to use irate():

• Detecting sudden spikes
• Alerting on rapid changes
• Short-term analysis
• Metrics that change dramatically

When to use rate():

• Dashboard visualizations
• Trend analysis
• Smooth charts
• Most monitoring use cases

---

Recording Rules

Follow Naming Convention

Format: level:metric:operations

# level: Aggregation level (job, service, cluster)
# metric: Base metric name
# operations: Functions applied (rate5m, p95, sum)

rules:
  # Good examples
  - record: job:http_requests:rate5m
    expr: sum by (job) (rate(http_requests_total[5m]))

  - record: job_endpoint:http_latency:p95
    expr: |
      histogram_quantile(0.95,
        sum by (job, endpoint, le) (rate(http_request_duration_seconds_bucket[5m]))
      )

  - record: cluster:cpu_usage:ratio
    expr: |
      sum(rate(node_cpu_seconds_total{mode!="idle"}[5m]))
      /
      sum(rate(node_cpu_seconds_total[5m]))

Pre-Aggregate Expensive Queries

# Instead of running this expensive query repeatedly:
# histogram_quantile(0.95, sum by (le) (rate(latency_bucket[5m])))

# Create a recording rule:
- record: :http_request_duration:p95
  expr: |
    histogram_quantile(0.95,
      sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
    )

# Then use the recorded metric:
# :http_request_duration:p95

Layer Recording Rules

Build complex metrics in stages:

# Layer 1: Basic rates
- record: instance:requests:rate5m
  expr: rate(http_requests_total[5m])

# Layer 2: Job-level aggregation
- record: job:requests:rate5m
  expr: sum by (job) (instance:requests:rate5m)

# Layer 3: Derived metrics
- record: job:error_ratio:rate5m
  expr: |
    sum by (job) (instance:requests:rate5m{status_code=~"5.."})
    /
    job:requests:rate5m

---

Alerting Best Practices

Make Alert Expressions Boolean

Alert expressions should return 1 (firing) or 0 (not firing).

# ✅ Good: Boolean expression
(
  sum(rate(errors_total[5m]))
  /
  sum(rate(requests_total[5m]))
) > 0.05

# ✅ Good: Explicit comparison
http_requests_rate < 10

# ✅ Good: Complex boolean
(cpu_usage > 80) and (memory_usage > 90)

Use for Duration for Stability

Avoid alerting on transient spikes.

# Alert only after condition persists for 10 minutes
- alert: HighErrorRate
  expr: |
    (
      sum(rate(errors_total[5m]))
      /
      sum(rate(requests_total[5m]))
    ) > 0.05
  for: 10m
  annotations:
    summary: "Error rate above 5% for 10+ minutes"

`for` duration guidelines:

• Short-lived issues: 5m
• Sustained problems: 10m to 15m
• Avoid false positives: 30m+
• Critical immediate alerts: 0m (no for)

Include Context in Alert Queries

# ✅ Good: Include labels that identify the problem
sum by (service, environment) (
  rate(errors_total[5m])
) > 100

# Alerts will show which service and environment

Avoid Alerting on Absence Without Context

# ❌ Bad: Too generic
absent(up)

# ✅ Good: Specific service
absent(up{job="critical-service"})

# ✅ Good: With timeout
absent_over_time(up{job="critical-service"}[10m])

---

Query Readability

Format Complex Queries

Use multi-line formatting for readability:

# ✅ Good: Multi-line with indentation
histogram_quantile(0.95,
  sum by (service, le) (
    rate(http_request_duration_seconds_bucket{
      environment="production",
      job="api-server"
    }[5m])
  )
)

# ❌ Bad: Single line, hard to read
histogram_quantile(0.95, sum by (service, le) (rate(http_request_duration_seconds_bucket{environment="production", job="api-server"}[5m])))

Use Comments in Recording Rules

rules:
  # Calculate p95 latency for all API endpoints
  # Used by: API dashboard, SLO calculations, latency alerts
  - record: api:http_latency:p95
    expr: |
      histogram_quantile(0.95,
        sum by (endpoint, le) (
          rate(http_request_duration_seconds_bucket{job="api"}[5m])
        )
      )

Name Recording Rules Descriptively

# ✅ Good: Clear purpose from name
- record: api:error_rate:ratio5m
- record: db:query_duration:p99
- record: cluster:memory_usage:bytes

# ❌ Bad: Unclear names
- record: metric1
- record: temp_calc
- record: x

---

Common Anti-Patterns

Anti-Pattern 1: No Label Filters

# ❌ Anti-pattern
rate(http_requests_total[5m])

# ✅ Fix
rate(http_requests_total{job="api-server", environment="prod"}[5m])

Anti-Pattern 2: Regex for Exact Match

# ❌ Anti-pattern
metric{label=~"value"}

# ✅ Fix
metric{label="value"}

Anti-Pattern 3: rate() on Gauges

# ❌ Anti-pattern
rate(memory_usage_bytes[5m])

# ✅ Fix
avg_over_time(memory_usage_bytes[5m])

Anti-Pattern 4: Missing rate() on Counters

# ❌ Anti-pattern
http_requests_total

# ✅ Fix
rate(http_requests_total[5m])

Anti-Pattern 5: Averaging Quantiles

# ❌ Anti-pattern
avg(http_duration{quantile="0.95"})

# ✅ Fix
histogram_quantile(0.95,
  sum by (le) (rate(http_duration_bucket[5m]))
)

Anti-Pattern 6: Missing Aggregation in histogram_quantile

# ❌ Anti-pattern
histogram_quantile(0.95, rate(latency_bucket[5m]))

# ✅ Fix
histogram_quantile(0.95,
  sum by (le) (rate(latency_bucket[5m]))
)

Anti-Pattern 7: High-Cardinality Aggregation

# ❌ Anti-pattern
sum by (user_id) (requests)  # millions of series

# ✅ Fix
sum(requests)  # single series
# Or use low-cardinality labels
sum by (service) (requests)

---

Testing and Validation

Test Queries Before Production

1. Check cardinality:

   count(your_query)

2. Verify result makes sense:

• Check value range
• Verify labels in output
• Compare with expected results

3. Test edge cases:

• What if metric doesn't exist?
• What if all instances are down?
• What during counter resets?

Validate Time Ranges

# Test with different ranges
rate(metric[1m])
rate(metric[5m])
rate(metric[1h])

# Verify results are reasonable

Check for Missing Data

# Verify metric exists
count(metric_name) > 0

# Check for gaps
absent_over_time(metric_name[10m])

---

Summary Checklist

Before deploying a PromQL query, verify:

• [ ] Uses specific label filters (at least job)
• [ ] Uses exact match (=) instead of regex when possible
• [ ] Uses appropriate function for metric type
• [ ] rate() for counters
• [ ] Direct value or *_over_time() for gauges
• [ ] histogram_quantile() with sum by (le) for histograms
• [ ] Includes proper aggregation
• [ ] Uses reasonable time range (typically [5m])
• [ ] Avoids high-cardinality labels
• [ ] Formatted for readability
• [ ] Tested and returns expected results
• [ ] Considers using recording rule if expensive and frequently accessed
• [ ] Includes descriptive naming (for recording rules/alerts)
• [ ] Documented with comments (for complex queries)

---

Resources

• Official Prometheus Querying Documentation
• Prometheus Best Practices
• PromQL Functions Reference
• Common Query Patterns

Prometheus Metric Types

Comprehensive guide to the four Prometheus metric types: Counter, Gauge, Histogram, and Summary.

Table of Contents

1. Overview 2. Counter 3. Gauge 4. Histogram 5. Summary 6. Choosing the Right Type 7. Metric Naming Conventions

---

Overview

Prometheus has four core metric types, each designed for specific use cases:

---

Counter

Definition

A counter is a cumulative metric that only increases over time (or resets to zero on restart). Counters are used for counting events.

Characteristics

• Only increases (or resets to 0)
• Cumulative - represents total count since start
• Not meaningful as raw value - always use with rate() or increase()
• Handles restarts - rate functions automatically detect and handle counter resets

Examples

# Total HTTP requests since process started
http_requests_total

# Total errors since process started
http_errors_total

# Total bytes sent since process started
bytes_sent_total

# Total database queries executed
db_queries_total{operation="select"}

Naming Convention

Counters should end with _total:

• http_requests_total
• errors_total
• bytes_processed_total
• cache_hits_total

Common PromQL Functions

rate() - Per-Second Average Rate

# Requests per second over last 5 minutes
rate(http_requests_total[5m])

# Errors per second
rate(errors_total[2m])

# Bytes sent per second
rate(bytes_sent_total[1m])

When to use: Graphing trends, calculating throughput, most monitoring use cases

irate() - Instant Rate

# Instant requests per second
irate(http_requests_total[5m])

When to use: Detecting spikes, alerting on sudden changes, real-time dashboards

increase() - Total Increase

# Total requests in the last hour
increase(http_requests_total[1h])

# Total errors in the last day
increase(errors_total[24h])

When to use: Calculating totals over periods, capacity planning, billing

Best Practices

# ✅ Good: Use rate() for per-second values
rate(http_requests_total{job="api"}[5m])

# ✅ Good: Use increase() for totals
increase(http_requests_total{job="api"}[1h])

# ❌ Bad: Don't use raw counter values
http_requests_total

# ❌ Bad: Don't use rate() without time range
rate(http_requests_total)

Use Cases

• Request counting: http_requests_total, grpc_requests_total
• Error tracking: errors_total, failed_requests_total
• Throughput: bytes_sent_total, messages_processed_total
• Cache hits/misses: cache_hits_total, cache_misses_total
• Database operations: db_queries_total, db_transactions_total

---

Gauge

Definition

A gauge is a metric that represents a single numerical value that can go up or down. Gauges represent current state or level.

Characteristics

• Can increase or decrease
• Represents current value - meaningful as-is
• Snapshot - shows state at time of measurement
• No cumulative behavior

Examples

# Current memory usage in bytes
memory_usage_bytes

# Current CPU temperature
cpu_temperature_celsius

# Current number of items in queue
queue_length

# Current number of active connections
active_connections

# Current disk space available
disk_available_bytes

Naming Convention

Gauges should describe the measured value and include units:

• memory_usage_bytes
• temperature_celsius
• queue_depth
• active_threads
• cpu_usage_ratio (for percentages expressed as 0-1)

Common PromQL Functions

Direct Usage

# Current memory usage
memory_usage_bytes

# Current queue length
queue_depth{service="worker"}

*_over_time Functions

# Average memory usage over 5 minutes
avg_over_time(memory_usage_bytes[5m])

# Maximum queue depth in last hour
max_over_time(queue_depth[1h])

# Minimum available disk space in last day
min_over_time(disk_available_bytes[24h])

# Count of samples (how many times scraped)
count_over_time(metric[5m])

Statistical Analysis

# Standard deviation of response time
stddev_over_time(response_time_seconds[5m])

# Quantile of gauge values over time
quantile_over_time(0.95, metric[5m])

# Rate of change (derivative)
deriv(queue_length[10m])

Best Practices

# ✅ Good: Use gauge directly for current value
memory_usage_bytes

# ✅ Good: Use *_over_time for analysis
avg_over_time(memory_usage_bytes[5m])

# ❌ Bad: Don't use rate() on gauges
rate(memory_usage_bytes[5m])

# ❌ Bad: Don't use increase() on gauges
increase(memory_usage_bytes[1h])

# ✅ Good: Use deriv() for rate of change
deriv(disk_usage_bytes[1h])

Use Cases

• Resource usage: memory_usage_bytes, cpu_usage_percent, disk_usage_bytes
• Temperatures: cpu_temperature_celsius, disk_temperature_celsius
• Queue metrics: queue_length, pending_jobs
• Connection counts: active_connections, idle_connections
• Thread counts: active_threads, blocked_threads
• Current state: replica_count, node_count, pod_count

---

Histogram

Definition

A histogram samples observations (like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values.

Characteristics

• Buckets - predefined upper bounds (le = "less than or equal")
• Cumulative - each bucket includes all observations ≤ its upper bound
• Three metrics:
• _bucket - counter for each bucket
• _sum - sum of all observed values
• _count - total number of observations
• Calculate quantiles - use histogram_quantile()
• Flexible - can calculate any quantile from the same data

Structure

For metric http_request_duration_seconds, you get:

http_request_duration_seconds_bucket{le="0.1"}   # ≤ 0.1s
http_request_duration_seconds_bucket{le="0.5"}   # ≤ 0.5s
http_request_duration_seconds_bucket{le="1"}     # ≤ 1s
http_request_duration_seconds_bucket{le="5"}     # ≤ 5s
http_request_duration_seconds_bucket{le="+Inf"}  # All observations
http_request_duration_seconds_sum                # Sum of all durations
http_request_duration_seconds_count              # Total count

Examples

# Request duration histogram
http_request_duration_seconds_bucket

# Response size histogram
http_response_size_bytes_bucket

# Database query duration histogram
db_query_duration_seconds_bucket

Naming Convention

Histograms should describe what is being measured and include units:

• http_request_duration_seconds
• response_size_bytes
• db_query_duration_seconds
• batch_processing_time_seconds

The instrumentation library automatically adds _bucket, _sum, and _count suffixes.

Common PromQL Functions

histogram_quantile() - Calculate Percentiles

# 95th percentile request duration
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

# Multiple percentiles
histogram_quantile(0.50, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))  # P50
histogram_quantile(0.90, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))  # P90
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))  # P99

# Percentile by service
histogram_quantile(0.95,
  sum by (service, le) (rate(http_request_duration_seconds_bucket[5m]))
)

Average from Histogram

# Average request duration
sum(rate(http_request_duration_seconds_sum[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))

# Average by endpoint
sum by (endpoint) (rate(http_request_duration_seconds_sum[5m]))
/
sum by (endpoint) (rate(http_request_duration_seconds_count[5m]))

Request Rate from Histogram

# Requests per second (from histogram)
sum(rate(http_request_duration_seconds_count[5m]))

# Same as using counter
sum(rate(http_requests_total[5m]))

Fraction of Observations

# Percentage of requests under 100ms
(
  sum(rate(http_request_duration_seconds_bucket{le="0.1"}[5m]))
  /
  sum(rate(http_request_duration_seconds_count[5m]))
) * 100

# SLO: 95% of requests must be under 500ms
(
  sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m]))
  /
  sum(rate(http_request_duration_seconds_count[5m]))
) >= 0.95

Best Practices

# ✅ Good: Always use rate() on buckets
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

# ✅ Good: Always include sum by (le)
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

# ✅ Good: Can include other labels for grouping
histogram_quantile(0.95,
  sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))
)

# ❌ Bad: Missing aggregation
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# ❌ Bad: Missing le in aggregation
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket[5m]))
)

# ❌ Bad: Missing rate()
histogram_quantile(0.95,
  sum by (le) (http_request_duration_seconds_bucket)
)

Use Cases

• Request latency: http_request_duration_seconds, grpc_request_duration_seconds
• Response sizes: http_response_size_bytes, message_size_bytes
• Database query times: db_query_duration_seconds
• Batch processing times: batch_processing_duration_seconds
• Any measurement where you need percentiles: response times, processing durations, sizes

Advantages

• Flexible: Calculate any quantile from same data
• Aggregatable: Can aggregate across dimensions
• Resource efficient: Client-side bucketing, not all observations
• Suitable for alerting: Consistent with rate() calculations

Bucket Configuration

Choose buckets that cover your expected range:

// Example: HTTP request duration (Go client)
[]float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10}
// 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s

// Example: Response size in bytes
[]float64{100, 1000, 10000, 100000, 1000000, 10000000}
// 100B, 1KB, 10KB, 100KB, 1MB, 10MB

---

Summary

Definition

A summary is similar to a histogram but calculates quantiles on the client side and streams pre-calculated percentiles to Prometheus.

Characteristics

• Pre-calculated quantiles - computed by client
• Three metrics:
• {quantile="0.5"} - 50th percentile
• {quantile="0.9"} - 90th percentile
• {quantile="0.99"} - 99th percentile
• _sum - sum of all observed values
• _count - total number of observations
• Not aggregatable - quantiles can't be averaged or summed
• Less flexible - can only view pre-configured quantiles

Structure

For metric http_request_duration_seconds, you get:

http_request_duration_seconds{quantile="0.5"}   # 50th percentile (median)
http_request_duration_seconds{quantile="0.9"}   # 90th percentile
http_request_duration_seconds{quantile="0.99"}  # 99th percentile
http_request_duration_seconds_sum               # Sum of all durations
http_request_duration_seconds_count             # Total count

Examples

# Pre-calculated 95th percentile
http_request_duration_seconds{quantile="0.95"}

# Pre-calculated 50th percentile (median)
rpc_duration_seconds{quantile="0.5"}

Common PromQL Functions

Using Pre-Calculated Quantiles

# Use quantile directly (no calculation needed)
http_request_duration_seconds{quantile="0.95"}

# By service
http_request_duration_seconds{service="api", quantile="0.95"}

Calculate Average

# Average from summary
sum(rate(http_request_duration_seconds_sum[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))

Best Practices

# ✅ Good: Use quantile directly
http_request_duration_seconds{quantile="0.95"}

# ✅ Good: Calculate average from _sum and _count
sum(rate(http_request_duration_seconds_sum[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))

# ❌ Bad: Don't average quantiles across instances
avg(http_request_duration_seconds{quantile="0.95"})

# ❌ Bad: Don't sum quantiles
sum(http_request_duration_seconds{quantile="0.95"})

# ❌ Bad: Don't use histogram_quantile() on summaries
histogram_quantile(0.95, http_request_duration_seconds)

Use Cases

• When client-side quantiles are acceptable
• Single instance metrics (not aggregated across multiple instances)
• Legacy systems (histograms are generally preferred now)
• Specific quantile requirements that won't change

Limitations

1. Cannot aggregate across instances/labels - quantiles can't be averaged 2. Fixed quantiles - can't calculate new percentiles from existing data 3. More client resources - quantile calculation happens on client 4. Not suitable for alerting - quantiles calculated differently than rates

Histogram vs Summary

Recommendation: Use histograms for new instrumentation. Summaries are mainly for legacy compatibility.

---

Choosing the Right Type

Decision Tree

Are you counting events that only increase?
├─ Yes → Counter (e.g., requests_total, errors_total)
└─ No → Is it a current state that can go up or down?
    ├─ Yes → Gauge (e.g., memory_bytes, queue_length)
    └─ No → Do you need percentiles/distributions?
        ├─ Yes → Histogram (e.g., duration_seconds, size_bytes)
        └─ No → Consider if you really need metrics for this

Use Case Matrix

---

Metric Naming Conventions

General Rules

1. Use base units: seconds (not milliseconds), bytes (not kilobytes) 2. Include units in name: _seconds, _bytes, _ratio, _percent 3. Use descriptive names: http_request_duration_seconds not http_req_dur_s 4. Counters end in `_total`: requests_total, errors_total 5. Ratios use `_ratio` suffix: cpu_usage_ratio (0-1 range) 6. Avoid stuttering: http_requests_total not http_http_requests_total

Unit Suffixes

Namespace Structure

<namespace>_<subsystem>_<metric_name>_<unit>

# Good examples
http_request_duration_seconds
http_response_size_bytes
db_query_duration_seconds
process_resident_memory_bytes
node_cpu_seconds_total

# Component structure
prometheus_http_requests_total     # namespace: prometheus, subsystem: http
node_network_receive_bytes_total   # namespace: node, subsystem: network

---

Summary Comparison

Most common: Counter and Histogram cover 90% of use cases.

---

References

• Prometheus Metric Types
• Prometheus Best Practices - Naming
• Histograms and Summaries