Prometheus

Prometheus

Choose the right metric type, name it clearly, label it sparingly. Prometheus is a pull-based monitoring system built on a dimensional data model — every metric is a time series identified by a name and key-value label pairs. Getting this right at instrumentation time prevents expensive rework later.

References

Metric Type Selection

Choose correctly at instrumentation time — changing later requires migration of dashboards, alerts, and recording rules.

Counter

Monotonically increasing value — resets to zero only on restart.

• Use for: requests served, errors occurred, bytes transferred, tasks completed
• API: inc(), inc(v) where v >= 0
• Always suffix with _total: http_requests_total
• Always apply rate() or increase() in queries — raw values are meaningless
• Prometheus handles counter resets automatically in rate()
• Never use a counter for values that can decrease — that is a gauge

Gauge

Value that goes up and down arbitrarily.

• Use for: temperature, memory usage, in-progress requests, queue depth, timestamps
• API: inc(), dec(), set(v), set_to_current_time()
• No _total suffix
• Never apply rate() to a gauge — use deriv() or delta()
• Timestamp pattern: store Unix epoch seconds as myapp_last_success_timestamp_seconds;

compute elapsed time with time() - metric in PromQL

• Info metric pattern: myapp_build_info{version="1.2.3", commit="abc"} 1 — metadata

as labels with constant value 1

Histogram

Samples observations into configurable buckets. Produces _bucket{le="..."}, _sum, _count.

• Use for: request latencies, response sizes, any distribution needing percentiles or

cross-instance aggregation

• API: observe(v)
• Buckets are cumulative — le="0.5" includes all observations <= 0.5
• Must include +Inf bucket (equal to _count)
• Choose buckets matching expected value range; place more buckets near SLO boundaries
• Buckets cannot be changed after metric creation
• Use histogram_quantile() in PromQL to calculate percentiles
• Aggregatable across instances — the primary advantage over summary

Summary

Calculates streaming quantiles on the client side. Produces {quantile="..."}, _sum, _count.

• Cannot be aggregated across instances — avg(x{quantile="0.95"}) is statistically

invalid

• _sum and _count without quantiles is a valid and useful configuration

Use summary over histogram only when ALL of these are true: 1. You need accurate quantiles (not approximate) 2. From a single instance (no cross-instance aggregation) 3. You know the exact quantiles at instrumentation time 4. You accept that adding new quantiles requires code changes

Default choice: histogram. See ${CLAUDE_SKILL_DIR}/references/metric-types.md for detailed comparison.

Naming

Format: <namespace>_<subsystem>_<name>_<unit>_<suffix>. Not all parts required — minimum is namespace + meaningful name + unit/suffix.

Naming Rules

1. Use snake_case — lowercase with underscores, matching [a-zA-Z_:][a-zA-Z0-9_:]* 2. Colons (:) are reserved for recording rules — never use in direct instrumentation 3. Double underscore prefix (__) is reserved for Prometheus internals 4. Every metric MUST have a namespace prefix identifying its origin 5. Always use base units — seconds not milliseconds, bytes not megabytes. Let visualization tools handle conversion. 6. Append unit to metric name in plural form: http_request_duration_seconds 7. Suffix counters with _total, counter-with-unit as _<unit>_total (e.g., process_cpu_seconds_total) 8. Suffix info metrics with _info, timestamps with _timestamp_seconds 9. A metric MUST represent the same logical thing across all its label dimensions — sum() or avg() across all dimensions should be meaningful. If nonsensical, split into separate metrics.

See ${CLAUDE_SKILL_DIR}/references/naming.md for base units table, component ordering, and full examples.

Labels

Use labels for dimensions you will filter or aggregate by: http_requests_total{method="GET", status="200"} — not separate metrics per status. Do not put label names in metric names.

When NOT to Use Labels

• Unbounded values — user IDs, email addresses, full URLs, query strings
• High cardinality — anything above ~100 unique values per metric

Cardinality

Every unique label combination is a new time series. Each costs RAM, CPU, disk, and network. Cardinality math: total series = metric cardinality x number of targets.

Label Best Practices

1. Start with no labels. Add as concrete use cases emerge. 2. Keep cardinality below 10 per metric as a default target. 3. Initialize all label combinations you know upfront to avoid missing metrics — export 0 for known label sets. 4. Use stable label values. Avoid labels that change frequently. 5. Never include a "total" label value — rely on Prometheus sum().

Instrumentation Patterns

Online-Serving Systems (HTTP servers, APIs, databases)

Key metrics: request rate (_total), error rate, latency (histogram), in-progress (gauge).

• Count requests at completion (not start) — aligns with error and latency stats
• Always have a total requests counter alongside error counters (for ratio calculation)

Offline Processing (queues, pipelines, ETL)

Key metrics per stage: items in (_total), items out (_total), in progress (gauge), last processed timestamp (gauge), processing duration (histogram).

• Export heartbeat timestamps to detect stalled processing

Batch Jobs (cron, scheduled tasks)

Key metrics (push to Pushgateway): last success timestamp (gauge), last completion timestamp (gauge), duration (gauge — single run, not distribution).

• Batch job durations are gauges (single event), not histograms
• Jobs running more often than every 15 minutes should be converted to daemons

Libraries

Instrument transparently — users get metrics without configuration. Minimum for external resource access: request count (counter), error count (counter), latency (histogram).

Subsystem Patterns

• Logging: maintain log_messages_total{level="..."} counter per log level
• Failures: always pair failure counter with total attempts counter for ratio calculation
• Caches: cache_requests_total{result="hit|miss"}, evictions (counter), size (gauge),

lookup latency (histogram). Also instrument the downstream system.

See ${CLAUDE_SKILL_DIR}/references/instrumentation.md for threadpool patterns, custom collectors, and performance tuning in hot paths.

PromQL

Rate and Increase (Counters Only)

• rate(counter[5m]) — per-second rate. Use for alerts and dashboards.
• increase(counter[5m]) — total increase. Sugar for rate() * range_seconds.
• irate(counter[5m]) — instant rate from last two samples. Only for graphing volatile

counters.

• `rate()` first, then aggregate: sum(rate(x[5m])), never rate(sum(x)[5m]). Rate

must see individual counter resets.

• Never rate() a gauge — use deriv() or delta().

Histogram Quantiles

• histogram_quantile(0.95, rate(metric_bucket[5m])) — single histogram
• When aggregating histogram buckets, always preserve le in the by clause:

histogram_quantile(0.95, sum by (job, le) (rate(metric_bucket[5m])))

• Average duration: rate(metric_sum[5m]) / rate(metric_count[5m])

PromQL Gotchas

• Staleness: most recent sample within lookback period (default 5 min). Series

disappears if not scraped within that window.

• Rate window size: rate() needs at least 2 samples. Range should be at least 4x

scrape interval. With 15s scrape, use rate(x[5m]) or wider.

• Expensive queries: bare metric names can expand to thousands of series. Always filter

or aggregate before graphing. Use recording rules for expensive expressions.

See ${CLAUDE_SKILL_DIR}/references/promql.md for data types, selectors, aggregation operators, vector matching, over-time aggregation, binary operators, and operator precedence.

Alerting Rules

Alert on symptoms (user-visible impact), not causes. Use dashboards to pinpoint causes after an alert fires.

Only page on latency at one point in the stack — if overall user latency is fine, don't page on a slow sub-component. Avoid noisy alerts — if an alert fires and there's nothing to do, remove it.

See ${CLAUDE_SKILL_DIR}/references/alerting-and-rules.md for alert design, naming conventions, and recording rule details.

Recording Rules

Pre-compute frequently used or expensive expressions. Format: level:metric:operations.

• Aggregate ratios correctly — aggregate numerator and denominator separately, then divide.

Never average a ratio or average an average.

• Use without for aggregation — preserves all labels except those being removed. Prefer

over by.

• Use recording rules for dashboard queries that are expensive and queried frequently,

expressions used in multiple alerts, or complex multi-step aggregations.

See ${CLAUDE_SKILL_DIR}/references/alerting-and-rules.md for full naming convention and recording rule anti-patterns.

Exporters

Write an exporter when the target system does not expose Prometheus metrics natively. For your own code, use a client library directly.

• Prefix all metrics with exporter name: haproxy_up, mysql_global_status_threads_connected
• Create fresh metric instances per scrape — do NOT use global metric variables updated

each scrape (race conditions, stale labels)

• Drop pre-computed rates, min/max since start, stddev from source systems — export raw

counters and current values; let Prometheus rate() handle the rest

See ${CLAUDE_SKILL_DIR}/references/exporters.md for architecture, collectors, help strings, label rules, and push-based sources.

Application

When writing Prometheus instrumentation:

• Apply all conventions silently — don't narrate each rule being followed.
• Choose metric types based on the decision criteria above.
• If an existing codebase contradicts a convention, follow the codebase and flag the

divergence once.

When writing PromQL queries:

• Always wrap counters in rate() or increase() before further operations.
• Prefer without over by for aggregation.

When writing alerting or recording rules:

• Follow level:metric:operations naming.
• Alert on symptoms, not causes.
• Aggregate ratios correctly (numerator and denominator separately).

When reviewing Prometheus code:

• Cite the specific violation and show the fix inline.
• Don't lecture — state what's wrong and how to fix it.

Integration

The coding skill governs workflow; this skill governs Prometheus implementation choices.

Alerting and Recording Rules

Alerting Philosophy

Keep alerting simple. Alert on symptoms (user-visible impact), not causes. Have good dashboards to pinpoint causes after an alert fires.

What to Alert On

Online-serving systems:

• High latency (user-facing, as high in the stack as possible)
• High error rate (user-visible errors)
• Only page on latency at one point in the stack — if the overall user latency

is fine, don't page on a slow sub-component

Offline processing:

• Data taking too long to get through the system

Batch jobs:

• Job has not succeeded recently enough to avoid user impact
• Threshold: at least 2x the normal job run cycle

Capacity:

• Approaching resource limits that will cause outage without intervention

Meta-monitoring:

• Prometheus, Alertmanager, Pushgateway are healthy
• Prefer blackbox tests (end-to-end) over individual component checks

Alert Naming

Use CamelCase for alert names (community convention):

- alert: HighRequestLatency
  expr: histogram_quantile(0.95, sum by (le, job) (rate(http_request_duration_seconds_bucket[5m]))) > 0.5
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "High request latency on {{ $labels.job }}"
    description: "95th percentile latency is {{ $value }}s (threshold: 0.5s)"

Alert Design Guidelines

1. Allow for slack. Use for duration to accommodate small blips. 2. Link to relevant dashboards in annotations. 3. Include threshold in description so on-call knows the boundary. 4. Avoid noisy alerts. If an alert fires and there's nothing to do, remove it. 5. Different alert types for different request characteristics — low-traffic endpoints may need different thresholds than high-traffic ones.

Recording Rules

Recording rules pre-compute frequently used or expensive expressions. They run at evaluation intervals and store the result as a new time series.

Naming Convention

level:metric:operations

• level — Aggregation level and labels of the output
• metric — Original metric name (strip _total when using rate())
• operations — Applied operations, newest first

# rate of requests per instance and path
- record: instance_path:requests:rate5m
  expr: rate(requests_total{job="myjob"}[5m])

# aggregate away instance
- record: path:requests:rate5m
  expr: sum without (instance)(instance_path:requests:rate5m{job="myjob"})

Aggregation Rules

1. Aggregate ratios correctly. Aggregate numerator and denominator separately, then divide. Never average a ratio or average an average.

# Failure ratio — aggregate numerator and denominator separately
- record: instance_path:request_failures:rate5m
  expr: rate(request_failures_total{job="myjob"}[5m])

- record: instance_path:request_failures_per_requests:ratio_rate5m
  expr: |2
      instance_path:request_failures:rate5m{job="myjob"}
    /
      instance_path:requests:rate5m{job="myjob"}

# Aggregate up — divide after aggregation
- record: path:request_failures_per_requests:ratio_rate5m
  expr: |2
      sum without (instance)(instance_path:request_failures:rate5m{job="myjob"})
    /
      sum without (instance)(instance_path:requests:rate5m{job="myjob"})

2. Use `without` for aggregation. Preserves all labels except the ones you're removing, avoiding accidental label loss.

3. Average latency from summary/histogram — use mean operation name:

- record: instance_path:request_latency_seconds_count:rate5m
  expr: rate(request_latency_seconds_count{job="myjob"}[5m])

- record: instance_path:request_latency_seconds_sum:rate5m
  expr: rate(request_latency_seconds_sum{job="myjob"}[5m])

- record: instance_path:request_latency_seconds:mean5m
  expr: |2
      instance_path:request_latency_seconds_sum:rate5m{job="myjob"}
    /
      instance_path:request_latency_seconds_count:rate5m{job="myjob"}

4. Level labels must match. The labels removed via without should be reflected in the output level. If without (instance) is applied, the output level should not include instance.

When to Use Recording Rules

• Dashboard queries that are expensive and queried frequently
• Expressions used in multiple alerts
• Complex multi-step aggregations (build up in layers)
• When histogram_quantile() on large datasets causes query timeouts

Recording Rule Anti-Patterns

Exporters

Exporters bridge third-party systems into the Prometheus ecosystem. When instrumenting your own code, use a client library directly instead.

When to Write an Exporter

• The system you need to monitor does not expose Prometheus metrics natively
• You need to transform metrics from another monitoring system into Prometheus format
• The system is a black box (hardware, closed-source software)

Architecture

One Exporter Per Instance

Each exporter monitors exactly one application instance, deployed beside it on the same machine. Service discovery happens in Prometheus, not in exporters.

Exceptions: 1. Black-box monitoring (SNMP, IPMI) — can't run on the target device. Prometheus passes the target via URL parameter. 2. Random instance queries — pulling aggregate stats from a load-balanced pool where you don't care which instance answers.

Pull, Don't Push

Metrics are collected synchronously on scrape. Do not run scrapes on internal timers. Let Prometheus control the timing.

• Do not set timestamps on exposed metrics
• If collection takes > 10s (default scrape timeout), document this
• If collection is expensive (> 1 minute), cache results and note this in HELP

Naming Metrics

Follow all naming conventions (see ${CLAUDE_SKILL_DIR}/references/naming.md) plus:

1. Prefix with exporter name — haproxy_up, mysql_global_status_threads_connected 2. Use base units — seconds, bytes. Let grafana convert. 3. Don't include label names in metric names — http_requests_total{method="GET"} not http_get_requests_total 4. Colons reserved for recording rules — never in exporter metrics 5. `_total` for counters — always 6. Don't use `_sum`, `_count`, `_bucket` suffixes unless producing a histogram or summary

Labels

1. Avoid `type` as a label — too generic. Use specific names. 2. Avoid target-like labels — region, cluster, env belong in Prometheus scrape configuration, not in the exporter. 3. Don't include a "total" label value — rely on Prometheus sum(). 4. Separate read/write into different metrics — users typically care about one at a time. 5. Minimal labels — every label is a dimension users must handle in PromQL.

Metric Types

Match source metric behavior to Prometheus types:

• Source value only goes up → Counter
• Source value goes up and down → Gauge
• Source provides distribution data → Histogram or Summary
• Unknown or ambiguous → Untyped (safe default)

If a source counter can be decremented (e.g., Dropwizard metrics), it's actually a gauge. Use UNTYPED rather than misleading with GAUGE.

Collectors

Create New Metrics per Scrape

Do NOT use global metric variables that you update on each scrape. This causes race conditions between concurrent scrapes and stale label values.

Instead, create fresh metric instances each scrape:

• Go: MustNewConstMetric in Collect() method
• Python: Custom collector returning new metrics
• Java: Return List<MetricFamilySamples> in collect()

Scrape Meta-Metrics

myexporter_scrape_duration_seconds   (gauge — per-scrape duration)
myexporter_scrape_errors_total       (counter — collection failures)

Scrape duration is a gauge (single event measurement), not a histogram.

Up Metric

Expose myexporter_up (0 or 1) to indicate whether the target is reachable. This is preferred over returning 5xx when the target is down, because it allows partial metric export even when the target is unhealthy.

Help Strings

Include the original metric name, collector/exporter name, and any transformation rules in the HELP string. This helps users trace metrics back to their source.

# HELP haproxy_server_bytes_in_total Total bytes received from server.
# Derived from HAProxy stat: bin

Drop Unnecessary Stats

Source systems often expose pre-computed rates (1m, 5m, 15m averages), min/max since start, and standard deviations. Drop all of these:

• Prometheus computes rates more accurately via rate()
• Min/max have unknown time windows
• Standard deviation is statistically useless without context

Export raw counters and current values. Let Prometheus do the math.

Push-Based Sources

For systems that push metrics (StatsD, Graphite, collectd):

1. Expiry: Set a TTL for pushed metrics. Collectd includes expiry time; Graphite needs a flag. 2. Counters: Prefer raw counters over deltas — matches the Prometheus model. 3. Batch jobs: Push to Pushgateway and exit. Don't manage state in the exporter.

Instrumentation

Instrument everything. Every library, subsystem, and service should have at least a few metrics. Instrumentation should be an integral part of your code — define metrics in the same file you use them.

Service Types

Online-Serving Systems

HTTP servers, databases, APIs — anything where a human or system expects an immediate response.

Key metrics:

• Request rate: http_requests_total{method, status, handler}
• Error rate: http_requests_total{status=~"5.."} or separate http_errors_total
• Latency: http_request_duration_seconds (histogram)
• In-progress: http_requests_in_progress (gauge)

Guidelines:

• Monitor both client and server side when possible
• Count requests at completion (not start) — aligns with error and latency stats
• Use a histogram for latency — enables percentile calculations and aggregation
• Always have a total requests counter alongside error counters (for ratio calculation)

Offline Processing

Queues, pipelines, ETL jobs — processing happens asynchronously.

Key metrics per stage:

• Items in: pipeline_items_received_total{stage}
• Items out: pipeline_items_processed_total{stage}
• In progress: pipeline_items_in_progress{stage} (gauge)
• Last processed timestamp: pipeline_last_processed_timestamp_seconds{stage} (gauge)
• Processing duration: pipeline_processing_duration_seconds{stage} (histogram)

Guidelines:

• Track items at each stage to detect bottlenecks and stalls
• Export heartbeat timestamps to detect stalled processing
• If batching, also track batch count and size

Batch Jobs

Cron jobs, scheduled tasks — do not run continuously.

Key metrics (push to Pushgateway):

• Last success: job_last_success_timestamp_seconds (gauge)
• Last completion: job_last_completion_timestamp_seconds (gauge)
• Duration: job_duration_seconds (gauge — represents single run, not distribution)
• Records processed: job_records_processed_total (counter)

Guidelines:

• Push to Pushgateway at job completion
• Batch job durations are gauges (single event), not histograms
• For jobs running > few minutes, also expose pull-based metrics for live monitoring
• Jobs running more often than every 15 minutes should be converted to daemons

Subsystem Patterns

Libraries

Instrument transparently — users should get metrics without configuration.

Minimum for external resource access:

• Request count (counter)
• Error count (counter)
• Latency (histogram)

Distinguish uses with labels where appropriate (e.g., database connection pool should label by database name).

Logging

For every log level, maintain a counter:

log_messages_total{level="info"}
log_messages_total{level="warning"}
log_messages_total{level="error"}

Check for significant changes in log rates as part of release validation.

Failures

Every failure increments a counter. Always pair with a total attempts counter:

http_requests_total{handler="/api"}          # total
http_request_errors_total{handler="/api"}    # failures

Failure ratio: rate(http_request_errors_total[5m]) / rate(http_requests_total[5m])

Threadpools

threadpool_tasks_queued        (gauge)
threadpool_threads_active      (gauge)
threadpool_threads_total       (gauge)
threadpool_tasks_completed_total  (counter)
threadpool_task_duration_seconds  (histogram)
threadpool_queue_wait_seconds     (histogram)

Caches

cache_requests_total{result="hit"}
cache_requests_total{result="miss"}
cache_evictions_total
cache_size                     (gauge — current number of entries)
cache_request_duration_seconds (histogram — cache lookup latency)

Plus the downstream system's metrics (the system the cache sits in front of).

Custom Collectors

When implementing a non-trivial custom collector:

mycollector_scrape_duration_seconds  (gauge — time to collect)
mycollector_scrape_errors_total      (counter — collection failures)

Collector durations are gauges (per-scrape measurement), not histograms.

Things to Watch Out For

Use Labels, Not Separate Metrics

# Bad — separate metrics per status code
http_responses_500_total
http_responses_403_total

# Good — single metric with label
http_responses_total{code="500"}
http_responses_total{code="403"}

Don't Overuse Labels

Most metrics should have no labels. Start with none and add as concrete use cases emerge. Keep cardinality below 10 per metric as a default target.

Timestamps, Not Durations

# Bad — requires update logic, stale if process hangs
time_since_last_success_seconds

# Good — compute elapsed time in PromQL: time() - last_success_timestamp_seconds
last_success_timestamp_seconds

Avoid Missing Metrics

Initialize metrics with default values (typically 0) at startup. Most client libraries do this automatically for metrics without labels. For labeled metrics, call the label combination once with a zero value.

Performance in Hot Paths

Counter increments cost ~12-17ns (Java benchmark). For code called >100K times per second:

• Limit metrics incremented in the inner loop
• Cache label lookup results (e.g., With() return value in Go)
• Avoid time-based observations in tight loops (syscall overhead)

Metric Types

Four core metric types. Choose correctly at instrumentation time — changing later requires migration of dashboards, alerts, and recording rules.

Decision Tree

Can the value decrease?
├── No  → Counter
└── Yes → Is it a snapshot of current state?
    ├── Yes → Gauge
    └── No  → Are you observing a distribution (latency, size)?
        ├── Yes → Do you need to aggregate across instances?
        │   ├── Yes → Histogram
        │   └── No  → Summary (but histogram still preferred)
        └── No  → Gauge

Counter

A monotonically increasing value that can only go up or reset to zero on restart.

Use for: requests served, errors occurred, bytes transferred, tasks completed.

API: inc(), inc(v) where v >= 0.

Rules:

• Always suffix with _total: http_requests_total
• Always apply rate() or increase() in queries — raw values are meaningless
• Starts at 0, never decreases
• Prometheus handles counter resets (process restarts) automatically in rate()

Common mistake: Using a counter for values that can decrease (e.g., current connections). Use a gauge instead.

# Good: counter for cumulative events
http_requests_total{method="GET", status="200"} 14832

# In PromQL: per-second rate over 5 minutes
rate(http_requests_total[5m])

Gauge

A value that can go up and down arbitrarily.

Use for: temperature, memory usage, in-progress requests, queue depth, configuration values, timestamps.

API: inc(), dec(), set(v), set_to_current_time().

Rules:

• No _total suffix
• Never apply rate() to a gauge — use deriv() or delta() if needed
• Use for "what is the current state?" questions
• For timestamps, store Unix epoch seconds: last_success_timestamp_seconds

Two special gauge patterns: 1. Timestamps — myapp_last_success_timestamp_seconds with Unix epoch. Query elapsed time with time() - myapp_last_success_timestamp_seconds. 2. Info metrics — myapp_build_info{version="1.2.3", commit="abc"} 1. Pseudo-metric exposing metadata as labels with value 1.

# Good: gauge for current state
node_memory_MemAvailable_bytes 4.123e+09

# Good: timestamp gauge
batch_job_last_success_timestamp_seconds 1.7e+09

Histogram

Samples observations into configurable buckets. Produces multiple time series: _bucket{le="..."}, _sum, and _count.

Use for: request latencies, response sizes, any distribution where you need percentiles or aggregation across instances.

API: observe(v).

Rules:

• Buckets are cumulative — le="0.5" includes all observations <= 0.5
• Must include +Inf bucket (equal to _count)
• Choose buckets matching expected value range (e.g., `0.005, 0.01, 0.025, 0.05,

0.1, 0.25, 0.5, 1, 2.5, 5, 10` for HTTP latency in seconds)

• Use histogram_quantile() in PromQL to calculate percentiles
• Aggregatable across instances — the primary advantage over summary

Choosing buckets:

• Cover the expected range of values
• Place more buckets near your SLO boundaries for higher accuracy
• Default buckets work for many HTTP latency use cases
• Buckets cannot be changed after metric creation

# Histogram exposes multiple series:
http_request_duration_seconds_bucket{le="0.1"}  24054
http_request_duration_seconds_bucket{le="0.25"} 33342
http_request_duration_seconds_bucket{le="0.5"}  100392
http_request_duration_seconds_bucket{le="1"}    129389
http_request_duration_seconds_bucket{le="+Inf"} 133988
http_request_duration_seconds_sum               53423
http_request_duration_seconds_count             133988

# PromQL: 95th percentile across all instances
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# PromQL: average request duration
rate(http_request_duration_seconds_sum[5m])
/
rate(http_request_duration_seconds_count[5m])

# PromQL: Apdex score (target 300ms, tolerable 1.2s)
(
  sum(rate(http_request_duration_seconds_bucket{le="0.3"}[5m])) by (job)
+
  sum(rate(http_request_duration_seconds_bucket{le="1.2"}[5m])) by (job)
) / 2 / sum(rate(http_request_duration_seconds_count[5m])) by (job)

Summary

Calculates streaming quantiles on the client side. Produces {quantile="..."}, _sum, and _count.

Use for: accurate quantiles from a single instance when you know the exact quantiles needed at instrumentation time.

API: observe(v).

Rules:

• Quantiles are pre-configured and cannot be changed at query time
• Cannot be aggregated across instances — avg(x{quantile="0.95"}) is

statistically invalid

• Client-side computation is more expensive than histogram
• _sum and _count without quantiles is a valid and useful configuration

When to use summary over histogram: Almost never. Use histogram unless ALL of these are true: 1. You need accurate quantiles (not just approximate) 2. From a single instance (no cross-instance aggregation needed) 3. You know the exact quantiles at instrumentation time 4. You accept that you cannot add new quantiles later without code changes

Histogram vs Summary Comparison

Default choice: histogram. Switch to summary only with a specific, justified reason.

Naming Conventions

Naming is the contract between instrumentation and consumption. A well-named metric is self-documenting — someone unfamiliar with the system should guess what it measures.

Metric Name Structure

<namespace>_<subsystem>_<name>_<unit>_<suffix>

• namespace — Application or domain prefix: http, myapp, process
• subsystem — Component within the application: request, db, cache
• name — What is being measured: duration, size, total
• unit — Base unit in plural: seconds, bytes, meters
• suffix — Type indicator: _total for counters, _info for info metrics

Not all parts are required. The minimum is namespace + meaningful name + unit/suffix.

Rules

Character Set

• Use [a-zA-Z_:][a-zA-Z0-9_:]* for maximum compatibility
• Prefer snake_case — lowercase with underscores
• Colons (:) are reserved for recording rules — never in direct instrumentation
• Prefixes __ (double underscore) are reserved for Prometheus internals

Application Prefix

Every metric MUST have a namespace prefix identifying its origin:

prometheus_notifications_total       # Prometheus server metrics
process_cpu_seconds_total            # Standard process metrics
http_request_duration_seconds        # Generic HTTP metrics
myapp_orders_processed_total         # Application-specific metrics

Base Units

Always use base units. Let visualization tools handle conversion.

Unit Suffix

Append the unit to the metric name in plural form:

http_request_duration_seconds        # time in seconds
node_memory_usage_bytes              # memory in bytes
disk_usage_ratio                     # ratio 0-1

Type Suffix

Name Ordering for Sorting

Order components so related metrics sort together lexicographically:

# Good — common prefix groups related metrics
prometheus_tsdb_head_truncations_closed_total
prometheus_tsdb_head_truncations_established_total
prometheus_tsdb_head_truncations_failed_total
prometheus_tsdb_head_truncations_total

Semantic Consistency

A metric MUST represent the same logical thing across all its label dimensions. Test: sum() or avg() across all dimensions should be meaningful.

# Good — all dimensions measure the same thing (request duration)
http_request_duration_seconds{method="GET"}
http_request_duration_seconds{method="POST"}

# Bad — mixing different things under one name
resource_usage{type="cpu"}      # percentage
resource_usage{type="memory"}   # bytes

If sum() across dimensions is nonsensical, split into separate metrics.

Label Rules

When to Use Labels

Use labels to differentiate characteristics of the thing being measured:

api_http_requests_total{operation="create"}
api_http_requests_total{operation="update"}
api_http_requests_total{operation="delete"}

When NOT to Use Labels

• Unbounded values — user IDs, email addresses, full URLs, query strings
• High cardinality — anything above ~100 unique values per metric
• Label names in metric names — http_requests_by_method_total is redundant

if there's a method label

Cardinality Guidelines

Cardinality math: Total time series = metric cardinality x number of targets. A metric with 100 label combinations across 1000 targets = 100,000 time series.

Reserved Labels

• __* (double underscore prefix) — Prometheus internal use
• le — Histogram bucket boundary
• quantile — Summary quantile value
• job, instance — Set by Prometheus scrape configuration

Label Best Practices

1. Minimal labels. Every label is a dimension users must consider in PromQL. 2. Stable label values. Avoid labels that change frequently. 3. Initialize all combinations. Export 0 for known label sets to prevent missing metrics. 4. Separate read/write. Use separate metrics rather than a direction label — users typically care about one at a time. 5. No "total" label value. Don't include a total or empty aggregation label — rely on Prometheus sum() instead.

PromQL

Prometheus Query Language — a functional language for selecting and aggregating time series data. Understanding PromQL's data types, operators, and functions is essential for dashboards, alerts, and recording rules.

Data Types

Range vectors cannot be graphed directly — they must be passed through a function like rate() that returns an instant vector.

Selectors

Instant Vector

http_requests_total                                    # by name
http_requests_total{job="api", method="GET"}           # with label matchers
http_requests_total{status=~"5.."}                     # regex match
http_requests_total{method!="OPTIONS"}                 # negative match

Label matchers:

• = exact match
• != not equal
• =~ regex match (fully anchored: "foo" becomes "^foo$")
• !~ negative regex match

Range Vector

Append [duration] to select a time range:

http_requests_total{job="api"}[5m]      # last 5 minutes
http_requests_total[1h]                 # last 1 hour

Duration units: ms, s, m, h, d, w, y

Combine: 1h30m, 12h34m56s

Modifiers

http_requests_total offset 5m           # 5 minutes ago
http_requests_total @ 1609746000        # at specific Unix timestamp
rate(http_requests_total[5m] offset 1w) # rate one week ago

Key Functions

Rate and Increase (Counters)

# Per-second rate over 5 minutes — use for alerts, dashboards
rate(http_requests_total[5m])

# Total increase over 5 minutes — use for human-readable counts
increase(http_requests_total[5m])

# Instant rate from last two samples — use for volatile, fast-moving counters only
irate(http_requests_total[5m])

Rules:

• rate() first, then aggregate: sum(rate(x[5m])) not rate(sum(x)[5m])
• rate() for alerts and slow counters; irate() only for graphing volatile counters
• increase() is syntactic sugar for rate() * range_seconds
• Use rate() in recording rules for consistent per-second tracking

Histogram Quantiles

# 95th percentile from a single histogram
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# 95th percentile aggregated across instances, by job
histogram_quantile(0.95, sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))

# Average request duration
rate(http_request_duration_seconds_sum[5m])
/
rate(http_request_duration_seconds_count[5m])

Critical: When aggregating histogram buckets, always preserve le in the by clause — histogram_quantile() requires it.

Gauges

# Current value
node_memory_MemAvailable_bytes

# Change over time (not rate — gauges are not counters)
delta(cpu_temp_celsius[2h])

# Derivative (per-second change)
deriv(node_memory_MemAvailable_bytes[5m])

# Predict future value
predict_linear(node_filesystem_avail_bytes[1h], 4*3600)  # 4 hours from now

Existence Checks

# Returns 1 if metric is absent — useful for alerting on missing metrics
absent(up{job="myservice"})

# Same but over a time range
absent_over_time(up{job="myservice"}[5m])

Aggregation Over Time

avg_over_time(metric[1h])       # average over last hour
max_over_time(metric[1h])       # max over last hour
min_over_time(metric[1h])       # min over last hour
count_over_time(metric[1h])     # number of samples
quantile_over_time(0.95, metric[1h])  # 95th percentile over time

Aggregation Operators

All aggregation operators take an instant vector and return a new vector with fewer elements.

sum(v)              # sum across dimensions
avg(v)              # average
min(v) / max(v)     # extremes
count(v)            # count of series
topk(k, v)          # top k by value
bottomk(k, v)       # bottom k by value
quantile(0.95, v)   # quantile across series
stddev(v)           # standard deviation
group(v)            # returns 1 for each group (existence check)
count_values("label", v)  # count unique values

Dimension Control

# Keep only specified labels
sum by (job, method) (rate(http_requests_total[5m]))

# Remove specified labels (keep everything else)
sum without (instance) (rate(http_requests_total[5m]))

Prefer without when aggregating away a few labels — it preserves all other labels including job, avoiding conflicts.

Binary Operators

Arithmetic

+, -, *, /, %, ^

# Unused memory in MiB
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / 1024 / 1024

Comparison (filtering)

==, !=, >, <, >=, <=

# Only series where value > 100
http_requests_total > 100

# Returns 0 or 1 with bool modifier
http_requests_total > bool 100

Logical/Set

vector1 and vector2       # intersection
vector1 or vector2        # union
vector1 unless vector2    # complement (in v1 but not v2)

Vector Matching

# One-to-one: match on specific labels
method_code:http_errors:rate5m{code="500"} / ignoring(code) method:http_requests:rate5m

# Many-to-one with group_left
method_code:http_errors:rate5m / ignoring(code) group_left method:http_requests:rate5m

`on(labels)` — match only on listed labels. `ignoring(labels)` — match on all labels except listed. `group_left` / `group_right` — enable many-to-one matching.

Operator Precedence (highest to lowest)

1. ^ 2. *, /, %, atan2 3. +, - 4. ==, !=, <=, <, >=, > 5. and, unless 6. or

All left-associative except ^ (right-associative).

Common Query Patterns

# Request rate by job
sum by (job) (rate(http_requests_total[5m]))

# Error ratio
sum(rate(http_request_errors_total[5m])) / sum(rate(http_requests_total[5m]))

# P99 latency by handler
histogram_quantile(0.99, sum by (handler, le) (rate(http_request_duration_seconds_bucket[5m])))

# Top 5 CPU consumers
topk(5, sum by (app) (rate(process_cpu_seconds_total[5m])))

# Disk space prediction — time until full
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600) < 0

# Alert on missing scrape target
absent(up{job="critical-service"} == 1)

Gotchas

Staleness

Prometheus returns the most recent sample within the lookback period (default 5 minutes). If a series stops being scraped, it goes stale and disappears from queries.

Rate Window Size

rate() needs at least two samples in the range. With a 15s scrape interval, rate(x[30s]) may have only 2 points. Use rate(x[5m]) or wider for reliability.

Rule of thumb: range should be at least 4x the scrape interval.

Avoid Expensive Queries

• Bare metric names like http_requests_total can expand to thousands of series
• Always filter or aggregate before graphing
• Use recording rules to pre-compute expensive expressions