Promql Cli

promql-cli — Prometheus Query CLI Skill

promql-cli (github.com/nalbury/promql-cli) is a Go CLI for querying, analyzing, and visualizing Prometheus metrics, plus PromQL fundamentals.

Reference Files

Read the relevant reference file(s) before executing tasks:

For most tasks, read references/usage.md. For PromQL help, read references/promql-reference.md. When debugging, read both references/debugging.md and references/promql-reference.md.

Setup Check

Before running any query, verify that a host is configured:

promql 'up'   # succeeds if host is reachable; fails with connection error if not configured
# or
promql --host xxx 'up'

Recognize these errors as a configuration/auth problem and refer to references/installation.md:

If any of these appear, do not create config files on behalf of the user — config files may contain credentials (tokens, passwords) that must never pass through an LLM. Instead, guide the user to set it up themselves:

"Please create ~/.promql-cli.yaml manually with your Prometheus host (and credentials if needed). See references/installation.md for the exact format. Let me know once it's ready."

Only after the user confirms the config is in place should you proceed with queries.

Quick Command Reference

promql 'up'                                          # instant query
promql 'rate(http_requests_total[5m])' --start 1h    # range query (ASCII graph)
promql 'up' --output csv                             # CSV output
promql 'up' --output json                            # JSON output
promql metrics                                       # list all metric names
promql labels <metric>                               # list labels for a metric
promql meta <metric>                                 # show metric type and help
promql --config ~/.promql-cli-prod.yaml 'up'         # target a specific host

Key Principles

1. Use `rate()` on counters, never raw values — raw counters only ever increase; the absolute value is meaningless. rate() gives the per-second change rate, which is what you actually care about. 2. When debugging, isolate a single instance — aggregating across replicas masks per-instance anomalies. A single overloaded pod hidden behind healthy peers won't show up in averages. 3. Filter early with label matchers in the innermost selector — Prometheus evaluates selectors before functions, so filtering late means scanning all time series. Early filters reduce data scanned and query latency. 4. For histograms, keep `le` in the `by` clause before histogram_quantile() — the function needs all le buckets to interpolate percentiles; dropping le early produces NaN or wrong results. 5. Prefer `--output graph` for range queries — ASCII sparklines convey trend direction (rising, falling, spiking) in a compact format that LLMs parse well; raw timestamp tables require mental modeling. 6. Store credentials in `~/.promql-cli.yaml` and `~/.promql_token`, chmod 600 — passing tokens as CLI args exposes them in shell history and process listings.

This skill is not exhaustive. Please refer to the official promql-cli documentation and examples for up-to-date information. Context7 can help as a discoverability platform.

If you encounter a bug or unexpected behavior in promql-cli itself, open an issue at https://github.com/nalbury/promql-cli/issues.

promql-cli — Debugging Methodology

Core Rule: Isolate Before Aggregating

The most common debugging mistake is starting with aggregated metrics. Aggregation masks individual bad actors — a single overloaded pod looks fine when its metrics are averaged with 9 healthy ones. Always start by narrowing to a single instance, then broaden once you understand what's happening.

USE Method (Utilization, Saturation, Errors)

For infrastructure components (CPU, memory, disk, network):

# Utilization — how busy is the resource?
promql 'avg by(instance)(rate(node_cpu_seconds_total{mode!="idle"}[5m]))' --start 1h --output graph

# Saturation — is the resource overloaded? (queue depth, wait time)
promql 'node_load1 / on(instance) count by(instance)(node_cpu_seconds_total{mode="idle"})' --start 1h

# Errors — are requests failing?
promql 'rate(node_disk_io_time_seconds_total[5m])' --start 1h --output graph

RED Method (Rate, Errors, Duration)

For services and HTTP endpoints:

# Rate — requests per second
promql 'sum by(job)(rate(http_requests_total[5m]))' --start 1h --output graph

# Errors — error ratio
promql 'sum by(job)(rate(http_requests_total{status=~"5.."}[5m])) / sum by(job)(rate(http_requests_total[5m]))' --start 1h

# Duration — latency (requires histogram metric)
promql 'histogram_quantile(0.99, sum by(le, job)(rate(http_request_duration_seconds_bucket[5m])))' --start 1h --output graph

Step-by-Step: Investigating a Latency Spike

1. Identify the time window

promql 'histogram_quantile(0.99, sum by(le)(rate(http_request_duration_seconds_bucket[5m])))' --start 6h --output graph

2. Isolate by instance — don't aggregate yet

# Remove the sum(), keep instance labels
promql 'histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{job="api"}[5m]))' --start 2h --output graph

3. Check resource pressure on the worst instance

promql 'rate(node_cpu_seconds_total{mode!="idle", instance="bad-host:9100"}[5m])' --start 2h --output graph
promql 'node_memory_MemAvailable_bytes{instance="bad-host:9100"}' --start 2h --output graph

4. Correlate with upstream dependencies

promql 'rate(db_query_duration_seconds_sum{instance="bad-host:5432"}[5m]) / rate(db_query_duration_seconds_count{instance="bad-host:5432"}[5m])' --start 2h --output graph

Step-by-Step: Investigating an Error Rate Spike

1. Get the overall error rate

promql 'sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))' --start 3h --output graph

2. Break down by endpoint

promql 'sum by(handler)(rate(http_requests_total{status=~"5.."}[5m]))' --start 1h --output csv | sort -t, -k2 -rn | head -10

3. Check for upstream failures (dependency error rates)

promql 'rate(grpc_client_handled_total{grpc_code!="OK"}[5m])' --start 1h --output graph

Diagnose: Useful Checks

Diagnose: 1- promql 'up' --output table — check which scrape targets are down; a missing instance explains missing metrics 2- promql metrics | grep <service> — discover what metrics a service actually exposes before querying 3- promql meta <metric> — confirm the metric type (counter vs gauge) before applying rate() or increase()

Finding the Right Metrics

When you don't know which metrics or thresholds are meaningful for a given component, start with promql metrics and promql meta to discover what your Prometheus instance exposes. For reference, the Awesome Prometheus Alerts project (samber/awesome-prometheus-alerts) maintains a curated collection of battle-tested PromQL alert rules organized by exporter, covering:

Standard alert expressions are valid PromQL — you can use similar patterns with promql-cli to inspect current values. Example workflow:

# 1. Discover relevant metrics in your Prometheus instance
# 2. Write a PromQL expression based on common alerting patterns
# 3. Run it to see the current value
promql 'node_filesystem_avail_bytes{fstype!="tmpfs"} / node_filesystem_size_bytes{fstype!="tmpfs"} * 100' --output table

# 4. Run as a range query to see the trend
promql 'node_filesystem_avail_bytes{fstype!="tmpfs"} / node_filesystem_size_bytes{fstype!="tmpfs"} * 100' --start 6h --output graph

Exporter documentation — when alert rules aren't enough, check the official exporter docs for the full list of exposed metrics and their semantics. Each exporter's README lists all metric names and labels.

Listing Available Metrics

Use promql metrics to discover what's actually exposed in your Prometheus instance. On large production setups this can return thousands of metric names — always filter immediately or the output becomes unmanageable.

# ✗ Bad — dumps every metric name, potentially thousands of lines
promql metrics

# ✓ Good — filter by exporter prefix or keyword
promql metrics | grep '^node_'          # node-exporter metrics
promql metrics | grep '^container_'     # cAdvisor / Kubernetes metrics
promql metrics | grep '^pg_'            # PostgreSQL exporter
promql metrics | grep '^redis_'         # Redis exporter
promql metrics | grep '^kafka_'         # Kafka exporter
promql metrics | grep http              # any metric mentioning http

Once you have a metric name, drill into its labels and type:

promql labels <metric>         # list all label names
promql labels <metric> job     # list all values for a specific label
promql meta <metric>           # show metric type (counter/gauge/histogram) and help text

Label values help you understand the cardinality before running a query — a metric with hundreds of instance values will return a large result set unless filtered:

# Check how many instances exist before querying
promql labels http_requests_total instance

# Then filter down to the relevant one
promql 'rate(http_requests_total{instance="api-1:8080"}[5m])' --output table

promql-cli — ASCII Graphs

Use --output graph for an in-terminal ASCII sparkline of range queries. No external dependencies required.

promql 'rate(http_requests_total[5m])' --start 1h --output graph
promql 'node_memory_MemAvailable_bytes' --start 6h --step 5m --output graph

PromQL aggregations return multiple series — each becomes a separate line in the chart:

# One line per job
promql 'sum by(job)(rate(http_requests_total[5m]))' --start 1h --output graph

# One line per instance
promql 'rate(http_requests_total{job="api"}[5m])' --start 1h --output graph

Save to file by redirecting stdout:

promql 'node_load1' --start 6h --output graph > load_trend.txt

--output graph is the default for range queries (when --start is set) — omitting --output has the same effect.

promql-cli — Installation

Latest release: v0.3.0 — macOS and Linux only (no official Windows binary). Please check the releases to find the latest version.

macOS

Pre-built binary (recommended)

# Intel (x86_64)
curl -L https://github.com/nalbury/promql-cli/releases/download/v0.3.0/promql-v0.3.0-darwin-amd64.tar.gz | tar xz
sudo mv promql /usr/local/bin/

# Apple Silicon (M1/M2/M3/M4/M+)
curl -L https://github.com/nalbury/promql-cli/releases/download/v0.3.0/promql-v0.3.0-darwin-arm64.tar.gz | tar xz
sudo mv promql /usr/local/bin/

Build from source (requires Go 1.13+)

git clone https://github.com/nalbury/promql-cli.git
cd promql-cli
OS=darwin ARCH=amd64 INSTALL_PATH=/usr/local/bin make install   # Intel
OS=darwin ARCH=arm64 INSTALL_PATH=/usr/local/bin make install   # Apple Silicon

go install

go install github.com/nalbury/promql-cli@latest
# Binary lands in $(go env GOPATH)/bin — ensure that's on your PATH

Linux

Pre-built binary (recommended)

# x86_64 (amd64)
curl -L https://github.com/nalbury/promql-cli/releases/download/v0.3.0/promql-v0.3.0-linux-amd64.tar.gz | tar xz
sudo mv promql /usr/local/bin/

# ARM64 (Raspberry Pi 4+, AWS Graviton, etc.)
curl -L https://github.com/nalbury/promql-cli/releases/download/v0.3.0/promql-v0.3.0-linux-arm64.tar.gz | tar xz
sudo mv promql /usr/local/bin/

Build from source (requires Go 1.13+)

git clone https://github.com/nalbury/promql-cli.git
cd promql-cli
OS=linux ARCH=amd64 INSTALL_PATH=/usr/local/bin make install    # x86_64
OS=linux ARCH=arm64 INSTALL_PATH=/usr/local/bin make install    # ARM64

go install

go install github.com/nalbury/promql-cli@latest

Windows

No official Windows binary is provided. Options:

WSL2 (recommended): Install the Linux binary inside your WSL2 environment — full feature support.

Build from source:

git clone https://github.com/nalbury/promql-cli.git
cd promql-cli
$env:GOOS="windows"; $env:GOARCH="amd64"; go build -o promql.exe ./
# Move promql.exe to a directory on your PATH

Verify Installation

Test against your local Prometheus instance (default: http://localhost:9090):

promql --version
promql 'up' --host http://localhost:9090
promql metrics --host http://localhost:9090 | head -20

Configuration

By default promql-cli connects to http://localhost:9090. Override with --host or a config file:

# ~/.promql-cli.yaml
host: http://prometheus.acme.org:9090
timeout: 30s

chmod 600 ~/.promql-cli.yaml   # file may contain tokens — restrict access

Multi-Host Setup

Use separate config files per environment, switch with --config:

# ~/.promql-cli-prod.yaml
host: https://prometheus-prod.acme.org:9090

# ~/.promql-cli-staging.yaml
host: https://prometheus-staging.acme.org:9090

promql --config ~/.promql-cli-prod.yaml 'up'
promql --config ~/.promql-cli-staging.yaml 'up'

Authentication

Never pass tokens or passwords as CLI arguments — they appear in shell history (~/.bash_history, ~/.zsh_history), in process listings (ps aux) and are sent to an LLM. Always store secrets in files under $HOME with restricted permissions.

Bearer token:

echo "your-token" > ~/.promql_token
chmod 600 ~/.promql_token

# ~/.promql-cli.yaml  (chmod 600)
host: https://prometheus.acme.org:9090
token-file: ~/.promql_token # ✓ path to file — token never appears in CLI args

Basic auth:

echo "your-password" > ~/.promql_password
chmod 600 ~/.promql_password

# ~/.promql-cli.yaml  (chmod 600)
host: https://prometheus.acme.org:9090
username: admin
password-file: ~/.promql_password # ✓ path to file — password never appears in CLI args

TLS / mTLS:

host: https://prometheus.acme.org:9090
ca-cert: /path/to/ca.crt
client-cert: /path/to/client.crt # only for mTLS
client-key: /path/to/client.key # only for mTLS
insecure-skip-verify: false

PromQL Reference

Metric Types

Using rate() on a gauge produces meaningless results. Using raw counter values ignores resets and gives absolute counts rather than rates — always check the metric type with promql meta <metric> before querying.

Key Functions

rate() and irate()

rate(http_requests_total[5m])    # average per-second rate over 5m window (smoothed)
irate(http_requests_total[5m])   # instantaneous rate (last 2 samples) — spiky, reactive

Use rate() for dashboards and alerts. Use irate() only when you need to detect very short spikes.

increase()

increase(http_requests_total[1h])   # total increase over 1h (extrapolated)

increase() is rate() * duration — useful for "how many requests in the last hour?" rather than per-second rates.

histogram_quantile()

# p99 latency across all instances
histogram_quantile(0.99, sum by(le)(rate(http_request_duration_seconds_bucket[5m])))

# p99 per job — keep job and le in by()
histogram_quantile(0.99, sum by(le, job)(rate(http_request_duration_seconds_bucket[5m])))

The le label must stay in the by() clause — it identifies the buckets that histogram_quantile() needs to interpolate percentiles. Dropping it produces NaN.

Aggregation Operators

sum by(job)(rate(http_requests_total[5m]))       # sum, grouped by job
avg without(instance)(node_memory_MemFree_bytes) # avg, dropping instance label
max by(instance)(node_load1)                     # max per instance
min(node_memory_MemAvailable_bytes)              # global min
count by(job)(up)                                # count of series
topk(5, rate(http_requests_total[5m]))           # top 5 series by value
bottomk(3, node_memory_MemAvailable_bytes)       # bottom 3

Label Matchers

http_requests_total{job="api"}                   # exact match
http_requests_total{job!="api"}                  # not equal
http_requests_total{status=~"5.."}               # regex match (RE2)
http_requests_total{status!~"2.."}               # regex not-match
http_requests_total{job="api", status=~"5.."}    # multiple matchers (AND)

Filter as early as possible — put matchers in the innermost vector selector, not after functions. This reduces the number of time series Prometheus evaluates.

Binary Operators

# Error ratio
rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])

# CPU usage ratio
1 - avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m]))

# Matching labels across metrics (one-to-one)
http_requests_total / on(instance, job) http_request_duration_seconds_sum

offset

rate(http_requests_total[5m]) offset 1h    # same query, 1 hour ago (for comparison)

Common Pitfalls

promql-cli — Usage Reference

Instant Queries

Return a single value at the current time (or a specified time):

promql 'up'                                                      # all targets
promql 'up{job="node"}'                                          # filter by label
promql 'up' --time 2024-01-15T10:00:00Z                          # at a specific time
promql 'rate(http_requests_total[5m])'                           # computed rate

Range Queries

Return a time series over a window. --start accepts durations or RFC3339:

promql 'rate(http_requests_total[5m])' --start 1h                # last 1 hour
promql 'up' --start 2h --end 1h                                  # 2h ago to 1h ago
promql 'up' --start 2024-01-15T00:00:00Z --end 2024-01-15T12:00:00Z
promql 'up' --start 1h --step 5m                                 # custom step (default: auto)

Output Formats

promql 'up'                          # default: table (aligned columns)
promql 'up' --output table           # explicit table
promql 'up' --output csv             # CSV — pipe to files, spreadsheets
promql 'up' --output json            # JSON — pipe to jq for processing
promql 'up' --output raw             # raw PromQL API response
promql 'rate(http_requests_total[5m])' --start 1h --output graph  # ASCII sparkline — preferred for range queries

Prefer `--output graph` for range queries when working with an LLM. ASCII sparklines convey trend direction (rising, falling, spiking, flat) in a compact format that LLMs parse well — far better than a table of raw timestamps and values.

Metric Discovery

promql metrics                       # list all metric names in Prometheus
promql metrics | grep http           # filter metrics by name
promql labels http_requests_total    # list all label names for a metric
promql labels http_requests_total job  # list values for a specific label
promql meta http_requests_total      # show metric type (counter/gauge/etc.) and HELP text

Targeting a Specific Host

promql --host http://prometheus.acme.org:9090 'up'
promql --config ~/.promql-cli-prod.yaml 'up'

Full Flag Reference

Combining with Shell Tools

# Find the top 5 jobs by request rate
promql 'sum by(job)(rate(http_requests_total[5m]))' --output csv | sort -t, -k2 -rn | head -5

# Export a time series to a file
promql 'node_cpu_seconds_total' --start 24h --output csv > cpu_export.csv

# Parse JSON with jq
promql 'up' --output json | jq '.data.result[] | {metric: .metric, value: .value[1]}'