
Prometheus
- 48 installs
- 6 repo stars
- Updated July 22, 2026
- julianobarbosa/claude-code-skills
Query Prometheus metrics and create PromQL alerts for system monitoring
About
Provides HTTP API access for querying Prometheus metrics and PromQL expressions. Used when you need to extract monitoring data, create dashboards, or trigger alerts based on metrics.
- PromQL query construction and execution
- HTTP API integration for metric retrieval
Prometheus by the numbers
- 48 all-time installs (skills.sh)
- +1 installs in the week ending Aug 2, 2026 (Skillselion tracking)
- Ranked #743 of 1,435 DevOps & CI/CD skills by installs in the Skillselion catalog
- Data as of Aug 3, 2026 (Skillselion catalog sync)
npx skills add https://github.com/julianobarbosa/claude-code-skills --skill prometheusAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 48 |
|---|---|
| repo stars | ★ 6 |
| Last updated | July 22, 2026 |
| Repository | julianobarbosa/claude-code-skills ↗ |
What it does
Query Prometheus metrics and create PromQL alerts for system monitoring
Files
Prometheus API Skill
Query Prometheus monitoring systems via HTTP API at /api/v1.
Quick Reference
Instant Query
curl 'http://<prometheus>:9090/api/v1/query?query=<promql>&time=<timestamp>'Range Query
curl 'http://<prometheus>:9090/api/v1/query_range?query=<promql>&start=<ts>&end=<ts>&step=<duration>'Response Format
All responses return JSON:
{
"status": "success" | "error",
"data": <result>,
"errorType": "<string>",
"error": "<string>",
"warnings": ["<string>"]
}HTTP codes: 400 (bad params), 422 (expression error), 503 (timeout).
Query Endpoints
| Endpoint | Purpose | Key Parameters |
|---|---|---|
/api/v1/query | Instant query | query, time, timeout, limit |
/api/v1/query_range | Range query | query, start, end, step, timeout, limit |
/api/v1/format_query | Format PromQL | query |
/api/v1/series | Find series by labels | match[], start, end, limit |
/api/v1/labels | List label names | start, end, match[], limit |
/api/v1/label/<name>/values | Label values | start, end, match[], limit |
/api/v1/query_exemplars | Query exemplars | query, start, end |
Metadata & Status Endpoints
| Endpoint | Purpose |
|---|---|
/api/v1/targets | Target discovery status (`state=active\ |
/api/v1/targets/metadata | Metric metadata from targets |
/api/v1/metadata | All metric metadata |
/api/v1/rules | Alerting/recording rules |
/api/v1/alerts | Active alerts |
/api/v1/alertmanagers | Alertmanager discovery |
/api/v1/status/config | Current config YAML |
/api/v1/status/flags | CLI flags |
/api/v1/status/runtimeinfo | Runtime info |
/api/v1/status/buildinfo | Build info |
/api/v1/status/tsdb | TSDB cardinality stats |
/api/v1/status/walreplay | WAL replay progress |
Admin Endpoints (require --web.enable-admin-api)
| Endpoint | Method | Purpose |
|---|---|---|
/api/v1/admin/tsdb/snapshot | POST | Create TSDB snapshot |
/api/v1/admin/tsdb/delete_series | POST | Delete series (match[], start, end) |
/api/v1/admin/tsdb/clean_tombstones | POST | Clean deleted data |
Common PromQL Patterns
# Rate of counter over 5m
rate(http_requests_total[5m])
# Sum by label
sum by (job) (rate(http_requests_total[5m]))
# Percentile from histogram
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
# Filter by label
up{job="prometheus", instance=~".*:9090"}
# Increase over time
increase(http_requests_total[1h])
# Average over time range
avg_over_time(process_cpu_seconds_total[5m])Result Types
- vector:
[{"metric": {...}, "value": [timestamp, "value"]}] - matrix:
[{"metric": {...}, "values": [[ts, "val"], ...]}] - scalar:
[timestamp, "value"] - string:
[timestamp, "string"]
Scripts
Query script: scripts/prom_query.py
# Instant query
python scripts/prom_query.py http://localhost:9090 'up'
# Range query
python scripts/prom_query.py http://localhost:9090 'rate(http_requests_total[5m])' \
--start '2024-01-01T00:00:00Z' --end '2024-01-01T01:00:00Z' --step '1m'
# Output: table, json, csv
python scripts/prom_query.py http://localhost:9090 'up' --format tableHealth check: scripts/prom_health.py
python scripts/prom_health.py http://localhost:9090Detailed Reference
For complete API documentation: references/api_reference.md
For PromQL functions: references/promql_functions.md
---
Gotchas
- `rate()` over a counter that resets too often: math is correct but meaningless — use
increase()and divide by interval explicitly when counters don't survive scrapes. - `up{}` per-target gauge: a flaky target shows up=0 but doesn't trigger alerts unless
foris met. Set shortforfor liveness, long for noise. - Recording rules evaluate at fixed interval; missed evaluations don't backfill — gaps in the recording series during incidents.
- Federation `match[]` parameter requires ALL matchers to match — an empty matcher returns no series, which looks like a working query with no data.
- Stale-marker semantics: a series stops being scraped → stale marker after 5 min by default → queries see "no data" not "0". Affects alerts on
absent(). - Service Discovery + relabel_config: a bad regex in
keepaction silently drops all targets — verify with/api/v1/targetsafter each config change.
Prometheus HTTP API Reference
Complete API documentation for Prometheus /api/v1 endpoints.
Table of Contents
1. Expression Queries 2. Metadata Endpoints 3. Targets & Service Discovery 4. Rules & Alerts 5. Status Endpoints 6. Admin APIs 7. Response Formats
---
Expression Queries
Instant Query
Evaluate expression at single point in time.
GET/POST /api/v1/queryParameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
query | string | Yes | PromQL expression |
time | rfc3339/unix | No | Evaluation timestamp (default: now) |
timeout | duration | No | Query timeout |
limit | number | No | Max series returned (0=disabled) |
Example:
curl 'http://localhost:9090/api/v1/query?query=up&time=2024-01-01T00:00:00Z'Response:
{
"status": "success",
"data": {
"resultType": "vector",
"result": [
{
"metric": {"__name__": "up", "job": "prometheus", "instance": "localhost:9090"},
"value": [1704067200, "1"]
}
]
}
}Range Query
Evaluate expression over time range.
GET/POST /api/v1/query_rangeParameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
query | string | Yes | PromQL expression |
start | rfc3339/unix | Yes | Start timestamp (inclusive) |
end | rfc3339/unix | Yes | End timestamp (inclusive) |
step | duration/float | Yes | Resolution step width |
timeout | duration | No | Query timeout |
limit | number | No | Max series returned |
Example:
curl 'http://localhost:9090/api/v1/query_range?query=up&start=2024-01-01T00:00:00Z&end=2024-01-01T01:00:00Z&step=15s'Response:
{
"status": "success",
"data": {
"resultType": "matrix",
"result": [
{
"metric": {"__name__": "up", "job": "prometheus"},
"values": [
[1704067200, "1"],
[1704067215, "1"],
[1704067230, "1"]
]
}
]
}
}Format Query
Prettify PromQL expression.
GET/POST /api/v1/format_queryExample:
curl 'http://localhost:9090/api/v1/format_query?query=foo/bar'
# Returns: "foo / bar"---
Metadata Endpoints
Find Series
Return time series matching label sets.
GET/POST /api/v1/seriesParameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
match[] | series_selector | Yes | Repeated selector (at least one) |
start | rfc3339/unix | No | Start timestamp |
end | rfc3339/unix | No | End timestamp |
limit | number | No | Max series returned |
Example:
curl -g 'http://localhost:9090/api/v1/series?match[]=up&match[]=process_start_time_seconds{job="prometheus"}'Label Names
Return all label names.
GET/POST /api/v1/labelsParameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
start | rfc3339/unix | No | Start timestamp |
end | rfc3339/unix | No | End timestamp |
match[] | series_selector | No | Filter by series |
limit | number | No | Max labels returned |
Example:
curl 'http://localhost:9090/api/v1/labels'
# Returns: ["__name__", "instance", "job", ...]Label Values
Return values for specific label.
GET /api/v1/label/<label_name>/valuesParameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
start | rfc3339/unix | No | Start timestamp |
end | rfc3339/unix | No | End timestamp |
match[] | series_selector | No | Filter by series |
limit | number | No | Max values returned |
Example:
curl 'http://localhost:9090/api/v1/label/job/values'
# Returns: ["prometheus", "node", "alertmanager"]Metric Metadata
Return metadata for metrics.
GET /api/v1/metadataParameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
limit | number | No | Max metrics returned |
limit_per_metric | number | No | Max metadata per metric |
metric | string | No | Filter by metric name |
Example:
curl 'http://localhost:9090/api/v1/metadata?metric=http_requests_total'Response:
{
"status": "success",
"data": {
"http_requests_total": [
{"type": "counter", "help": "Total HTTP requests", "unit": ""}
]
}
}---
Targets & Service Discovery
Targets
Return target discovery state.
GET /api/v1/targetsParameters:
| Parameter | Type | Description |
|---|---|---|
state | string | Filter: active, dropped, any |
scrapePool | string | Filter by scrape pool name |
Example:
curl 'http://localhost:9090/api/v1/targets?state=active'Response:
{
"status": "success",
"data": {
"activeTargets": [
{
"discoveredLabels": {"__address__": "127.0.0.1:9090"},
"labels": {"instance": "127.0.0.1:9090", "job": "prometheus"},
"scrapePool": "prometheus",
"scrapeUrl": "http://127.0.0.1:9090/metrics",
"lastScrape": "2024-01-01T00:00:00Z",
"lastScrapeDuration": 0.05,
"health": "up"
}
],
"droppedTargets": []
}
}Target Metadata
Return metric metadata from targets.
GET /api/v1/targets/metadataParameters:
| Parameter | Type | Description |
|---|---|---|
match_target | label_selector | Filter targets |
metric | string | Filter by metric name |
limit | number | Max targets to match |
Example:
curl -G 'http://localhost:9090/api/v1/targets/metadata' \
--data-urlencode 'metric=go_goroutines' \
--data-urlencode 'match_target={job="prometheus"}'---
Rules & Alerts
Rules
Return alerting and recording rules.
GET /api/v1/rulesParameters:
| Parameter | Type | Description |
|---|---|---|
type | string | alert or record |
rule_name[] | string | Filter by rule name |
rule_group[] | string | Filter by group name |
file[] | string | Filter by file path |
exclude_alerts | bool | Exclude active alerts |
match[] | label_selector | Filter by configured labels |
Example:
curl 'http://localhost:9090/api/v1/rules?type=alert'Response:
{
"status": "success",
"data": {
"groups": [
{
"name": "example",
"file": "/rules.yaml",
"interval": 60,
"rules": [
{
"name": "HighRequestLatency",
"query": "job:request_latency_seconds:mean5m > 0.5",
"duration": 600,
"labels": {"severity": "page"},
"annotations": {"summary": "High request latency"},
"alerts": [],
"health": "ok",
"type": "alerting"
}
]
}
]
}
}Alerts
Return active alerts.
GET /api/v1/alertsExample:
curl 'http://localhost:9090/api/v1/alerts'Alertmanagers
Return Alertmanager discovery state.
GET /api/v1/alertmanagersResponse:
{
"status": "success",
"data": {
"activeAlertmanagers": [{"url": "http://127.0.0.1:9093/api/v1/alerts"}],
"droppedAlertmanagers": []
}
}---
Status Endpoints
Config
GET /api/v1/status/configReturns currently loaded configuration as YAML.
Flags
GET /api/v1/status/flagsReturns CLI flag values.
Runtime Info
GET /api/v1/status/runtimeinfoReturns runtime properties (startTime, goroutineCount, storageRetention, etc).
Build Info
GET /api/v1/status/buildinfoReturns version, revision, branch, buildDate, goVersion.
TSDB Stats
GET /api/v1/status/tsdb?limit=<n>Returns cardinality statistics:
headStats: numSeries, chunkCount, minTime, maxTimeseriesCountByMetricNamelabelValueCountByLabelNamememoryInBytesByLabelNameseriesCountByLabelValuePair
WAL Replay
GET /api/v1/status/walreplayReturns WAL replay progress (available before server ready).
---
Admin APIs
Note: Require --web.enable-admin-api flag.
Snapshot
POST /api/v1/admin/tsdb/snapshot?skip_head=<bool>Create TSDB snapshot. Returns snapshot directory name.
Delete Series
POST /api/v1/admin/tsdb/delete_seriesParameters:
| Parameter | Type | Required | Description |
|---|---|---|---|
match[] | series_selector | Yes | Series to delete |
start | rfc3339/unix | No | Start time (default: min) |
end | rfc3339/unix | No | End time (default: max) |
Example:
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=up&match[]=process_start_time_seconds{job="prometheus"}'Clean Tombstones
POST /api/v1/admin/tsdb/clean_tombstonesRemove deleted data from disk. Returns 204 on success.
---
Response Formats
Result Types
Instant Vector (vector):
[
{
"metric": {"__name__": "up", "job": "prometheus"},
"value": [1704067200, "1"]
}
]Range Vector (matrix):
[
{
"metric": {"__name__": "up", "job": "prometheus"},
"values": [[1704067200, "1"], [1704067215, "1"]]
}
]Scalar:
[1704067200, "42"]String:
[1704067200, "hello"]Native Histogram (experimental)
{
"count": "100",
"sum": "45.5",
"buckets": [
[0, "-0.5", "0.5", "25"],
[1, "0.5", "1.0", "50"]
]
}Boundary rules: 0=open-left, 1=open-right, 2=open-both, 3=closed-both.
---
Time & Duration Formats
Timestamps: RFC3339 (2024-01-01T00:00:00Z) or Unix epoch (1704067200)
Durations: ms, s, m, h, d, w, y (e.g., 5m, 1h30m, 7d)
Series Selectors: metric_name{label="value", label2=~"regex.*"}
Label matchers:
=exact match!=not equal=~regex match!~regex not match
PromQL Functions Reference
Quick reference for commonly used PromQL functions and patterns.
Table of Contents
1. Aggregation Operators 2. Rate Functions 3. Aggregation Over Time 4. Math Functions 5. Date/Time Functions 6. Histogram Functions 7. Label Functions 8. Common Patterns
---
Aggregation Operators
Aggregate across series dimensions.
| Operator | Description |
|---|---|
sum | Sum values |
min | Minimum value |
max | Maximum value |
avg | Average value |
group | All values are 1 |
stddev | Standard deviation |
stdvar | Variance |
count | Count series |
count_values | Count series by value |
bottomk | Smallest k elements |
topk | Largest k elements |
quantile | Calculate quantile |
Syntax:
<aggr_op>([parameter,] <vector>) [without|by (<label_list>)]Examples:
# Sum by job
sum by (job) (http_requests_total)
# Average excluding instance
avg without (instance) (node_cpu_seconds_total)
# Top 5 by memory
topk(5, container_memory_usage_bytes)
# Count series per job
count by (job) (up)
# 95th percentile
quantile(0.95, http_request_duration_seconds)---
Rate Functions
Calculate rates for counters (monotonically increasing values).
| Function | Description |
|---|---|
rate(v[d]) | Per-second average rate over duration |
irate(v[d]) | Instantaneous rate (last two points) |
increase(v[d]) | Total increase over duration |
delta(v[d]) | Difference (for gauges) |
idelta(v[d]) | Instantaneous difference |
deriv(v[d]) | Per-second derivative (gauges) |
Examples:
# Requests per second
rate(http_requests_total[5m])
# Bytes received increase in 1h
increase(node_network_receive_bytes_total[1h])
# CPU delta over 5m
delta(process_cpu_seconds_total[5m])
# Instantaneous rate (for spiky graphs)
irate(http_requests_total[5m])Best Practice: Use rate() for smooth graphs and alerting. Use irate() only when you need to see spikes.
---
Aggregation Over Time
Aggregate single series across time.
| Function | Description |
|---|---|
avg_over_time(v[d]) | Average over time |
min_over_time(v[d]) | Minimum over time |
max_over_time(v[d]) | Maximum over time |
sum_over_time(v[d]) | Sum over time |
count_over_time(v[d]) | Count samples |
quantile_over_time(q,v[d]) | Quantile over time |
stddev_over_time(v[d]) | Std deviation |
stdvar_over_time(v[d]) | Variance |
last_over_time(v[d]) | Most recent value |
present_over_time(v[d]) | 1 if any value exists |
Examples:
# Average CPU over 1h
avg_over_time(node_cpu_seconds_total[1h])
# Max memory in last day
max_over_time(container_memory_usage_bytes[1d])
# 95th percentile latency over 10m
quantile_over_time(0.95, http_request_duration_seconds[10m])
# Count samples in window
count_over_time(up[1h])---
Math Functions
| Function | Description |
|---|---|
abs(v) | Absolute value |
ceil(v) | Round up |
floor(v) | Round down |
round(v, to) | Round to nearest |
clamp(v, min, max) | Clamp between bounds |
clamp_min(v, min) | Lower bound |
clamp_max(v, max) | Upper bound |
exp(v) | Exponential |
ln(v) | Natural logarithm |
log2(v) | Base-2 logarithm |
log10(v) | Base-10 logarithm |
sqrt(v) | Square root |
sgn(v) | Sign (-1, 0, 1) |
Examples:
# Round to nearest GB
round(container_memory_usage_bytes / 1024 / 1024 / 1024, 0.1)
# Clamp values
clamp(cpu_usage, 0, 100)
# Absolute difference
abs(predicted_value - actual_value)---
Date/Time Functions
| Function | Description |
|---|---|
time() | Current Unix timestamp |
timestamp(v) | Timestamp of samples |
day_of_month() | Day (1-31) |
day_of_week() | Day (0=Sun to 6=Sat) |
day_of_year() | Day (1-366) |
days_in_month() | Days in month |
hour() | Hour (0-23) |
minute() | Minute (0-59) |
month() | Month (1-12) |
year() | Year |
Examples:
# Seconds since last scrape
time() - timestamp(up)
# Filter by business hours (9-17)
http_requests_total and on() hour() >= 9 < 17
# Filter weekdays only
up and on() day_of_week() >= 1 <= 5---
Histogram Functions
| Function | Description |
|---|---|
histogram_quantile(φ, v) | Calculate φ-quantile from histogram |
histogram_count(v) | Extract count from native histogram |
histogram_sum(v) | Extract sum from native histogram |
histogram_avg(v) | Calculate average from histogram |
histogram_fraction(l,u,v) | Fraction between bounds |
histogram_stddev(v) | Std deviation from histogram |
histogram_stdvar(v) | Variance from histogram |
Examples:
# 99th percentile latency
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
# Median (50th percentile) by job
histogram_quantile(0.5, sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))
# Average request size
histogram_avg(http_request_size_bytes)Note: For classic histograms, aggregate buckets first with sum by (le).
---
Label Functions
| Function | Description |
|---|---|
label_join(v, dst, sep, src...) | Join labels |
label_replace(v, dst, repl, src, regex) | Regex replace |
Examples:
# Create full address label
label_join(up, "address", ":", "instance", "port")
# Extract host from instance
label_replace(up, "host", "$1", "instance", "([^:]+):.*")
# Rename label
label_replace(metric, "new_name", "$1", "old_name", "(.*)")---
Common Patterns
Error Rate
# Error percentage
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) * 100Availability (SLI)
# Percentage of successful requests
sum(rate(http_requests_total{status=~"2.."}[5m]))
/ sum(rate(http_requests_total[5m])) * 100Saturation
# CPU saturation
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
# Memory usage percentage
100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)Latency (Apdex Score)
# Apdex with 0.5s target
(
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m]))
+ sum(rate(http_request_duration_seconds_bucket{le="2"}[5m]))
) / 2 / sum(rate(http_request_duration_seconds_count[5m]))Prediction
# Disk will fill in 4 hours?
predict_linear(node_filesystem_avail_bytes[1h], 4*3600) < 0Changes Detection
# Config reloads in last hour
changes(prometheus_config_last_reload_successful[1h])
# Restarts
resets(process_start_time_seconds[1d])Absent Alerting
# Alert if metric missing
absent(up{job="myservice"})
# Alert if no data for duration
absent_over_time(up{job="myservice"}[5m])Vector Matching
# Divide by matching labels
http_requests_total / on(instance, job) group_left http_requests_limit
# Join with ignoring
node_memory_MemFree_bytes / ignoring(device) node_memory_MemTotal_bytes---
Operators
Arithmetic
+, -, *, /, % (modulo), ^ (power)
Comparison
==, !=, >, <, >=, <=
Add bool for 0/1 result: http_requests_total > bool 100
Logical/Set
and, or, unless
# Series in both
http_requests_total and http_errors_total
# Series in first but not second
up unless on(instance) alerts
# Either
metric_a or metric_b#!/usr/bin/env python3
"""
Prometheus Health Check Tool
Check Prometheus server health, targets status, and basic statistics.
Usage:
python prom_health.py <prometheus_url> [options]
Examples:
python prom_health.py http://localhost:9090
python prom_health.py http://localhost:9090 --targets
python prom_health.py http://localhost:9090 --rules
python prom_health.py http://localhost:9090 --all
"""
import argparse
import json
import sys
import urllib.request
from typing import Any
def fetch_endpoint(base_url: str, endpoint: str) -> dict[str, Any]:
"""Fetch data from Prometheus endpoint."""
url = f"{base_url.rstrip('/')}{endpoint}"
try:
with urllib.request.urlopen(url, timeout=10) as response:
return json.loads(response.read().decode())
except urllib.error.HTTPError as e:
return {"status": "error", "error": f"HTTP {e.code}", "errorType": "http_error"}
except urllib.error.URLError as e:
return {"status": "error", "error": str(e.reason), "errorType": "connection_error"}
except Exception as e:
return {"status": "error", "error": str(e), "errorType": "unknown"}
def check_ready(base_url: str) -> bool:
"""Check if Prometheus is ready."""
url = f"{base_url.rstrip('/')}/-/ready"
try:
with urllib.request.urlopen(url, timeout=5) as response:
return response.status == 200
except Exception:
return False
def check_healthy(base_url: str) -> bool:
"""Check if Prometheus is healthy."""
url = f"{base_url.rstrip('/')}/-/healthy"
try:
with urllib.request.urlopen(url, timeout=5) as response:
return response.status == 200
except Exception:
return False
def print_section(title: str) -> None:
"""Print section header."""
print(f"\n{'='*60}")
print(f" {title}")
print(f"{'='*60}")
def print_status(label: str, status: bool, details: str = "") -> None:
"""Print status line with icon."""
icon = "✓" if status else "✗"
status_text = "OK" if status else "FAIL"
detail_str = f" ({details})" if details else ""
print(f" [{icon}] {label}: {status_text}{detail_str}")
def check_build_info(base_url: str) -> dict[str, Any] | None:
"""Get build info."""
result = fetch_endpoint(base_url, "/api/v1/status/buildinfo")
if result.get("status") == "success":
return result.get("data", {})
return None
def check_runtime_info(base_url: str) -> dict[str, Any] | None:
"""Get runtime info."""
result = fetch_endpoint(base_url, "/api/v1/status/runtimeinfo")
if result.get("status") == "success":
return result.get("data", {})
return None
def check_targets(base_url: str) -> None:
"""Check and display target status."""
result = fetch_endpoint(base_url, "/api/v1/targets")
if result.get("status") != "success":
print(f" Error fetching targets: {result.get('error', 'Unknown')}")
return
data = result.get("data", {})
active = data.get("activeTargets", [])
dropped = data.get("droppedTargets", [])
# Count by health status
health_counts = {"up": 0, "down": 0, "unknown": 0}
for target in active:
health = target.get("health", "unknown")
health_counts[health] = health_counts.get(health, 0) + 1
print(f" Active targets: {len(active)}")
print(f" - Up: {health_counts.get('up', 0)}")
print(f" - Down: {health_counts.get('down', 0)}")
print(f" - Unknown: {health_counts.get('unknown', 0)}")
print(f" Dropped targets: {len(dropped)}")
# List down targets
down_targets = [t for t in active if t.get("health") == "down"]
if down_targets:
print("\n Down targets:")
for target in down_targets[:10]: # Limit to 10
labels = target.get("labels", {})
job = labels.get("job", "unknown")
instance = labels.get("instance", "unknown")
error = target.get("lastError", "No error info")
print(f" - {job}/{instance}: {error[:60]}")
if len(down_targets) > 10:
print(f" ... and {len(down_targets) - 10} more")
def check_rules(base_url: str) -> None:
"""Check and display rules status."""
result = fetch_endpoint(base_url, "/api/v1/rules")
if result.get("status") != "success":
print(f" Error fetching rules: {result.get('error', 'Unknown')}")
return
groups = result.get("data", {}).get("groups", [])
total_rules = 0
alerting_rules = 0
recording_rules = 0
firing_alerts = 0
unhealthy_rules = 0
for group in groups:
for rule in group.get("rules", []):
total_rules += 1
rule_type = rule.get("type")
if rule_type == "alerting":
alerting_rules += 1
alerts = rule.get("alerts", [])
firing = [a for a in alerts if a.get("state") == "firing"]
firing_alerts += len(firing)
elif rule_type == "recording":
recording_rules += 1
if rule.get("health") != "ok":
unhealthy_rules += 1
print(f" Rule groups: {len(groups)}")
print(f" Total rules: {total_rules}")
print(f" - Alerting: {alerting_rules}")
print(f" - Recording: {recording_rules}")
print(f" Firing alerts: {firing_alerts}")
print(f" Unhealthy rules: {unhealthy_rules}")
def check_tsdb(base_url: str) -> None:
"""Check TSDB statistics."""
result = fetch_endpoint(base_url, "/api/v1/status/tsdb")
if result.get("status") != "success":
print(f" Error fetching TSDB stats: {result.get('error', 'Unknown')}")
return
data = result.get("data", {})
head_stats = data.get("headStats", {})
print(f" Series count: {head_stats.get('numSeries', 'N/A')}")
print(f" Chunk count: {head_stats.get('chunkCount', 'N/A')}")
# Top metrics by series count
series_by_metric = data.get("seriesCountByMetricName", [])
if series_by_metric:
print("\n Top 5 metrics by series count:")
for item in series_by_metric[:5]:
print(f" - {item.get('name', 'unknown')}: {item.get('value', 0)}")
def check_alerts(base_url: str) -> None:
"""Check active alerts."""
result = fetch_endpoint(base_url, "/api/v1/alerts")
if result.get("status") != "success":
print(f" Error fetching alerts: {result.get('error', 'Unknown')}")
return
alerts = result.get("data", {}).get("alerts", [])
# Group by state
by_state = {}
for alert in alerts:
state = alert.get("state", "unknown")
by_state[state] = by_state.get(state, 0) + 1
print(f" Total active alerts: {len(alerts)}")
for state, count in sorted(by_state.items()):
print(f" - {state}: {count}")
# List firing alerts
firing = [a for a in alerts if a.get("state") == "firing"]
if firing:
print("\n Firing alerts:")
for alert in firing[:10]:
name = alert.get("labels", {}).get("alertname", "unknown")
severity = alert.get("labels", {}).get("severity", "unknown")
print(f" - {name} (severity: {severity})")
if len(firing) > 10:
print(f" ... and {len(firing) - 10} more")
def main():
parser = argparse.ArgumentParser(
description="Check Prometheus server health and status",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument("url", help="Prometheus server URL")
parser.add_argument("--targets", "-t", action="store_true", help="Show detailed target info")
parser.add_argument("--rules", "-r", action="store_true", help="Show detailed rules info")
parser.add_argument("--tsdb", "-d", action="store_true", help="Show TSDB statistics")
parser.add_argument("--alerts", "-a", action="store_true", help="Show active alerts")
parser.add_argument("--all", action="store_true", help="Show all information")
args = parser.parse_args()
if args.all:
args.targets = args.rules = args.tsdb = args.alerts = True
base_url = args.url.rstrip("/")
# Basic health checks
print_section("Health Checks")
ready = check_ready(base_url)
healthy = check_healthy(base_url)
print_status("Ready", ready)
print_status("Healthy", healthy)
if not (ready and healthy):
print("\n ⚠ Prometheus is not fully operational")
sys.exit(1)
# Build info
build_info = check_build_info(base_url)
if build_info:
print_section("Build Information")
print(f" Version: {build_info.get('version', 'N/A')}")
print(f" Revision: {build_info.get('revision', 'N/A')[:12]}")
print(f" Go version: {build_info.get('goVersion', 'N/A')}")
# Runtime info
runtime_info = check_runtime_info(base_url)
if runtime_info:
print_section("Runtime Information")
print(f" Start time: {runtime_info.get('startTime', 'N/A')}")
print(f" Time series: {runtime_info.get('timeSeriesCount', 'N/A')}")
print(f" Goroutines: {runtime_info.get('goroutineCount', 'N/A')}")
print(f" Storage retention: {runtime_info.get('storageRetention', 'N/A')}")
reload_success = runtime_info.get("reloadConfigSuccess")
if reload_success is not None:
print_status("Config reload", reload_success)
# Optional detailed sections
if args.targets:
print_section("Targets")
check_targets(base_url)
if args.rules:
print_section("Rules")
check_rules(base_url)
if args.alerts:
print_section("Alerts")
check_alerts(base_url)
if args.tsdb:
print_section("TSDB Statistics")
check_tsdb(base_url)
print()
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""
Prometheus Metadata Tool
Query metadata from Prometheus: series, labels, label values, and metric info.
Usage:
python prom_metadata.py <prometheus_url> <command> [options]
Commands:
series Find series matching label selectors
labels List all label names
values Get values for a specific label
metadata Get metric metadata
targets List scrape targets
Examples:
python prom_metadata.py http://localhost:9090 series 'up' 'process_start_time_seconds{job="prometheus"}'
python prom_metadata.py http://localhost:9090 labels
python prom_metadata.py http://localhost:9090 values job
python prom_metadata.py http://localhost:9090 metadata --metric http_requests_total
python prom_metadata.py http://localhost:9090 targets --state active
"""
import argparse
import json
import sys
import urllib.parse
import urllib.request
from typing import Any
def fetch_api(base_url: str, endpoint: str, params: dict[str, Any] | None = None) -> dict[str, Any]:
"""Fetch from Prometheus API."""
url = f"{base_url.rstrip('/')}{endpoint}"
if params:
# Handle repeated parameters (like match[])
param_parts = []
for key, value in params.items():
if isinstance(value, list):
for v in value:
param_parts.append(f"{urllib.parse.quote(key)}={urllib.parse.quote(str(v))}")
elif value is not None:
param_parts.append(f"{urllib.parse.quote(key)}={urllib.parse.quote(str(value))}")
if param_parts:
url += "?" + "&".join(param_parts)
try:
with urllib.request.urlopen(url, timeout=30) as response:
return json.loads(response.read().decode())
except urllib.error.HTTPError as e:
body = e.read().decode()
try:
return json.loads(body)
except json.JSONDecodeError:
return {"status": "error", "error": body, "errorType": str(e.code)}
except urllib.error.URLError as e:
return {"status": "error", "error": str(e.reason), "errorType": "connection_error"}
def cmd_series(base_url: str, matchers: list[str], start: str | None, end: str | None, limit: int | None) -> int:
"""Find series by label matchers."""
params = {"match[]": matchers}
if start:
params["start"] = start
if end:
params["end"] = end
if limit:
params["limit"] = limit
result = fetch_api(base_url, "/api/v1/series", params)
if result.get("status") != "success":
print(f"Error: {result.get('error', 'Unknown error')}", file=sys.stderr)
return 1
series = result.get("data", [])
print(f"Found {len(series)} series:\n")
for s in series:
name = s.get("__name__", "")
labels = {k: v for k, v in s.items() if k != "__name__"}
label_str = ", ".join(f'{k}="{v}"' for k, v in sorted(labels.items()))
if name:
print(f"{name}{{{label_str}}}" if labels else name)
else:
print(f"{{{label_str}}}")
return 0
def cmd_labels(base_url: str, matchers: list[str] | None, start: str | None, end: str | None, limit: int | None) -> int:
"""List all label names."""
params = {}
if matchers:
params["match[]"] = matchers
if start:
params["start"] = start
if end:
params["end"] = end
if limit:
params["limit"] = limit
result = fetch_api(base_url, "/api/v1/labels", params if params else None)
if result.get("status") != "success":
print(f"Error: {result.get('error', 'Unknown error')}", file=sys.stderr)
return 1
labels = result.get("data", [])
print(f"Found {len(labels)} labels:\n")
for label in sorted(labels):
print(f" {label}")
return 0
def cmd_values(base_url: str, label_name: str, matchers: list[str] | None, start: str | None, end: str | None, limit: int | None) -> int:
"""Get values for a label."""
params = {}
if matchers:
params["match[]"] = matchers
if start:
params["start"] = start
if end:
params["end"] = end
if limit:
params["limit"] = limit
result = fetch_api(base_url, f"/api/v1/label/{urllib.parse.quote(label_name)}/values", params if params else None)
if result.get("status") != "success":
print(f"Error: {result.get('error', 'Unknown error')}", file=sys.stderr)
return 1
values = result.get("data", [])
print(f"Found {len(values)} values for label '{label_name}':\n")
for value in sorted(values):
print(f" {value}")
return 0
def cmd_metadata(base_url: str, metric: str | None, limit: int | None, limit_per_metric: int | None) -> int:
"""Get metric metadata."""
params = {}
if metric:
params["metric"] = metric
if limit:
params["limit"] = limit
if limit_per_metric:
params["limit_per_metric"] = limit_per_metric
result = fetch_api(base_url, "/api/v1/metadata", params if params else None)
if result.get("status") != "success":
print(f"Error: {result.get('error', 'Unknown error')}", file=sys.stderr)
return 1
metadata = result.get("data", {})
print(f"Found metadata for {len(metadata)} metrics:\n")
for metric_name, info_list in sorted(metadata.items()):
print(f"{metric_name}:")
for info in info_list:
print(f" Type: {info.get('type', 'unknown')}")
print(f" Help: {info.get('help', 'N/A')}")
if info.get("unit"):
print(f" Unit: {info.get('unit')}")
print()
return 0
def cmd_targets(base_url: str, state: str | None, scrape_pool: str | None) -> int:
"""List scrape targets."""
params = {}
if state:
params["state"] = state
if scrape_pool:
params["scrapePool"] = scrape_pool
result = fetch_api(base_url, "/api/v1/targets", params if params else None)
if result.get("status") != "success":
print(f"Error: {result.get('error', 'Unknown error')}", file=sys.stderr)
return 1
data = result.get("data", {})
active = data.get("activeTargets", [])
dropped = data.get("droppedTargets", [])
if active:
print(f"Active targets ({len(active)}):\n")
for target in active:
labels = target.get("labels", {})
job = labels.get("job", "unknown")
instance = labels.get("instance", "unknown")
health = target.get("health", "unknown")
health_icon = "✓" if health == "up" else "✗" if health == "down" else "?"
print(f" [{health_icon}] {job}/{instance}")
print(f" URL: {target.get('scrapeUrl', 'N/A')}")
print(f" Pool: {target.get('scrapePool', 'N/A')}")
print(f" Last scrape: {target.get('lastScrape', 'N/A')}")
if target.get("lastError"):
print(f" Error: {target.get('lastError')}")
print()
if dropped and (state is None or state in ("dropped", "any")):
print(f"\nDropped targets ({len(dropped)}):\n")
for target in dropped[:20]: # Limit output
discovered = target.get("discoveredLabels", {})
job = discovered.get("job", "unknown")
address = discovered.get("__address__", "unknown")
print(f" - {job}: {address}")
if len(dropped) > 20:
print(f" ... and {len(dropped) - 20} more")
return 0
def main():
parser = argparse.ArgumentParser(
description="Query Prometheus metadata",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument("url", help="Prometheus server URL")
subparsers = parser.add_subparsers(dest="command", required=True)
# Series command
series_parser = subparsers.add_parser("series", help="Find series by label matchers")
series_parser.add_argument("matchers", nargs="+", help="Series selectors (e.g., 'up', 'http_requests_total{job=\"api\"}')")
series_parser.add_argument("--start", help="Start timestamp")
series_parser.add_argument("--end", help="End timestamp")
series_parser.add_argument("--limit", type=int, help="Max series to return")
# Labels command
labels_parser = subparsers.add_parser("labels", help="List all label names")
labels_parser.add_argument("--match", action="append", dest="matchers", help="Filter by series selector")
labels_parser.add_argument("--start", help="Start timestamp")
labels_parser.add_argument("--end", help="End timestamp")
labels_parser.add_argument("--limit", type=int, help="Max labels to return")
# Values command
values_parser = subparsers.add_parser("values", help="Get values for a label")
values_parser.add_argument("label_name", help="Label name")
values_parser.add_argument("--match", action="append", dest="matchers", help="Filter by series selector")
values_parser.add_argument("--start", help="Start timestamp")
values_parser.add_argument("--end", help="End timestamp")
values_parser.add_argument("--limit", type=int, help="Max values to return")
# Metadata command
metadata_parser = subparsers.add_parser("metadata", help="Get metric metadata")
metadata_parser.add_argument("--metric", "-m", help="Filter by metric name")
metadata_parser.add_argument("--limit", type=int, help="Max metrics to return")
metadata_parser.add_argument("--limit-per-metric", type=int, help="Max metadata per metric")
# Targets command
targets_parser = subparsers.add_parser("targets", help="List scrape targets")
targets_parser.add_argument("--state", choices=["active", "dropped", "any"], help="Filter by state")
targets_parser.add_argument("--scrape-pool", help="Filter by scrape pool name")
args = parser.parse_args()
if args.command == "series":
return cmd_series(args.url, args.matchers, args.start, args.end, args.limit)
elif args.command == "labels":
return cmd_labels(args.url, args.matchers, args.start, args.end, args.limit)
elif args.command == "values":
return cmd_values(args.url, args.label_name, args.matchers, args.start, args.end, args.limit)
elif args.command == "metadata":
return cmd_metadata(args.url, args.metric, args.limit, getattr(args, "limit_per_metric", None))
elif args.command == "targets":
return cmd_targets(args.url, args.state, getattr(args, "scrape_pool", None))
if __name__ == "__main__":
sys.exit(main())
#!/usr/bin/env python3
"""
Prometheus Query Tool
Execute instant and range queries against Prometheus HTTP API.
Usage:
python prom_query.py <prometheus_url> <query> [options]
Examples:
# Instant query
python prom_query.py http://localhost:9090 'up'
# Range query
python prom_query.py http://localhost:9090 'rate(http_requests_total[5m])' \
--start '2024-01-01T00:00:00Z' --end '2024-01-01T01:00:00Z' --step '1m'
# Output formats
python prom_query.py http://localhost:9090 'up' --format json
python prom_query.py http://localhost:9090 'up' --format csv
python prom_query.py http://localhost:9090 'up' --format table
"""
import argparse
import json
import sys
import urllib.parse
import urllib.request
from datetime import datetime
from typing import Any
def query_prometheus(
base_url: str,
query: str,
time: str | None = None,
start: str | None = None,
end: str | None = None,
step: str | None = None,
timeout: str | None = None,
limit: int | None = None,
) -> dict[str, Any]:
"""Execute a Prometheus query."""
# Determine query type
is_range = start is not None and end is not None
endpoint = "/api/v1/query_range" if is_range else "/api/v1/query"
# Build parameters
params = {"query": query}
if is_range:
params["start"] = start
params["end"] = end
params["step"] = step or "1m"
elif time:
params["time"] = time
if timeout:
params["timeout"] = timeout
if limit:
params["limit"] = str(limit)
# Build URL
url = f"{base_url.rstrip('/')}{endpoint}?{urllib.parse.urlencode(params)}"
# Execute request
try:
with urllib.request.urlopen(url, timeout=30) as response:
data = json.loads(response.read().decode())
return data
except urllib.error.HTTPError as e:
error_body = e.read().decode()
try:
error_data = json.loads(error_body)
return error_data
except json.JSONDecodeError:
return {"status": "error", "error": error_body, "errorType": str(e.code)}
except urllib.error.URLError as e:
return {"status": "error", "error": str(e.reason), "errorType": "connection_error"}
def format_timestamp(ts: float) -> str:
"""Format Unix timestamp as ISO 8601."""
return datetime.utcfromtimestamp(ts).strftime("%Y-%m-%dT%H:%M:%SZ")
def format_metric(metric: dict[str, str]) -> str:
"""Format metric labels as string."""
name = metric.get("__name__", "")
labels = {k: v for k, v in metric.items() if k != "__name__"}
if not labels:
return name or "{}"
label_str = ", ".join(f'{k}="{v}"' for k, v in sorted(labels.items()))
return f"{name}{{{label_str}}}" if name else f"{{{label_str}}}"
def output_json(data: dict[str, Any]) -> None:
"""Output as JSON."""
print(json.dumps(data, indent=2))
def output_table(data: dict[str, Any]) -> None:
"""Output as formatted table."""
if data.get("status") != "success":
print(f"Error: {data.get('error', 'Unknown error')}")
return
result_type = data.get("data", {}).get("resultType")
results = data.get("data", {}).get("result", [])
if not results:
print("No results")
return
if result_type == "vector":
# Instant vector
print(f"{'METRIC':<60} {'TIMESTAMP':<25} {'VALUE':>15}")
print("-" * 100)
for item in results:
metric = format_metric(item.get("metric", {}))
ts, value = item.get("value", [0, ""])
print(f"{metric:<60} {format_timestamp(ts):<25} {value:>15}")
elif result_type == "matrix":
# Range vector
for item in results:
metric = format_metric(item.get("metric", {}))
print(f"\n{metric}")
print(f"{'TIMESTAMP':<25} {'VALUE':>15}")
print("-" * 40)
for ts, value in item.get("values", []):
print(f"{format_timestamp(ts):<25} {value:>15}")
elif result_type == "scalar":
ts, value = data.get("data", {}).get("result", [0, ""])
print(f"Scalar: {value} at {format_timestamp(ts)}")
elif result_type == "string":
ts, value = data.get("data", {}).get("result", [0, ""])
print(f"String: {value} at {format_timestamp(ts)}")
# Print warnings if any
warnings = data.get("warnings", [])
if warnings:
print("\nWarnings:")
for w in warnings:
print(f" - {w}")
def output_csv(data: dict[str, Any]) -> None:
"""Output as CSV."""
if data.get("status") != "success":
print(f"error,{data.get('error', 'Unknown error')}")
return
result_type = data.get("data", {}).get("resultType")
results = data.get("data", {}).get("result", [])
if result_type == "vector":
print("metric,timestamp,value")
for item in results:
metric = format_metric(item.get("metric", {})).replace(",", ";")
ts, value = item.get("value", [0, ""])
print(f'"{metric}",{format_timestamp(ts)},{value}')
elif result_type == "matrix":
print("metric,timestamp,value")
for item in results:
metric = format_metric(item.get("metric", {})).replace(",", ";")
for ts, value in item.get("values", []):
print(f'"{metric}",{format_timestamp(ts)},{value}')
elif result_type in ("scalar", "string"):
print("timestamp,value")
ts, value = data.get("data", {}).get("result", [0, ""])
print(f"{format_timestamp(ts)},{value}")
def main():
parser = argparse.ArgumentParser(
description="Query Prometheus HTTP API",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
Instant query:
%(prog)s http://localhost:9090 'up'
%(prog)s http://localhost:9090 'rate(http_requests_total[5m])' --time '2024-01-01T00:00:00Z'
Range query:
%(prog)s http://localhost:9090 'up' --start '2024-01-01T00:00:00Z' --end '2024-01-01T01:00:00Z' --step '1m'
Output formats:
%(prog)s http://localhost:9090 'up' --format table
%(prog)s http://localhost:9090 'up' --format json
%(prog)s http://localhost:9090 'up' --format csv
"""
)
parser.add_argument("url", help="Prometheus server URL (e.g., http://localhost:9090)")
parser.add_argument("query", help="PromQL query expression")
parser.add_argument("--time", "-t", help="Evaluation timestamp (RFC3339 or Unix)")
parser.add_argument("--start", "-s", help="Range query start time")
parser.add_argument("--end", "-e", help="Range query end time")
parser.add_argument("--step", help="Range query step (default: 1m)")
parser.add_argument("--timeout", help="Query timeout duration")
parser.add_argument("--limit", type=int, help="Max series to return")
parser.add_argument(
"--format", "-f",
choices=["json", "table", "csv"],
default="table",
help="Output format (default: table)"
)
args = parser.parse_args()
# Validate range query args
if (args.start is None) != (args.end is None):
parser.error("--start and --end must be used together for range queries")
# Execute query
result = query_prometheus(
base_url=args.url,
query=args.query,
time=args.time,
start=args.start,
end=args.end,
step=args.step,
timeout=args.timeout,
limit=args.limit,
)
# Output result
if args.format == "json":
output_json(result)
elif args.format == "csv":
output_csv(result)
else:
output_table(result)
# Exit with error code if query failed
if result.get("status") != "success":
sys.exit(1)
if __name__ == "__main__":
main()