Logql Generator

LogQL Query Generator

Interactive Query Planning Workflow

CRITICAL: Always engage the user in collaborative planning before generating queries.

Stages 1-3: Gather Requirements (use AskUserQuestion)

Ask about: goal (error analysis, alerting, debugging), use case, log sources (labels, format), query type (log/metric), filtering needs, parsing method, aggregation, and time range.

Stage 4: Plan, Validate & Consult References

Before generating code, present a plain-English plan and confirm with the user via AskUserQuestion:

## LogQL Query Plan

**Goal**: [Description]
**Query Structure**:
1. Select streams: `{label="value"}`
2. Filter lines: [operations]
3. Parse logs: [parser]
4. Aggregate: [function]

**Does this match your intentions?**

Once confirmed, MANDATORY: consult references before generating. Do NOT rely on prior knowledge.

Local References (Read tool)

Example paths:

Read(".claude/skills/logql-generator/assets/common_queries.logql")
Read(".claude/skills/logql-generator/references/best_practices.md")

External Documentation (context7 MCP / WebSearch)

Use when local references don't cover the topic:

Stage 5: Generate Query

Best Practices

1. Specific Stream Selectors: {namespace="prod", app="api", level="error"} not just {namespace="prod"} 2. Filter Order: Line filter → parse → label filter (fastest to slowest) 3. Parser Performance: pattern > logfmt > json > regexp

Core Query Patterns

Log Filtering:

{job="app"} |= "error" |= "timeout"        # Contains both
{job="app"} |~ "error|fatal|critical"       # Regex match
{job="app"} != "debug"                      # Exclude

JSON/logfmt Parsing:

{app="api"} | json | level="error" | status_code >= 500
{app="app"} | logfmt | caller="database.go"

Pattern Extraction:

{job="nginx"} | pattern "<ip> - - [<_>] \"<method> <path>\" <status> <size>"

Metrics:

# Rate
rate({job="app"} | json | level="error" [5m])

# Count by label
sum by (app) (count_over_time({namespace="prod"} | json [5m]))

# Error percentage
sum(rate({app="api"} | json | level="error" [5m])) / sum(rate({app="api"}[5m])) * 100

# Latency percentiles
quantile_over_time(0.95, {app="api"} | json | unwrap duration [5m])

# Top N
topk(10, sum by (error_type) (count_over_time({job="app"} | json | level="error" [1h])))

Formatting:

{job="app"} | json | line_format "{{.level}}: {{.message}}"
{job="app"} | json | label_format env="{{.environment}}"

IP Filtering (prefer label filter after parsing for precision):

{job="nginx"} | logfmt | remote_addr = ip("192.168.4.0/24")

Stage 5a: Incremental Query Building (Educational/Debugging)

When to use this stage:

• User is learning LogQL
• Complex multi-stage queries
• Debugging query issues
• User explicitly requests step-by-step explanation

Present the query construction incrementally:

## Building Your Query Step-by-Step

### Step 1: Stream Selector (verify logs exist)

{app="api"}

Test this first to confirm logs are flowing

### Step 2: Add Line Filter (fast pre-filtering)

{app="api"} |= "error"

Reduces data before parsing

### Step 3: Add Parser (extract fields)

{app="api"} |= "error" | json

Now you can filter on extracted labels

### Step 4: Add Label Filter (precise filtering)

{app="api"} |= "error" | json | level="error"

Final filter on parsed data

### Step 5: Add Aggregation (if metric query)

sum(count_over_time({app="api"} |= "error" | json | level="error" [5m]))

Complete metric query

Use AskUserQuestion to offer incremental mode:

• Option: "Show step-by-step construction" vs "Show final query only"

Stage 6: Provide Usage

1. Final Query with explanation 2. How to Use: Grafana panel, Loki alerting rules, logcli query, HTTP API 3. Customization: Labels to modify, thresholds to tune

Advanced Techniques

Multiple Parsers

{app="api"} | json | regexp "user_(?P<user_id>\\d+)"

Unwrap for Numeric Metrics

sum(sum_over_time({app="api"} | json | unwrap duration [5m]))

Pattern Match Operators (Loki 3.0+, 10x faster than regex)

{service_name=`app`} |> "<_> level=debug <_>"

Logical Operators

{app="api"} | json | (status_code >= 400 and status_code < 500) or level="error"

Offset Modifier

sum(rate({app="api"} | json | level="error" [5m])) - sum(rate({app="api"} | json | level="error" [5m] offset 1d))

Label Operations

{app="api"} | json | keep namespace, pod, level
{app="api"} | json | drop pod, instance

Note: LogQL has no dedup or distinct operators. Use metric aggregations like sum by (field) for programmatic deduplication.

Loki 3.x Key Features

Structured Metadata

High-cardinality data without indexing (trace_id, user_id, request_id):

# Filter AFTER stream selector, NOT in it
{app="api"} | trace_id="abc123" | json | level="error"

Query Acceleration (Bloom Filters)

Place structured metadata filters BEFORE parsers:

# ACCELERATED
{cluster="prod"} | detected_level="error" | logfmt | json
# NOT ACCELERATED
{cluster="prod"} | logfmt | json | detected_level="error"

approx_topk (Probabilistic)

approx_topk(10, sum by (endpoint) (rate({app="api"}[5m])))

vector() for Alerting

sum(count_over_time({app="api"} | json | level="error" [5m])) or vector(0)

Automatic Labels

• service_name: Auto-populated from container name
• detected_level: Auto-detected when discover_log_levels: true (stored as structured metadata)

Function and Parser Quick Reference

For comprehensive function and parser documentation, see references/function_reference.md:

• Log range aggregations: rate(), count_over_time(), bytes_rate(), absent_over_time()
• Unwrapped aggregations: sum_over_time(), quantile_over_time(), etc.
• Aggregation operators: sum, topk, approx_topk, with by/without grouping
• Parsers: json, logfmt with options
• Template functions for line_format/label_format

Alerting Rules

# Alert when error rate exceeds 5%
(sum(rate({app="api"} | json | level="error" [5m])) / sum(rate({app="api"}[5m]))) > 0.05

# With vector() to avoid "no data"
sum(rate({app="api"} | json | level="error" [5m])) or vector(0) > 10

Anti-Patterns

NEVER use a broad stream selector without label filters

• WHY: {job="app"} on high-volume streams reads all log lines before any filtering, causing massive I/O and query timeouts in production Loki.
• BAD: {job="app"} | json | level="error"
• GOOD: {job="app", namespace="prod"} |= "error" | json | level="error" — line filter before parser reduces data volume

NEVER place parsers before line filters

• WHY: Parsers (json, logfmt, regexp) are CPU-intensive. Placing a fast line filter (|=, !=, |~) before the parser pre-filters data and reduces parsing overhead by up to 10x.
• BAD: {app="api"} | json | level="error"
• GOOD: {app="api"} |= "error" | json | level="error"

NEVER use regexp when logfmt or json applies

• WHY: Parser performance order is: pattern > logfmt > json > regexp. Regexp is 5-20x slower than structured parsers and breaks on format changes.
• BAD: {app="api"} | regexp "level=(?P<level>[^ ]+)"
• GOOD: {app="api"} | logfmt (if logs are logfmt-formatted)

NEVER put high-cardinality values in stream selectors

• WHY: Loki indexes stream selectors. High-cardinality labels (user_id, trace_id, request_id) in stream selectors create millions of streams, causing index bloat and degraded query performance.
• BAD: {user_id="12345", app="api"} | json
• GOOD: {app="api"} | trace_id="abc123" | json — use structured metadata for high-cardinality values (Loki 3.x)

NEVER use count_over_time when rate is needed for alerting

• WHY: count_over_time returns a raw count, not a rate. Alert thresholds set on raw counts break when the time range changes. Use rate() for stable per-second thresholds.
• BAD: count_over_time({app="api"} | json | level="error" [5m]) > 100
• GOOD: rate({app="api"} | json | level="error" [5m]) > 2 (2 errors/second)

Error Handling

Guidelines

1. Always plan interactively - Present plain-English plan before generating 2. Use AskUserQuestion - Gather requirements and confirm plans 3. Consult references - See Stage 4 for mandatory reference consultation (local + external) 4. Offer incremental building - See Stage 5a for step-by-step construction 5. Explain queries - What it does, how to interpret results 6. Prioritize performance - Specific selectors, filter early, simpler parsers

Version Notes

• Loki 3.0+: Bloom filters, structured metadata, pattern match operators (|>, !>)
• Loki 3.3+: approx_topk function
• Loki 3.5+: Promtail deprecated (use Grafana Alloy)
• Loki 3.6+: Horizontally scalable compactor, Loki UI as Grafana plugin

Deprecations: Promtail (use Alloy), BoltDB store (use TSDB with v13 schema)

References

• Common Queries — comprehensive query examples covering filtering, parsing, metrics, alerting, and Loki 3.x patterns
• Best Practices — 39+ LogQL best practices, performance optimization, and anti-patterns
• Function Reference — quick reference tables for all LogQL functions and parsers

Scenario 01: Nginx Access Log Analysis

User Prompt

An e-commerce platform runs Nginx as its edge reverse proxy in Kubernetes. The platform is intermittently seeing slow response times, and the SRE team wants to investigate whether specific upstream services are causing 5xx errors. The Nginx instances emit access logs in a custom semi-structured format:

10.1.2.3 - alice [01/Mar/2026:10:15:00 +0000] "GET /api/products?page=2 HTTP/1.1" 502 1024 "https://shop.example.com" "Mozilla/5.0..."

The logs are collected into Grafana Loki with the following labels available on the streams: job="nginx-access", namespace="prod", cluster="us-east".

The SRE team needs queries to help with two things:

1. Filter to 5xx errors only — return raw log lines so engineers can read them and spot patterns (endpoint, upstream, source IP). 2. Count the rate of 5xx responses over time — a metric query that can be graphed in a Grafana dashboard panel.

For the rate query, the team cares about separating counts by HTTP method (GET, POST, etc.) so they can tell whether writes or reads are failing more.

Produce a Markdown file named queries.md that contains:

• The two LogQL queries (clearly labelled as the log filter query and the metric query)
• A brief explanation of each query and how to interpret the results
• Notes on which labels or values to change to adapt the queries to different environments or time windows

Do not include setup or installation instructions.

Expected Behavior

1. Place a line filter (e.g. |= "50" or |~ "5[0-9]{2}") before the parser stage 2. Use the pattern parser to extract fields from the semi-structured Nginx log format 3. Include at least two label matchers in the stream selector (e.g. job and namespace) 4. After parsing, filter the status code using a label filter (e.g. status >= 500) 5. Use rate() for the metric query (not count_over_time()) 6. Aggregate the metric query using sum by (method) to separate HTTP method counts 7. Add explanations for each query and customization notes for adapting to different environments

Success Criteria

• Line filter before parser: The 5xx log-filter query places a line filter (e.g. |= "50" or |~ "5[0-9]{2}") BEFORE the parser stage — not after it
• Pattern parser used: Uses the pattern parser (| pattern "...") to extract fields from the nginx access log format rather than regexp or json
• Specific stream selector: Stream selector includes at least two label matchers (e.g. job and namespace, or job and cluster) rather than a single broad label
• Status code via label filter: After parsing, the status code is filtered via a label filter (e.g. status >= 500) rather than only relying on the line filter
• rate() for metric query: The metric (rate) query uses rate() not count_over_time()
• sum by method grouping: The metric query aggregates using sum by (method) or equivalent grouping on the extracted HTTP method label
• Customization notes present: The output document includes notes identifying which labels or values to change (e.g. namespace, cluster, time window)
• Query explanations present: Each query is accompanied by a plain-English explanation of what it returns and how to interpret results

Failure Conditions

• Places line filter after the parser stage instead of before it
• Uses regexp or json parser instead of the pattern parser for the semi-structured format
• Uses only a single label matcher in the stream selector
• Does not apply a label filter on the status code after parsing
• Uses count_over_time() instead of rate() for the metric query
• Omits HTTP method grouping from the metric aggregation
• Produces queries without customization notes
• Produces queries without explanations

Scenario 02: Payment Service Error Alerting

User Prompt

A fintech startup processes card payments through a dedicated payment-service microservice. After several incidents where engineers only noticed elevated error rates after customers complained, the team has decided to proactively set up Grafana Loki alerting rules.

The payment-service logs are structured as JSON with a level field ("error", "warn", "info") and a transaction_type field ("charge", "refund", "void"). Logs flow into Loki under labels app="payment-service" and env="production".

The team wants two alerting expressions:

1. High error rate alert — fires when the absolute number of error-level log events per second exceeds a threshold of 2 errors/second, sustained over a 5-minute window. 2. Error ratio alert — fires when error-level events exceed 5% of all payment-service log events in the same window.

A previous intern wrote some alerts using raw event counts that kept misfiring after someone changed the dashboard time range. The team wants to make sure the new alerts won't have that problem. They also want the high error rate alert to never show "no data" in Grafana — it should evaluate to 0 when there are no errors.

Produce a file named alert_expressions.md containing:

• Both LogQL alert expressions (clearly labelled)
• A brief explanation of why each expression is structured the way it is
• One sentence identifying the failure mode of the previous intern's approach and how the new alerts avoid it

Expected Behavior

1. Use rate() (not count_over_time()) in the high error rate alert expression 2. Apply the or vector(0) pattern to the high error rate expression to prevent "no data" gaps 3. Build the error ratio expression using rate() on both the numerator and denominator 4. Place line filters (e.g. |= "error") before the json parser stage in both expressions 5. Set the high error rate threshold at > 2 (per-second rate semantics, not a raw count) 6. Set the error ratio threshold at 0.05 (5% as a fraction, not 5) 7. Explain why count_over_time() breaks when the time range changes and how rate() avoids it 8. Clearly label which expression is the absolute rate alert and which is the ratio alert

Success Criteria

• rate() for absolute alert: The high-error-rate alert expression uses rate() not count_over_time()
• vector(0) fallback: The high error rate expression uses ... or vector(0) to ensure the metric evaluates to 0 when no errors are present
• Error ratio uses rate(): The error ratio expression divides rate() of errors by rate() of all events (not count_over_time() on either side)
• Line filter before parser: Both expressions place a line filter (e.g. |= "error") before the json parser stage
• Threshold set correctly: The high error rate alert uses a threshold of > 2 (not > 100 or another count-based value) consistent with per-second rate semantics
• Error ratio threshold 5%: The error ratio expression compares against 0.05 (5%) not 5 or another non-fractional value
• count_over_time failure explained: The output document contains a sentence or note explaining that count_over_time breaks when the time range changes (raw count vs rate semantics)
• Expressions labelled: The output document clearly labels which expression is the absolute rate alert and which is the ratio alert

Failure Conditions

• Uses count_over_time() instead of rate() in alert expressions
• Omits the or vector(0) fallback from the high error rate alert
• Builds the ratio expression using count_over_time() on either side of the division
• Places line filters after the json parser instead of before it
• Sets a count-based threshold (e.g. > 100) instead of a per-second rate threshold
• Compares the ratio against 5 instead of 0.05
• Fails to explain the failure mode of raw count_over_time() alerting
• Omits labels distinguishing the two alert expressions

Scenario 03: Distributed Trace Correlation in Loki 3.x

User Prompt

A large SaaS platform has recently migrated its logging infrastructure to Loki 3.x and has enabled structured metadata support. The platform's microservices emit JSON-formatted logs. Each log line carries a trace_id (UUID), user_id (numeric, high cardinality), and request_id (UUID) alongside a level field and a message field. The Loki configuration has discover_log_levels: true enabled.

The platform runs an order-service in Kubernetes. Relevant labels available on the streams are: app, namespace, cluster, and pod.

A customer support engineer needs help with two queries:

1. Trace investigation — Find all error-level log lines from the order-service for a specific trace ID ("a1b2c3d4-e5f6-7890-abcd-ef1234567890") to reconstruct what happened during a failed checkout. The query should work efficiently with Loki 3.x.

2. Per-namespace error breakdown — A metric query that shows the rate of detected error-level events per namespace across the entire cluster (not limited to order-service), grouped by namespace. This should also leverage Loki 3.x automatic level detection.

Produce a file named trace_queries.md containing:

• Both LogQL queries (clearly labelled)
• A brief explanation of each query, including which Loki 3.x capabilities they leverage
• A note about what would go wrong if the trace ID were placed differently in the first query

Expected Behavior

1. Keep trace_id out of the stream selector braces (do not use {trace_id="..."}) 2. Filter trace_id as a label filter after the stream selector (e.g. {app="order-service"} | trace_id="a1b2c3d4...") 3. Place the trace_id or detected_level filter before the json parser for bloom filter acceleration 4. Use detected_level (the Loki 3.x automatic label) in at least one query instead of parsing level from JSON 5. Keep user_id out of the stream selector braces as well 6. Use sum by (namespace) or equivalent grouping for the per-namespace metric query 7. Include a note warning that placing high-cardinality IDs in the stream selector causes index bloat 8. Identify at least one specific Loki 3.x feature (structured metadata, bloom filters, detected_level, etc.) by name

Success Criteria

• trace_id not in stream selector: The trace investigation query does NOT place trace_id inside the stream selector braces {trace_id="..."}
• trace_id as post-stream filter: The trace investigation query filters trace_id as a label filter AFTER the stream selector (e.g. {app="order-service"} | trace_id="a1b2c3d4...")
• Structured metadata before parser: In the trace investigation query, the trace_id or detected_level filter appears BEFORE the json parser (for bloom filter acceleration)
• detected_level used: At least one query uses detected_level (the Loki 3.x automatic label) rather than parsing level from JSON to detect error events
• user_id not in stream selector: Neither query places user_id inside the stream selector braces
• sum by namespace grouping: The per-namespace metric query uses sum by (namespace) or equivalent grouping on the namespace label
• High-cardinality warning note: The output document contains a note explaining that placing high-cardinality IDs (trace_id, user_id) in the stream selector causes index bloat or degraded performance
• Loki 3.x capabilities identified: The explanation mentions at least one specific Loki 3.x feature by name (structured metadata, bloom filters, detected_level, approx_topk, or pattern match operators)

Failure Conditions

• Places trace_id inside the stream selector braces, causing index bloat
• Filters trace_id only via the stream selector and not as a post-stream label filter
• Places the structured metadata filter after the json parser, bypassing bloom filter acceleration
• Uses | json | level="error" instead of leveraging the detected_level auto-label
• Places user_id inside the stream selector braces
• Omits namespace grouping from the per-namespace metric query
• Provides no warning about high-cardinality label placement
• Fails to name any specific Loki 3.x feature in the explanation

Scenario 04: API Gateway Latency and Top Endpoints Dashboard

User Prompt

A cloud API gateway team wants to populate a Grafana dashboard with four panels to monitor their gateway-service. The service runs in a Kubernetes cluster and emits logfmt-formatted logs. Each log line includes fields such as level, method, path, status, duration_ms (integer, milliseconds), and upstream. The service logs flow into Loki under labels app="gateway-service", namespace="platform", and env="prod".

The four panels they need:

1. p95 request latency over time — the 95th-percentile request duration in milliseconds, across all requests, sampled over 5-minute windows.

2. Top 10 slowest endpoints — a snapshot of which path values have the highest average response time right now (over the last 15 minutes), showing the top 10.

3. Error rate by upstream — the per-second rate of requests where status >= 500, broken down by the upstream label, over 5-minute windows.

4. Anomaly detection: today vs yesterday — the difference between the current error rate and the error rate exactly 24 hours ago, so on-call engineers can quickly see whether error volumes are trending up.

For all panels, the team cares about correctness and query performance given the high volume of traffic on this service.

Produce a file named dashboard_queries.md with all four LogQL queries, each clearly labelled (Panel 1 through Panel 4). Include a short explanation beneath each query describing how the result should be interpreted in a Grafana panel.

Expected Behavior

1. Use the logfmt parser (| logfmt) for all queries that extract fields — not regexp or json 2. Use quantile_over_time(0.95, ... | unwrap duration_ms ...) for the p95 latency panel 3. Use | unwrap duration_ms to extract the numeric field before quantile_over_time 4. Use topk(10, ...) to select the top 10 slowest endpoints for Panel 2 5. Group Panel 3's error rate using sum by (upstream) or equivalent 6. Use rate() (not count_over_time()) for Panels 3 and 4 7. Apply the offset modifier (e.g. [5m] offset 1d) to reference the same metric 24 hours ago in Panel 4 8. Subtract the offset expression from the current expression using the minus operator for Panel 4

Success Criteria

• logfmt parser used: All queries that extract fields use the logfmt parser (| logfmt), not regexp or json
• quantile_over_time for p95: Panel 1 uses quantile_over_time(0.95, ... | unwrap duration_ms ...) to compute the 95th percentile latency
• unwrap used for numeric field: The latency query uses | unwrap duration_ms to extract the numeric field before quantile_over_time
• topk for top endpoints: Panel 2 uses topk(10, ...) to select the top 10 slowest endpoints
• sum by upstream for error rate: Panel 3 groups the error rate using sum by (upstream) or equivalent grouping on upstream
• rate() for error rate panels: Panels 3 and 4 use rate() not count_over_time() for computing error rates
• offset modifier for comparison: Panel 4 uses the offset modifier (e.g. [5m] offset 1d) to reference the same metric 24 hours ago
• Subtraction for delta: Panel 4 subtracts the offset expression from the current expression using the minus operator

Failure Conditions

• Uses regexp or json parser instead of logfmt for logfmt-formatted logs
• Uses avg_over_time or a non-quantile function instead of quantile_over_time for p95
• Omits | unwrap duration_ms before the quantile calculation
• Uses sort_desc or manual filtering instead of topk(10, ...) for Panel 2
• Omits upstream grouping from the error rate panel
• Uses count_over_time() instead of rate() for Panels 3 or 4
• Omits the offset modifier from Panel 4's historical comparison
• Fails to subtract the offset expression in Panel 4

Scenario 05: LogQL Query for a New Team Member

User Prompt

A junior platform engineer just joined a team that uses Grafana Loki for log management. They have been asked to build a query to investigate a recurring issue: sporadic database connection failures in the auth-service. The service runs in Kubernetes and logs in JSON format. Relevant log fields include level, component (value: "db" for database logs), error_type, and message. Stream labels available are app="auth-service", namespace="backend", and env="staging".

The engineer is not yet confident with LogQL and wants to understand how the query is constructed, not just receive a final answer. They have asked for help understanding what each part of the query does before seeing the full version, and want to be able to take away a reference they can consult later.

The final goal is a log-filter query that returns only error-level lines from the database component of the auth-service, formatted to show the error_type and message fields prominently.

Produce a file named learning_session.md that:

1. Shows the query being built one stage at a time — each step should be on its own, runnable, and there should be at least 3 intermediate steps before the final query. 2. Explains what each step adds and why it is done in that order. 3. Shows the final complete query. 4. Includes a section explaining how to run the final query (at least two different methods, e.g. Grafana UI and a CLI tool). 5. Includes a note identifying which labels in the stream selector or filters the engineer would need to change when moving to production.

Expected Behavior

1. Present at least 3 distinct intermediate queries that build incrementally (stream selector → line filter → parser → label filter) 2. Ensure the line filter step (|= "error" or similar) appears before the json parser step 3. Accompany each intermediate step with a plain-English explanation of what it adds and why 4. Use line_format or label_format in the final query to surface error_type and message prominently 5. In the final query, maintain correct order: line filter before json parser, label filters after parser 6. Provide at least two ways to run the query (e.g. Grafana Explore UI and logcli, or HTTP API) 7. Identify at least one label or value to change when moving from staging to production 8. Include at least two label matchers in the final query's stream selector

Success Criteria

• Incremental steps present: The document shows at least 3 distinct intermediate queries, each building on the previous step (stream selector → line filter → parser → label filter)
• Line filter before parser in steps: The step that adds a line filter (|= "error" or similar) appears BEFORE the step that adds the json parser
• Each step explained: Each intermediate step is accompanied by a plain-English explanation of what it adds and why
• line_format used for output: The final query uses line_format or label_format to surface the error_type and message fields prominently
• Filter ordering correct in final query: In the final query, line filter comes before json parser, and label filters (level, component) come after the parser
• Two usage methods provided: The document mentions at least two ways to run the query (e.g. Grafana Explore UI and logcli, or HTTP API)
• Customization notes present: The document identifies at least one label or value to change when moving from staging to production
• Specific stream selector: The final query stream selector includes at least two label matchers

Failure Conditions

• Provides fewer than 3 intermediate steps, jumping straight to the final query
• Introduces the json parser before the line filter in the step sequence
• Omits explanations for one or more intermediate steps
• Final query does not use line_format or label_format to format output fields
• Final query places the line filter after the json parser
• Only describes one way to run the query
• Provides no customization notes for moving to production
• Final stream selector uses only a single label matcher

LogQL Best Practices

This document outlines best practices for writing efficient, maintainable, and performant LogQL queries in Grafana Loki.

Query Structure and Performance

1. Use Specific Stream Selectors

Always use the most specific label selectors possible to reduce the number of streams Loki needs to search.

Good:

{namespace="production", app="api-server", environment="prod"}

Bad:

{namespace="production"}  # Too broad, searches many streams

Why: Loki indexes logs by label combinations (streams). More specific selectors mean fewer streams to search, resulting in faster queries.

2. Order Operations Efficiently

Apply filters in the most efficient order: stream selector → line filters → parser → label filters → aggregations.

Good:

{job="nginx"} |= "error" | json | status_code >= 500 | sum(count_over_time([5m]))

Bad:

{job="nginx"} | json | status_code >= 500 |= "error"  # Parse before line filter

Why: Line filters are fast and work on raw log lines. Parsers are more expensive. Apply cheap operations first to reduce data early.

3. Use Line Filters Before Parsing

Filter out irrelevant log lines before parsing to reduce computational overhead.

Good:

{app="api"} |= "error" | json | level="error"

Bad:

{app="api"} | json | level="error"  # Parses all logs, not just errors

Why: Line filters (|=, !=, |~, !~) are extremely fast string operations. Parsing (json, logfmt, regexp) is more expensive.

4. Avoid Complex Regex When Simple Matching Works

Use exact string matching when possible instead of regex.

Good:

{job="app"} |= "ERROR:"  # Fast string match

Bad:

{job="app"} |~ "ERROR:"  # Slower regex match for simple string

Why: Regex matching requires compilation and more complex pattern matching. Simple string contains is significantly faster.

5. Use Appropriate Time Ranges

Use the shortest time range that satisfies your requirements.

Good:

rate({app="api"}[1m])  # For real-time dashboards
rate({app="api"}[1h])  # For trend analysis

Bad:

rate({app="api"}[24h])  # Unnecessarily long for real-time monitoring

Why: Larger time ranges mean more data to process. Match the range to your use case.

Label Management

6. Understand Label vs Line Filter Trade-offs

Use labels for indexed dimensions, line filters for unique values.

Good (using line filter for unique ID):

{app="api"} |= "trace_id=abc123"

Bad (would create high cardinality if trace_id was a label):

{app="api", trace_id="abc123"}  # Don't do this!

Why: Labels create separate streams and indexes. High cardinality labels (user IDs, trace IDs, session IDs) create too many streams, degrading performance.

7. Keep Cardinality Low

Avoid using high-cardinality data as labels in stream selectors.

High cardinality fields (use line filters instead):

• user_id
• trace_id
• request_id
• session_id
• ip_address (individual IPs)
• timestamp

Good cardinality fields (suitable for labels):

• namespace
• app
• environment
• cluster
• level (error, warn, info)
• pod (in moderation)
• job
• host (in moderation)

Why: Each unique combination of labels creates a new stream. Too many streams overwhelm Loki's indexing.

8. Use Label Operations Wisely

Drop unnecessary labels to reduce series cardinality in metric queries.

Good:

{app="api"} | json | drop instance, pod | sum by (namespace, app) (rate([5m]))

Why: Fewer labels in results = fewer time series = better performance and lower memory usage.

Parsing Best Practices

9. Choose the Right Parser

Use the most appropriate parser for your log format.

Performance order (fastest to slowest): pattern > logfmt > json > regexp

Why this order matters:

• pattern: Simple string matching with placeholders, fastest execution
• logfmt: Optimized key=value parsing, very efficient
• json: Full JSON parsing, moderate overhead
• regexp: Regex compilation and matching, slowest but most flexible

Why: Simpler parsers are faster. JSON and logfmt are optimized. Pattern is faster than regex for simple cases.

9a. Use logfmt Parser Flags When Needed

The logfmt parser supports optional flags for handling edge cases:

`--strict` flag:

# Fail on malformed key=value pairs (stops scanning on error)
{app="api"} | logfmt --strict

# Use when you need to detect malformed log entries
{app="api"} | logfmt --strict | __error__ != ""

`--keep-empty` flag:

# Retain standalone keys as labels with empty string value
{app="api"} | logfmt --keep-empty

# Combine flags
{app="api"} | logfmt --strict --keep-empty

When to use:

• --strict: When log quality matters and you want to detect malformed entries
• --keep-empty: When logs have standalone keys (no values) that need to be preserved

Why: By default, logfmt is non-strict (skips invalid tokens) which is more lenient but may hide log quality issues.

9b. Use JSON Parser Parameter Extraction for Performance

Extract only the fields you need instead of parsing entire JSON:

Good (extract specific fields):

{app="api"} | json status="response.code", method="request.method"

Less efficient (parse all fields):

{app="api"} | json

Supported access patterns:

• Dot notation: | json method="request.method"
• Bracket notation: | json ua="headers[\"User-Agent\"]"
• Array access: | json first="items[0]"
• Combined: | json item="data.items[0].name"

Why: Extracting fewer fields reduces parsing overhead and memory usage.

10. Parse Only What You Need

If you only need specific fields, extract just those fields.

Good:

{app="api"} | json level, message, status_code

Better than:

{app="api"} | json  # Parses all fields

Why: Extracting fewer fields reduces parsing overhead and memory usage.

11. Use Pattern Parser for Simple Cases

Pattern parser is faster than regex for straightforward field extraction.

Good:

{job="nginx"} | pattern "<ip> - - [<timestamp>] \"<method> <path> <_>\" <status>"

Avoid (unless necessary):

{job="nginx"} | regexp "(?P<ip>\\S+) .* (?P<method>\\w+) (?P<path>\\S+).*"

Why: Pattern parser is simpler and faster for structured formats.

Aggregation Best Practices

12. Use Appropriate Aggregation Functions

Choose the right function for your metric type.

13. Aggregate Early and Often

Reduce data volume as early as possible.

Good:

sum by (namespace) (
  count_over_time({app="api"} | json | level="error" [5m])
)

Why: Aggregating reduces the number of time series, improving query performance.

14. Use by Instead of without When Possible

Explicitly specify labels to keep rather than labels to remove.

Good:

sum by (namespace, app) (rate({job="kubernetes-pods"}[5m]))

Less efficient:

sum without (pod, instance, node) (rate({job="kubernetes-pods"}[5m]))

Why: by is more explicit and often results in fewer output series.

Query Optimization

15. Avoid Expensive Operations in Inner Loops

Don't use regex or complex parsing inside frequently-evaluated contexts.

Good:

sum(rate({app="api"} |= "error" [5m]))  # Filter first

Bad:

sum(rate({app="api"} | regexp "complex.*pattern" [5m]))  # Regex on every line

16. Use Metric Queries for Dashboards

For dashboard panels, use metric queries (aggregations) rather than log queries.

Good (for time series panel):

rate({app="api"}[5m])

Bad (for time series panel):

{app="api"}  # Returns log lines, not metrics

Why: Metric queries return time series data suitable for graphing.

17. Limit Log Query Results

When querying for log lines (not metrics), limit the result set.

Important: The limit is an API parameter, not a LogQL pipeline operator. Set it via:

• API: /loki/api/v1/query_range?query={...}&limit=100
• Grafana UI: "Line limit" field in the query editor (default: 1000)
• logcli: --limit=100 flag

Good:

# Using logcli
logcli query '{app="api"} | json | level="error"' --limit=100

# Using API
curl -G "http://localhost:3100/loki/api/v1/query_range" \
  --data-urlencode 'query={app="api"} | json | level="error"' \
  --data-urlencode 'limit=100'

Why: Returning thousands of log lines is slow and resource-intensive. Always set appropriate limits for log queries.

18. Use __error__="" to Filter Parse Errors

When parsing, filter out lines that fail to parse to get clean results.

Good:

{app="api"} | json | __error__="" | level="error"

Why: Parse errors create __error__ labels. Filtering them out gives you only successfully parsed logs.

Alerting Best Practices

19. Use Metric Queries for Alerts

Alerts require numeric values. Always use metric queries (aggregations).

Good:

sum(rate({app="api"} | json | level="error" [5m])) > 10

Bad:

{app="api"} | json | level="error"  # Returns logs, not metrics

20. Include Meaningful Thresholds

Set explicit, meaningful thresholds for alerting.

Good:

(
  sum(rate({app="api"} | json | level="error" [5m]))
  /
  sum(rate({app="api"}[5m]))
) > 0.05  # Alert if error rate > 5%

Why: Thresholds should be based on SLOs or historical baselines.

21. Use absent_over_time for Missing Logs

Detect when logs stop coming (potential service outage).

Good:

absent_over_time({app="critical-service"}[5m])

Why: This returns 1 when no logs match in the time range, indicating a potential problem.

Security and Sensitive Data

22. Don't Log Sensitive Information

Avoid logging sensitive data that could appear in LogQL query results.

Avoid in logs:

• Passwords
• API keys
• Tokens
• Credit card numbers
• PII (personally identifiable information)

If you must log sensitive data:

• Use structured metadata (not indexed)
• Redact before ingestion
• Use Loki's data retention policies
• Restrict access with Loki's multi-tenancy

23. Use Structured Metadata for High-Cardinality Data

Store high-cardinality data as structured metadata, not labels.

Good:

# In your log shipper config
structured_metadata:
  trace_id: ${TRACE_ID}
  user_id: ${USER_ID}

Then query:

{app="api"} | trace_id="abc123"

Why: Structured metadata is not indexed, avoiding cardinality issues.

Maintenance and Debugging

24. Test Queries Incrementally

Build complex queries step by step, testing each stage.

Approach:

# Step 1: Test stream selector
{app="api"}

# Step 2: Add line filter
{app="api"} |= "error"

# Step 3: Add parser
{app="api"} |= "error" | json

# Step 4: Add label filter
{app="api"} |= "error" | json | status_code >= 500

# Step 5: Add aggregation
sum(count_over_time({app="api"} |= "error" | json | status_code >= 500 [5m]))

Why: Incremental testing helps identify issues early and understand query behavior.

25. Use line_format for Debugging

Format log output to see extracted fields during development.

Debugging query:

{app="api"} | json | line_format "level={{.level}} status={{.status_code}} message={{.message}}"

Why: Makes it easy to see what fields were extracted and their values.

26. Comment Complex Queries

Use LogQL comments to document complex queries.

Good:

# Calculate 5xx error rate as percentage
# Alerts when > 5% for SLO compliance
(
  sum(rate({app="api"} | json | status_code >= 500 [5m]))
  /
  sum(rate({app="api"}[5m]))
) * 100 > 5

Why: Comments help team members understand query intent and logic.

Performance Tuning

27. Use Query Splitting for Large Time Ranges

For very large time ranges, consider splitting queries or using downsampling.

Instead of:

sum(count_over_time({app="api"}[30d]))  # Very expensive

Consider:

• Using Loki's query splitting (automatic in recent versions)
• Using recording rules for frequently-queried metrics
• Adjusting retention policies

28. Leverage Loki's Query Parallelization

Recent Loki versions automatically parallelize queries. Structure queries to take advantage:

Good (parallelizable):

sum by (namespace) (rate({job="kubernetes-pods"}[5m]))

Why: Loki can process different namespaces in parallel.

29. Use Appropriate Step Sizes

For metric queries over long time ranges, use appropriate step sizes.

Good:

# For 24h dashboard, use 1m step
rate({app="api"}[5m])  # With 1m step in Grafana

# For 7d dashboard, use 5m or 15m step
rate({app="api"}[15m])  # With 5m step

Why: Smaller steps = more data points = slower queries. Match resolution to your needs.

Structured Metadata (Loki 3.x)

35. Use Structured Metadata for High-Cardinality Data

Structured metadata is metadata attached to logs without indexing. Introduced in Loki 3.0.

What it is:

• Metadata attached to logs that is NOT indexed
• Ideal for high-cardinality data (trace_id, user_id, request_id, pod names)
• Avoids index bloat and cardinality explosion
• Automatically extracted as labels in query results

Key differences from labels:

• Labels are indexed → fast stream selection, but high cardinality is expensive
• Structured metadata is NOT indexed → no cardinality impact, but requires scanning

Query syntax:

# Filter by structured metadata (AFTER stream selector, not inside it!)
{app="api"} | trace_id="abc123"

# Combine multiple structured metadata filters
{app="api"} | trace_id="abc123" | user_id="user456"

# Use with other filters
{app="api"} | trace_id="abc123" | json | level="error"

WRONG (structured metadata is not a label):

{app="api", trace_id="abc123"}  # This won't work!

When to use:

• OpenTelemetry data (trace IDs, span IDs)
• High-cardinality identifiers (user IDs, request IDs, session IDs)
• Kubernetes metadata (pod UIDs, container IDs)
• Any data that would create too many unique label combinations

Configuration (requires Loki 3.0+ with schema v13+):

limits_config:
  allow_structured_metadata: true

36. Query Acceleration with Structured Metadata

Loki 3.x can accelerate queries using bloom filters when structured metadata filters are placed correctly.

CRITICAL: Filter Order Matters for Acceleration

Accelerated (bloom filters used):

{cluster="prod"} | detected_level="error" | logfmt | json

The structured metadata filter comes BEFORE parsers.

NOT Accelerated (bloom filters NOT used):

{cluster="prod"} | logfmt | json | detected_level="error"

The filter comes AFTER parsers, preventing acceleration.

Rules for query acceleration: 1. Use string equality filters: | key="value" 2. Place structured metadata filters BEFORE any parser expressions 3. Filters BEFORE logfmt, json, pattern, regexp, label_format, label_replace

Supported filter patterns:

# Simple equality (accelerated)
{app="api"} | trace_id="abc123" | json

# Multiple filters with OR (accelerated)
{app="api"} | detected_level="error" or detected_level="warn" | json

# Multiple filters with AND (accelerated)
{app="api"} | service="api" and environment="prod" | json

Why this matters:

• Bloom filters can skip chunks that definitely don't contain the data
• Significant performance improvement for "needle in haystack" queries
• Essential for large-scale deployments (75TB+ monthly logs)

__error__ Label Debugging

37. Debug Parse Errors with __error__ Label

When parsing fails, Loki creates an __error__ label with the error type.

Show only lines that failed to parse:

{app="api"} | json | __error__ != ""

Show only successfully parsed lines (filter OUT errors):

{app="api"} | json | __error__=""

Common error values:

• JSONParserErr - Invalid JSON
• LogfmtParserErr - Invalid logfmt
• PatternParserErr - Pattern didn't match
• RegexpParserErr - Regex didn't match

Debugging workflow:

# Step 1: See which lines are failing
{app="api"} | json | __error__ != "" | line_format "ERROR: {{.__error__}} LINE: {{.__line__}}"

# Step 2: Count errors by type
sum by (__error__) (count_over_time({app="api"} | json | __error__ != "" [5m]))

# Step 3: Production query (exclude errors)
{app="api"} | json | __error__="" | level="error"

Why this matters:

• Silent parse failures can cause missing data
• Always filter __error__="" in production dashboards
• Use error queries to debug log format issues

Recording Rules

38. Use Recording Rules for Expensive Queries

Recording rules precompute expensive queries and store results as metrics.

When to use recording rules:

• Dashboard queries that run frequently
• Complex aggregations over large datasets
• Queries that would otherwise time out
• Per-tenant alerting in multi-tenant systems

Example recording rule configuration:

# /tmp/loki/rules/<tenant-id>/rules.yaml
groups:
  - name: error_rates
    interval: 1m
    rules:
      # Record error rate per app
      - record: app:error_rate:1m
        expr: |
          sum by (app) (
            rate({job="kubernetes-pods"} | json | level="error" [1m])
          )
        labels:
          source: loki_recording_rule

      # Record request rate per namespace
      - record: namespace:request_rate:5m
        expr: |
          sum by (namespace) (
            rate({job="kubernetes-pods"}[5m])
          )

  - name: alerting_rules
    interval: 1m
    rules:
      - alert: HighErrorRate
        expr: |
          (
            sum by (app) (rate({job="app"} | json | level="error" [5m]))
            /
            sum by (app) (rate({job="app"}[5m]))
          ) > 0.05
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High error rate for {{ $labels.app }}"
          description: "Error rate is {{ $value | printf \"%.2f\" }}%"

Ruler configuration:

ruler:
  storage:
    type: local
    local:
      directory: /tmp/loki/rules
  rule_path: /tmp/scratch
  alertmanager_url: http://alertmanager:9093
  enable_api: true
  ring:
    kvstore:
      store: inmemory

Benefits:

• Reduces query load on Loki
• Faster dashboard loading
• Consistent results across queries
• Enables alerting on complex conditions

39. Use vector() for Reliable Alerting

The vector() function ensures alerting rules always return a value.

Problem: When no logs match, the query returns nothing, causing "no data" alert states.

Solution:

# Always returns a value (0 when no matches)
sum(count_over_time({app="api"} | json | level="error" [5m])) or vector(0)

# Use in alerting rule
sum(rate({app="api"} | json | level="error" [5m])) or vector(0) > 10

Why this matters:

• Prevents flapping alerts due to "no data" states
• Provides consistent behavior for sparse logs
• Essential for reliable alerting on low-volume services

Anti-Patterns to Avoid

30. Don't Use High-Cardinality Labels

Never do this:

{app="api", user_id="12345"}  # user_id is high cardinality!

Do this instead:

{app="api"} | json | user_id="12345"

31. Don't Parse Multiple Times

Inefficient:

{app="api"} | json | json | json  # Multiple parsers

Efficient:

{app="api"} | json  # Once is enough

32. Don't Use Regex for Simple String Matching

Inefficient:

{app="api"} |~ "GET"  # Regex for simple string

Efficient:

{app="api"} |= "GET"  # Fast string contains

33. Don't Aggregate Without Labels

Inefficient (no grouping):

sum(rate({app="api"}[5m]))  # Single time series

Better (grouped by useful dimensions):

sum by (namespace, app, environment) (rate({app="api"}[5m]))

34. Don't Use Very Long Time Ranges in range vectors

Inefficient:

rate({app="api"}[24h])  # 24 hours of data per calculation

Efficient:

rate({app="api"}[5m])  # 5 minutes of data per calculation

Why: Range vectors determine how much historical data each point calculation needs.

Important Notes About Non-Existent Features

LogQL Does NOT Have dedup or distinct Operators

No `| dedup` syntax: Deduplication is handled at the UI level in Grafana's Explore panel, not in LogQL itself.

No `| distinct` syntax: A distinct operator was proposed in PR #8662 but was reverted before public release due to issues with query splitting, sharding, and metric query compatibility. The proposed syntax {job="app"} | distinct label is NOT available in current Loki versions.

For programmatic deduplication, use metric aggregations:

# Count unique messages
sum by (message) (count_over_time({app="api"} | json [5m])) > 0

# Count distinct values of a label
count(count by (user_id) ({app="api"} | json))

LogQL limit is an API Parameter, NOT a Pipeline Operator

There is no | limit 100 syntax in LogQL. The limit is set via:

• API parameter: &limit=100
• Grafana UI: "Line limit" field
• logcli: --limit=100 flag

See Best Practice #17 for details.

Summary Checklist

When writing LogQL queries, ensure:

• [ ] Stream selectors are as specific as possible
• [ ] Line filters come before parsers
• [ ] Exact string matching is used instead of regex when possible
• [ ] Time ranges are appropriate for the use case
• [ ] High-cardinality data is not used as labels
• [ ] The right parser is chosen for the log format
• [ ] Only necessary fields are extracted
• [ ] Aggregations are used for metric queries
• [ ] Results are limited for log queries
• [ ] Queries are tested incrementally
• [ ] Complex queries are documented with comments
• [ ] sort or sort_desc used for ordered results
• [ ] label_replace used for regex-based label manipulation in metrics
• [ ] vector(0) used as fallback in alerting rules

Additional Resources

• Grafana Loki Best Practices
• LogQL Documentation
• Loki Operations Guide

Related Skills

• loki-config-generator: For configuring Loki server
• promql-generator: For PromQL queries (similar concepts)
• fluentbit-generator: For log collection pipelines

LogQL Function and Parser Reference

Quick Function Reference

Log Range Aggregations (most common)

Unwrapped Range Aggregations (most common)

Aggregation Operators

sum, avg, min, max, count, stddev, topk, bottomk, approx_topk, sort, sort_desc

With grouping: sum by (label1, label2) or sum without (label1)

Conversion Functions

label_replace()

label_replace(rate({job="api"} |= "err" [1m]), "foo", "$1", "service", "(.*):.*")

Parser Reference

logfmt

| logfmt [--strict] [--keep-empty]

• --strict: Error on malformed entries
• --keep-empty: Keep standalone keys

JSON

| json                                           # All fields
| json method="request.method", status="response.status"  # Specific fields
| json servers[0], headers="request.headers[\"User-Agent\"]"  # Nested/array

Template Functions

Common functions for line_format and label_format:

String: trim, upper, lower, replace, trunc, substr, printf, contains, hasPrefix Math: add, sub, mul, div, addf, subf, floor, ceil, round Date: date, now, unixEpoch, toDate, duration_seconds Regex: regexReplaceAll, count Other: fromJson, default, int, float64, __line__, __timestamp__

See assets/common_queries.logql for detailed usage examples.