Rag Perf

RAG-Perf — config-driven perf benchmark CLI

Purpose

Drive a deployed NVIDIA RAG Blueprint server with a YAML config, run a server-side profiling pass (per-stage timing, citation quality, bottleneck inference) and an optional aiperf load test (TTFT / E2E / token & request throughput / error rate), and write a unified report. The CLI is intentionally minimal: rag-perf -c <config> plus --help / --version. Behaviour is fully config-driven; field variations belong in YAML.

Scope

• Accuracy / RAGAS scoring of answer quality → use the rag-eval skill.
• Deploying, repairing, or configuring services (compose, helm, NIM env vars) → use the rag-blueprint skill.
• Production monitoring / alerting — rag-perf is a one-shot benchmark tool.
• Runtime requirement: a deployed RAG server reachable on the network.

Prerequisites

• Repo cloned; run commands from the repo root (config paths in the presets are repo-root-relative).
• Python 3.11+ and uv on PATH.
• Install rag-perf into its own uv-managed venv: uv sync --project scripts/rag-perf.
• For unit tests: install dev extras as well — uv sync --project scripts/rag-perf --extra dev (otherwise pytest-asyncio is missing and async tests error out at collection time).
• A reachable RAG server (default http://localhost:8081). For the aiperf phase, the bundled nvidia_rag endpoint plugin must be installed — pip install -e ./scripts/rag-perf registers it via the aiperf.plugins entry point.
• For synthetic queries: an OpenAI-compatible chat-completions endpoint reachable at synthetic.llm_url (default http://localhost:8999/v1/chat/completions).
• rag-perf itself runs without NVIDIA_API_KEY (unlike rag-eval). The synthetic LLM endpoint may require its own auth — that's the deployment's concern.

Instructions

1. Pick a preset. The three under `scripts/rag-perf/configs/` are:

• quick_profile.yaml — profile-only, ~30 s. Skips load test. For fast iteration on retrieval / reranker tuning.
• single_run.yaml — one concurrency level, profiling + aiperf, ~2 min. Regression checks.
• sweep.yaml — multi-axis sweep. load.concurrency, rag.vdb_top_k, rag.reranker_top_k are all int | list[int]; any of them as a list becomes a sweep axis (Cartesian product).

2. Edit the preset. Required: replace rag.collection_names: ["<collection_name>"] with a real collection on the deployed ingestor server. Verify the collection exists via GET /v1/collections on the ingestor. The placeholder <collection_name> validates fine but every request will fail at retrieval. Use a copied YAML preset for variants; the CLI surface is intentionally config-only.

3. Run. From repo root:

   uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/single_run.yaml

Same form for the other presets. The CLI accepts only -c / --config (required), --help, --version.

4. Read stdout. Every invocation prints, in order: a startup banner, a one-line summary, the fully resolved config as YAML (so the run is reproducible from terminal output), per-grid-point progress with the shlex-joined aiperf command in copy-pastable form, a rich per-point summary table (stage breakdown with bars, citation quality, bottleneck, load-test block), and finally a side-by-side comparison table auto-labelled by whichever axis varied. See `references/output-and-analysis.md`.

5. Inspect artifacts. Layout depends on run shape — flat for single-point + iterations=1, nested under iter_<i>/<point>/... otherwise. See `references/output-and-analysis.md` for the full directory tree, file purposes, and how to parse results.json / results.csv / report.md.

6. Summarise for the user. When reporting back, follow the playbook in `references/output-and-analysis.md#summarising-results-to-the-user`: pick the canonical result file for the run shape, build a headline table (concurrency × top-k axes × TTFT × throughput × bottleneck × citation quality), compute scaling efficiency on sweeps, always flag zero citations / non-zero error rate / suspect llm_ttft_ms / small-sample p99, and propose a concrete next-experiment YAML.

7. Tune. Schema is fully documented in `docs/performance-benchmarking.md` and the deeper-dive references below. Common knobs: turn aiperf.enabled: false for profile-only mode, increase load.iterations for variance estimation, set load.sleep_between_points_s: 60 for overnight Cartesian sweeps.

Examples

Profile-only (quickest signal on retrieval / reranker tuning):

uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/quick_profile.yaml

Output: rag-perf-results/quick_profile/run_<ts>/{profile_report.md, profile_results.json, profiling/}. The aiperf_rag_on/ directory is omitted. Filenames are profile_* because aiperf.enabled: false.

Single benchmark point with full report:

uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/single_run.yaml

Output: flat run_<ts>/{report.md, results.json, results.csv, profiling/, aiperf_rag_on/}.

Concurrency sweep:

uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/sweep.yaml

Output: nested run_<ts>/iter_1/<CR:_VDB-K:_RERANKER-K:_…>/{profiling,aiperf_rag_on}/ per point, plus aggregate report.md / results.json / results.csv at the run root.

Run unit tests:

uv sync --project scripts/rag-perf --extra dev   # one-time, installs pytest-asyncio
uv run --project scripts/rag-perf python -m pytest tests/unit/test_rag_perf/

Limitations

• The CLI is config-only: author or copy YAML to vary a parameter.
• load.concurrency / rag.vdb_top_k / rag.reranker_top_k accept int | list[int]; the validator requires unique list values because each value names a unique point dir.
• input.file and input.synthetic follow an XOR rule — both set fails validation. When neither is set, synthetic auto-fills with defaults so a bare config still validates.
• File-based input format is inferred from extension only (.jsonl or .csv); other extensions are rejected.
• Synthetic generation streams each query to disk as it completes (failure-resilient) but fails fast on the first LLM error — partial JSONL is preserved. Re-run after fixing the endpoint.
• Reasoning models (Nemotron Omni, Qwen-Reasoning) require synthetic.disable_thinking: true (the default). Without it the model exhausts the token budget on chain-of-thought and content returns empty — the generator now raises with a clear message instead of substituting reasoning_content for the answer.
• aiperf-specific knobs outside the YAML surface (request rate distribution, GPU telemetry config, etc.) require editing AiperfRunner._base_aiperf_cmd in scripts/rag-perf/rag_perf/runner.py.
• Procedural detail lives under `references/` to keep this file concise.

Troubleshooting

Gotchas

• Run from repo root. Preset configs reference scripts/rag-perf/examples/queries.jsonl and scripts/rag-perf/prompts/default_prompts.yaml with repo-root-relative paths. Running from inside scripts/rag-perf/ will fail those file lookups.
• CLI is config-only. Edit the YAML or copy a preset for URL, concurrency, collection, and similar fields.
• Always edit `rag.collection_names` before the first run. The presets ship with ["<collection_name>"] as a deliberate placeholder. Validation passes, retrieval fails silently for every request — manifests as Citation count (mean): 0 everywhere.
• `load.concurrency_list`, `rag.vdb_top_k_list`, `rag.reranker_top_k_list` are read-only properties that normalise scalar-or-list to a list. Use them when reasoning about the grid; the underlying YAML field is whatever the user wrote.
• `aiperf.enabled: false` changes filenames. The top-level outputs become profile_report.md / profile_results.json / profile_results.csv. The aggregate sweep table also suppresses load-test rows and the "Optimal throughput" footer.
• Resolved-config dump is verbose (50+ lines) — expected. It's what makes terminal output a self-contained reproducer; don't filter it out in scripts.
• The aiperf shell command is logged before each subprocess. Look for \n $ python -m aiperf profile -m ... --endpoint-type nvidia_rag ... in stdout — copy-paste runnable for reproducing a single point outside rag-perf.
• `--endpoint-type nvidia_rag` comes from the bundled plugin at scripts/rag-perf/rag_perf/plugin/nvidia_rag.py. It teaches aiperf about the RAG /v1/generate request shape and parses citations + per-stage metrics out of the SSE stream. If aiperf can't resolve nvidia_rag, rag-perf needs editable installation in the venv — re-run uv sync --project scripts/rag-perf (or uv pip install -e ./scripts/rag-perf).
• Sweep-mode point-name collision. When two points differ only in concurrency (e.g. [1, 4] × single vdb_top_k), the dir name encodes everything: CR:1_ISL:50_OSL:512_VDB-K:20_RERANKER-K:4_Model:.... Cluster / GPU / experiment_name (output.cluster, output.gpu, output.experiment_name) are appended too — useful for diff-friendly artifact paths across machines.
• `load.iterations > 1` repeats the entire grid. Each repetition writes to its own iter_<i>/. Aggregate CSV row count = n_points × iterations.

Source of truth

Agent playbook

1. Sync deps: uv sync --project scripts/rag-perf (one-time per checkout). 2. Pick & customise a preset: copy scripts/rag-perf/configs/<preset>.yaml if you want a variant; always set rag.collection_names to a real collection. 3. Run: uv run --project scripts/rag-perf rag-perf -c <config> from repo root. 4. Read the per-point + aggregate tables on stdout. Bottleneck inference is in the per-point profiling section; comparison across points is the final aggregate table. 5. Parse artifacts under output.dir/run_<ts>/ — see `references/output-and-analysis.md`. For multi-point runs, results.csv has one row per (point × iteration). 6. Summarise for the user using the playbook in `references/output-and-analysis.md#summarising-results-to-the-user` — headline table, scaling-efficiency math for sweeps, mandatory flags for zero citations / non-zero errors / suspect llm_ttft_ms / low sample size, and a concrete next-experiment YAML. 7. Tune retrieval / reranker: flip to quick_profile.yaml or aiperf.enabled: false for fast iteration, then return to single_run.yaml / sweep.yaml when characterising under load. 8. Triage failures: see Troubleshooting above and `references/output-and-analysis.md` for empty-citation / bottleneck=N/A patterns.

Anti-Patterns

• Optimizing before recording a baseline: Without a starting point, there is no trustworthy performance story.
• Using toy traffic to justify production tuning: Tiny prompts or empty corpora hide the real bottleneck.
• Accepting lower answer quality as an untracked side effect of a latency win.

Verification Protocol

Before claiming "skill applied successfully":

1. Pass/fail: The performance investigation starts from a named baseline and a concrete bottleneck hypothesis. 2. Pass/fail: The workload shape, corpus size, and concurrency assumptions match the path being optimized. 3. Pass/fail: Any latency or throughput claim is paired with a correctness or quality guardrail. 4. Pressure-test scenario: Re-run the workflow on a retrieval stack that speeds up only because caching masked a stale index. 5. Success metric: The user gets a reproducible benchmark path and a tuning change that improves the intended metric without hidden regressions.

<!-- PORTABILITY:START -->

Cross-Client Portability

This skill is written to stay usable across GitHub Copilot, Claude Code, Codex, and Gemini CLI.

• GitHub Copilot: keep the folder in a Copilot-visible skill or plugin path, or wrap the workflow as project instructions if the host does not support portable skill folders directly.
• Claude Code: keep the folder in a local skills directory or a compatible plugin or marketplace source.
• Codex: install or sync the folder into $CODEX_HOME/skills/<skill-name> and restart Codex after major changes.
• Gemini CLI: this repository generates a project command named /skills:rag-perf from this skill. Rebuild commands with python scripts/export-gemini-skill.py rag-perf and then run /commands reload inside Gemini CLI.

<!-- PORTABILITY:END -->

<!-- MCP:START -->

MCP Availability And Fallback

Preferred MCP Server: None required

• Fallback prompt: "Use the rag-perf skill without MCP. Rely on the local SKILL.md, bundled references or scripts, and manual verification. Show the exact commands, evidence, and final checks you used before concluding."
• If the current host does not expose a matching server, use the bundled references, scripts, native toolchain, and manual workflow already described in this skill.
• Treat direct local verification, rendered output, logs, tests, or screenshots as the fallback evidence path before completion.

<!-- MCP:END -->

Related Skills

• devops-tooling: Use it when the performance work also needs container, CI, or infrastructure diagnostics.
• cloud-design-patterns: Use it when bottlenecks reveal larger scaling or architecture tradeoffs.
• documentation-verification: Use it when benchmark claims must be published or preserved in repo docs.

Evaluation Report

Evaluation of the rag-perf skill before publication through NVSkills-Eval.

This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.

Evaluation Summary

• Skill: rag-perf
• Evaluation date: 2026-05-29
• NVSkills-Eval profile: external
• Overall verdict: PASS
• Tier 3 live agent evaluation: not available in this report

Agents Used

• Tier 3 agent details were not available in this report.

Metrics Used

Reported benchmark dimensions:

• Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
• Correctness: checks whether the agent follows the expected workflow and produces the correct final output.
• Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
• Effectiveness: checks whether the agent performs measurably better with the skill than without it.
• Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.

Underlying evaluation signals used in this run:

• No Tier 3 evaluation signal details were available in this report.

Test Tasks

Tier 3 evaluation task details were not available in this report.

Results

Tier 3 dimension rollup was not available in this report.

Tier 1: Static Validation Summary

Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 5 total findings.

Top findings:

• MEDIUM PII/phone_numbers: US phone number pattern (references/synthetic-generation.md:85)
• MEDIUM QUALITY/quality_efficiency: Deeply nested references in config-schema.md (skills/rag-perf/SKILL.md)
• LOW QUALITY/quality_discoverability: Description very long (241 chars, recommend 50-150) (skills/rag-perf/SKILL.md)
• LOW SCHEMA/unexpected_file: Unexpected 'BENCHMARK.md' in skill root (skills/rag-perf/BENCHMARK.md)
• LOW SCHEMA/unexpected_file: Unexpected 'eval' in skill root (skills/rag-perf/eval)

Tier 2: Deduplication Summary

Tier 2 validation passed. NVSkills-Eval ran 2 checks and found 0 total findings.

Notable observations:

• Context Deduplication: Collected 4 file(s)
• Inter-Skill Deduplication: Parsed skill 'rag-perf': 241 char description

Publication Recommendation

The skill is suitable to proceed toward NVSkills-Eval publication based on this benchmark. Skill owners should keep this file with the skill and refresh it when the evaluation dataset, skill behavior, or target agents materially change.

Changelog

All notable changes to the rag-perf skill will be documented in this file.

[2026-06-09] - Initial Import and Catalog Normalization

Added

• Imported rag-perf from https://github.com/NVIDIA/skills at skills/rag-perf pinned to 129a1087a1853f32a950e2f7bbc0fd7d57b9d422.
• Added repo-standard Anti-Patterns, Verification Protocol, portability, MCP fallback, and related-skills sections.
• Preserved the upstream benchmark, signature, skill card, and bundled references or scripts for provenance and later refreshes.

Changed

• Normalized SKILL.md frontmatter to the shared catalog schema with version: "1.2" and last_updated: 2026-06-09.
• Moved upstream-only top-level metadata into the nested metadata block so validation, export, and downstream sync stay consistent.

Fixed

• Aligned the imported skill with this repository's maintained-skill requirements and downstream sync workflow.

Config schema reference

Load this when the user is authoring a new YAML, debugging a Configuration errors message, or asking which knob controls a behaviour. Schema is defined in `scripts/rag-perf/rag_perf/config.py` (RunConfig + sub-models, Pydantic v2). User-facing prose is in `docs/performance-benchmarking.md`.

Top-level shape

target:    {...}
aiperf:    {...}
load:      {...}
rag:       {...}
generation: {...}
input:     {...}
output:    {...}
model_name: "nvidia/nemotron-3-super-120b-a12b"   # passed to aiperf via -m
tokenizer:  ""                                     # optional HF tokenizer for token counting

There is no sweep: block any more — sweep axes live where they belong (load.concurrency, rag.vdb_top_k, rag.reranker_top_k) and run-orchestration moved under load (iterations, sleep_between_points_s).

target

aiperf

load

Drives the aiperf load-test phase and the orchestration of the grid.

Helper: LoadConfig.concurrency_list returns [scalar] or the list — use this when iterating.

rag

Forwarded verbatim into the /v1/generate request body. Per-query overrides in JSONL/CSV win over these defaults.

Helpers: RagParams.vdb_top_k_list, RagParams.reranker_top_k_list mirror the concurrency_list pattern.

generation

`min_tokens: null` handling. rag-perf strips None-valued generation fields before merging into the request body — the server's Prompt.min_tokens: int rejects an explicit null (would be a 422). This is in `QueryLoader._build_request`.

input

Set exactly one of file or synthetic. They are mutually exclusive — both → validation error. Neither → synthetic auto-fills with defaults.

File-based input details

• `.jsonl`: one JSON object per line, {"query": "...", ...}. Any field also defined under rag.* or generation.* is treated as a per-query override.
• `.csv`: must have a query column. Other columns matching rag.* / generation.* field names become per-query overrides; CSV cell values are JSON-parsed when possible (so ["finance"] is a list, not a string).

output

Polymorphic axes & the grid

Three fields are scalar-or-list:

• load.concurrency
• rag.vdb_top_k
• rag.reranker_top_k

The full grid is the Cartesian product across whichever are lists. Each point yields a fresh RunConfig with all three resolved to scalars (see BenchmarkRunner._iter_grid_points in `runner.py`). Run shape:

When aiperf.enabled: false, top-level files become profile_report.md / profile_results.json / profile_results.csv.

Validation invariants (worth remembering)

• load.concurrency rejects [], scalar <1, list with <1 entries, and duplicates.
• rag.vdb_top_k / reranker_top_k enforce range (1–400 / 1–25), reject duplicates in lists, reject empty lists.
• load.warmup_requests >= 1 (aiperf rejects 0).
• input XOR rule: both file and synthetic set → fail; neither set → auto-fill synthetic with defaults.
• input.file extension must be .jsonl or .csv; anything else → fail.
• For synthetic.mode: dataset_based, either dataset_file or dataset_name must be set.

These all run at YAML load time in _load_config (cli.py). Errors print a per-field bullet list and exit 1 — no benchmark code runs, no output dir is created.

Programmatic overrides

For tests / scripted invocations:

from rag_perf.config import RunConfig
cfg = RunConfig.from_yaml("scripts/rag-perf/configs/single_run.yaml")
cfg = cfg.with_overrides(load__concurrency=[1, 4, 8], rag__vdb_top_k=50)

Double-underscore = nested key. with_overrides re-runs Pydantic validation on the merged config. There is no equivalent on the CLI — see SKILL.md "CLI is config-only" gotcha.

Output layout and result analysis

Load this when the user asks where artifacts went, how to interpret a metric, or what a column in results.csv means. Driver code: `scripts/rag-perf/rag_perf/runner.py` (BenchmarkRunner.run, _write_aggregate_outputs) and `scripts/rag-perf/rag_perf/reporting.py` (MetricsAggregator, Reporter, RagMetricsSummary).

Stdout sequence (in order)

1. Banner: ASCII "RAG PERF" logo + version. 2. Run-info summary: target URL, collection, vdb_top_k / reranker_top_k, input source, concurrency, total_requests, aiperf on/off. One-line per field, ~7 lines. 3. Resolved configuration: the full RunConfig dumped as YAML via RunConfig.to_yaml_str(). Verbose (~50 lines) by design — makes terminal output a self-contained reproducer. Don't strip in scripts. 4. Per grid point:

• Section rule: ─── Point N/M: conc=... vdb_top_k=... rr_top_k=... ───
• → Running profiling pass (collecting server-side metrics)...
• → Running aiperf load test (concurrency=..., requests=...)... (only when aiperf.enabled: true)
• aiperf's own per-iteration log lines (logger.INFO output from the subprocess)
• Copy-pastable shell command: \n $ python -m aiperf profile -m ... --endpoint-type nvidia_rag ...\n — useful for reproducing a single point outside rag-perf
• aiperf summary (its own table)

5. Per-point summary table (rich format, after each point completes in multi-point mode): "RAG-Perf Results — conc=N vdb_top_k=N rr_top_k=N" with stage breakdown bars, citation quality, bottleneck, load-test block. 6. Aggregate sweep table (multi-point only): "RAG-Perf Sweep — \<varying axis\>" side-by-side comparison. Auto-detects which axes vary; column header reflects the varying axis (concurrency / vdb_top_k / reranker_top_k / iter#). Footer: Optimal throughput: <axis>=<value> (X req/s) and Best p99 TTFT < 30s: <axis>=<value>.

If aiperf.enabled: false, the load-test rows in step 5/6 are suppressed and the optimal-throughput footer is hidden.

On-disk layout

Top level always: output.dir/run_<ts>/ (UTC timestamp YYYYMMDDTHHMMSS).

Single point + iterations=1 + aiperf.enabled=true

run_<ts>/
├── report.md            # markdown summary of this point
├── results.csv          # one-row CSV
├── results.json         # single RagMetricsSummary dict
├── profiling/
│   └── profiler_records.jsonl
└── aiperf_rag_on/
    ├── inputs.json
    ├── profile_export_aiperf.csv
    ├── profile_export_aiperf.json
    ├── profile_export.jsonl
    └── logs/aiperf.log

Single point + iterations=1 + aiperf.enabled=false

run_<ts>/
├── profile_report.md
├── profile_results.json
├── (no profile_results.csv if "csv" not in output.formats)
└── profiling/
    └── profiler_records.jsonl

No aiperf_rag_on/. profile_* filename prefix is the visual indicator.

Multi-point or iterations > 1

run_<ts>/
├── report.md            # aggregate, summarises all points
├── results.csv          # one row per (point × iteration)
├── results.json         # list of RagMetricsSummary dicts (or single dict if N=1)
└── iter_<i>/
    └── CR:<conc>_ISL:<isl>_OSL:<osl>_VDB-K:<vdb>_RERANKER-K:<rr>_Model:<model_clean>[_Cluster:<x>][_GPU:<y>][_Experiment:<z>]/
        ├── profiling/
        │   └── profiler_records.jsonl
        └── aiperf_rag_on/
            └── ... (same files as above)

<isl> is synthetic.min_query_tokens for synthetic mode, literal var for file-based mode (where ISL varies per query). <osl> is generation.max_tokens. <model_clean> is model_name with / replaced by -.

RagMetricsSummary fields (results.json / results.csv)

Defined in `scripts/rag-perf/rag_perf/reporting.py`.

Stage breakdown (profiling pass)

Citation quality

Citations land on the first SSE chunk. The profiler latches them on the first non-empty citations.results payload (server attaches them alongside the initial empty content delta, not the final chunk). Don't change this.

Client-side timing (profiling pass)

aiperf load-test fields

All None when aiperf.enabled: false (suppressed in tables).

Run metadata

Quick analysis recipes

Pretty-print a single-point summary:

python3 -m json.tool rag-perf-results/<dir>/run_<ts>/results.json

One-row-per-point view of a sweep:

column -ts',' rag-perf-results/<dir>/run_<ts>/results.csv | less -S

Compare two sweep runs:

diff <(cat rag-perf-results/before/run_*/results.csv) \
     <(cat rag-perf-results/after/run_*/results.csv)

Replay a single aiperf invocation outside rag-perf: copy the \n $ python -m aiperf profile ... line from rag-perf's stdout — it's a self-contained shlex-joined shell command using the same temp queries JSONL.

Summarising results to the user

After a run finishes, follow this playbook to produce a tight report instead of dumping raw JSON.

1. Locate the canonical result file

Depends on run shape:

Discover the latest run dir with:

ls -td rag-perf-results/<preset>/run_* | head -1

2. Extract the headline numbers

For each point pull these into a table:

If aiperf.enabled: false, load_* are all None — note "profile-only run" and skip the load-test column group.

3. Compute the unaccounted-time gap

unaccounted = rag_ttft_ms − (retrieval_ms + reranking_ms + llm_ttft_ms)

If unaccounted > a stage's reported time, the breakdown isn't telling the whole story (most often: llm_ttft_ms is mismeasured server-side and reads near zero, leaving most of the TTFT unattributed). Mention this in the summary as a caveat — don't let the user infer "the LLM is free."

4. Compute scaling efficiency (sweeps only)

For a concurrency sweep, compute throughput ratio vs concurrency ratio between the lowest and highest points:

scaling_efficiency = (req/s_max / req/s_min) / (concurrency_max / concurrency_min)

Linear scaling = 1.0; sub-linear < 1.0 indicates saturation. Pair with TTFT p99 ratio — >2× p99 worsening for <1.5× throughput gain is the canonical congestion signature; flag the knee location.

5. Signals worth calling out

Always flag in the summary, not just in passing:

• `Citation count (mean): 0` everywhere — collection mismatch. Suggest verifying with curl http://<ingestor>:8082/v1/collections.
• `load_error_rate > 0` — non-zero error rate in a benchmark is a finding, not a footnote. State the absolute count and the likely cause (saturation? timeouts?).
• `stage_breakdown.llm_ttft_ms < 1 ms` — almost certainly a measurement bug, not a real number. Caveat any LLM-stage conclusions.
• `profile_requests_failed > 0` — partial profiling pass; the per-stage means may be skewed if the failures clustered.
• Bottleneck stays constant across the sweep — informative: tells the user that scaling that axis doesn't shift the bottleneck (e.g. reranker stays dominant whether vdb_top_k=20 or 100 → reranker model is the real cost, not the chunk-count).
• Tail-latency p99 from very low `total_requests` (< 50) — explicitly note that the tail is not statistically robust at that sample size; recommend bumping total_requests for follow-up.

6. Suggest concrete next experiments

Tie suggestions to the data, not generic advice. Examples:

• "Reranker is the bottleneck at 23% of TTFT — try enable_reranker: false as a baseline to see how much accuracy you'd give up to drop that 164 ms."
• "Throughput plateau between conc=4 and conc=8 — add concurrency: [1, 2, 4, 6, 8] to find the knee precisely."
• "TTFT p99 jumps 3× for 1.7× throughput gain at conc=4 — the system is saturating; back off to conc=2 for SLA-bound traffic and use conc≥4 only when batched throughput matters more than tail latency."
• "Citation score mean 0.58 with p90 0.80 is fine; if you want higher precision try reranker_top_k=2 and watch the per-citation score change."

7. Format the summary

Use a small fixed structure:

1. Run shape — preset, point count, iterations, profile-only or full. 2. Headline table — one row per point, columns from §2. 3. Findings — 3–5 bullets pointing at numbers in the table (cite the column). 4. Caveats — sample size, suspect metrics, anything in §5. 5. Recommended next config — concrete YAML diff or a "try this preset" line.

Aim for ~30 lines total. Long-form interpretation belongs in a follow-up if the user asks; the first response should be scannable.

Common patterns in results

Synthetic query generation

Load this when input.synthetic is in play, when reasoning-model query leakage is suspected, when generation fails midway, or when the user wants to reproduce a query set across runs.

Implementation lives in `scripts/rag-perf/rag_perf/query.py` (SyntheticQueryGenerator). Default prompts are in `scripts/rag-perf/prompts/default_prompts.yaml`.

Pipeline

When input.synthetic is set, rag-perf — before the benchmark phase even starts — does this:

1. Resolves the LLM model (synthetic.llm_model, or auto-discover via GET /v1/models). 2. Loads prompt templates (synthetic.prompts_file, or bundled defaults). 3. For mode: dataset_based, loads reference questions from synthetic.dataset_file or synthetic.dataset_name (auto-lookup under ./datasets/<name>/{train,data}.json). For mode: random, no reference material. 4. Builds N per-query user messages. 5. Fans out concurrent LLM calls (bounded by synthetic.generation_concurrency, default 8) using asyncio.gather over asyncio.to_thread wrappers around the sync httpx.post. 6. Streams each successful query to disk as it completes — under an asyncio.Lock, with flush() after every line. The file at synthetic.jsonl_output_path is opened in "w" mode and written line-by-line. 7. Returns the in-memory list (also persisted on disk). 8. Hands off to QueryLoader._load_jsonl and the benchmark runs from the now-static file.

The key consequence: a mid-generation failure preserves all queries that completed before it. The exception still propagates (asyncio.gather cancels remaining tasks on first failure) and the run aborts — no automatic retry.

All synthetic knobs

For dataset_based, validation requires either dataset_file or dataset_name. Both unset → ValidationError.

Reasoning-model gotcha (read this if generation looks corrupted)

Symptom: the synthetic JSONL contains entries like:

{"query": "We need to output a single question, at least 384 words long. Must be specific and self-contained. Only the question, no extra text. So we need a long question (384+ words). Must be a question that could be answered..."}

The LLM's chain-of-thought is leaking into the query text.

Cause: Nemotron Omni / Qwen-Reasoning / similar models, in reasoning mode, put their final answer in message.content and the deliberation in message.reasoning_content. With min_tokens near the model's reasoning budget, content can come back empty — the model exhausted the budget on CoT.

Why rag-perf used to leak this: an old version of _call_llm fell back to reasoning_content when content was empty. We removed that fallback — _call_llm now reads only message.content and raises if empty, with a clear hint:

LLM returned empty content (reasoning_content was populated — model exhausted its
budget on chain-of-thought; raise min_query_tokens or set
synthetic.disable_thinking=true).

Fix paths:

1. Default already correct: synthetic.disable_thinking: true injects chat_template_kwargs: {enable_thinking: false}. The model skips reasoning and writes the answer directly to content. 2. For non-reasoning endpoints: set disable_thinking: false to avoid sending the unsupported kwarg. 3. Last resort: raise min_query_tokens substantially so the model has budget for both reasoning and answer.

Failure recovery (partial JSONL)

If generation fails at query 47 of 100:

• Queries 1–46 (or however many had completed; order is completion-order, not request-order, since calls are concurrent) are on disk at synthetic.jsonl_output_path.
• The exception in stdout looks like: RuntimeError: Random synthetic query generation failed at query N: <root cause>.

Recovery options:

• Fix the LLM endpoint and re-run: the file is overwritten ("w" mode) — old partial is lost.
• Use the partial directly: swap input.synthetic for input.file: <jsonl_output_path> and the benchmark runs from whatever made it to disk.
• Lower `num_queries` so the new total stays under what you previously generated; combine with input.file pointing at the partial.

Prompt templates

Default templates (`prompts/default_prompts.yaml`) are deliberately strict to keep content clean: forbid markdown, numbering, "Question:" / "Here is" / "Sure," prefixes, planning/thinking text, restating instructions. They require exactly one ? at the end.

If swapping in custom prompts via synthetic.prompts_file, preserve the same output discipline or expect leaked planning text in the JSONL — the rag-perf side does only minimal cleanup (q.lstrip("0123456789.). ").strip() to drop leading numbering).

Variables interpolated into the templates:

• {word_target} — int(min_query_tokens * 0.75), lower-bound 10.
• {index} — 1-based query index (for "make this unique" hints).
• {ref} — reference question (dataset_based mode only).

Reproducibility

• synthetic.jsonl_output_path is the canonical artefact. Commit it to a known location and switch to input.file: <that path> for subsequent runs to keep the load identical while iterating on retrieval / reranker config.
• Generation is concurrent → completion order is non-deterministic. The dataset is reproducible across runs only if you pin and reuse the JSONL — not by re-running generation with the same config. (Even with the same seed, async scheduling is non-deterministic.)

Description: <br>

Performance benchmarking for a deployed NVIDIA RAG Blueprint server: profiling pass plus aiperf load test driven by a single YAML config. <br>

This skill is ready for commercial/non-commercial use. <br>

Owner

NVIDIA <br>

License/Terms of Use: <br>

Apache-2.0 <br>

Use Case: <br>

Developers and engineers use this skill to benchmark latency, throughput, and bottleneck characteristics of a deployed NVIDIA RAG Blueprint server under configurable load patterns. <br>

Deployment Geography for Use: <br>

Global <br>

Known Risks and Mitigations: <br>

Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. <br> Mitigation: Review and scan skill before deployment. <br>

Reference(s): <br>

• Config Schema Reference <br>
• Output Layout and Analysis <br>
• Synthetic Query Generation <br>
• Performance Benchmarking Documentation <br>

Skill Output: <br>

Output Type(s): [Shell commands, Analysis] <br> Output Format: [Markdown with inline bash code blocks] <br> Output Parameters: [1D] <br> Other Properties Related to Output: [None] <br>

Evaluation Tasks: <br>

Evaluated via NVSkills-Eval 3-Tier evaluation framework with external profile. <br>

Evaluation Metrics Used: <br>

Reported benchmark dimensions: <br>

• Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. <br>
• Correctness: Checks whether the agent follows the expected workflow and produces the correct final output. <br>
• Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant. <br>
• Effectiveness: Checks whether the agent performs measurably better with the skill than without it. <br>
• Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work. <br>

Skill Version(s): <br>

2.6.0 (source: frontmatter) <br>

Ethical Considerations: <br>

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>

(For Release on NVIDIA Platforms Only) <br> Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns here. <br>