Rag Eval

On-disk RAG evaluation (corpus/ + train.json)

Purpose

Guide agents through NVIDIA RAG Blueprint filesystem benchmarks: preparing corpus/ and train.json, running scripts/eval/evaluate_rag.py, tuning retrieval and generation flags for quality comparisons, interpreting RAGAS JSON outputs, and triaging failures (HTTP/stream errors, empty contexts, collection mismatch, judge API).

For latency, throughput, and load testing, use the rag-perf skill (scripts/rag-perf, docs/performance-benchmarking.md) — not this skill.

When not to use

Do not use this skill for: deploying or repairing services (use rag-blueprint); evaluating APIs without the corpus/ + train.json layout; general ML experimentation unrelated to this evaluator; production monitoring/alerting; or latency/throughput benchmarking (use rag-perf).

Prerequisites

• Repo cloned; run commands from repo root (imports and paths assume this).
• Python 3.11+ and uv; eval deps: uv sync --project scripts/eval.
• Reachable RAG server and ingestor (defaults often localhost:8081 / 8082).
• `NVIDIA_API_KEY` for RAGAS (see credential hygiene); optional `RAG_EVAL_JUDGE_MODEL`.
• Dataset roots passed to --dataset-paths each contain `corpus/` and `train.json`.

Instructions

1. Prepare data — Ensure each dataset directory matches the layout and train.json rules in `references/dataset-and-conversion.md`. When sources arrive as public links (sites or dataset pages), materialize documents under corpus/—prefer PDF for multimodal content so images stay embedded; convert CSV/JSONL/etc. using the patterns there. 2. Run eval — uv run --project scripts/eval python scripts/eval/evaluate_rag.py with --dataset-paths, --host, and --port. See `references/benchmark-execution.md` for command examples, outputs, and errors. Use `references/evaluate-rag-cli.md` for flag-level detail. 3. Tune quality — Adjust --top_k / --vdb_top_k, reranker and query-rewriting toggles, and generation overrides (--temperature, --top-p, --max-tokens) as documented in `references/benchmark-execution.md` when comparing retrieval/generation configs for RAGAS scores. 4. Analyze results — Use `references/result-analysis.md` for scripts; scan rag_*_evaluation_summary.json for headline RAGAS metrics. 5. Triage errors — Use the error signal table and the Troubleshooting section below.

Examples

Set API key without putting secrets in shell history (preferred patterns): load from a gitignored env file or secrets manager; avoid committing .env; rotate keys if exposed. Details: `references/benchmark-execution.md#credential-hygiene-nvidia_api_key`.

Minimal eval (key already in environment):

uv sync --project scripts/eval
uv run --project scripts/eval python scripts/eval/evaluate_rag.py \
  --dataset-paths /path/to/my_dataset \
  --host localhost \
  --port 8081

Pretty-print summary JSON:

python3 -m json.tool results/my_dataset/rag_my_dataset_evaluation_summary.json

More examples (skip ingestion, quality sweeps): `references/benchmark-execution.md`.

Limitations

• Evaluator behavior is fixed to the filesystem contract and evaluate_rag.py; it does not substitute for custom offline judges or non-RAG benchmarks.
• Vector DB / embedding choices follow deployed ingestor and RAG env — not overridden by this CLI alone.
• Scores depend on retrieval quality, judge model availability, and NVIDIA_API_KEY; empty contexts yield partial RAGAS metrics (see references).
• Large procedural detail lives under `references/` to keep routing concise; read those files when the user needs step-by-step conversion, full flags, or error tables.

Troubleshooting

Full signal table: `references/benchmark-execution.md#common-error-cases-and-signals`.

Gotchas

• Run from repo root: paths and imports in scripts/eval/evaluate_rag.py assume this; a wrong directory silently breaks imports.
• `--ingestor_server_url`: pass http://host:port without /v1—the code appends /v1/ automatically. Including /v1 causes 404s on ingestor calls.
• Vector DB / embedding settings: not set by this CLI; configure via the deployed ingestor and RAG server env vars (e.g. APP_VECTORSTORE_URL, embedding model).
• `--model` / `--llm_endpoint`: forwarded verbatim only when explicitly set; omit to keep the server's configured LLM.
• Stale collections: a previous run's ingested data persists unless you use --force_ingestion. Use --collection with a unique name when comparing quality across isolated runs.
• Empty context metrics: if all generated_contexts are empty, RAGAS scores only nv_accuracy and leaves the other two metrics blank—this is not a silent success.

Source of truth

Agent playbook

1. Run eval — uv sync --project scripts/eval then uv run --project scripts/eval python scripts/eval/evaluate_rag.py with required --dataset-paths, --host, and --port (and env NVIDIA_API_KEY). Argument --ingestor_server_url is optional (defaults to http://localhost:8082); pass it only when overriding the ingestor endpoint. 2. Quality tuning — See `references/benchmark-execution.md`: --top_k/--vdb_top_k, reranker and query-rewriting toggles, --temperature, --top-p, --max-tokens. 3. Data conversion — Follow `references/dataset-and-conversion.md`. 4. Analyze results — `references/result-analysis.md`; quick scan: python3 -m json.tool results/<dataset>/rag_<dataset>_evaluation_summary.json. 5. Error triage — `references/benchmark-execution.md#common-error-cases-and-signals`.

Anti-Patterns

• Changing the eval dataset while comparing runs: It destroys the baseline and makes improvements meaningless.
• Confusing latency smoke tests with answer-quality evaluation: Fast responses can still be wrong or ungrounded.
• Claiming gains without showing the baseline, scorer, and prompt or config deltas that changed the outcome.

Verification Protocol

Before claiming "skill applied successfully":

1. Pass/fail: The evaluation plan names the dataset, scorer, and baseline run before comparing variants. 2. Pass/fail: Retrieval and generation quality are separated so failures are attributed to the correct stage. 3. Pass/fail: Reported improvements include reproducible commands, configs, or artifacts that another maintainer can rerun. 4. Pressure-test scenario: Re-evaluate a RAG change where latency improves but groundedness falls on the held-out set. 5. Success metric: Quality claims survive a rerun on the same eval slice with no hidden configuration drift.

<!-- PORTABILITY:START -->

Cross-Client Portability

This skill is written to stay usable across GitHub Copilot, Claude Code, Codex, and Gemini CLI.

• GitHub Copilot: keep the folder in a Copilot-visible skill or plugin path, or wrap the workflow as project instructions if the host does not support portable skill folders directly.
• Claude Code: keep the folder in a local skills directory or a compatible plugin or marketplace source.
• Codex: install or sync the folder into $CODEX_HOME/skills/<skill-name> and restart Codex after major changes.
• Gemini CLI: this repository generates a project command named /skills:rag-eval from this skill. Rebuild commands with python scripts/export-gemini-skill.py rag-eval and then run /commands reload inside Gemini CLI.

<!-- PORTABILITY:END -->

<!-- MCP:START -->

MCP Availability And Fallback

Preferred MCP Server: None required

• Fallback prompt: "Use the rag-eval skill without MCP. Rely on the local SKILL.md, bundled references or scripts, and manual verification. Show the exact commands, evidence, and final checks you used before concluding."
• If the current host does not expose a matching server, use the bundled references, scripts, native toolchain, and manual workflow already described in this skill.
• Treat direct local verification, rendered output, logs, tests, or screenshots as the fallback evidence path before completion.

<!-- MCP:END -->

Related Skills

• development-workflow: Use it when the eval work needs a scoped implementation plan with explicit quality gates.
• documentation-verification: Use it when the output is an evaluation report or benchmark note that must stay source-backed.
• cloud-design-patterns: Use it when evaluation results drive bigger architecture changes in the RAG stack.

Evaluation Report

Evaluation of the rag-eval skill before publication through NVSkills-Eval.

This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.

Evaluation Summary

• Skill: rag-eval
• Evaluation date: 2026-05-29
• NVSkills-Eval profile: external
• Overall verdict: FAIL
• Tier 3 live agent evaluation: not available in this report

Agents Used

• Tier 3 agent details were not available in this report.

Metrics Used

Reported benchmark dimensions:

• Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
• Correctness: checks whether the agent follows the expected workflow and produces the correct final output.
• Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
• Effectiveness: checks whether the agent performs measurably better with the skill than without it.
• Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.

Underlying evaluation signals used in this run:

• No Tier 3 evaluation signal details were available in this report.

Test Tasks

Tier 3 evaluation task details were not available in this report.

Results

Tier 3 dimension rollup was not available in this report.

Tier 1: Static Validation Summary

Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 3 total findings.

Top findings:

• MEDIUM QUALITY/quality_efficiency: Deeply nested references in benchmark-execution.md (skills/rag-eval/SKILL.md)
• LOW SCHEMA/unexpected_file: Unexpected 'BENCHMARK.md' in skill root (skills/rag-eval/BENCHMARK.md)
• LOW SCHEMA/unexpected_file: Unexpected 'eval' in skill root (skills/rag-eval/eval)

Tier 2: Deduplication Summary

Tier 2 validation reported findings. NVSkills-Eval ran 2 checks and found 2 total findings.

Top findings:

• HIGH DUPLICATE/duplicate: Duplicate content found across references/benchmark-execution.md and references/evaluate-rag-cli.md:

"### Toggle pipeline stages" in references/benchmark-execution.md (lines 94-104) vs "### Pipeline stage toggles" in references/evaluate-rag-cli.md (lines 37-47) (references/benchmark-execution.md:94)

• HIGH DUPLICATE/duplicate: Duplicate content found within references/result-analysis.md:

"## Per-query table with worst-accuracy rows" in references/result-analysis.md (lines 7-34) vs "## Markdown table of worst queries" in references/result-analysis.md (lines 58-75) (references/result-analysis.md:7)

Publication Recommendation

The skill should be reviewed before NVSkills-Eval publication. Skill owners should address the findings above and rerun NVSkills-Eval to refresh this benchmark.

Changelog

All notable changes to the rag-eval skill will be documented in this file.

[2026-06-09] - Initial Import and Catalog Normalization

Added

• Imported rag-eval from https://github.com/NVIDIA/skills at skills/rag-eval pinned to 129a1087a1853f32a950e2f7bbc0fd7d57b9d422.
• Added repo-standard Anti-Patterns, Verification Protocol, portability, MCP fallback, and related-skills sections.
• Preserved the upstream benchmark, signature, skill card, and bundled references or scripts for provenance and later refreshes.

Changed

• Normalized SKILL.md frontmatter to the shared catalog schema with version: "1.2" and last_updated: 2026-06-09.
• Moved upstream-only top-level metadata into the nested metadata block so validation, export, and downstream sync stay consistent.

Fixed

• Aligned the imported skill with this repository's maintained-skill requirements and downstream sync workflow.

Benchmark runs, outputs, and error signals

Load this for full command examples, artifact descriptions, quality interpretation, retrieval/generation flags, and the error-signal table.

For latency, throughput, and load testing, use the rag-perf skill — not this document.

Credential hygiene (NVIDIA_API_KEY)

• Prefer a secrets manager or a sourced env file that is not committed; ensure .env and key files are in .gitignore.
• Shell history may record export ... lines — avoid pasting real keys on the command line; rotate the key if it was exposed.
• Do not hardcode API keys in scripts or commit them to version control.

After the key is available in the environment, run commands from the repo root.

Output artifacts

Under --output_dir (default results), each dataset gets a subdirectory named after the dataset directory basename. Files share the same <label> (the dataset folder name):

Analysis tips: If evaluation_data has fewer rows than train.json, some queries failed (exceptions print during the run). After drops, use id / query_id to align rows rather than positional index. For "worst questions," pair index i in evaluation_results score lists with the ith object in evaluation_data.

Interpreting RAGAS quality metrics

• `nv_accuracy` — answer accuracy (LLM judge vs ground-truth answer).
• `nv_context_relevance` and `nv_response_groundedness` — scored when retrieved contexts exist.
• If no non-empty generated_contexts are present across the run, the code scores answer accuracy only—do not treat empty context metrics as a silent success.

Running the benchmark

Set NVIDIA_API_KEY (see credential hygiene above). Optionally set RAG_EVAL_JUDGE_MODEL for the RAGAS judge LLM id. Then from repo root:

Minimal full-run example

uv run --project scripts/eval python scripts/eval/evaluate_rag.py \
  --dataset-paths /path/to/my_dataset \
  --host localhost \
  --port 8081 \
  --ingestor_server_url http://localhost:8082 \
  --output_dir results

(NVIDIA_API_KEY must already be exported or injected by your environment.)

Skip ingestion (collection already populated)

uv run --project scripts/eval python scripts/eval/evaluate_rag.py \
  --dataset-paths /path/to/my_dataset \
  --host localhost \
  --port 8081 \
  --ingestor_server_url http://localhost:8082 \
  --skip_ingestion

Ingestion only (no RAGAS scoring)

uv run --project scripts/eval python scripts/eval/evaluate_rag.py \
  --dataset-paths /path/to/my_dataset \
  --host localhost \
  --port 8081 \
  --ingestor_server_url http://localhost:8082 \
  --skip_evaluation

Force re-ingest (delete existing collection first)

uv run --project scripts/eval python scripts/eval/evaluate_rag.py \
  --dataset-paths /path/to/my_dataset \
  --host localhost --port 8081 \
  --ingestor_server_url http://localhost:8082 \
  --force_ingestion

Retrieval and generation options (quality comparisons)

Use these flags when comparing pipeline configs for RAGAS scores. Omit any flag to leave the RAG server default.

Retrieval depth

--top_k 5          # sent as reranker_top_k to the generate endpoint
--vdb_top_k 20     # vector DB candidate pool size

Toggle pipeline stages

--enable-reranker          # send enable_reranker=true on /v1/generate
--disable-reranker         # send enable_reranker=false
--enable-query-rewriting   # send enable_query_rewriting=true
--disable-query-rewriting  # send enable_query_rewriting=false

Omitting these flags does not send the field—the RAG server uses its own configured default. --enable-reranker and --disable-reranker are mutually exclusive; same for the query-rewriting pair.

Generation parameters

--temperature 0.0    # deterministic output for repeatable benchmarks
--top-p 0.95
--max-tokens 512     # cap answer length

These are forwarded verbatim to /v1/generate; omit to use the server default.

Example: quality comparison across configs

uv run --project scripts/eval python scripts/eval/evaluate_rag.py \
  --dataset-paths /path/to/my_dataset \
  --host localhost --port 8081 \
  --ingestor_server_url http://localhost:8082 \
  --skip_ingestion \
  --disable-reranker \
  --disable-query-rewriting \
  --temperature 0.0 \
  --max-tokens 512 \
  --output_dir results/baseline_no_rerank

Use a distinct --collection or --force_ingestion when you need an isolated corpus for each config.

Result analysis

For ready-to-run Python scripts, read `result-analysis.md`. It contains: per-query worst-accuracy table, CSV export, and markdown report table.

Quick headline scan:

python3 -m json.tool results/my_dataset/rag_my_dataset_evaluation_summary.json

Rows with has_context=N and low nv_accuracy signal retrieval problems (ingestion gap or collection mismatch), not generation problems.

Common error cases and signals

Pre-flight checklist

1. Each dataset root: corpus/ + train.json (corpus/ preferably PDF, including sources where the upstream link does not name a file explicitly). 2. train.json: top-level array of objects (dict-shaped root is rejected). Run the quick validation in `dataset-and-conversion.md` after any conversion. 3. Rows include question and answer for meaningful RAGAS scores. 4. NVIDIA_API_KEY available before invoking the script (optional RAG_EVAL_JUDGE_MODEL if not using the default judge). 5. For config comparisons: use a distinct --collection or --force_ingestion / --skip_ingestion so each run sees the intended corpus state.

Dataset layout, train.json, and conversion

Load this when shaping corpus/ + train.json or converting external benchmarks.

Dataset layout

Each --dataset-paths entry is a directory containing:

1. corpus/ — files indexed recursively for ingestion. 2. train.json — evaluation questions and answers.

train.json schema

The driver accepts a top-level JSON array of objects only. Required per row: question, answer. Optional: id or query_id.

Field rules:

• id: integer from the source row index. Do not use prefixed strings (e.g. "dataset-0").
• is_impossible: include as a boolean if the source dataset carries it; use false for benchmarks that have no unanswerable questions.
• contexts: optional array of objects — one entry per supporting document. `filename` (required on each object) is the file’s basename under corpus/ exactly as on disk (including any percent-encoding in the name). `text` is optional: include it when you have a ground-truth span; omit it when you only need to tie the row to corpus files by name (for example multimodal PDFs where no span was curated).
• Omit benchmark-internal metadata fields (reasoning category labels, source tags, etc.) that are not question, answer, id, is_impossible, or contexts.

[
  {
    "id": 0,
    "question": "...",
    "answer": "...",
    "is_impossible": false,
    "contexts": [
      { "filename": "Article_Title" },
      { "filename": "Another%20Article", "text": "…" }
    ]
  }
]

Multiple context entries per row are allowed. Plain strings (["...", "..."]) remain acceptable for minimal bundles without per-file tagging.

Quick validation

python3 -c "import json,sys; d=json.load(open(sys.argv[1])); assert isinstance(d, list) and all(isinstance(x, dict) for x in d), 'train.json must be a list of objects'" train.json

Run this after any conversion step to catch shape errors before the eval.

Corpus format when converting external benchmarks

Prefer putting sources in corpus/ as PDF. That matches typical production RAG on documents, aligns with the evaluator default --file-type pdf, and unlocks PDF page counts in ingestion metrics.

Materializing corpus/ from public links (datasets, sites, and mirrors)

Eval requires a real `corpus/` tree on disk. When the only inputs are public links—dataset landing pages, file listings, paper or supplement URLs, or arbitrary websites—download or render into `corpus/` as documents the ingestor can index, do not point the eval at URLs alone.

For multimodal material (figures, tables, charts, photos, diagrams, or screenshots that carry meaning), standardize on PDF as the file format under corpus/ whenever practical so images and layout stay inside the same artifact the retriever will chunk and embed. Goals:

• Preserve visuals: Use the publisher’s official PDF download or export when it exists. Avoid workflows that rebuild PDFs from plain text only (for example simple text-to-PDF libraries): those often drop graphics and produce a corpus that no longer matches multimodal retrieval expectations.
• Web-only pages: Prefer full-fidelity print paths (browser print-to-PDF, or headless Chromium / Playwright rendering) so embedded and inline images survive in the PDF. HTML or .txt alone usually discard or isolate visuals from the indexed blob you need side-by-side with questions.
• One logical source → one primary file: Keep a stable basename under corpus/ and reference that same basename in train.json contexts[].filename (see below). If a source truly splits into separate image files plus text, still align names with how citations and ingestion expose document_name.

After materializing files, pass `--file-type` to evaluate_rag.py according to what sits under corpus/ (for example keep the default when the corpus is mostly PDF).

Image-heavy web articles: When upstream pages mix text and images, still prefer a PDF export or faithful render over generating PDFs with text-only toolkits. If an API offers binary PDF download, use it before HTML-to-text shortcuts.

If the upstream artifact only gives URLs or document pointers that do not name a concrete file (common in published benchmarks), assume PDF as the target format. Use plain text or HTML only when converting to PDF is impractical; then set --file-type to match what dominates under corpus/.

Each contexts object’s `filename` must match the actual corpus file basename (same as the file’s name in corpus/, e.g. Report_2023 for corpus/Report_2023 or corpus/subdir/Report_2023). `text`, when present, should be the reference span or excerpt; when omitted, only the filename association is carried through.

Deriving corpus filenames from URLs

When the benchmark provides a URL per source document, derive the corpus filename and contexts[].filename using this rule — it preserves the source URL's identity exactly and ensures downstream citation matching works:

stem = path_last_segment + "#" + fragment   (if URL has a fragment)
stem = path_last_segment                     (if no fragment)

The file you write under corpus/ must start with stem and follow the same naming pattern as the rest of that dataset so --file-type and document_name from ingestion stay consistent.

Where:

• path_last_segment = last /-separated component of urllib.parse.urlparse(url).path.
• Do not call `urllib.parse.unquote()` on the segment — keep percent-encoding exactly as it appears in the URL.
• fragment = urllib.parse.urlparse(url).fragment — include verbatim if non-empty.
• Do not pass the segment through any slug or sanitize function that strips or replaces characters (%, ', ., #, -, non-ASCII bytes, etc.). Any such transformation breaks alignment between the corpus file, the train.json context reference, and the ingestor's document_name.

If the content must be fetched via an API that requires a decoded title (e.g. a REST endpoint that does not accept percent-encoded paths), decode only for that API call: urllib.parse.unquote(path_last_segment). The on-disk filename stays encoded.

Bringing external data into this layout

Benchmarks packaged elsewhere (CSV, JSONL, parquet, archives, APIs, annotation exports, etc.) are not consumed directly. Convert them so each eval root has corpus/ documents and a train.json that follows the schema. Keep corpus/ filenames consistent with how the ingestor and citations surface document_name so retrieval and scoring align.

Conversion checklist:

1. Normalize source encodings to UTF-8. 2. train.json: top-level array of objects, each with at minimum question and answer. 3. id: integer from the source row index — not a prefixed or composite string. 4. is_impossible: carry over from the source if present; add as false if the benchmark has no unanswerable questions. 5. Corpus filenames: if derived from URLs, use the stem rule above (raw path last segment + #fragment if any, no decoding, no sanitization). 6. contexts entries: filename must equal the corpus file basename; text is optional (add when you have a gold span). 7. Drop any benchmark-internal fields that are not part of the schema (question, answer, id, is_impossible, contexts). 8. Run the quick train.json validation above after any conversion.

Conversion patterns

JSONL → train.json

import json, pathlib

rows = [json.loads(l) for l in pathlib.Path("source.jsonl").read_text().splitlines() if l.strip()]
train = [{"id": r.get("id"), "question": r["question"], "answer": r["answer"]} for r in rows]
pathlib.Path("my_dataset/train.json").write_text(json.dumps(train, indent=2, ensure_ascii=False))

CSV → train.json

import csv, json, pathlib

with open("source.csv", newline="", encoding="utf-8") as f:
    rows = list(csv.DictReader(f))

train = [{"question": r["question"], "answer": r["answer"]} for r in rows]
pathlib.Path("my_dataset/train.json").write_text(json.dumps(train, indent=2, ensure_ascii=False))

Map source column names to question / answer as needed. Add "id" from the source if available to aid per-query traceability.

evaluate_rag.py CLI flag reference

Complete argument tables for scripts/eval/evaluate_rag.py. Load this when the user asks about a specific flag, its default value, or fixed evaluator behavior not covered in the main skill.

For latency, throughput, and load testing, use the rag-perf skill — not the --thread / --timeout knobs here (they exist on the CLI for operational reliability only).

Arguments

Required

Dataset and ingestion

Retrieval

Pipeline stage toggles

Omitting either pair entirely does not send the field — the RAG server uses its own configured default.

Generation overrides

Output and run control

Fixed behavior (not CLI flags)

• The evaluator does not send vdb_endpoint, embedding dimension, or related overrides to the ingestor or /v1/generate; services use their configured defaults (environment / server config).
• Ingestion uploads always use blocking: true for a synchronous ingestor response.
• The client does not send split_options on document upload; chunk size and overlap are controlled by the ingestor server configuration.
• RAG queries use POST /v1/generate with a single user turn per benchmark row; enable_filter_generator is not sent (server default applies).
• RAG_EVAL_JUDGE_MODEL env var sets the RAGAS judge model id (ChatNVIDIA); defaults to mistralai/mixtral-8x22b-instruct-v0.1 when unset or empty.

Result analysis scripts

Ready-to-run Python patterns for analyzing evaluate_rag.py RAGAS outputs. Load when the user wants per-row queries, worst-accuracy tables, or CSV export.

All paths assume default --output_dir results; substitute your actual dataset basename for my_dataset.

Per-query table with worst-accuracy rows

import json

data   = json.load(open("results/my_dataset/rag_my_dataset_evaluation_data.json"))
scores = json.load(open("results/my_dataset/rag_my_dataset_evaluation_results.json"))

rows = []
for i, (d, acc) in enumerate(zip(data, scores.get("nv_accuracy", []))):
    rows.append({
        "i": i,
        "id": d.get("id"),
        "question": d["question"][:80],
        "nv_accuracy": acc,
        "has_context": bool(d.get("generated_contexts")),
        "answer_len": len(d.get("generated_answer", "")),
    })

rows.sort(key=lambda r: r["nv_accuracy"])
print(f"{'i':>3}  {'acc':>5}  {'ctx':>3}  question")
print("-" * 70)
for r in rows[:10]:
    print(f"{r['i']:>3}  {r['nv_accuracy']:>5.2f}  {'Y' if r['has_context'] else 'N':>3}  {r['question']}")

has_context=N with low nv_accuracy → retrieval problem (ingestion gap or collection mismatch), not generation.

Export to CSV

import csv, json

data   = json.load(open("results/my_dataset/rag_my_dataset_evaluation_data.json"))
scores = json.load(open("results/my_dataset/rag_my_dataset_evaluation_results.json"))

acc  = scores.get("nv_accuracy",  [None]*len(data))
ctxr = scores.get("nv_context_relevance", [None]*len(data))
grd  = scores.get("nv_response_groundedness", [None]*len(data))

with open("eval_out.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=["id","question","answer","generated_answer",
                                       "nv_accuracy","nv_context_relevance","nv_response_groundedness"])
    w.writeheader()
    for i, d in enumerate(data):
        w.writerow({"id": d.get("id",""), "question": d["question"],
                    "answer": d["answer"], "generated_answer": d.get("generated_answer",""),
                    "nv_accuracy": acc[i], "nv_context_relevance": ctxr[i],
                    "nv_response_groundedness": grd[i]})

Markdown table of worst queries

Paste into a PR description or evaluation report:

import json

data   = json.load(open("results/my_dataset/rag_my_dataset_evaluation_data.json"))
scores = json.load(open("results/my_dataset/rag_my_dataset_evaluation_results.json"))

pairs = sorted(zip(scores.get("nv_accuracy", []), data), key=lambda x: x[0])
print("| id | acc | question | generated_answer |")
print("|----|-----|----------|-----------------|")
for acc, d in pairs[:5]:
    q = d["question"][:60].replace("|", "\\|")
    a = d.get("generated_answer", "")[:80].replace("|", "\\|")
    print(f"| {d.get('id','')} | {acc:.2f} | {q} | {a} |")

Description: <br>

Filesystem RAG benchmarks: corpus/, train.json, evaluate_rag.py (RAGAS quality). <br>

This skill is ready for commercial/non-commercial use. <br>

Owner

NVIDIA <br>

License/Terms of Use: <br>

Apache 2.0 <br>

Use Case: <br>

Developers and engineers use this skill to run filesystem-based RAGAS quality benchmarks against NVIDIA RAG Blueprint deployments, evaluating retrieval and generation quality through dataset preparation, evaluation execution, and result analysis. <br>

Deployment Geography for Use: <br>

Global <br>

Known Risks and Mitigations: <br>

Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. <br> Mitigation: Review and scan skill before deployment. <br>

Reference(s): <br>

• Benchmark Execution <br>
• Dataset and Conversion <br>
• Evaluate RAG CLI <br>
• Result Analysis <br>
• NVIDIA RAG Blueprint <br>

Skill Output: <br>

Output Type(s): [Shell commands, Analysis] <br> Output Format: [Markdown with inline bash code blocks] <br> Output Parameters: [1D] <br> Other Properties Related to Output: [None] <br>

Evaluation Metrics Used: <br>

Reported benchmark dimensions: <br>

• Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. <br>
• Correctness: Checks whether the agent follows the expected workflow and produces the correct final output. <br>
• Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant. <br>
• Effectiveness: Checks whether the agent performs measurably better with the skill than without it. <br>
• Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work. <br>

Skill Version(s): <br>

2.6.0 (source: frontmatter) <br>

Ethical Considerations: <br>

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>

(For Release on NVIDIA Platforms Only) <br> Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns here. <br>