Nemo Retriever

nemo-retriever

The retriever CLI indexes a folder of PDFs into LanceDB (retriever ingest) and serves vector search over it (retriever query). For any task about searching/answering questions across a folder of PDFs, use this CLI — do not write a custom RAG.

Beyond PDFs and beyond semantic search. retriever ingest also handles images, Office, HTML, TXT, audio, and video — see references/setup.md for the per-format recipe and references/install.md for the install extras ([multimedia], libreoffice, ffmpeg). For non-semantic operations — page filter, verbatim quote with citation, corpus-level aggregate, chart/image caption hits — see references/query.md. Don't fall back to native Read/Grep/Python on non-PDF inputs.

Install (if retriever is missing)

If command -v retriever returns nothing, follow references/install.md to install the NeMo Retriever Library before proceeding. It prints RETRIEVER_VENV=<path>; substitute that path for <RETRIEVER_VENV> in every example in this skill (setup, query, troubleshooting, and the CLI references).

Workflow — read the reference for the current phase, then execute

For the full retriever ingest / retriever query CLI specs, see references/cli/ingest.md and references/cli/query.md. You do not need these for routine turns — <RETRIEVER_VENV>/bin/retriever <subcommand> --help is faster.

Before ingesting a mixed folder, inventory extensions (find <dir> -name '*.*' | sed 's/.*\.//' | sort -u) — --input-type=auto silently drops anything outside the supported set. See references/troubleshooting.md "Unsupported file types".

Hard limits (apply to every turn)

• Setup turn: build the index in one shell command (see references/setup.md). STOP after the index lands.
• Query turn: at most 2 Bash calls — 1 retriever query, +1 optional targeted text-extract per references/query.md. Reply and then STOP.
• No narration between tool calls. Tokens you emit between calls become input + cached input for every later turn — quadratic cost. Go straight from reading the summary to writing the JSON file.
• Banned: TodoWrite, Glob, Grep, Read of whole PDFs, re-running setup, spawning subagents, speculative "confirmation" calls.

Long query turns (5+ tool calls, 1M+ cache-read tokens) cost ~5× a disciplined turn and almost always still produce the wrong answer. Answering partially beats timing out.

Anti-Patterns

• Indexing content before clarifying corpus boundaries, freshness, or ownership: Retrieval quality collapses when the source of truth is unstable.
• Treating embedding, chunking, and backend choices as invisible defaults: They change recall, latency, and storage cost in user-visible ways.
• Claiming grounded answers without checking the retrieved passages that supported them.

Verification Protocol

Before claiming "skill applied successfully":

1. Pass/fail: The workflow names the corpus, index or backend choice, and the query path before answering deployment or QA questions. 2. Pass/fail: Retrieval checks include at least one real query and inspection of the supporting passages or scores. 3. Pass/fail: Ingestion or indexing advice keeps corpus freshness and reindex cost visible instead of implicit. 4. Pressure-test scenario: Apply the workflow to a retriever that answers quickly but returns stale passages after a corpus update. 5. Success metric: The user gets a reproducible retriever setup or debugging path with live retrieval evidence.

<!-- PORTABILITY:START -->

Cross-Client Portability

This skill is written to stay usable across GitHub Copilot, Claude Code, Codex, and Gemini CLI.

• GitHub Copilot: keep the folder in a Copilot-visible skill or plugin path, or wrap the workflow as project instructions if the host does not support portable skill folders directly.
• Claude Code: keep the folder in a local skills directory or a compatible plugin or marketplace source.
• Codex: install or sync the folder into $CODEX_HOME/skills/<skill-name> and restart Codex after major changes.
• Gemini CLI: this repository generates a project command named /skills:nemo-retriever from this skill. Rebuild commands with python scripts/export-gemini-skill.py nemo-retriever and then run /commands reload inside Gemini CLI.

<!-- PORTABILITY:END -->

<!-- MCP:START -->

MCP Availability And Fallback

Preferred MCP Server: None required

• Fallback prompt: "Use the nemo-retriever skill without MCP. Rely on the local SKILL.md, bundled references or scripts, and manual verification. Show the exact commands, evidence, and final checks you used before concluding."
• If the current host does not expose a matching server, use the bundled references, scripts, native toolchain, and manual workflow already described in this skill.
• Treat direct local verification, rendered output, logs, tests, or screenshots as the fallback evidence path before completion.

<!-- MCP:END -->

Related Skills

• notebooklm-management: Use it when retrieval-backed research needs a notebook-style grounding workflow.
• development-workflow: Use it when the retriever work also needs scoped implementation and validation checkpoints.
• cloud-design-patterns: Use it when the retriever deployment choice also needs storage, scaling, or service-boundary analysis.

Evaluation Report

Evaluation of the nemo-retriever skill before publication through NVSkills-Eval.

This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.

Evaluation Summary

• Skill: nemo-retriever
• Evaluation date: 2026-05-29
• NVSkills-Eval profile: external
• Environment: local
• Dataset: 4 evaluation tasks
• Attempts per task: 2
• Pass threshold: 50%
• Overall verdict: PASS

Agents Used

• claude-code
• codex

Metrics Used

Reported benchmark dimensions:

• Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
• Correctness: checks whether the agent follows the expected workflow and produces the correct final output.
• Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
• Effectiveness: checks whether the agent performs measurably better with the skill than without it.
• Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.

Underlying evaluation signals used in this run:

• security (Security): checks for unsafe operations, secret leakage, and unauthorized access.
• skill_execution (Skill Execution): verifies that the agent loaded the expected skill and workflow.
• skill_efficiency (Efficiency): checks routing quality, decoy avoidance, and redundant tool usage.
• accuracy (Accuracy): grades final-answer correctness against the reference answer.
• goal_accuracy (Goal Accuracy): checks whether the overall user task completed successfully.
• behavior_check (Behavior Check): verifies expected behavior steps, including safety expectations.
• token_efficiency (Token Efficiency): compares token usage with and without the skill.

Test Tasks

The benchmark dataset contained 4 evaluation tasks:

• Positive tasks: 3 tasks where the skill was expected to activate.
• Negative tasks: 1 tasks where no skill was expected.
• Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred.

Task composition is derived from the evaluation dataset when possible. Entries with expected_skill set are treated as positive skill-activation cases, while entries with expected_skill: null are treated as negative activation cases.

Results

Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.

Tier 1: Static Validation Summary

Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 19 total findings.

Top findings:

• MEDIUM QUALITY/quality_correctness: No documented scripts in table format (skills/nemo-retriever/SKILL.md)
• MEDIUM QUALITY/quality_correctness: Instructions don't mention 'run_script' (skills/nemo-retriever/SKILL.md)
• MEDIUM QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.author' (skills/nemo-retriever/SKILL.md)
• MEDIUM QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.tags' (skills/nemo-retriever/SKILL.md)
• MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (skills/nemo-retriever/SKILL.md)

Tier 2: Deduplication Summary

Tier 2 validation passed. NVSkills-Eval ran 2 checks and found 0 total findings.

Notable observations:

• Context Deduplication: Collected 9 file(s)
• Inter-Skill Deduplication: Parsed skill 'nemo-retriever': 432 char description

Publication Recommendation

The skill is suitable to proceed toward NVSkills-Eval publication based on this benchmark. Skill owners should keep this file with the skill and refresh it when the evaluation dataset, skill behavior, or target agents materially change.

Changelog

All notable changes to the nemo-retriever skill will be documented in this file.

[2026-06-09] - Initial Import and Catalog Normalization

Added

• Imported nemo-retriever from https://github.com/NVIDIA/skills at skills/nemo-retriever pinned to 129a1087a1853f32a950e2f7bbc0fd7d57b9d422.
• Added repo-standard Anti-Patterns, Verification Protocol, portability, MCP fallback, and related-skills sections.
• Preserved the upstream benchmark, signature, skill card, and bundled references or scripts for provenance and later refreshes.

Changed

• Normalized SKILL.md frontmatter to the shared catalog schema with version: "1.2" and last_updated: 2026-06-09.
• Moved upstream-only top-level metadata into the nested metadata block so validation, export, and downstream sync stay consistent.

Fixed

• Aligned the imported skill with this repository's maintained-skill requirements and downstream sync workflow.

retriever ingest

End-to-end ingestion of supported documents and media into a LanceDB table — runs the full extract -> embed -> vector-DB flow in a single command.

If flags below look stale, re-check retriever ingest --help.

When to use this

• You have one or more supported files (or a directory/glob of files) and want them

searchable via retriever query.

• You want an auto-routed ingest: supported file families are detected from

the manifest, then routed through document/image/text/audio/video extraction branches before embedding and LanceDB insert.

Use a different command when:

• You only need a single stage (e.g. just extract text, no embeddings) →

retriever pdf, retriever chart, retriever image, etc.

• You need a long-running service rather than one-shot CLI → retriever service.
• You're benchmarking throughput → retriever benchmark.
• You're iterating on the pipeline locally and want a non-distributed runner →

retriever local.

Canonical invocations

Ingest a single file into the default table (lancedb/nv-ingest.lance):

<RETRIEVER_VENV>/bin/retriever ingest data/multimodal_test.pdf

Default PDF ingest:

<RETRIEVER_VENV>/bin/retriever ingest data/corpus/

Large text-only PDF fallback:

retriever ingest data/pdfs/ --profile fast-text

Optional local VLM captioning:

retriever ingest data/pdfs/ --caption \
  --caption-infographics

Add --caption-invoke-url only when a remote OpenAI-compatible VLM endpoint is already deployed.

Ingest a directory of supported files:

retriever ingest data/corpus/

Ingest via glob:

retriever ingest "data/**/*"

Write to a custom DB / table:

<RETRIEVER_VENV>/bin/retriever ingest data/multimodal_test.pdf \
  --lancedb-uri ./my-lancedb \
  --table-name my-corpus

Inputs

• Positional `DOCUMENTS...` — one or more file paths, directories, or

shell globs. Required, repeatable.

• Supported input types — pdf, doc (.docx, .pptx), txt, html,

image (.jpg, .jpeg, .png, .tiff, .tif, .bmp, .svg), audio (.mp3, .wav, .m4a), and video (.mp4, .mov, .mkv).

Outputs

• A LanceDB dataset at <lancedb-uri>/<table-name>.lance. Default:

./lancedb/nemo-retriever.lance.

• One row per extracted primitive (text chunk, table, chart, image region),

each with: text, source, page_number, metadata (JSON: type, bbox, …), and the embedding vector.

Key flags

Pipeline shape

The default ingest entrypoint expands inputs, builds a manifest, resolves the selected profile into normal params, and calls GraphIngestor.extract(...). The manifest planner routes PDF/document, image, text, HTML, audio, and video branches without relying on retriever pipeline.

For text, HTML, image, audio, video, or mixed auto inputs, ingest routes through the same GraphIngestor extraction paths used by retriever pipeline.

Common failure modes

• `Clamping num_partitions from 16 to 7` — informational, not an error.

LanceDB IVF index needs num_partitions < row_count; happens on very small ingests.

• First run is slow (~60s+ before any pages process) — vLLM model load and

CUDA-graph capture for the embedder. Subsequent runs in the same process are fast; one-shot CLI invocations always pay this cost.

• `No existing dataset at …/nemo-retriever.lance, it will be created` — expected

on the first ingest into a new DB. Subsequent ingests append.

• HuggingFace download on first run — the embedder and page-element

detector pull weights to ~/.cache/huggingface. Needs network the first time; cached afterwards.

Related

• [[query]] — search the table this command writes.
• retriever vector-store --help — utilities for inspecting/moving LanceDB

tables.

retriever query

Embed a text query and return the top-k nearest rows from a LanceDB table previously written by retriever ingest (or any compatible pipeline).

If flags below look stale, re-check retriever query --help.

When to use this

• You have already ingested documents and want to retrieve relevant

chunks/primitives for a natural-language query.

• You want a one-shot CLI lookup — no service, no UI.

Use a different command when:

• You want recall metrics over a labelled query set → retriever recall.
• You want to grade end-to-end QA quality → retriever eval.
• You want a long-running query endpoint → retriever service.
• You want to compare two retrieval runs → retriever compare.

Canonical invocations

Top-10 search against the default table:

<RETRIEVER_VENV>/bin/retriever query "what is in chart 1?"

Top-3, custom table:

<RETRIEVER_VENV>/bin/retriever query "average frequency ranges for tweeters" \
  --top-k 3 \
  --lancedb-uri ./my-lancedb \
  --table-name my-corpus

Inputs

• Positional `QUERY` — single text string. Required. Quote it in the shell

to keep multi-word queries intact.

Outputs

• JSON array on stdout, one object per hit, sorted by ascending _distance

(lower = more similar). Each hit includes:

• _distance — vector distance in the embedding space.
• text — the retrieved primitive's text content.
• source / path / source_id — origin document path.
• page_number, pdf_basename, pdf_page — locator.
• metadata — JSON string with type (text / table / chart / image)

and, where applicable, a normalised bbox_xyxy_norm.

Pipe through Python for filtering, e.g. only chart hits:

<RETRIEVER_VENV>/bin/retriever query "gadget costs" | <RETRIEVER_VENV>/bin/python -c 'import json,sys; hits=json.load(sys.stdin); print(json.dumps([h for h in hits if json.loads(h["metadata"]).get("type")=="chart"], indent=2))'

Key flags

Distance interpretation

• The embedder (llama-nemotron-embed-vl-1b-v2) returns mean-pooled vectors;

LanceDB returns L2 distance by default. Typical relevant hits are in the ~1.0–1.7 range for this model on prose queries; treat _distance as ranking-only, not a calibrated similarity score.

• The query uses the VL variant of the embedder so text queries can match

ingested image/chart embeddings as well as text. Expect mixed-modality hits in the result list.

Common failure modes

• Empty result array — table is empty (no ingest run yet) or

--table-name / --lancedb-uri don't match where ingest wrote.

• `Table 'nemo-retriever' was not found` — same root cause: wrong table/URI,

or ingest hasn't been run.

• First query is slow (~10–15s) — vLLM startup for the query embedder.

Subsequent queries in the same process are sub-second; one-shot CLI invocations always pay this cost.

• Surprisingly low-relevance top hit — for very short corpora, even

unrelated queries return something with a non-huge distance. Inspect _distance gaps between hits rather than absolute values.

Related

• [[ingest]] — populate the table this command reads.
• retriever recall --help — batch query → recall@k against ground truth.
• retriever eval --help — end-to-end QA evaluation.

Install NeMo Retriever Library

One-time bootstrap to make the retriever CLI available. Skip if command -v retriever already prints a path.

The recipe below detects the host capabilities and picks the right install:

• GPU present and CUDA 13.x → installs the local-GPU torch wheels from

the cu130 index plus the [local] extra, so the bundled nvidia/llama-nemotron-embed-1b-v2 embedder can run locally on GPU.

• No GPU, or a non-CUDA-13 driver → installs the package without

[local]. Torch is pulled from PyPI defaults; the local-GPU embedder is unavailable. Provide a remote NIM endpoint at query/ingest time via --embed-invoke-url (or set EMBED_INVOKE_URL).

When to use this

• You're in a fresh container or host and command -v retriever returns

nothing.

• You need to bump to a newer commit and want to reinstall from a fresh

source tree.

Recipe

# Use the current checkout if cwd is already the NeMo-Retriever repo; else
# clone to a shared cache. Override the cache path with NRL_SRC=... if needed.
if [ -f "pyproject.toml" ] && grep -q '^name = "nemo-retriever"' pyproject.toml; then
  NRL_PKG="$PWD"                              # already in nemo_retriever/
elif [ -f "nemo_retriever/pyproject.toml" ] && grep -q '^name = "nemo-retriever"' nemo_retriever/pyproject.toml; then
  NRL_PKG="$PWD/nemo_retriever"               # at repo root
else
  NRL_SRC="${NRL_SRC:-$HOME/.cache/nemo-retriever/source}"
  if [ ! -d "$NRL_SRC/.git" ]; then
    mkdir -p "$(dirname "$NRL_SRC")"
    git clone https://github.com/NVIDIA/NeMo-Retriever.git "$NRL_SRC"
  fi
  NRL_PKG="$NRL_SRC/nemo_retriever"
fi

# Detect GPU + CUDA 13 to choose the install flavor.
USE_LOCAL=0
if command -v nvidia-smi >/dev/null 2>&1 && nvidia-smi >/dev/null 2>&1; then
  CUDA_MAJOR=$(nvidia-smi | sed -n 's/.*CUDA Version: \([0-9]\+\)\..*/\1/p' | head -1)
  [ "$CUDA_MAJOR" = "13" ] && USE_LOCAL=1
fi
echo "use_local=$USE_LOCAL (cuda_major=${CUDA_MAJOR:-none})"

uv python install 3.12
uv venv retriever --python 3.12
VENV=$PWD/retriever
(
  cd "$NRL_PKG"
  EPOCH=$(date +%s)
  if [ "$USE_LOCAL" = "1" ]; then
    env SOURCE_DATE_EPOCH=$EPOCH uv pip install -q --python "$VENV/bin/python" "torch~=2.11.0" "torchvision>=0.26.0,<0.27" -i https://download.pytorch.org/whl/cu130
    env SOURCE_DATE_EPOCH=$EPOCH uv pip install -q --python "$VENV/bin/python" ".[local]"
  else
    env SOURCE_DATE_EPOCH=$EPOCH uv pip install -q --python "$VENV/bin/python" "."
  fi
)
echo "RETRIEVER_VENV=$VENV"   # record this absolute path — substitute it for <RETRIEVER_VENV> in every later example

Notes

• SOURCE_DATE_EPOCH is passed inline via env so uv forwards it to the

PEP-517 build subprocess; a bare export was being dropped and the resulting dev-suffix mismatch between wheel filename and metadata broke the install.

• -q keeps uv pip install silent on the happy path; errors and a

non-zero exit code still surface.

• The cache path defaults to $HOME/.cache/nemo-retriever/source so every

cwd you launch from shares one copy. The block intentionally does not git fetch on reuse, so installs are reproducible — run git -C ~/.cache/nemo-retriever/source pull manually to bump.

• Only add further extras ([nemotron-parse], [multimedia], [llm]) when

a later step actually demands one — append them inside the brackets, e.g. ".[local,multimedia]".

In the examples in SKILL.md and other reference docs, substitute <RETRIEVER_VENV> with the absolute path printed by the final echo (e.g. /workspace/retriever).

Optional extras (install only when the user's input demands it)

Stack extras with the base flavor, e.g. ".[local,multimedia]". Base install already covers PDF, image, HTML, TXT.

Query turn — the WHOLE workflow

<RETRIEVER_VENV>/bin/retriever query "<the user's question>" --top-k 10 --embed-model-name nvidia/llama-nemotron-embed-1b-v2 --rerank \
  | tee /tmp/hits.json \
  | <RETRIEVER_VENV>/bin/python -c "import json,sys; [print(f'rank={h.get(\"rank\",0)} page={h[\"page_number\"]} pdf={h[\"pdf_basename\"]} type={h.get(\"metadata\",{}).get(\"type\",\"?\")}') for h in json.load(sys.stdin)]"

Run that exactly as a single pipeline — do not split it into HITS=$(...) + echo "$HITS" | <RETRIEVER_VENV>/bin/python -c ... (the assignment swallows stdout, the pipe sees nothing, you waste 3 bash calls recovering). Stdout is clean JSON (model-init logs are silenced at the CLI layer); leave stderr unredirected so real errors surface on the first call. The summary above lists only rank/page/pdf/type — to read hit text for synthesizing final_answer, parse /tmp/hits.json directly. The top hit's text is one one-liner away: <RETRIEVER_VENV>/bin/python -c "import json; print(json.load(open('/tmp/hits.json'))[0]['text'])" (or [i] for the rank-(i+1) hit). Fetch only what you need — pulling all 10 hits' text into context inflates cached prompt size on every subsequent turn.

That's your FIRST tool call on every query turn. Do not Read, Glob, Grep, or list PDFs before this — those duplicate what retriever query already did.

No narration between tool calls. Do not write "Let me search…", "I'll now analyze…", "The retriever returned…", or any other commentary. Every assistant token you emit with the retriever query Bash call becomes input tokens (and cached input tokens) for every subsequent turn in this session — quadratic cost. Go straight from reading the summary to writing the JSON file. The only assistant text in a query turn should be the tool calls themselves.

Each hit has: text, pdf_basename, page_number (int, 1-indexed: the first page of a PDF is page 1), pdf_page (string composite key "<basename>_<page_number>" — not a number, don't use it as one), _distance, and metadata (JSON with type ∈ text|table|chart|image).

Keyword/regex search across the corpus

If you need exact text matches that semantic retriever query may have skipped — e.g. "find every mention of 'mRNA-1273' across all PDFs" — use:

<RETRIEVER_VENV>/bin/python <skill_dir>/scripts/grep_corpus.py "<regex>" [--max-hits 50]

It scans the LanceDB table the retriever already built — no PDF re-extraction. Output is <pdf>:p<page>:<type>: ...<snippet>... per hit; NO_MATCH if nothing. Counts against the same "one optional follow-up call" budget as the targeted text-extract (mutually exclusive — pick one).

Don't reach for pdftotext, pdftohtml, or pdfgrep — they're system tools that aren't guaranteed installed on the user's machine. The retriever venv bundles pdfium and lancedb; grep_corpus.py and retriever pdf stage page-elements --method pdfium cover the same use cases without that dependency.

Compose your reply from the hits

• final_answer: synthesize from the top hits' text. Include the exact number / name / date / row / column the question asks for, plus the source PDF and 0-indexed page. One paragraph. No restating the question, no hedging caveats. If the chunks talk around the fact but don't state it, run ONE <RETRIEVER_VENV>/bin/retriever pdf stage page-elements ./pdfs --method pdfium --json-output-dir /tmp/pdf_text --compact-json and Read /tmp/pdf_text/<top_pdf>.pdf.pdf_extraction.json for the rank-1 page (or rank-2 if rank-1 is metadata) — that almost always surfaces the exact figure. Then synthesize. If after both calls the asked-for fact still isn't in the evidence, write `final_answer` that says so explicitly — e.g. "The retrieved pages do not state [X] for [entity]; the closest content is [Y]." Do NOT invent, extrapolate, or generate plausible-sounding content from adjacent material. A confidently-wrong answer scores worse than an honest "not in the retrieved pages".
• ranked_retrieved: one entry per hit in the order retriever query returned: {"doc_id": "<pdf_basename without .pdf>", "page_number": <int>, "rank": <i+1>}. Up to 10. Duplicate (doc, page) is fine. Indexing: the retriever's page_number is 1-indexed. If the task's output schema says 0-indexed (e.g. "first page is page 0"), emit hit.page_number - 1; if the task says 1-indexed or doesn't specify, emit hit.page_number as-is.

Before writing `final_answer`, re-read the question. If it lists multiple entities, years, or categories, your answer must address each one explicitly — even if for some of them the chunks say "not provided" or contain no data. Missing entities lose more judge points than imprecise numbers.

Charts and images — the single biggest source of judge=2/3 trials

When metadata.type of a hit is chart or image, its text field is a model-generated transcription that frequently:

• reverses direction words (increase↔decrease, rose↔fell, surge↔drop), and
• rounds or misreads exact percentages (e.g. transcribing 12% as 20%).

If a question asks for an exact percentage or a directional claim and the evidence is only a chart/image hit (no text-type hit corroborates the same number or direction):

1. Run the targeted <RETRIEVER_VENV>/bin/retriever pdf stage page-elements --method pdfium text-extract on the rank-1 PDF (this counts as your second tool call) and look for the number in prose. 2. If prose confirms the chart number, assert it confidently. 3. If prose doesn't mention it, quote the chart transcription verbatim with an explicit hedge in `final_answer`: "The chart on page N indicates [verbatim phrase] (chart-derived, not verified against prose)." Do NOT restate the chart's number as a confident fact.

When both a chart hit and a text hit cover the same fact, always prefer the text hit's number. After your reply, STOP. No print, no summary, no further tool calls.

Non-semantic operations (use these, don't fall back to native tools)

Page filter — "what's on page N of doc.pdf" → filter LanceDB directly, no Read:

<RETRIEVER_VENV>/bin/python -c "import lancedb; t=lancedb.connect('./lancedb').open_table('nv-ingest'); df=t.to_pandas(); print('\n'.join(df[(df.pdf_basename=='APPLE_2022_10K.pdf')&(df.page_number==14)].text))"

Verbatim quote with `[page]` citation — quote retrieved chunks with [page N] markers in final_answer; don't paraphrase.

Corpus-level aggregate — "list distinct sources", "count chunks per source" → no ls/grep/find:

<RETRIEVER_VENV>/bin/python -c "import lancedb; df=lancedb.connect('./lancedb').open_table('nv-ingest').to_pandas(); print(sorted(df.pdf_basename.unique())); print(df.pdf_basename.value_counts().to_dict())"

Image / chart captioning — when the user asks to describe / caption an image (prose summary, not OCR text): retriever ingest already produces chart/image-type hits whose text field is the model-generated caption (see "Charts and images" above). Workflow: ingest the image folder (setup.md image recipe), then retriever query with a topic-related question — the hits with metadata.type=chart|image carry the caption in text. Use that as final_answer. No separate captioning CLI command.

Setup turn (when ./lancedb/nv-ingest.lance doesn't exist)

retriever ingest ./pdfs/ runs the full pipeline (text extraction + page-element detection + OCR + embedding + LanceDB insert). On corpora >~800 pages this often won't fit a typical setup turn budget (10 min) — the OCR + page-element stages dominate and scale roughly linearly with page count. Always build an index — pick the recipe by corpus size:

TOTAL_PAGES=$(<RETRIEVER_VENV>/bin/python -c "import pypdfium2, glob; print(sum(len(pypdfium2.PdfDocument(p)) for p in glob.glob('./pdfs/*.pdf')))" 2>/dev/null || echo 0)
echo "total_pages=$TOTAL_PAGES"
if [ "$TOTAL_PAGES" -le 800 ]; then
  <RETRIEVER_VENV>/bin/retriever ingest ./pdfs/ --embed-model-name nvidia/llama-nemotron-embed-1b-v2
else
  <RETRIEVER_VENV>/bin/retriever pipeline run ./pdfs/ --run-mode inprocess --method pdfium --no-extract-tables --no-extract-charts --no-extract-page-as-image --evaluation-mode none --embed-model-name nvidia/llama-nemotron-embed-1b-v2 --quiet
fi

retriever ingest is quiet by default; the else (retriever pipeline run) branch needs --quiet passed explicitly. Quiet mode suppresses progress bars, HuggingFace download logs, vLLM init noise, Ray worker stdout, and INFO-level pipeline status lines on success, while still flushing captured output to stderr on error. Without it the pipeline run branch burns thousands of tokens on irrelevant progress output. On success you only see one line: Ingested N document(s) into LanceDB lancedb/nv-ingest. (for retriever ingest) or Pipeline complete: N page(s) → lancedb lancedb/nv-ingest (T.Ts). (for retriever pipeline run).

The else branch skips page-element detection, OCR, table extraction, and chart extraction — only pdfium text extraction + embedding. Embedding runs locally via the bundled HuggingFace model by default (no remote NIM needed). It's strictly better to have a text-only index than no index at all: the per-query pdfium text-extract fallback re-extracts a full PDF per query, which is both slow and expensive. Page-element detection may emit warning logs when its remote endpoint isn't reachable; the warnings are non-fatal as long as the embedding step itself succeeds (and are silenced by --quiet on a successful run).

Don't pre-OCR, don't pre-chunk, don't write Python wrappers — the CLI handles extraction + (optionally) page-element detection + OCR + embedding + LanceDB insert in one shot.

After the setup command returns successfully, STOP. Don't run smoke queries to "warm up" — the first query turn does that naturally.

Other input shapes

Same retriever ingest command, different --input-type and (for non-PDF) install extras. Install extras live in references/install.md "Optional extras".

Images / scanned forms / charts (.jpg .png .tiff .bmp):

<RETRIEVER_VENV>/bin/retriever ingest ./images/ --input-type image --ocr-version v2 --ocr-lang english

For mixed-script docs (bilingual contracts, multilingual forms) use --ocr-lang multi. Chart understanding (axis/legend/data) runs inline — no separate call.

HTML / TXT — ingest even though Read could work; the chunking + citation matters:

<RETRIEVER_VENV>/bin/retriever ingest ./docs/

Office (.docx .pptx) — requires libreoffice (host package, not pip):

<RETRIEVER_VENV>/bin/retriever ingest ./office/ --input-type doc

Audio / video — requires the [multimedia] extra and ffmpeg (host pkg). Both audio and video go through the same extra:

<RETRIEVER_VENV>/bin/retriever ingest ./media/ --input-type audio   # or --input-type video

Audio is .mp3 / .wav / .m4a only — .flac is silently filtered. Inventory first.

Troubleshooting and recovery

Read this only after you hit one of the named errors below. Don't read it pre-emptively.

If the index is missing or retriever query returns []

Means ingest didn't complete (e.g. the text-only pipeline still hit the turn wall, or the table is empty). Tight fallback using the retriever's own pdfium-based extractor (always available — same binary the agent just used for retriever query):

1. ls ./pdfs/ (one call) to see filenames. 2. Pick the SINGLE PDF whose name best matches the question. 3. ONE call: <RETRIEVER_VENV>/bin/retriever pdf stage page-elements ./pdfs --method pdfium --json-output-dir /tmp/pdf_text --compact-json. This emits a JSON sidecar per PDF at /tmp/pdf_text/<basename>.pdf.pdf_extraction.json containing per-page text primitives — pdfium only, no OCR, no NIM, fast. 4. Read /tmp/pdf_text/<name>.pdf.pdf_extraction.json for the chosen PDF and synthesize from the per-page text. If the answer isn't there, still write your best guess based on the filename + extracted pages plus a one-sentence acknowledgement of uncertainty in final_answer. Then stop.

Do NOT keep doing text-extract calls across many PDFs to hunt — that exhausts the turn budget. Better to answer partially than to time out. Never re-run retriever ingest.

For an unlisted subcommand: <RETRIEVER_VENV>/bin/retriever <subcommand> --help.

Failure modes (expected, not errors)

• First `ingest` takes ~60s+ — vLLM warmup. Expected.
• First `query` takes ~10–15s — embedder cold-start. Expected.
• Empty result — ingest didn't run. Use the fallback above.
• `Clamping num_partitions ...` — informational on tiny corpora, not an error.
• Low-relevance top hit on tiny corpus — look at _distance gaps between hits, not absolute values.
• Page-element-detection warnings during ingest — non-fatal as long as the embedding step itself succeeds (and they're silenced on a successful run, since ingest is quiet by default).

Unsupported file types (silent filter — the v2 regression mode)

retriever ingest --input-type=auto silently drops .flac, .rtf, .eml, .py, .jsonl, .zip, etc. The "Ingested N documents" line uses the count of supported files — N may be lower than the folder count with no error. Before ingest, inventory:

find <dir> -type f -name '*.*' | sed 's/.*\.//' | sort -u

If unsupported extensions appear, name them in your reply and ask the user whether to skip or convert. Don't let the count silently drop.

You ran more than 2 Bash calls on a query turn

Budget violation. Stop, write final_answer from what you have, end the turn. Long turns cost ~5× a disciplined turn and usually still produce the wrong answer.

Query-turn cost discipline (recap)

• ONE retriever query per turn. ONE optional targeted text-extract on the rank-1 PDF if the chunks miss the asked-for fact. That's the budget — it is a hard cap, not a soft preference.
• After your 2nd tool call, write final_answer with what you have and STOP. If both calls left the asked-for fact unresolved, write final_answer that explicitly states the retrieved pages don't contain the requested fact (naming the closest related content if any) — do not run more tool calls hunting for it, and do not extrapolate a plausible value.
• Don't read whole PDFs.
• Don't make speculative Read/Glob/Grep calls "to confirm". The retriever already found the relevant pages — trust the ranking.
• Don't spawn agents, write plans, or make todo lists. The workflow is the workflow.

Description: <br>

Use when the user wants to search, query, extract, transcribe, describe, quote, filter, or aggregate across documents — PDFs, scanned forms / images (.jpg .png .tiff), Office (.docx .pptx), text (.html .txt), audio (.mp3 .wav .m4a), or video (.mp4 .mov). <br>

This skill is ready for commercial/non-commercial use. <br>

Owner

NVIDIA <br>

License/Terms of Use: <br>

Apache 2.0 <br>

Use Case: <br>

Developers and engineers who need to search, query, extract, or aggregate information across multimodal document collections including PDFs, images, Office files, audio, and video for retrieval-augmented generation workflows. <br>

Deployment Geography for Use: <br>

Global <br>

Known Risks and Mitigations: <br>

Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. <br> Mitigation: Review and scan skill before deployment. <br>

Reference(s): <br>

• Install Guide <br>
• Setup Guide <br>
• Query Guide <br>
• Troubleshooting <br>
• CLI: ingest <br>
• CLI: query <br>
• NeMo Retriever Library Documentation <br>

Skill Output: <br>

Output Type(s): [Shell commands, JSON] <br> Output Format: [Markdown with inline bash code blocks and JSON query results] <br> Output Parameters: [1D] <br> Other Properties Related to Output: [None] <br>

Evaluation Agents Used: <br>

• Claude Code (claude-code) <br>
• Codex (codex) <br>

Evaluation Tasks: <br>

Evaluated against 4 evaluation tasks (3 positive skill-activation, 1 negative), 2 attempts per task, 50% pass threshold. Overall verdict: PASS. <br>

Evaluation Metrics Used: <br>

Reported benchmark dimensions: <br>

• Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. <br>
• Correctness: Checks whether the agent follows the expected workflow and produces the correct final output. <br>
• Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant. <br>
• Effectiveness: Checks whether the agent performs measurably better with the skill than without it. <br>
• Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work. <br>

Underlying evaluation signals used in this run: <br>

• security: Checks for unsafe operations, secret leakage, and unauthorized access. <br>
• skill_execution: Verifies that the agent loaded the expected skill and workflow. <br>
• skill_efficiency: Checks routing quality, decoy avoidance, and redundant tool usage. <br>
• accuracy: Grades final-answer correctness against the reference answer. <br>
• goal_accuracy: Checks whether the overall user task completed successfully. <br>
• behavior_check: Verifies expected behavior steps, including safety expectations. <br>
• token_efficiency: Compares token usage with and without the skill. <br>

Evaluation Results: <br>

Skill Version(s): <br>

b331d0f7 (source: git SHA, committed 2026-05-29) <br>

Ethical Considerations: <br>

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>

(For Release on NVIDIA Platforms Only) <br> Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns here. <br>