Rag Blueprint

NVIDIA RAG Blueprint

Purpose

Use this skill for NVIDIA RAG Blueprint operations: deployment, configuration, troubleshooting, shutdown, and feature management across Docker, Helm, and library deployments.

Instructions

1. Match the user request to the intent routing table below. 2. Read the referenced playbook before making changes. 3. Use repository docs and deployment config files as the source of truth. 4. Verify the affected service or workflow after changes.

Prerequisites

• NVIDIA RAG Blueprint repository checkout.
• Docker/Compose or Kubernetes/Helm for deployments.
• Python 3.11+ for library workflows.
• NVIDIA GPU tooling for self-hosted NIM services.

Autonomy Principles

• Auto-detect everything: GPU, VRAM, drivers, Docker, CUDA, disk, OS, ports, existing services, NGC key, repo state.
• If it can be checked with a command, check it — don't ask the user.
• Ask only when user action is required: providing an API key, confirming data deletion, or choosing between equally valid options.
• Once analysis is done, route to the correct workflow and execute.

Intent Detection

Determine what the user wants and route immediately:

If the intent is ambiguous, infer from context (e.g., "RAG isn't working" → troubleshoot; "get RAG running" → deploy). Only ask if genuinely unclear.

---

Configure

Requires a running RAG deployment. If services are not running, deploy first via references/deploy.md.

Match the user's request to a reference file, then read and follow it:

Configure Flow

1. Match the user's request to a reference file from the table above.

2. Detect what's running:

   echo "=== NIM ===" && docker ps --format '{{.Names}}' 2>/dev/null | grep -iE '(nim-llm|nemotron-(vlm-)?embedding|nemotron-ranking|nemotron-vlm|nemotron-3-nano-omni|page-elements|graphic-elements|table-structure|nemotron-ocr)' || echo "NO_LOCAL_NIMS"; echo "=== RAG ===" && docker ps --format '{{.Names}}' 2>/dev/null | grep -iE '(rag-server|ingestor-server|elasticsearch|milvus|seaweedfs|lancedb)' || echo "NO_DOCKER_RAG"; echo "=== K8S ===" && kubectl get pods -n rag 2>/dev/null | head -5 || echo "NO_K8S"; echo "=== LIBRARY ===" && ps aux 2>/dev/null | grep -E '(nvidia_rag|uvicorn.*rag)' | grep -v grep || echo "NO_LIBRARY"

3. Use this table to determine platform, deployment type, and where config lives:

Tell the user what you detected and ask to confirm. Example: "I see local NIM containers running (nim-llm-ms, nemotron-vlm-embedding-ms) — this is a self-hosted deployment. Config file is deploy/compose/.env. Correct?"

4. Check current feature state before changing anything — read the config location from step 3, then cross-check the live service:

• Docker: docker exec rag-server env 2>/dev/null | grep -E "<VAR_NAME>"
• Helm: kubectl get pod -n rag -l app=rag-server -o jsonpath='{.items[0].spec.containers[0].env}' 2>/dev/null

If the config file and live service disagree, tell the user the service has stale config and will need a restart.

5. If the feature needs extra GPUs, check availability against hardware restrictions (see below):

   nvidia-smi --query-gpu=index,name,memory.total,memory.used --format=csv,noheader 2>/dev/null || echo "NO_GPU"

6. Read the reference file and apply changes:

• Docker: edit the env file (uncomment to enable, re-comment to disable — the env file is the source of truth). Then restart the affected service:

     source <env-file> && docker compose -f deploy/compose/<compose-file> up -d

• Helm: edit values.yaml, then upgrade: helm upgrade rag <chart> -n rag -f values.yaml
• Library: edit notebooks/config.yaml, then restart the Python process

7. Verify:

• Docker: docker ps --format "table {{.Names}}\t{{.Status}}" | head -20; curl -s http://localhost:8081/v1/health?check_dependencies=true 2>/dev/null | head -1
• Helm: kubectl get pods -n rag; kubectl rollout status deployment/rag-server -n rag --timeout=120s
• Library: curl -s http://localhost:8081/v1/health 2>/dev/null | head -1

8. If restart fails, read references/troubleshoot.md. If multiple features requested, repeat from step 1 for each.

Examples

• "Deploy RAG" -> route to references/deploy.md.
• "Enable VLM" -> route to references/configure/vlm.md.
• "RAG is unhealthy" -> route to references/troubleshoot.md.
• "Stop RAG" -> route to references/shutdown.md.

Limitations

• Operational guidance only applies to this RAG Blueprint repository.
• Live deployment changes require a running Docker, Helm, or library target.
• Secrets such as NGC_API_KEY must be supplied by the user environment.

Troubleshooting

When User Says "Configure" Without Specifics

Run steps 2–3 above, then read the identified config file to list what's currently enabled:

grep -E "^(export )?(ENABLE_|APP_)" <config-file> 2>/dev/null | sort

Summarize what's running and enabled, then ask which feature to change.

---

Hardware Restrictions

Read docs/support-matrix.md for current GPU requirements per deployment mode. Read docs/service-port-gpu-reference.md for port mappings and GPU assignments.

Anti-Patterns

• Changing deployment knobs before identifying the active deployment mode: Compose, Helm, and library paths are not interchangeable.
• Treating retrieval, model, and infrastructure faults as the same class of problem: It wastes time and can hide the real failing layer.
• Stopping or tearing down services without checking persistence impact: Cleanup can destroy the exact evidence needed for recovery.

Verification Protocol

Before claiming "skill applied successfully":

1. Pass/fail: The workflow identifies the active deployment path and uses the matching upstream playbook before proposing changes. 2. Pass/fail: Any configuration change is tied to the exact file, chart value, or environment variable that owns the behavior. 3. Pass/fail: Health checks, logs, or a real retrieval request are used before claiming the stack is healthy again. 4. Pressure-test scenario: Apply the workflow to a half-running deployment where ingestion works but retrieval answers are empty. 5. Success metric: The requested RAG feature or service state is reproducible, observable, and verified with a live check path.

<!-- PORTABILITY:START -->

Cross-Client Portability

This skill is written to stay usable across GitHub Copilot, Claude Code, Codex, and Gemini CLI.

• GitHub Copilot: keep the folder in a Copilot-visible skill or plugin path, or wrap the workflow as project instructions if the host does not support portable skill folders directly.
• Claude Code: keep the folder in a local skills directory or a compatible plugin or marketplace source.
• Codex: install or sync the folder into $CODEX_HOME/skills/<skill-name> and restart Codex after major changes.
• Gemini CLI: this repository generates a project command named /skills:rag-blueprint from this skill. Rebuild commands with python scripts/export-gemini-skill.py rag-blueprint and then run /commands reload inside Gemini CLI.

<!-- PORTABILITY:END -->

<!-- MCP:START -->

MCP Availability And Fallback

Preferred MCP Server: None required

• Fallback prompt: "Use the rag-blueprint skill without MCP. Rely on the local SKILL.md, bundled references or scripts, and manual verification. Show the exact commands, evidence, and final checks you used before concluding."
• If the current host does not expose a matching server, use the bundled references, scripts, native toolchain, and manual workflow already described in this skill.
• Treat direct local verification, rendered output, logs, tests, or screenshots as the fallback evidence path before completion.

<!-- MCP:END -->

Related Skills

• cloud-design-patterns: Use it when the RAG deployment decision also needs broader distributed-system tradeoff analysis.
• devops-tooling: Use it when the work also needs repo, CI, or infrastructure automation steps.
• notebooklm-management: Use it when the user also needs retrieval-oriented research workflows outside the deployment stack.

Evaluation Report

Evaluation of the rag-blueprint skill before publication through NVSkills-Eval.

This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.

Evaluation Summary

• Skill: rag-blueprint
• Evaluation date: 2026-05-29
• NVSkills-Eval profile: external
• Overall verdict: FAIL
• Tier 3 live agent evaluation: not available in this report

Agents Used

• Tier 3 agent details were not available in this report.

Metrics Used

Reported benchmark dimensions:

• Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
• Correctness: checks whether the agent follows the expected workflow and produces the correct final output.
• Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
• Effectiveness: checks whether the agent performs measurably better with the skill than without it.
• Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.

Underlying evaluation signals used in this run:

• No Tier 3 evaluation signal details were available in this report.

Test Tasks

Tier 3 evaluation task details were not available in this report.

Results

Tier 3 dimension rollup was not available in this report.

Tier 1: Static Validation Summary

Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 5 total findings.

Top findings:

• LOW QUALITY/quality_discoverability: Description very long (368 chars, recommend 50-150) (skills/rag-blueprint/SKILL.md)
• LOW QUALITY/quality_discoverability: Description doesn't mention WHEN to use this skill (skills/rag-blueprint/SKILL.md)
• LOW QUALITY/quality_discoverability: Broad description without negative triggers may cause over-triggering (skills/rag-blueprint/SKILL.md)
• LOW SCHEMA/unexpected_file: Unexpected 'BENCHMARK.md' in skill root (skills/rag-blueprint/BENCHMARK.md)
• LOW SCHEMA/unexpected_file: Unexpected 'eval' in skill root (skills/rag-blueprint/eval)

Tier 2: Deduplication Summary

Tier 2 validation reported findings. NVSkills-Eval ran 2 checks and found 6 total findings.

Top findings:

• HIGH DUPLICATE/duplicate: Duplicate content found across references/configure/query-and-conversation.md and references/configure/reasoning-and-generation.md:

"## Process" in references/configure/query-and-conversation.md (lines 23-28) vs "## Process" in references/configure/reasoning-and-generation.md (lines 6-12) (references/configure/query-and-conversation.md:23)

• HIGH DUPLICATE/duplicate: Duplicate content found across references/configure/notebooks.md and references/deploy.md:

"### Deployment" in references/configure/notebooks.md (lines 47-51) vs "## Notebooks" in references/deploy.md (lines 114-116) (references/configure/notebooks.md:47)

• HIGH DUPLICATE/duplicate: Duplicate content found across references/configure/multimodal-query.md and references/configure/vlm.md:

"## When to Use" in references/configure/multimodal-query.md (lines 3-7) vs "## Notebooks" in references/configure/multimodal-query.md (lines 31-33) vs "## When to Use" in references/configure/vlm.md (lines 3-5) vs "## Notebooks" in references/configure/vlm.md (lines 51-53) (references/configure/multimodal-query.md:3)

• HIGH DUPLICATE/duplicate: Duplicate content found across references/deploy/library-full.md and references/deploy/library-lite.md and references/deploy/library.md:

"## Source Documentation" in references/deploy/library-full.md (lines 42-43) vs "## Source Documentation" in references/deploy/library-lite.md (lines 36-37) vs "## Source Documentation" in references/deploy/library.md (lines 53-54) (references/deploy/library-full.md:42)

• HIGH DUPLICATE/duplicate: Duplicate content found across references/configure/models-and-infrastructure.md and references/deploy.md and references/deploy/docker.md and references/deploy/library.md:

"### API Keys" in references/configure/models-and-infrastructure.md (lines 31-35) vs "## Verify NGC_API_KEY" in references/deploy/docker.md (lines 22-33) vs "## Verify NGC_API_KEY" in references/deploy/library.md (lines 18-27) vs "## Phase 2: NGC_API_KEY Handling" in references/deploy.md (lines 39-48) (references/configure/models-and-infrastructure.md:31)

Publication Recommendation

The skill should be reviewed before NVSkills-Eval publication. Skill owners should address the findings above and rerun NVSkills-Eval to refresh this benchmark.

Changelog

All notable changes to the rag-blueprint skill will be documented in this file.

[2026-06-09] - Initial Import and Catalog Normalization

Added

• Imported rag-blueprint from https://github.com/NVIDIA/skills at skills/rag-blueprint pinned to 129a1087a1853f32a950e2f7bbc0fd7d57b9d422.
• Added repo-standard Anti-Patterns, Verification Protocol, portability, MCP fallback, and related-skills sections.
• Preserved the upstream benchmark, signature, skill card, and bundled references or scripts for provenance and later refreshes.

Changed

• Normalized SKILL.md frontmatter to the shared catalog schema with version: "1.2" and last_updated: 2026-06-09.
• Moved upstream-only top-level metadata into the nested metadata block so validation, export, and downstream sync stay consistent.

Fixed

• Aligned the imported skill with this repository's maintained-skill requirements and downstream sync workflow.

Agentic RAG

When to Use

• User wants the LangGraph agentic pipeline/agentic rag, planning/execution, multi-hop reasoning, ambiguity handling, or verification.
• User asks about agentic, ENABLE_AGENTIC_RAG, agentic streaming, stage events, or agentic reasoning traces.

Restrictions

• Requires use_knowledge_base=true; otherwise the agentic path is not applied.
• Higher latency and more LLM calls than standard RAG. Prefer per-request enablement for latency-sensitive deployments.
• The agentic path does not use NeMo Guardrails, Self-Reflection, Query Decomposition, or VLM Inference.
• Verification is single-pass.

Process

1. Detect deployment mode. Docker: edit the active env file. Helm: edit values.yaml. Library/API callers can set request fields directly. 2. Read docs/agentic-rag.md for the current architecture, env vars, and limitations. 3. Prefer per-request enablement:

   {
     "messages": [{"role": "user", "content": "..."}],
     "use_knowledge_base": true,
     "collection_names": ["..."],
     "agentic": true
   }

4. For API/library clients that omit agentic, set ENABLE_AGENTIC_RAG=true and restart the RAG server. In the React UI, also select Pipeline → Agentic because the UI sends an explicit per-request value. 5. Optionally configure LLMs:

• One deployment-wide LLM: set APP_LLM_MODELNAME, APP_LLM_SERVERURL, and APP_LLM_APIKEY; Docker Compose chains each agentic role to these defaults.
• Role-specific LLMs: set AGENTIC_PLANNER_LLM_*, AGENTIC_TASK_LLM_*, AGENTIC_SEED_GEN_LLM_*, or AGENTIC_SYNTHESIS_LLM_*.
• One request only: pass model and/or llm_endpoint in /v1/generate; the runtime override applies to all agentic roles for that request.

6. Verify with /v1/generate: streaming agentic chunks include event_type, stage, and supplementary reasoning_content; final answer text still streams through content.

Decision Table

Agent-Specific Notes

• enable_streaming=true is the default. Agentic streaming emits stage events (stage_start, stage_end), intermediate reasoning/output, final answer chunks, agent events, and errors.
• enable_streaming=false makes the agent graph finish before returning a full answer chunk; standard RAG always streams.
• The React UI has only Standard and Agentic modes. Standard sends agentic: false, so ENABLE_AGENTIC_RAG=true alone does not override UI Standard mode.
• In the UI, agentic and standard reasoning traces render in the reasoning panel when the stream includes reasoning_content.
• Docker Compose chains AGENTIC_*_LLM_MODEL, AGENTIC_*_LLM_SERVERURL, and AGENTIC_*_LLM_APIKEY through APP_LLM_MODELNAME, APP_LLM_SERVERURL, and APP_LLM_APIKEY, so one standard LLM override propagates to all four agentic roles unless a role-specific value is set.
• Helm values list the role-specific envs explicitly. Keep them aligned with the main LLM values for one shared agentic model, or set per-role values when the planner, task, seed generation, or synthesis roles need different models.
• If a role model is empty in config, the builder falls back to the planner LLM, then the main RAG LLM. API keys fall back through the role config, main RAG LLM config, and the usual NVIDIA-hosted defaults.
• Per-request /v1/generate model and llm_endpoint values override every agentic role for that request; omit the fields to use deployment or role-specific configuration.
• If the result is slow or expensive, use per-request agentic instead of a global default, lower AGENTIC_CONTEXT_MAX_TOKENS, or leave verification disabled.

Source Documentation

• docs/agentic-rag.md — architecture, API usage, env vars, limitations
• docs/api-rag.md — /v1/generate request and streaming behavior
• deploy/compose/docker-compose-rag-server.yaml — Docker APP_LLM_* and AGENTIC_*_LLM_* fallback chain
• src/nvidia_rag/rag_server/agentic_rag/builder.py — role LLM fallback order and runtime override model
• frontend/src/hooks/useMessageSubmit.ts — UI request field behavior for agentic
• frontend/src/hooks/useChatStream.ts and frontend/src/components/chat/ReasoningPanel.tsx — reasoning trace rendering

API Reference

When to Use

• User needs to call RAG or Ingestor APIs directly
• User asks about endpoints, request/response formats, or task status tracking

Process

1. Read docs/api-rag.md for RAG server endpoints (port 8081) 2. Read docs/api-ingestor.md for Ingestor server endpoints (port 8082) 3. Consult OpenAPI schemas for exact request/response shapes

Agent-Specific Notes

• RAG Server runs on port 8081: /v1/generate, /v1/search, /v1/health, /v1/configuration, /v1/metrics, /v1/summary
• Ingestor Server runs on port 8082: /v1/documents, /v1/collection, /v1/collections, /v1/status
• POST /v1/documents returns a task_id — poll GET /v1/status?task_id=<id> for progress
• Task states: PENDING → FINISHED or FAILED (also UNKNOWN if not found)
• NV-Ingest extraction states: not_started → submitted → processing → completed or failed
• Max file size: 400 MB per document
• Full health check: GET /v1/health?check_dependencies=true
• Streaming /v1/generate chunks may include supplementary reasoning_content. Agentic RAG streaming chunks also include event_type and stage; final user-facing answer text remains in content.

Notebooks

• notebooks/ingestion_api_usage.ipynb — ingestion API usage examples
• notebooks/retriever_api_usage.ipynb — RAG retriever API: search and query examples

Source Documentation

• docs/api-rag.md -- RAG server API details
• docs/api-ingestor.md -- Ingestor server API details
• docs/api_reference/openapi_schema_rag_server.json -- RAG server OpenAPI schema
• docs/api_reference/openapi_schema_ingestor_server.json -- Ingestor server OpenAPI schema

Data Catalog

When to Use

• User wants to manage collection or document metadata for governance
• User asks about tagging, ownership, or lifecycle status of collections
• User wants to list or update collection metadata

Restrictions

• None — available automatically after deployment, no additional configuration needed
• Works with both Milvus and Elasticsearch (full feature parity)

Process

1. Read docs/data-catalog.md for full API reference, field definitions, and examples 2. All endpoints are on the ingestor server (port 8082) 3. Use PATCH endpoints for updates (merge updates — only provided fields change)

Decision Table

Agent-Specific Notes

• Auto-populated metrics (number_of_files, last_indexed, has_tables, etc.) are system-set — not user-editable
• date_created and last_updated timestamps are automatic
• PATCH is a merge update — omitted fields keep current values
• Different from custom metadata: catalog = governance/discovery, custom metadata = retrieval filtering

Notebooks

• notebooks/ingestion_api_usage.ipynb — ingestion and collection management examples

Source Documentation

• docs/data-catalog.md — full API reference, catalog fields, auto-populated metrics, Python client examples

Evaluation

When to Use

• The user wants to measure RAG pipeline quality.

• User asks about accuracy, relevancy, groundedness, or recall metrics.

• The user wants to run the filesystem benchmark evaluator (scripts/eval/evaluate_rag.py) with corpus/ plus train.json.

Process

1. Read docs/evaluate.md for full evaluation methodology and setup. 2. Choose the path:

• Notebooks — interactive RAGAS workflows against a running stack.
• CLI benchmark — on-disk datasets and evaluate_rag.py; follow skill rag-eval (skills/rag-eval/SKILL.md), scripts/eval/README.md, and the skill’s references/ for conversion, flags, runs, and result parsing.

3. Run evaluation against the deployed RAG pipeline.

When building a CLI eval bundle from a public benchmark, materialize corpus/ as PDF when you can (multimodal content keeps images embedded; matches default --file-type pdf). If the source only provides web links or no file extension, default to PDF rather than plain text. Details: rag-eval → `references/dataset-and-conversion.md` and scripts/eval/README.md.

Agent-Specific Notes

• Uses RAGAS framework for all metrics
• Answer Accuracy, Context Relevancy, and Groundedness are covered in one notebook
• Recall is measured separately at top-k cutoffs (1, 3, 5, 10)
• evaluate_rag.py ingests corpus/, queries /v1/generate, then runs RAGAS NVIDIA metrics (ragas.metrics); requires NVIDIA_API_KEY. Install CLI deps with uv sync --project scripts/eval (declared under scripts/eval/).

Notebooks

CLI benchmark (repo)

Source Documentation

• docs/evaluate.md — full evaluation guide and metric definitions
• RAGAS documentation
• NVIDIA RAGAS metrics

NeMo Guardrails

When to Use

• User wants content safety, topic control, or jailbreak prevention
• User asks to enable/disable guardrails

Restrictions

• Not available on B200 GPUs
• Requires 2 extra GPUs with 48GB+ each (H100, A100 SXM 80GB, or RTX PRO 6000)
• Not supported in library mode or Helm deployments
• Jailbreak detection model not yet available out-of-the-box

Process

1. Detect the deployment mode (guardrails are Docker-only — not supported on Helm or library mode). Edit the active env file for Docker 2. Read docs/nemo-guardrails.md for full setup and configuration 3. Choose deployment mode: self-hosted (local NIMs) or cloud-hosted (NVIDIA API) 4. For self-hosted: assign GPU IDs — read docs/service-port-gpu-reference.md for default GPU assignments and adjust for your system 5. Verify all three services healthy: nemo-guardrails-microservice, content-safety NIM, topic-control NIM 6. Enable in UI: Settings > Output Preferences > Guardrails toggle

Agent-Specific Notes

• Cloud mode (nemoguard_cloud config) skips local NIM containers — only the microservice is needed
• Per-request toggle via enable_guardrails in /generate body requires server-level ENABLE_GUARDRAILS=true first
• Override guardrails URL with NEMO_GUARDRAILS_URL if running on a different host
• Content-safety and topic-control models are trained on single-turn data — multi-turn conversations may get inconsistent safety classifications
• Current guardrails only produce simple refusal responses ("I'm sorry. I can't respond to that.")

Source Documentation

• docs/nemo-guardrails.md -- full setup, configuration, and customization of guardrail rules

Ingestion: Text-Only, Audio, Nemotron Parse, OCR & Batch

When to Use

User wants to configure ingestion mode (text-only, audio, Nemotron Parse), switch OCR engines, save extraction results to disk, use standalone NV-Ingest, tune ingestion performance, or run batch ingestion.

Restrictions

• Nemotron Parse: not available on B200 or RTX PRO 6000 GPUs (requires H100 or A100 SXM 80GB)
• Audio on Helm: not supported on RTX PRO 6000
• Nemotron Parse GPU conflict: read docs/service-port-gpu-reference.md for default GPU assignments. Nemotron Parse defaults to the same GPU as LLM — reassign on limited-GPU systems

Process

1. Detect the deployment mode (Docker self-hosted / NVIDIA-hosted / Helm / Library). Docker: edit the active env file. Helm: edit values.yaml. Library: edit notebooks/config.yaml 2. Read the relevant source doc for detailed configuration 3. Apply the required env vars to the active config, restart ingestor (and NIM services if enabling new profiles) 4. Verify: upload a test document and check ingestion status

Decision Table

Agent-Specific Notes

• Text-only mode: set COMPONENTS_TO_READY_CHECK="" in the active env file so NV-Ingest does not wait for disabled extraction services. If the compose file hardcodes COMPONENTS_TO_READY_CHECK=ALL, update it to ${COMPONENTS_TO_READY_CHECK:-ALL} so the env var takes effect
• Use --profile rag with nims.yaml to skip OCR/detection NIMs in text-only mode
• Audio formats supported: .mp3, .wav, .mp4, .avi, .mov, .mkv
• Riva ASR requires ~8GB VRAM
• Nemotron OCR is 2x+ faster than Paddle OCR but needs about 8GB vs 3GB VRAM
• Batch CLI: pip install -r scripts/requirements.txt first; idempotent (skips already-ingested files)
• MIG deployments: reduce batch sizes for large bulk ingestion jobs

Notebooks

• notebooks/ingestion_api_usage.ipynb — Ingestor API: collections, uploads, document management

Source Documentation

• docs/text_only_ingest.md — Text-only ingestion (skip OCR/detection)
• docs/audio_ingestion.md — Audio/video ingestion via ASR
• docs/nemotron-parse-extraction.md — Nemotron Parse PDF extraction
• docs/nemoretriever-ocr.md — Nemotron OCR configuration and switching
• docs/mount-ingestor-volume.md — Volume mount for extraction results
• docs/nv-ingest-standalone.md — Standalone NV-Ingest without ingestor server
• docs/accuracy_perf.md — Ingestion tuning settings (chunk size, overlap, batch params)
• docs/service-port-gpu-reference.md — OCR port mappings and GPU assignments

MCP Server & Client

When to Use

• User wants to expose RAG APIs as MCP tools for agentic workflows
• User asks about MCP transport modes, NeMo Agent Toolkit integration, or ReAct agents

Process

1. Read docs/mcp.md for full MCP server/client setup and configuration 2. Choose transport mode: sse, streamable_http, or stdio 3. Run MCP server from examples/nvidia_rag_mcp/mcp_server.py 4. For agentic RAG, see ReAct agent example in examples/rag_react_agent/

Agent-Specific Notes

• MCP wraps both RAG tools (generate, search, get_summary) and Ingestor tools (create_collection, upload_documents, etc.) via FastMCP
• stdio transport does not require a running server — client spawns it directly
• ReAct agent requires: Python 3.11+, NVIDIA_API_KEY, and data already ingested into Milvus
• Configure Milvus endpoint in examples/rag_react_agent/src/rag_react_agent/configs/config.yml or via APP_VECTORSTORE_URL

Notebooks

Source Documentation

• docs/mcp.md -- full MCP server/client documentation and transport configuration

Migration Guide

When to Use

• User is upgrading between RAG Blueprint versions
• User encounters breaking API changes or deprecated endpoints after an update

Process

1. Read docs/migration_guide.md for full version-by-version migration details 2. Identify the user's current and target versions 3. Apply changes sequentially for each version gap

Agent-Specific Notes

v2.2.0 → v2.3.0

• New confidence_threshold field in /generate and /search (0.0–1.0, default 0.0)
• New summary_options parameter with page_filter, shallow_summary, summarization_strategy
• SUMMARY_LLM_MAX_CHUNK_LENGTH and SUMMARY_CHUNK_OVERLAP changed from character-based to token-based — divide old values by ~4

v2.1.0 → v2.2.0

• Added generate_summary to /documents, new GET /summary endpoint
• POST /collection (singular) replaces POST /collections for single collection creation
• collection_names: List[str] replaces collection_name: str in /generate and /search

v2.0.0 → v2.1.0

• POST /documents gained blocking: bool (default True); use false + GET /status for async

v1.0.0 → v2.0.0 (Breaking)

• Single server split into RAG Server (port 8081) and Ingestion Server (port 8082)
• Collections must be explicitly created before uploading documents
• Default changed from cloud-hosted to on-prem models

Source Documentation

• docs/migration_guide.md — Full migration guide with examples and env var changes
• docs/release-notes.md — Release notes and version history
• docs/query-to-answer-pipeline.md — Query-to-answer pipeline architecture overview

Models, Vector DB & Service API Keys

When to Use

User wants to change LLM, embedding, or ranking models; switch vector DB (Elasticsearch/Milvus); configure Elasticsearch or Milvus auth, GPU mode, or custom endpoints; set service-specific API keys; or build a custom VDB operator.

Process

Detect the deployment mode before making changes. Docker: edit the active env file. Helm: edit values.yaml under nimOperator and envVars sections. Library: edit notebooks/config.yaml.

Change Models (LLM, Embedding, Ranking)

1. Read docs/change-model.md for full model change instructions 2. Read docs/model-profiles.md for NIM profile selection and GPU-specific profiles 3. Key env vars: APP_LLM_MODELNAME, APP_EMBEDDINGS_MODELNAME, APP_RANKING_MODELNAME 4. Embedding model change requires re-ingesting all documents — update APP_EMBEDDINGS_DIMENSIONS to match 5. Restart affected services (RAG server + ingestor for embedding changes) 6. Verify via health endpoint

Switch Vector DB

1. Read docs/change-vectordb.md for full setup (Docker and Helm) 2. Key env vars: APP_VECTORSTORE_URL, APP_VECTORSTORE_NAME 3. Data is not migrated — re-ingest all documents after switching 4. Elasticsearch is the default backend and uses rag-vol-elasticsearch in Docker Compose 5. Elasticsearch requires port 9200; check for conflicts

Milvus Configuration

1. Read docs/milvus-configuration.md for indexing, GPU, auth, and tuning 2. Read docs/milvus-schema.md for collection schema requirements 3. CPU mode: set APP_VECTORSTORE_ENABLEGPUSEARCH=False, APP_VECTORSTORE_ENABLEGPUINDEX=False, change Milvus image to non-GPU 4. Auth: download milvus.yaml, enable authorizationEnabled, set password before first deployment

API Keys

1. Read docs/api-key.md for NGC API key setup and per-service keys 2. Fallback order: service-specific key > NVIDIA_API_KEY > NGC_API_KEY 3. Per-service keys: APP_LLM_APIKEY, APP_EMBEDDINGS_APIKEY, APP_RANKING_APIKEY, APP_VLM_APIKEY, etc.

Decision Table

Agent-Specific Notes

• Current default model family uses nvidia/nemotron-3-super-120b-a12b, nvidia/llama-nemotron-embed-vl-1b-v2, and nvidia/llama-nemotron-rerank-1b-v2.
• Nemotron-3-Nano naming: nvidia/nemotron-3-nano-30b-a3b (NVIDIA-hosted) vs nvidia/nemotron-3-nano (self-hosted NIM) — same model, different names
• Helm model changes go in values.yaml under nimOperator and envVars sections
• Custom VDB operator requires implementing VDBRag base class — see docs/change-vectordb.md "Custom Vector Database Operator" section
• VDB auth tokens can be passed per-request via Authorization: Bearer <token> header; Elasticsearch runtime auth supports API keys
• Milvus password persists in etcd volume — to change after deployment, must delete volumes (destroys data)

Notebooks

• notebooks/building_rag_vdb_operator.ipynb — Custom VDB operator implementation (OpenSearch example)

Source Documentation

• docs/change-model.md — Model changes (LLM, embedding, ranking, NIM images)
• docs/change-vectordb.md — Vector DB switching, Elasticsearch setup, custom VDB operator
• docs/milvus-configuration.md — Milvus indexing, GPU config, auth, tuning
• docs/milvus-schema.md — Collection schema fields and requirements
• docs/model-profiles.md — NIM profile definitions and selection
• docs/api-key.md — NGC API key setup, per-service keys, fallback order
• docs/service-port-gpu-reference.md — Port mappings and GPU assignments for all services

Multimodal Query (Image + Text)

When to Use

• User wants to query knowledge base with images and text together
• User asks about VLM (Vision Language Model) deployment for RAG
• User wants image-based document understanding or visual Q&A

Restrictions

• Reranker must be disabled (ENABLE_RERANKER=false)
• On-prem: requires NVIDIA H100 or A100 SXM 80GB GPU
• Single-page retrieval only — image queries return content from one page per document

Process

1. Detect the deployment mode (Docker / Helm / Library). Docker: edit the active env file. Helm: edit values.yaml. Library: edit notebooks/config.yaml 2. Read docs/multimodal-query.md for full env var configuration and commands 3. Choose variant: self-hosted (Docker), NVIDIA-hosted (cloud), or Helm 4. Deploy VLM + VLM Embedding NIMs per source doc instructions 5. Set VLM env vars in the active config and switch embedding model to VLM embedding 6. Restart ingestor + RAG server (Docker: add --build flag) and verify

Agent-Specific Notes

• Must select a collection before querying — queries without collection return no results
• First VLM deployment: model downloads take 10–20 min (~10GB+)
• VLM_MS_GPU_ID — read docs/service-port-gpu-reference.md for the default GPU assignment and override if needed
• Cloud rate limits apply for ingestion of >10 files
• NVIDIA-hosted VLM endpoints should include the /v1 suffix, e.g. https://integrate.api.nvidia.com/v1
• For Helm with MIG: ensure dedicated MIG slice is assigned to VLM
• Image extraction must be enabled: APP_NVINGEST_EXTRACTIMAGES=True, APP_NVINGEST_IMAGE_ELEMENTS_MODALITY=image
• Helm multimodal deployments that disable nim-llm must set summary env vars under ingestor-server.envVars when generate_summary=true

Notebooks

• notebooks/image_input.ipynb — end-to-end multimodal query examples, image upload, VLM querying

Source Documentation

• docs/multimodal-query.md — full Docker/cloud/Helm configuration, env vars, API usage, limitations

Notebooks

When to Use

• Hands-on examples of NVIDIA RAG Blueprint features are needed
• There are questions about Jupyter notebooks, tutorials, or code samples

Process

1. Read docs/notebooks.md for full notebook descriptions and prerequisites. 2. Set up the environment: virtualenv, jupyterlab, and git lfs pull for test data. 3. Open JupyterLab at http://<server-ip>:8889.

Agent-Specific Notes

• Git LFS is required because several notebooks rely on large data files (git lfs install && git lfs pull).
• In Docker mode, deploy NVIDIA RAG Blueprint first, then run notebooks against the running services.
• In library mode, use rag_library_usage.ipynb (full) or rag_library_lite_usage.ipynb (containerless).
• The custom VDB operator notebook requires Docker for OpenSearch services.
• Agentic RAG examples are integrated into rag_library_usage.ipynb (library mode, agentic=True on generate()) and retriever_api_usage.ipynb (API streaming). For configuration, see references/configure/agentic-rag.md.

Notebook Catalog

Beginner

Intermediate

Advanced

Deployment

Source Documentation

• docs/notebooks.md — full notebook descriptions, setup, and prerequisites.

Observability

When to Use

• User wants tracing, metrics, or monitoring for the RAG pipeline
• User asks about latency debugging, Zipkin, Grafana, or Prometheus

Process

1. Detect the deployment mode. Docker: edit the active env file. Helm: edit values.yaml. Library: edit notebooks/config.yaml 2. Read docs/observability.md for full setup (Docker and Helm) 3. Set OPENTELEMETRY_CONFIG_FILE and APP_TRACING_ENABLED=True in the active config 4. Start observability stack and restart RAG server 5. Import Grafana dashboard from deploy/config/rag-metrics-dashboard.json

Agent-Specific Notes

• Library mode: set OPENTELEMETRY_CONFIG_FILE in the environment for tracing; the Docker-based Prometheus/Grafana stack is independent
• Helm: Prometheus Operator CRDs must be installed before deploying with observability enabled
• Default Grafana credentials: admin / admin
• Zipkin spans cover: query-rewriter, retriever, context-reranker, llm-stream
• Span I/O visible via traceloop.entity.input / traceloop.entity.output fields

Quick Latency Triage

Source Documentation

• docs/observability.md -- full Docker/Helm setup, env vars, metrics reference, and dashboard import

Query Rewriting, Query Decomposition, and Multi-Turn

Use these features when the user wants follow-up questions, conversation-aware retrieval, query rewriting, or decomposition of complex questions. For LangGraph agent planning/execution, use agentic-rag.md instead.

When to Use

• Enable multi-turn conversations or support follow-up questions.
• Improve retrieval with query rewriting.
• Break complex multi-hop questions into smaller retrieval subqueries.
• Configure or debug conversation history behavior.

Restrictions

• Query rewriting and multi-turn both require CONVERSATION_HISTORY > 0; with 0, query rewriting has no effect.
• Query decomposition works only when use_knowledge_base=true and with a single collection.
• Query decomposition is separate from Agentic RAG; do not enable both without reading docs/agentic-rag.md and docs/query_decomposition.md limitations.

Dependencies

Process

1. Detect deployment mode. Docker: edit the active env file. Helm: edit values.yaml. Library: edit notebooks/config.yaml. 2. Read the source doc for the feature. 3. Apply config changes and restart the RAG server. 4. Verify with a follow-up or multi-hop query against a known collection.

Query Rewriting

1. Read docs/multiturn.md for full configuration details. 2. To enable, set ENABLE_QUERYREWRITER=True. If CONVERSATION_HISTORY=0, set it to 5 or another positive value. 3. To disable, unset or comment out ENABLE_QUERYREWRITER. 4. Optional per request: set enable_query_rewriting: true in POST /v1/generate; CONVERSATION_HISTORY must still be positive.

Multi-Turn

1. Read docs/multiturn.md for retrieval strategies and API usage. 2. To enable, set CONVERSATION_HISTORY > 0 and choose the retrieval strategy. 3. To disable, set CONVERSATION_HISTORY=0.

Query Decomposition

1. Read docs/query_decomposition.md for the algorithm, limitations, and examples. 2. Set ENABLE_QUERY_DECOMPOSITION=true and MAX_RECURSION_DEPTH=3 or another depth that fits the use case.

Decision Table

Agent-Specific Notes

• MULTITURN_RETRIEVER_SIMPLE only applies when query rewriting is disabled; query rewriting takes precedence if both are configured.
• Query decomposition adds latency and is most useful for multi-hop questions that involve multiple entities or steps.
• In library mode, configure these settings in notebooks/config.yaml instead of environment variables.

Notebooks

• notebooks/retriever_api_usage.ipynb — RAG retriever API usage with search and end-to-end query examples.

Source Documentation

• docs/query_decomposition.md — decomposition algorithm and recursion depth guidance
• docs/multiturn.md — conversation history behavior, retrieval strategies, API usage, Helm configuration
• docs/agentic-rag.md — separate LangGraph agentic pipeline

Reasoning, Self-Reflection & Prompt Customization

When to Use

User wants to enable reasoning/thinking mode, stream or inspect reasoning_content, configure self-reflection, customize prompts, adjust generation parameters (max tokens, temperature, citations), or understand thinking budget options.

Process

1. Detect the deployment mode (Docker / Helm / Library). Docker: edit the active env file. Helm: edit values.yaml. Library: edit notebooks/config.yaml 2. Read the relevant source doc for the specific feature 3. Apply env vars to the active config or edit prompt files, restart RAG server 4. Prompt changes require --build flag (Docker); env var changes only need restart 5. Verify: test with a query and check for reasoning output or changed behavior

Decision Table

Agent-Specific Notes

• Prompt changes need --build flag on restart; env var changes do not
• Self-reflection: streaming not supported during groundedness checks
• Self-reflection uses same LLM by default; override with REFLECTION_LLM, REFLECTION_LLM_SERVERURL, REFLECTION_LLM_APIKEY
• Helm: only on-premises reflection is supported
• GPU requirements for reflection: see docs/self-reflection.md for optimal GPU configurations
• Debug reflection: set LOGLEVEL=INFO to observe iteration counts
• ENABLE_NEMOTRON_3_NANO_THINKING is deprecated; use LLM_ENABLE_THINKING
• With current streaming responses, reasoning is separated from the user-facing answer: choices[].delta.reasoning_content carries reasoning while choices[].delta.content carries final answer tokens
• FILTER_THINK_TOKENS=true keeps final-answer content clean but still preserves reasoning structurally in reasoning_content when the server is configured to preserve it
• 18 prompt templates available in prompt.yaml — custom file only overrides specified keys

Reasoning Model Comparison

Thinking Budget Recommendations

Notebooks

• notebooks/retriever_api_usage.ipynb — end-to-end query examples showing generation behavior

Source Documentation

• docs/enable-nemotron-thinking.md — Reasoning mode for all Nemotron models
• docs/self-reflection.md — Self-reflection configuration and thresholds
• docs/prompt-customization.md — Prompt template catalog and customization
• docs/llm-params.md — Generation parameters (temperature, max tokens, etc.)

Search & Retrieval: Hybrid Search, Multi-Collection, Metadata & Profiles

When to Use

User wants to enable hybrid search, query multiple collections, add custom metadata/filters, tune retrieval performance, configure reranker, enable natural language filter generation, or switch accuracy/performance profiles.

Process

1. Detect the deployment mode (Docker / Helm / Library). Docker: edit the active env file. Helm: edit values.yaml. Library: edit notebooks/config.yaml 2. Read the relevant source doc for detailed configuration 3. Apply the required env vars to the active config and restart affected services 4. Verify via search/generate API call

Decision Table

Agent-Specific Notes

• Hybrid search requires re-ingesting — existing collections created with dense must be re-created
• Multi-collection: limited to 5 collections per query; reranker is mandatory
• Multi-collection not supported when ENABLE_QUERY_DECOMPOSITION=true
• Elasticsearch is the default vector DB. Milvus is optional and requires re-ingestion when switching.
• Elasticsearch RRF is not supported in the open-source version — use weighted ranker for open-source Elasticsearch hybrid search
• Ingestor must be restarted alongside RAG server when enabling hybrid search
• RERANKER_CONFIDENCE_THRESHOLD is a legacy alias for RERANKER_SCORE_THRESHOLD
• Recommended RERANKER_SCORE_THRESHOLD range: 0.3–0.5 (too high filters out too many chunks)

Advanced Tuning (not fully documented elsewhere)

Partial Filtering

• Strict (default): fails if any collection doesn't support the filter
• Flexible (allow_partial_filtering: true in config.yaml): succeeds if at least one collection supports it

VDB Filter Support

Notebooks

• notebooks/retriever_api_usage.ipynb — RAG retriever API: search and end-to-end queries
• notebooks/nb_metadata.ipynb — Metadata ingestion, filtering, and extraction from queries

Source Documentation

• docs/hybrid_search.md — Hybrid dense + sparse search configuration
• docs/multi-collection-retrieval.md — Multi-collection querying
• docs/custom-metadata.md — Custom metadata schema, filtering expressions, filter generation
• docs/accuracy_perf.md — Best practices for tuning ingestion/retrieval/generation settings
• docs/python-client.md — Python library API for search and filtering

Document Summarization

When to Use

• User wants to generate summaries during document ingestion
• User asks about summarization strategies or options
• User wants to check summary status or progress

Restrictions

• Not supported in lite mode (containerless/library-only deployment)
• Requires Redis for status tracking and rate limiting
• Collection must exist before uploading with generate_summary: true

Process

1. Detect the deployment mode. Docker: edit the active env file. Helm: configure under ingestor-server.envVars in values.yaml. Library: use the upload API parameters directly (no env vars needed) 2. Read docs/summarization.md for full configuration, env vars, and prompt customization 3. Set generate_summary: true in the upload payload (per-request, no global toggle) 4. Optionally configure summary_options: strategy, shallow mode, page filter 5. Retrieve summary via GET /v1/summary?collection_name=...&file_name=...

Decision Table

Agent-Specific Notes

• CONVERSATION_HISTORY prerequisite does not apply — that's for query rewriting only
• SUMMARY_LLM_SERVERURL="" (empty) routes to NVIDIA cloud; "nim-llm:8000" for self-hosted
• SUMMARY_LLM_MAX_CHUNK_LENGTH should be below the model's context window to leave room for prompt + output
• Redis semaphore auto-resets on ingestor startup (prevents stale values from crashes)
• If Redis is unavailable, summaries still generate but no real-time status tracking
• Status entries have 24-hour TTL in Redis

Notebooks

• notebooks/summarization.ipynb — complete examples for all strategies, status polling, library mode usage

Source Documentation

• docs/summarization.md — env var reference, prompt customization, rate limiting, chunking details

User Interface

When to Use

• User asks about the RAG UI, uploading documents, settings, reasoning traces, or metadata filtering
• User wants to configure features via the web interface

Restrictions

• Sample/experimentation UI — not intended for production
• 100-file limit per upload batch; use multiple batches or API for bulk uploads
• 10 MB max per image attachment

Process

1. Read docs/user-interface.md for full UI documentation 2. Access at http://localhost:8090 (or http://<workstation-ip>:8090 for remote) 3. Configure RAG settings and feature toggles via Settings panel 4. Use Filter Bar above chat input for metadata-filtered queries 5. For reasoning-capable responses, inspect the collapsible reasoning panel above the assistant answer

Agent-Specific Notes

• VLM Inference must be enabled in Settings > Feature Toggles before image attachments work
• ECONNRESET errors on multi-file uploads — recommend API for bulk operations
• Document summaries generate asynchronously; UI shows "Generating summary..." until complete
• Document count in UI may lag slightly after ingestion
• Metadata filtering supports AND/OR logic between filters (toggle via logic button)
• The UI serializes filter_expr differently by backend: Milvus gets a string expression; Elasticsearch gets a list of Query DSL clauses. Backend is detected from /v1/health database service labels.
• The reasoning panel renders both Agentic RAG stage traces and standard RAG reasoning_content chunks.
• Custom metadata schema is set during collection creation via the Metadata Schema Editor

Source Documentation

• docs/user-interface.md -- full UI documentation including settings, file types, metadata, and health monitoring

VLM, VLM Embeddings & Image Captioning

When to Use

User wants image understanding, visual content analysis, VLM inference, multimodal embeddings, VLM reranking, VLM reasoning output, or image captioning during ingestion.

Restrictions

• Not available on B200 GPUs — use H100, A100 SXM 80GB, or RTX PRO 6000.
• Requires extra GPU (GPU 1+ for 2-GPU systems, GPU 2+ for 3+ GPUs with fallback)
• VLM embeddings: experimental, PDF-only, no summarization, no citations with page-as-image.
• Image captioning on Helm: on-prem only (modify values.yaml to enable)

Process

1. Detect the deployment mode (Docker / Helm / Library). Docker: edit the active env file. Helm: edit values.yaml. Library: edit notebooks/config.yaml 2. Read the relevant source doc for detailed steps:

• VLM generation: docs/vlm.md
• VLM embeddings and VLM reranker: docs/multimodal-retriever.md
• Image captioning: docs/image_captioning.md

3. Start VLM NIM (self-hosted) or configure cloud endpoint (NVIDIA-hosted) 4. Set the required variables in the active config:

• Enabling: ENABLE_VLM_INFERENCE=true and APP_NVINGEST_EXTRACTIMAGES=True
• Disabling: re-comment those variables in the env file

5. Restart affected services and verify with a health check + image-containing document query

Decision Table

Agent-Specific Notes

• --profile vlm-generation skips the LLM entirely — use --profile vlm-only for fallback mode
• VLM_TO_LLM_FALLBACK defaults to true, but vlm-generation profile does not start LLM
• Helm VLM: disable nim-llm and enable nim-vlm (VLM uses LLM's GPU allocation)
• Helm fallback: keep both nim-vlm and nim-llm enabled, set VLM_TO_LLM_FALLBACK: "true"
• VLM context window is limited — keep queries self-contained
• VLM reasoning streams final answer in content and reasoning in reasoning_content; VLM_FILTER_THINK_TOKENS is retained for compatibility and no longer wraps reasoning in text sentinels
• Image queries bypass reranking, including VLM reranking
• Image captioning known issue: files without graphs/charts/tables/plots fail to ingest when captioning is enabled

Key Env Vars (always needed)

• ENABLE_VLM_INFERENCE=true — master toggle
• APP_NVINGEST_EXTRACTIMAGES=True — extract images during ingestion
• VLM_MS_GPU_ID=<gpu-id> — self-hosted GPU assignment

Notebooks

• notebooks/image_input.ipynb — Multimodal queries with VLM (text + image)

Source Documentation

• docs/vlm.md — VLM generation (self-hosted, NVIDIA-hosted, Helm, Library)
• docs/multimodal-retriever.md — VLM embeddings (experimental)
• docs/image_captioning.md — Image captioning during ingestion
• docs/multimodal-query.md — Image + text querying
• docs/service-port-gpu-reference.md — default GPU assignments for VLM and other NIMs

RAG Blueprint Deployment

Phase 1: Environment Analysis

Run this single command to collect all environment information at once:

echo "=== GPU ===" && nvidia-smi --query-gpu=index,name,memory.total --format=csv,noheader 2>/dev/null || echo "NO_GPU"; echo "=== VRAM ===" && nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits 2>/dev/null | awk '{s+=$1} END {print s "MB total"}' || echo "0MB total"; echo "=== DRIVER ===" && cat /proc/driver/nvidia/version 2>/dev/null | head -1 || echo "NO_DRIVER"; echo "=== CUDA ===" && nvcc --version 2>/dev/null | grep "release" || echo "NO_CUDA_TOOLKIT"; echo "=== DOCKER ===" && docker --version 2>/dev/null || echo "NO_DOCKER"; echo "=== COMPOSE ===" && docker compose version 2>/dev/null || echo "NO_COMPOSE"; echo "=== NVIDIA_TOOLKIT ===" && docker info 2>/dev/null | grep -i "runtimes.*nvidia" || echo "NO_NVIDIA_TOOLKIT"; echo "=== PYTHON ===" && python3 --version 2>/dev/null || echo "NO_PYTHON"; echo "=== DISK ===" && df -h --output=avail / | tail -1; echo "=== OS ===" && cat /etc/os-release 2>/dev/null | grep -E "^(NAME|VERSION)="; echo "=== NGC_KEY ===" && if [ -n "$NGC_API_KEY" ]; then echo "NGC_KEY_SET"; elif [ -n "$NVIDIA_API_KEY" ]; then echo "NVIDIA_KEY_SET"; elif grep -Eh '^(export[[:space:]]+)?(NGC_API_KEY|NVIDIA_API_KEY)=' deploy/compose/.env deploy/compose/nvdev.env 2>/dev/null | grep -v "nvapi-your-key" | grep -q "nvapi-"; then echo "DOTENV_SET"; else echo "NOT_SET"; fi; echo "=== RUNNING ===" && docker ps --format "{{.Names}}" 2>/dev/null | grep -E "(rag-server|ingestor-server|nim-llm|nemotron-vlm-embedding|elasticsearch|milvus|seaweedfs)" | head -15 || echo "NO_RUNNING_SERVICES"; echo "=== PORTS ===" && (ss -tlnp 2>/dev/null || netstat -tlnp 2>/dev/null) | grep -E ":(8081|8082|8090|9200|9010|19530) " || echo "PORTS_FREE"; echo "=== REPO ===" && git rev-parse --show-toplevel 2>/dev/null && git describe --tags 2>/dev/null || echo "NO_GIT_REPO"; echo "=== CACHE ===" && du -sh ~/.cache/model-cache/ 2>/dev/null || echo "NO_CACHE"

Present a summary table:

Existing Services Warning

If RAG services are already running, tell the user briefly: "Existing RAG services detected (list). Proceeding will restart them." Continue unless the user objects.

If the user wants to switch deployment modes (e.g., NVIDIA-hosted → self-hosted, or Docker → library), shut down the existing deployment first via references/shutdown.md, then proceed with the new mode.

If ports are occupied by non-RAG processes, tell the user which ports conflict and suggest stopping the conflicting process. This is a blocker.

Phase 2: NGC_API_KEY Handling

Check in this order:

1. If NGC_API_KEY is set in the shell environment → proceed. 2. If NVIDIA_API_KEY is set (common in library mode) → proceed silently. 3. If NGC_API_KEY is in deploy/compose/.env or deploy/compose/nvdev.env (and not the placeholder nvapi-your-key) → load it and proceed. 4. If none found → tell the user: "NGC_API_KEY is required. Get one from https://org.ngc.nvidia.com/setup/api-keys and run: export NGC_API_KEY=\"nvapi-...\" — then tell me when done." 5. After user confirms → re-check silently. If still not set, write placeholder to .env and tell the user to edit it.

Phase 3: Blocker Checks

Automatically check and report all blockers at once (don't stop at the first one):

Read docs/support-matrix.md for current minimum versions and disk requirements, then check:

• Docker Compose below minimum: "Upgrade Docker Compose. See https://docs.docker.com/compose/install/linux/"
• NVIDIA Driver below minimum (if self-hosted): "Upgrade NVIDIA driver. See docs/support-matrix.md for required version."
• NVIDIA Container Toolkit missing (and self-hosted needed): "Install NVIDIA Container Toolkit. See https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html"
• Insufficient disk: "Check docs/support-matrix.md for disk requirements per deployment mode."
• No Docker and no Python 3.11+: "Install Docker or Python 3.11+ to proceed."

List all blockers together so the user can fix them in one pass — don't make them fix one, re-run, fix another.

Phase 4: Route to Deployment Mode

User explicitly requests a mode

• "library mode" / "lite mode" / "no docker" / "python mode" → read and follow deploy/library.md
• "docker" / "self-hosted" / "local" → read and follow deploy/docker.md with mode self-hosted
• "cloud" / "nvidia-hosted" / "hosted" → read and follow deploy/docker.md with mode nvidia-hosted
• "retrieval only" / "search only" / "no LLM" → read and follow deploy/docker.md with mode retrieval-only
• "kubernetes" / "k8s" / "helm" → read and follow deploy/helm.md
• "workbench" / "ai workbench" → tell user to follow deploy/workbench/README.md (AI Workbench uses its own UI-driven workflow)

Docker is available (Docker + Compose detected)

Self-hosted eligible — read docs/support-matrix.md ("Hardware Requirements (Docker)" section) for current GPU requirements. All of the following must also be true:

• GPU count and type matches the Docker self-hosted requirements from the support matrix
• ≥200 GB free disk (per docs/support-matrix.md "Disk Space Requirements")
• NVIDIA Container Toolkit detected
• NVIDIA driver meets minimum version from docs/support-matrix.md ("Driver Versions")

If self-hosted eligible → read and follow deploy/docker.md with mode self-hosted

Otherwise with Docker → read and follow deploy/docker.md with mode nvidia-hosted

Tell the user WHY if they have some GPU but not enough:

• "You have [X GPU] with [Y GB] VRAM. Self-hosted requires [requirements from docs/support-matrix.md]. Deploying with NVIDIA-hosted cloud NIMs instead — faster startup, no model download."

Docker is available but Compose is not

Tell the user: "Docker is installed but Docker Compose is below the minimum version (see docs/support-matrix.md). Install it: https://docs.docker.com/compose/install/linux/ — or use library mode instead."

If user chooses library mode → read and follow deploy/library.md

Docker is not available

• Python 3.11+ available → read and follow deploy/library.md with mode lite
• No Python → tell user to install Python 3.11+ or Docker

After Deployment

Once deployment completes, verify health:

echo "=== RAG Server ===" && curl -s http://localhost:8081/v1/health?check_dependencies=true 2>/dev/null || echo "RAG_SERVER_NOT_READY"; echo "=== Ingestor ===" && curl -s http://localhost:8082/v1/health?check_dependencies=true 2>/dev/null || echo "INGESTOR_NOT_READY"

If healthy, tell the user:

• "RAG Blueprint is running and healthy."
• "Ask me to configure features like VLM, query rewriting, guardrails, etc."
• "Ask me to shutdown when you're done."

If unhealthy, read references/troubleshoot.md and diagnose. Match error output against known issues, fix, and retry. Escalate to the user only if the fix requires their action (API key, data deletion).

Notebooks

• notebooks/launchable.ipynb — Cloud deployment via Brev (alternative to local deployment)

Source Documentation

• docs/support-matrix.md — GPU requirements, driver versions, disk space, supported platforms
• docs/service-port-gpu-reference.md — port mappings and GPU assignments for all services

Docker Deployment (NVIDIA-Hosted NIMs)

When to Use

• User wants fast deployment without local model downloads
• User has no GPU or limited GPU
• User asks about cloud-hosted or NVIDIA API deployment
• User wants to avoid 15–30 min NIM startup time

Restrictions

• Requires internet access (calls NVIDIA cloud APIs)
• NVIDIA-hosted endpoints have rate limits — large ingestions (>10 files) may hit 429 errors
• NGC_API_KEY required for cloud API access
• Docker and Compose minimum versions per docs/support-matrix.md

Process

1. Read docs/deploy-docker-nvidia-hosted.md for full commands and env configuration 2. Use deploy/compose/nvdev.env — pre-configured for cloud endpoints. Source it before compose commands: source deploy/compose/nvdev.env 3. Start vector DB → ingestor → RAG server + frontend (no NIM startup needed) 4. Verify: docker ps shows containers; UI at http://localhost:8090

Decision Table

Agent-Specific Notes

• First run: 5–10 min (image pulls only); subsequent: 1–2 min
• No nims.yaml startup — all model inference is cloud-hosted
• Persistent Docker data is in named rag-vol-* volumes, created automatically
• All subsequent configure/restart operations should source the same env file used for the initial deploy (deploy/compose/nvdev.env)
• For zero-GPU with Milvus specifically: switch Milvus to CPU-only by changing the GPU image tag to the equivalent non-GPU tag and setting APP_VECTORSTORE_ENABLEGPUSEARCH=False. Default Elasticsearch does not require this.
• Rate limit mitigation for large ingestions: reduce NV_INGEST_FILES_PER_BATCH, NV_INGEST_CONCURRENT_BATCHES, MAX_INGEST_PROCESS_WORKERS, NV_INGEST_MAX_UTIL to minimum values

Source Documentation

• docs/deploy-docker-nvidia-hosted.md — full step-by-step commands, env var blocks, CPU Milvus setup

Retrieval-Only Deployment

When to Use

• User wants search/retrieval without LLM generation
• User asks to deploy only embedding + reranking services
• User wants /search endpoint with an external LLM
• User wants a lightweight, low-GPU deployment

Restrictions

• /generate endpoint returns an error — no LLM is deployed
• Self-hosted: 1 GPU, ~24 GB memory
• NVIDIA-hosted: 0 GPUs (cloud embedding + reranking)

Process

1. Read docs/retrieval-only-deployment.md for full commands, env vars, and API examples 2. Choose variant: self-hosted (local NIMs), NVIDIA-hosted (cloud), or Helm 3. For self-hosted: start only embedding + ranking NIMs, skip LLM 4. For NVIDIA-hosted: set embedding/ranking server URLs to empty, skip NIM startup entirely 5. For Helm: set nimOperator.nim-llm.enabled=false 6. Start vector DB → ingestor → RAG server 7. Verify health: GET http://localhost:8081/v1/health?check_dependencies=true

Decision Table

Agent-Specific Notes

• Permission errors on model cache → try USERID=0 or chmod -R 755 ~/.cache/model-cache
• Empty search results → verify documents ingested: GET http://localhost:8082/v1/documents?collection_name=<name>
• Users can send /search results to their own external LLM for generation

Source Documentation

• docs/retrieval-only-deployment.md — full deployment commands, API examples, search payload options

Docker Deployment (Self-Hosted NIMs)

When to Use

• User wants full on-premises deployment with local NIM containers
• User has supported GPUs and wants models running locally
• User asks to deploy RAG Blueprint with Docker

Restrictions

Read docs/support-matrix.md for current GPU requirements. Feature restrictions per GPU type:

• Read docs/support-matrix.md for current minimum NVIDIA Driver, CUDA, Docker, and Compose versions
• NVIDIA Container Toolkit required (docker info shows nvidia runtime)
• Disk space per docs/support-matrix.md ("Disk Space Requirements")
• If any prerequisite is missing, tell the user what to install before proceeding

Process

1. Read docs/deploy-docker-self-hosted.md for full commands and env configuration 2. Read docs/support-matrix.md for GPU compatibility and supported model combinations 3. Verify container toolkit, prepare model cache directory, source .env 4. Apply GPU-specific config per source docs 5. Start NIMs → wait for healthy → start remaining services 6. Verify: docker ps shows all containers healthy; UI at http://localhost:8090

Decision Table

Agent-Specific Notes

• First run: 15–30 min (model downloads ~100–150 GB, no progress bar); subsequent: 2–5 min
• Monitor download progress: du -sh ~/.cache/model-cache/
• Permission error on model cache → try USERID=0 instead of USERID=$(id -u)
• Cloud NIM section in deploy/compose/.env must be commented out for self-hosted
• Rebuild after code changes: add --build flag to compose up commands

Source Documentation

• docs/deploy-docker-self-hosted.md — full step-by-step commands, env vars, GPU assignments
• docs/support-matrix.md — GPU compatibility, supported models, hardware requirements

RAG Docker Deployment

Determine Mode

If routed here from the deploy workflow, the mode (self-hosted, nvidia-hosted, or retrieval-only) was already decided. Use it.

If invoked directly without a mode, auto-detect:

echo "=== COMPOSE ===" && docker compose version 2>/dev/null || echo "NO_COMPOSE"; echo "=== GPU ===" && nvidia-smi --query-gpu=name,memory.total --format=csv,noheader 2>/dev/null || echo "NO_GPU"; echo "=== DISK ===" && df -h --output=avail / | tail -1; echo "=== RUNNING ===" && docker ps --format "{{.Names}}" 2>/dev/null | grep -E "(rag-server|ingestor-server|nim-llm|nemotron-vlm-embedding|elasticsearch|milvus)" | head -10 || echo "NONE_RUNNING"

If NO_COMPOSE: stop and tell the user to install Docker Compose (see docs/support-matrix.md for minimum version).

Read docs/support-matrix.md ("Hardware Requirements (Docker)" section) for current GPU requirements, then:

• GPU count/type meets self-hosted requirements from the support matrix, and 200+ GB free disk → self-hosted
• Any GPU or no GPU with ≥50 GB free disk → nvidia-hosted (default Elasticsearch does not require a GPU)
• User explicitly says "retrieval only" / "no LLM" / "search only" → retrieval-only

Auto-route based on hardware. Only ask if two modes are equally valid and the user's intent is ambiguous.

Verify NGC_API_KEY

Auto-check all possible locations before asking:

if [ -n "$NGC_API_KEY" ] || [ -n "$NVIDIA_API_KEY" ]; then echo "ENV_SET"; elif grep -Eh '^(export[[:space:]]+)?(NGC_API_KEY|NVIDIA_API_KEY)=' deploy/compose/.env deploy/compose/nvdev.env 2>/dev/null | grep -v "nvapi-your-key" | grep -q "nvapi-"; then echo "DOTENV_SET"; else echo "NOT_SET"; fi

• ENV_SET: proceed silently.
• DOTENV_SET: load the env file that contains the key and proceed.
• NOT_SET: ask the user to provide it. This is the only thing to ask for.

Docker Login

Auto-check if already logged in:

grep -q "nvcr.io" ~/.docker/config.json 2>/dev/null && echo "ALREADY_LOGGED_IN" || echo "NOT_LOGGED_IN"

If already logged in → proceed silently.

If not logged in → tell the user to run this themselves (the key gets expanded in agent logs):

Please run in your terminal: echo "${NGC_API_KEY}" | docker login nvcr.io -u '$oauthtoken' --password-stdin

Wait for confirmation only if login was needed.

Deploy

Based on the mode, read and follow the appropriate reference:

• Self-hosted: read and follow docker-self-hosted.md
• NVIDIA-hosted: read and follow docker-nvidia-hosted.md
• Retrieval-only: read and follow docker-retrieval-only.md

Docker Compose persistent data is stored in named rag-vol-* volumes. Do not look for new data under the legacy deploy/compose/volumes/ tree unless the user is migrating old data.

Post-Deploy Verification

Run health checks:

sleep 5; echo "=== RAG ===" && curl -s http://localhost:8081/v1/health?check_dependencies=true 2>/dev/null || echo "RAG_NOT_READY"; echo "=== INGESTOR ===" && curl -s http://localhost:8082/v1/health?check_dependencies=true 2>/dev/null || echo "INGESTOR_NOT_READY"; echo "=== CONTAINERS ===" && docker ps --format "table {{.Names}}\t{{.Status}}" 2>/dev/null | grep -E "(rag|elasticsearch|milvus|seaweedfs|nim|ingest|embedding|ranking)" | head -20

If services are still initializing, automatically poll every 30 seconds:

• NVIDIA-hosted: poll until healthy or 5 minutes elapsed (no model downloads needed).
• Self-hosted: poll until healthy or 15 minutes elapsed (model downloads on first run).
• Retrieval-only: poll until healthy or 5 minutes elapsed.

Show progress to the user during polling.

On Success

Tell the user:

• "RAG Blueprint is running and healthy. Open http://localhost:8090 to use the UI." (skip for retrieval-only)
• "Ask me to configure features (VLM, query rewriting, guardrails, etc.)"
• "Ask me to shutdown when you're done."

On Error

1. Read the error output from the failed command. 2. Read references/troubleshoot.md to match against common issues (port conflict, disk full, NGC auth, GPU OOM). 3. Apply the fix and retry. 4. If still failing, report the specific error to the user with the fix that was attempted.

Source Documentation

• docs/support-matrix.md — GPU requirements, hardware compatibility, disk space

MIG GPU Deployment

When to Use

• User wants fine-grained GPU allocation on Kubernetes using MIG slices
• User has H100 GPUs and wants to share them across RAG services
• User asks about Multi-Instance GPU deployment

Restrictions

• Requires H100 80GB HBM3 GPUs (MIG-compatible)
• MIG profiles in this guide are specific to H100 80GB — other GPUs need different profiles
• Requires cloned repository (MIG config files in deploy/helm/)
• All standard Helm prerequisites apply (GPU Operator, NIM Operator, StorageClass)
• Ingestion profile is scaled down with MIG — large bulk ingestion jobs may fail

Process

1. Read docs/mig-deployment.md for full configuration, commands, and MIG slice definitions 2. Enable MIG with mixed strategy on ClusterPolicy 3. Apply MIG ConfigMap and label the node 4. Verify node labels show mig.config.state: "success" before proceeding 5. Install Helm chart with -f mig-slicing/values-mig.yaml

Decision Table

Agent-Specific Notes

• Must wait for mig.config.state: "success" on the node before Helm install — if not present, wait and re-check
• Default H100 MIG layout (see docs/mig-deployment.md for current GPU count and slice definitions): GPU 0 → small slices, GPU 1 → mixed slices, GPU 2 → full-GPU slice
• LLM gets the largest slice (7g.80gb); embedding/Milvus/ingest share small slices
• RTX PRO 6000 variant: uncomment model section in values.yaml, then use both -f values.yaml -f mig-slicing/values-mig.yaml
• Uninstall follows standard Helm procedure (see Helm deployment docs)

Source Documentation

• docs/mig-deployment.md — full MIG config, ClusterPolicy patches, node labeling, verification, Helm install commands

Helm Deployment on OpenShift

When to Use

• Cluster is Red Hat OpenShift or OKD (clusterversion resource present, or route.openshift.io API available)
• User mentions OpenShift, OKD, or RHEL OpenShift in the deployment context
• User wants OpenShift Routes with edge TLS instead of kubectl port-forward for external access

Restrictions

Read docs/support-matrix.md for current Kubernetes, Helm, and OS version requirements.

• Requires GPU Operator + NIM Operator pre-installed on the OpenShift cluster
• Default StorageClass must be configured for PVC provisioning
• Disk space per docs/support-matrix.md (~200 GB per node for NIM cache + images + PVCs)
• NeMo Guardrails not available in Helm deployment
• OpenShift's default Route timeout is 30 s — the chart sets haproxy.router.openshift.io/timeout: 300s on the RAG-server Route, but manually-created Routes need this annotation

Process

1. Read docs/deploy-helm-openshift.md for full commands and overlay file usage. 2. Ensure prerequisites: GPU Operator, NIM Operator, StorageClass, NGC_API_KEY, and a namespace:

   export NAMESPACE="${NAMESPACE:-rag}"
   kubectl create namespace "$NAMESPACE" 2>/dev/null || true

3. Install the chart with the values-openshift.yaml overlay (the overlay inherits the base values.yaml, so it does not need to be passed separately):

   helm upgrade --install rag -n "$NAMESPACE" <chart> \
     -f values-openshift.yaml \
     --set imagePullSecret.password="$NGC_API_KEY" \
     --set ngcApiSecret.password="$NGC_API_KEY" \
     --timeout 15m

The overlay turns on openshift.enabled, which makes the chart create OpenShift Routes with edge TLS and an anyuid SCC RoleBinding for all required ServiceAccounts — no manual oc adm policy add-scc-to-user is needed. 4. Link the pull secret to the NIM cache ServiceAccount after it exists:

   oc secrets link nim-cache-sa ngc-secret --for=pull -n "$NAMESPACE"

5. Monitor pods and Routes, then access the UI via the frontend Route's external host (no port-forward required):

   kubectl get pods -n "$NAMESPACE"
   kubectl get route -n "$NAMESPACE"

Decision Table

Agent-Specific Notes

• OpenShift Routes provide external access directly — do not propose kubectl port-forward workflows once Routes exist
• If a NIM pod hits CrashLoopBackOff with SCC-related errors, confirm openshift.enabled: true is set in the active overlay
• If NIMCache jobs or pods hit ImagePullBackOff, confirm the NGC pull secret is linked to nim-cache-sa
• Route timeouts during long requests → annotate the affected Route with haproxy.router.openshift.io/timeout=300s
• helm uninstall does not remove PVCs — clean up with kubectl delete nimcache --all -n "$NAMESPACE" && kubectl delete pvc --all -n "$NAMESPACE"

Source Documentation

• docs/deploy-helm-openshift.md — OpenShift Routes, SCC, overlay usage, OpenShift-specific troubleshooting
• docs/deploy-helm.md — standard (non-OpenShift) Helm deployment for comparison
• deploy/helm/nvidia-blueprint-rag/values-openshift.yaml — the overlay itself
• deploy/helm/nvidia-blueprint-rag/values-openshift-test.yaml — reference overlay for constrained/API-hosted setups

Helm Deployment

When to Use

• User wants to deploy RAG Blueprint on Kubernetes
• User asks about Helm chart installation (from NGC or local repo)
• User mentions Kubernetes, k8s, or Helm in deployment context

Restrictions

Read docs/support-matrix.md for current Kubernetes, Helm, and OS version requirements.

• Requires GPU Operator + NIM Operator pre-installed
• Default StorageClass must be configured for PVC provisioning
• Disk space per docs/support-matrix.md
• NeMo Guardrails not available in Helm deployment
• Image captioning: on-prem only (requires values.yaml changes; see docs/image_captioning.md)

Process

Option A: Deploy from NGC (Remote Chart)

1. Read docs/deploy-helm.md for full commands and values 2. Ensure prerequisites: GPU Operator, NIM Operator, StorageClass, NGC_API_KEY 3. Install chart, monitor pods, port-forward frontend

Option B: Deploy from Repository (Local Chart)

1. Read docs/deploy-helm-from-repo.md for full commands and repo setup 2. Add required Helm repos, run helm dependency update, install from local path

RTX PRO 6000 Variant

1. Uncomment model section under nimOperator.nim-llm.model in values.yaml 2. See source docs for engine/precision/GPU settings

Decision Table

Agent-Specific Notes

• First deployment: 60–70 min (model cache download); subsequent: 10–15 min
• Pods in ContainerCreating/Init for extended time is normal during cache download
• PVCs are not removed by helm uninstall — delete manually: kubectl delete nimcache --all -n rag && kubectl delete pvc --all -n rag
• Port-forwarding may timeout for large file ingestion — not suitable for bulk uploads
• All configurable endpoints documented in deploy/helm/nvidia-blueprint-rag/endpoints.md

Source Documentation

• docs/deploy-helm.md — NGC remote chart deployment, prerequisites, monitoring
• docs/deploy-helm-from-repo.md — local chart deployment, repo setup, dependency management

RAG Helm Deployment

If routed here from the deploy workflow, proceed directly to Phase 1.

Phase 1: Prerequisites Check

Run all checks at once:

echo "=== KUBECTL ===" && kubectl version --client 2>/dev/null || echo "NO_KUBECTL"; echo "=== HELM ===" && helm version --short 2>/dev/null || echo "NO_HELM"; echo "=== STORAGECLASS ===" && kubectl get storageclass 2>/dev/null || echo "NO_STORAGECLASS"; echo "=== NODES ===" && kubectl get nodes -o wide 2>/dev/null || echo "NO_CLUSTER_ACCESS"; echo "=== GPU_OPERATOR ===" && kubectl get pods -n gpu-operator 2>/dev/null | grep -i running || echo "NO_GPU_OPERATOR"; echo "=== NIM_OPERATOR ===" && kubectl get pods -n nim-operator 2>/dev/null | grep -i running || echo "NO_NIM_OPERATOR"; echo "=== NAMESPACE ===" && kubectl get namespace rag 2>/dev/null && echo "NAMESPACE_EXISTS" || echo "NO_NAMESPACE"; echo "=== HELM_RELEASE ===" && helm list -n rag 2>/dev/null | grep rag || echo "NO_EXISTING_RELEASE"; echo "=== PODS ===" && kubectl get pods -n rag 2>/dev/null | head -10 || echo "NO_PODS"; echo "=== NGC_KEY ===" && [ -n "$NGC_API_KEY" ] && echo "NGC_API_KEY SET" || echo "NGC_API_KEY NOT_SET"; echo "=== GPU_RESOURCES ===" && kubectl get nodes -o json 2>/dev/null | grep -o '"nvidia.com/gpu": "[0-9]*"' || echo "NO_GPU_RESOURCES"

Read docs/support-matrix.md for current Kubernetes, Helm, and OS version requirements.

Report all missing prerequisites together so the user can fix everything in one pass.

If NGC_API_KEY is NOT_SET: this is the one thing we must ask the user for.

If an existing Helm release is detected: warn "Existing RAG Helm release found. Proceeding will upgrade it." Continue unless user objects.

Phase 2: Route to Reference

Auto-detect the GPU variant and cluster flavor from cluster nodes (not the local machine):

echo "=== GPU_LABELS ===" && kubectl get nodes -o json 2>/dev/null | grep -oE '"nvidia.com/gpu.product":\s*"[^"]*"' | sort -u || echo "NO_GPU_LABELS"; echo "=== MIG ===" && kubectl get nodes -o json 2>/dev/null | grep -oE '"nvidia.com/mig.strategy":\s*"[^"]*"' || echo "NO_MIG"; echo "=== OPENSHIFT ===" && (kubectl get clusterversion 2>/dev/null | grep -q . && echo "OPENSHIFT_DETECTED") || (kubectl api-resources 2>/dev/null | grep -qi "route.openshift.io" && echo "OPENSHIFT_DETECTED") || echo "NOT_OPENSHIFT"

Determine variant from node GPU labels and cluster flavor:

Route based on detection:

• OpenShift / OKD (clusterversion resource present, or route.openshift.io API available, or user mentions OpenShift / RHEL OpenShift) → read and follow helm-openshift.md
• MIG enabled → read and follow helm-mig.md
• RTX PRO 6000 → read and follow helm-standard.md (use the RTX values.yaml variant described there)
• Standard (everything else) → read and follow helm-standard.md

Ask the user only if the variant is genuinely ambiguous. Default to standard deployment.

Phase 3: Expected Timelines

Set expectations with the user:

Pods in ContainerCreating or Init state for extended periods is normal — models download in the background without progress indicators.

Phase 4: Verification

After deployment completes, verify:

echo "=== PODS ===" && kubectl get pods -n rag; echo "=== NIMCACHE ===" && kubectl get nimcache -n rag; echo "=== NIMSERVICE ===" && kubectl get nimservice -n rag

Wait for all pods to reach Running status. Poll every 60 seconds for up to 70 minutes (first deployment involves model downloads). Show progress.

Once pods are running, port-forward and verify health:

kubectl port-forward -n rag service/rag-server 8081:8081 --address 0.0.0.0 & kubectl port-forward -n rag service/rag-frontend 3000:3000 --address 0.0.0.0 & sleep 3 && curl -s http://localhost:8081/v1/health?check_dependencies=true 2>/dev/null || echo "RAG_NOT_READY"

Phase 5: Uninstall

If the user wants to tear down:

helm uninstall rag -n rag
kubectl delete nimcache --all -n rag
kubectl delete pvc --all -n rag

On Success

Tell the user:

• "RAG Blueprint is running on Kubernetes. Access the UI at http://localhost:3000 (via port-forward)."
• "Ask me to configure features (VLM, query rewriting, guardrails, etc.)"
• "Ask me to shutdown when you're done."

On Error

1. Check pod status and events: kubectl describe pod <failing-pod> -n rag and kubectl get events -n rag --sort-by='.lastTimestamp' | tail -20. 2. Read pod logs: kubectl logs <failing-pod> -n rag --tail 50. 3. Read references/troubleshoot.md to match against common issues (PVC pending, OOM, image pull failure, port conflict). 4. Apply the fix and retry. If the fix requires data deletion (PVCs, namespace), confirm with user first.

Source Documentation

• docs/support-matrix.md — Kubernetes/Helm version requirements, GPU compatibility
• docs/deploy-helm.md — standard Helm deployment from NGC
• docs/deploy-helm-from-repo.md — Helm deployment from local repo
• docs/deploy-helm-openshift.md — Red Hat OpenShift deployment with Routes, SCC, and the values-openshift.yaml overlay

Library Mode (Full)

When to Use

• User wants programmatic Python access to RAG via nvidia_rag package
• User prefers code-level configuration over Docker-based servers
• User asks about library mode, Python client, or NvidiaRAG/NvidiaRAGIngestor

Restrictions

• Python 3.11+ (< 3.14)
• Docker still required for backend services (Milvus, NV-Ingest, Redis, optionally NIMs)
• Self-hosted NIMs require supported GPUs (see docs/support-matrix.md)

Process

1. Read docs/python-client.md for full API reference, configuration, and backend setup 2. Create virtual environment and install nvidia-rag[all] 3. Start backend services via Docker (Milvus, NV-Ingest + Redis, optionally NIMs) 4. Load config from notebooks/config.yaml using NvidiaRAGConfig.from_yaml() 5. Create NvidiaRAGIngestor and NvidiaRAG instances 6. Use ingestor.create_collection(), ingestor.upload_documents(), rag.generate(), rag.search()

Decision Table

Agent-Specific Notes

• Config file: notebooks/config.yaml; env file: notebooks/.env_library
• Docker login is interactive — tell user to run docker login nvcr.io themselves
• For cloud deployment: override config.embeddings.server_url, config.llm.server_url, etc. in code
• Config changes take effect immediately (no container restart needed, unlike Docker mode)
• Prompt customization via constructor: NvidiaRAG(config=config, prompts="custom_prompts.yaml")
• upload_documents() is async — returns task_id for status polling
• NV-Ingest cloud endpoints must be exported before starting NV-Ingest container

Notebooks

• notebooks/rag_library_usage.ipynb — complete walkthrough: setup, ingestion, querying, search, summaries

Source Documentation

• docs/python-client.md — full API reference, backend setup, configuration, cloud/self-hosted options

Library Mode (Lite / Containerless)

When to Use

• Quick prototyping with zero infrastructure (no Docker, no GPU)
• User wants the fastest path to try RAG
• CI/CD pipelines needing lightweight RAG testing

Restrictions

• No image/table/chart citations
• No document summarization
• Subject to NVIDIA API rate limits (cloud-hosted inference)
• Requires Python 3.11+ (< 3.14), internet access, and NGC_API_KEY

Process

1. Read docs/python-client.md for full library mode documentation 2. Create virtualenv and install: pip install nvidia-rag[all] 3. Ensure NGC_API_KEY is exported — maps to NVIDIA_API_KEY internally 4. Run the lite notebook: jupyter lab notebooks/rag_library_lite_usage.ipynb

Agent-Specific Notes

• NVIDIA_API_KEY (used by nvidia_rag package) must be set from NGC_API_KEY. In the notebook, copy the NGC key into the NVIDIA key variable: NVIDIA_API_KEY = value of NGC_API_KEY (falling back to empty string if unset)
• Lite config lives in notebooks/config.yaml; override server_url for embeddings to the NVIDIA API Catalog endpoint (see docs/python-client.md for current URL), and set LLM/ranking URLs to empty string for cloud defaults
• Milvus Lite runs embedded (no container), NV-Ingest runs as subprocess (no container)
• Also install python-dotenv jupyterlab for notebook support

When Not to Use

• Production workloads — use Docker or Kubernetes
• Large-scale ingestion — rate limits apply
• Need citations from images/tables/charts or document summarization

Notebooks

Source Documentation

• docs/python-client.md -- full library mode documentation (lite and full)

RAG Library Mode Setup

Determine Mode

If routed here from the deploy workflow, the mode (full or lite) may already be decided. Use it.

If invoked directly, auto-detect:

echo "=== DOCKER ===" && docker --version 2>/dev/null || echo "NO_DOCKER"; echo "=== GPU ===" && nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null || echo "NO_GPU"; echo "=== PYTHON ===" && python3 --version 2>/dev/null || echo "NO_PYTHON"; echo "=== PKG_MANAGER ===" && which uv 2>/dev/null && echo "UV_AVAILABLE" || (which pip3 2>/dev/null && echo "PIP_AVAILABLE" || echo "NO_PKG_MANAGER"); echo "=== VENV ===" && ls -d .venv/ venv/ nvidia-rag-env/ 2>/dev/null || echo "NO_EXISTING_VENV"; echo "=== INSTALLED ===" && pip3 show nvidia_rag 2>/dev/null | head -3 || echo "NOT_INSTALLED"

• Docker available → full (Python API + Docker backend services)
• No Docker or user explicitly says "lite" / "no docker" / "containerless" → lite

Auto-route based on Docker availability. Only ask if both modes are equally valid.

Verify NGC_API_KEY

Auto-check all locations:

if [ -n "$NGC_API_KEY" ]; then echo "NGC_KEY_SET"; elif [ -n "$NVIDIA_API_KEY" ]; then echo "NVIDIA_KEY_SET"; else echo "NOT_SET"; fi

If NOT_SET: ask the user. Otherwise proceed silently.

Deploy

Based on the mode:

• Full: read and follow library-full.md
• Lite: read and follow library-lite.md

On Success

Tell the user:

• Which mode was set up and how to start using it (notebook or Python script)
• "Ask me to configure features, change models, etc."
• "Ask me to shutdown backend services when done (if full mode)."

On Error

1. Read the error output (pip install failure, import error, service connection error). 2. Read references/troubleshoot.md to match against common issues. 3. Common fixes to try:

• pip install failure → try uv pip install or check Python version ≥3.11.
• Import error → check if virtual environment is activated.
• Connection error to backend services → check Docker containers are running.

4. Retry the failed step after fixing. 5. If still failing, report the specific error to the user.

Source Documentation

• docs/python-client.md — Python library API, installation, full and lite mode setup

RAG Shutdown

Stopping containers and processes does not require confirmation. Deleting data (volumes, cache, images) does.

Step 1: Detect What Is Running

Detect all deployment modes — Docker, K8s, and library:

echo "=== DOCKER ===" && docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Image}}" 2>/dev/null || echo "NO_DOCKER"; echo "=== LIBRARY ===" && ps aux | grep -E "(nvidia_rag|uvicorn|jupyter)" | grep -v grep || echo "NO_LIBRARY_PROCESSES"; echo "=== K8S ===" && kubectl get pods -n rag 2>/dev/null | head -10 || echo "NO_K8S"; echo "=== HELM ===" && helm list -n rag 2>/dev/null | grep rag || echo "NO_HELM_RELEASE"

Based on what's detected, execute the appropriate shutdown path below. If multiple modes are active (e.g., Docker + library), stop all of them.

Step 2: Stop Services (Reverse Startup Order)

Stop in this order — reverse of deployment. Only stop what is actually running (detected in Step 1).

2a: Optional Services

Stop these first if they are running:

docker compose -f deploy/compose/docker-compose-nemo-guardrails.yaml down 2>/dev/null; docker compose -f deploy/compose/observability.yaml down 2>/dev/null

2b: Application Services

docker compose -f deploy/compose/docker-compose-rag-server.yaml down; docker compose -f deploy/compose/docker-compose-ingestor-server.yaml down

2c: Vector DB

docker compose -f deploy/compose/vectordb.yaml down

If a profile-specific vector DB stack was started and containers remain, include the profile explicitly:

docker compose -f deploy/compose/vectordb.yaml --profile elasticsearch down

2d: NIMs (Self-Hosted Only)

Only present if self-hosted deployment was used:

docker compose -f deploy/compose/nims.yaml down

This stops ALL NIM containers (LLM, embedding, ranking, OCR, detection, and any profile-specific NIMs like VLM, audio, nemotron-parse).

2e: Library Mode Processes

If library mode is active (detected Python processes):

pkill -f "nvidia_rag" 2>/dev/null; pkill -f "uvicorn.*rag" 2>/dev/null; docker compose -f deploy/compose/docker-compose-ingestor-server.yaml down 2>/dev/null; docker compose -f deploy/compose/vectordb.yaml down 2>/dev/null

2f: Kubernetes (Helm) Deployment

If K8s deployment was detected, use the release name and namespace from helm list output in step 1:

helm uninstall <release-name> -n <namespace> 2>/dev/null

To also clean up persistent data (only if user requests full cleanup):

kubectl delete nimcache --all -n <namespace> 2>/dev/null; kubectl delete pvc --all -n <namespace> 2>/dev/null

Step 3: Verify Everything Stopped

echo "=== REMAINING ===" && docker ps --format "table {{.Names}}\t{{.Status}}" 2>/dev/null; echo "=== K8S ===" && kubectl get pods -n rag 2>/dev/null | head -10 || echo "NOT_K8S"; helm list -n rag 2>/dev/null || true

If any RAG-related containers remain, force remove:

docker ps -a --format "{{.Names}}" | grep -E "(rag|milvus|nim|ingest|redis|nemo|grafana|prometheus|embedding|ranking|vlm|ocr|page-elements|graphic-elements|table-structure)" | xargs -r docker rm -f

If pods remain after helm uninstall, force delete:

kubectl delete pods --all -n rag --force --grace-period=0 2>/dev/null

Step 4: Optional Cleanup

Ask the user if they want to clean up data/volumes:

• Remove Docker volumes (deletes ingested data, vector DB indices, object-store data, and ingestor scratch):

  docker volume ls -q --filter "name=^rag-vol-" | xargs -r docker volume rm

These named volumes include Elasticsearch, Milvus/etcd, SeaweedFS, and ingestor scratch data. Prefer deleting only the specific rag-vol-* volume the user requested.

• Remove model cache (frees 100-200 GB for self-hosted):

  rm -rf ~/.cache/model-cache/

• Remove Docker images (frees disk space):

  docker images | grep -E "nvcr.io/nvidia|milvusdb" | awk '{print $3}' | xargs -r docker rmi

Only perform cleanup if the user explicitly requests it.

Quick One-Liner (All Docker Services)

If the user wants a fast full teardown:

cd "$(git rev-parse --show-toplevel)" && \
docker compose -f deploy/compose/docker-compose-nemo-guardrails.yaml down 2>/dev/null; \
docker compose -f deploy/compose/observability.yaml down 2>/dev/null; \
docker compose -f deploy/compose/docker-compose-rag-server.yaml down 2>/dev/null; \
docker compose -f deploy/compose/docker-compose-ingestor-server.yaml down 2>/dev/null; \
docker compose -f deploy/compose/vectordb.yaml down 2>/dev/null; \
docker compose -f deploy/compose/nims.yaml down 2>/dev/null; \
echo "All RAG services stopped."

Source Documentation

• docs/troubleshooting.md — if services won't stop or containers hang

RAG Troubleshooting

Auto-Triage: Run First

Start with this diagnostic sweep:

echo "=== CONTAINERS ===" && docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" 2>/dev/null | grep -E "(rag|elasticsearch|milvus|seaweedfs|nim|ingest|redis|etcd|embedding|ranking)" | head -25; echo "=== HEALTH ===" && curl -s http://localhost:8081/v1/health?check_dependencies=true 2>/dev/null || echo "RAG_UNREACHABLE"; curl -s http://localhost:8082/v1/health?check_dependencies=true 2>/dev/null || echo "INGESTOR_UNREACHABLE"; echo "=== LOGS ===" && for svc in rag-server ingestor-server nim-llm-ms nemotron-vlm-embedding-ms nemotron-embedding-ms nemotron-ranking-ms elasticsearch seaweedfs; do echo "--- $svc ---"; docker logs --tail 20 "$svc" 2>/dev/null | grep -iE "(error|fail|exception|timeout|oom|permission)" || echo "OK"; done; echo "=== GPU ===" && nvidia-smi 2>/dev/null | head -20 || echo "NO_GPU"; echo "=== DISK ===" && df -h / | tail -1; echo "=== DOCKER_DISK ===" && docker system df 2>/dev/null; echo "=== VOLUMES ===" && docker volume ls --filter "name=^rag-vol-" 2>/dev/null; echo "=== K8S ===" && kubectl get pods -n rag 2>/dev/null | head -20 || echo "NOT_K8S"

Analyze all output, then diagnose and fix. If Auto-Triage doesn't reveal the cause, dig deeper into the specific failing service's logs (docker logs <service> --tail 100 or kubectl logs <pod> -n rag --tail 100).

Confirm with the user before deleting data (volumes, collections, model cache), changing deployment mode, or modifying API keys.

Source Documentation for Detailed Diagnosis

Read these docs to find specific issue descriptions, causes, and fixes:

• docs/troubleshooting.md — primary reference: all common issues with detailed symptoms/fixes
• docs/debugging.md — Pipeline debugging: monitoring deployment, verifying endpoints, tracing requests
• docs/service-port-gpu-reference.md — Complete port/GPU mapping table for all services

Expected Deployment Times

If user reports "deployment is taking too long," compare against these baselines:

If deployment exceeds these times, check NIM container logs: docker logs nim-llm-ms --tail 50 and model cache disk usage: watch -n 10 'du -sh ~/.cache/model-cache/'.

Service Health Endpoints

Read docs/service-port-gpu-reference.md for the complete port/GPU mapping. Quick check:

Kubernetes Monitoring Commands

kubectl get nimcache -n rag
kubectl get pods -n rag
kubectl logs -f <pod-name> -n rag
kubectl get pvc -n rag
kubectl get events -n rag --sort-by='.lastTimestamp'

Pods in ContainerCreating or Init state during model download is expected. Use kubectl get nimcache -n rag -w to watch download progress.

Enable Debug Logging

export LOGLEVEL=DEBUG
docker compose -f deploy/compose/docker-compose-ingestor-server.yaml up -d --no-deps ingestor-server
docker compose -f deploy/compose/docker-compose-rag-server.yaml up -d --no-deps rag-server

---

Symptom-to-Fix Quick Index

Match the symptom from Auto-Triage output, then read docs/troubleshooting.md for the detailed fix. For pipeline debugging steps, read docs/debugging.md.

---

Troubleshooting Checklists

Ingestion Checklist

• [ ] All required containers running (ingestor-server, nv-ingest-ms-runtime, milvus, redis)
• [ ] Vector database accessible (curl http://localhost:9200/_cluster/health for default Elasticsearch, or curl http://localhost:9091/healthz for Milvus)
• [ ] Embedding service healthy (curl http://localhost:9081/v1/health/ready for default VLM embedding, or curl http://localhost:9080/v1/health/ready for text-embed)
• [ ] File format supported and size <= 400 MB
• [ ] Sufficient disk space (df -h /)
• [ ] GPU resources available (nvidia-smi)

Retrieval Checklist

• [ ] RAG server running and healthy
• [ ] LLM service accessible (curl http://localhost:8999/v1/health/ready)
• [ ] Vector database contains data (collection exists with documents)
• [ ] Collection name is correct
• [ ] Query format is valid

Quality Checklist

• [ ] Reranker is enabled and healthy
• [ ] Top-K values are appropriate
• [ ] Collection has sufficient relevant data
• [ ] Query rewriting configured correctly
• [ ] Prompt template appropriate for use case

---

Full Reset

Destroys all data (volumes, images, caches). Confirm with the user before running.

If nothing else works and the user confirms:

cd "$(git rev-parse --show-toplevel)"
docker compose -f deploy/compose/docker-compose-nemo-guardrails.yaml down 2>/dev/null
docker compose -f deploy/compose/observability.yaml down 2>/dev/null
docker compose -f deploy/compose/docker-compose-rag-server.yaml down 2>/dev/null
docker compose -f deploy/compose/docker-compose-ingestor-server.yaml down 2>/dev/null
docker compose -f deploy/compose/vectordb.yaml down 2>/dev/null
docker compose -f deploy/compose/nims.yaml down 2>/dev/null

docker volume ls -q --filter "name=^rag-vol-" | xargs -r docker volume rm
docker system prune -af

Then deploy fresh using the deploy workflow.

Description: <br>

NVIDIA RAG Blueprint — deploy, configure, troubleshoot, and manage RAG pipelines across Docker Compose, Helm, and library deployments. <br>

This skill is ready for commercial/non-commercial use. <br>

Owner

NVIDIA <br>

License/Terms of Use: <br>

Apache 2.0 <br>

Use Case: <br>

Developers and engineers deploying, configuring, troubleshooting, and managing NVIDIA RAG Blueprint pipelines with Docker, Helm, or Python library workflows. <br>

Deployment Geography for Use: <br>

Global <br>

Known Risks and Mitigations: <br>

Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. <br> Mitigation: Review and scan skill before deployment. <br>

Reference(s): <br>

• NVIDIA RAG Blueprint GitHub <br>
• Deployment Guide <br>
• Troubleshooting <br>
• Shutdown <br>
• Agentic RAG <br>
• Guardrails <br>
• Models and Infrastructure <br>
• Search and Retrieval <br>
• Observability <br>
• MCP Server and Client <br>

Skill Output: <br>

Output Type(s): [Shell commands, Configuration instructions, Diagnostic analysis] <br> Output Format: [Markdown with inline bash code blocks] <br> Output Parameters: [1D] <br> Other Properties Related to Output: [None] <br>

Evaluation Tasks: <br>

NVSkills-Eval 3-tier evaluation (external profile): 9 static validation checks (Tier 1) and 2 deduplication checks (Tier 2). Tier 3 live agent evaluation not available in this report. <br>

Evaluation Metrics Used: <br>

Reported benchmark dimensions: <br>

• Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. <br>
• Correctness: Checks whether the agent follows the expected workflow and produces the correct final output. <br>
• Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant. <br>
• Effectiveness: Checks whether the agent performs measurably better with the skill than without it. <br>
• Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work. <br>

Skill Version(s): <br>

2.6.0 (source: frontmatter) <br>

Ethical Considerations: <br>

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>

(For Release on NVIDIA Platforms Only) <br> Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns here. <br>