
Tao Generate Referring Expressions
- 1.4k installs
- 2.8k repo stars
- Updated August 4, 2026
- nvidia/skills
tao-generate-referring-expressions is an NVIDIA TAO agent skill that generates referring-expression datasets and model workflows for developers who need to train vision-language models that ground natural language phrase
About
tao-generate-referring-expressions is an agent skill in the NVIDIA/skills TAO Toolkit catalog that plans and executes referring-expression dataset generation for vision-language grounding models. Through the TAO Execution SDK, a coding agent reads the SKILL.md contract, collects image corpora and annotation requirements, and dispatches TAO data workflows that pair natural language referring expressions with bounding regions or masks. The skill sits in the TAO Toolkit product covering 100+ pretrained vision AI models exported for edge or cloud inference. Developers reach for tao-generate-referring-expressions when building datasets for region-grounded VLMs—such as visual search, robotics perception, or agent tools that map phrases like the red cup to pixels. Use it after gap analysis surfaces missing grounding labels or when bootstrapping a new referring-expression corpus from raw images.
- NVIDIA TAO toolkit usage
- Referring expression dataset generation
- Vision-language grounding
- Training pipeline setup
- Computer vision NLP bridge
Tao Generate Referring Expressions by the numbers
- 1,449 all-time installs (skills.sh)
- +23 installs in the week ending Aug 5, 2026 (Skillselion tracking)
- Ranked #186 of 2,064 Data Science & ML skills by installs in the Skillselion catalog
- Data as of Aug 5, 2026 (Skillselion catalog sync)
npx skills add https://github.com/nvidia/skills --skill tao-generate-referring-expressionsAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 1.4k |
|---|---|
| repo stars | ★ 2.8k |
| Last updated | August 4, 2026 |
| Repository | nvidia/skills ↗ |
How do you build referring-expression datasets for VLM grounding?
Generate referring-expression datasets and model workflows with NVIDIA TAO to train vision-language models that ground natural language to image regions.
Who is it for?
Computer-vision engineers building region-grounded VLMs who need TAO-managed referring-expression dataset generation pipelines.
Skip if: Teams with finished referring-expression JSON or COCO-style annotations who only need to launch an existing training config.
When should I use this skill?
User asks to generate referring expressions, build VLM grounding datasets, or create phrase-to-region labels with NVIDIA TAO.
What you get
Referring-expression dataset files and TAO workflow configuration ready for vision-language grounding model training.
- referring-expression dataset
- TAO workflow configuration
By the numbers
- Listed in NVIDIA/skills TAO Toolkit row covering 100+ pretrained vision AI models
Files
Image Referring Expression Pipeline
Generate referring-expression and grounding annotations from images with KITTI-format bounding box labels. A single VLM (Gemini or any OpenAI-compatible endpoint) runs four steps: per-object region descriptions, holistic image captions, grouped grounding expressions tied to bboxes, and an optional double-check verification pass.
Purpose
Transform (image, KITTI labels) pairs into a unified annotations.jsonl containing rich, grounded referring expressions. The VLM acts as a "teacher" annotator: Steps 0-1 see the image; Step 2 groups Step 0 outputs into grouping phrases with bbox lists; Step 3 (optional) re-examines those bboxes against the image and corrects mismatches.
Pipeline Architecture
Step 0: Region expression ──┐
├──▶ Step 2: Grounding expression ──▶ [Step 3: Double check]
Step 1: Image caption ──────┘ (optional)- Step 0 (region_expr) — VLM emits one short discriminative phrase per KITTI bbox (
bbox_2d,type,color,description). - Step 1 (image_caption) — VLM emits a holistic, location-agnostic scene caption.
- Step 2 (grounding_expr) — VLM groups Step 0 objects into grouping phrases and returns one bbox list per group, optionally using Step 1's caption as extra context.
- Step 3 (double_check) — VLM re-checks each Step 2 bbox against the image; bad matches are removed, slightly-off boxes get tightened.
Steps 0 and 1 run in parallel within a single thread pool (they only depend on the seed records). Each step writes its own step_<N>_*/annotations.jsonl and skips already-processed images on re-run unless workflow.force_reprocess: true.
Instructions
Initial setup
When a user wants to run this pipeline, walk through these steps:
1. Images: Ask for data.image_dir, the directory containing .jpg, .jpeg, or .png images. 2. KITTI labels: Ask for data.kitti_label_dir, the directory containing one .txt label file per image. Each label line must use KITTI format: <type> <truncated> <occluded> <alpha> <bbox_left> <bbox_top> <bbox_right> <bbox_bottom> .... Lines with fewer than 8 fields are silently skipped. Set this even for Step 1-only runs because Steps 0 and 2 require it. 3. Resume from existing annotations: If the user already has a unified annotations.jsonl from a previous run, set data.input_annotations_jsonl to that file instead of seeding from data.image_dir and data.kitti_label_dir. 4. API access: Ask the user which VLM endpoint they want to use. Present these five options and act on the choice: 1. Gemini — set vlm.backend: "gemini"; require GOOGLE_API_KEY (env var or vlm.gemini.api_key). 2. NIM (e.g. https://inference-api.nvidia.com/v1) — set vlm.backend: "openai"; collect base_url, model_name, and api_key. 3. TAO inference microservice (self-hosted, OpenAI-compatible). Confirm whether the server is already running:
- Running — collect
base_url,model_name, and (optionally)api_key; setvlm.backend: "openai". - Not running — guide the user through the
skills/applications/tao-run-inference-serviceskill, which stands up a local TAO inference microservice with an OpenAI-compatible API. Before promising a specific model, checkskills/applications/tao-run-inference-service/references/service.yamlforvalid_network_arch_config_basenames. Once the server is up, collectbase_url,model_name, and (optionally)api_key; setvlm.backend: "openai".
4. vLLM (self-hosted, OpenAI-compatible). Confirm whether the server is already running:
- Running — collect
base_url,model_name, and (optionally)api_key; setvlm.backend: "openai". - Not running — follow references/vllm_server.md to install and launch a vLLM server, then collect
base_url,model_name, and (optionally)api_key; setvlm.backend: "openai".
5. Custom (any other OpenAI-compatible endpoint) — set vlm.backend: "openai"; collect base_url, model_name, and (optionally) api_key.
If the user has no endpoint and does not want to set one up, stop and help resolve API access first. 5. Workflow steps: Choose one of:
- Full pipeline:
["0", "1", "2", "3"] - No caption generation:
["0", "2", "3"], where Step 2 falls back to image-only context - No verification:
["0", "1", "2"] - Custom subset: any supported subset of steps
6. Output format: Choose one of:
jsonl: unified schema onlylegacy: byte-compatible.txt.stepNfiles onlyboth: writes both formats and is the default for downstream tooling
Running the pipeline
The pipeline runs inside the TAO Toolkit container via the auto_label CLI:
auto_label generate -e /path/to/spec.yaml \
results_dir=/results \
image_referring_expression.data.image_dir=/data/images \
image_referring_expression.data.kitti_label_dir=/data/labels \
image_referring_expression.vlm.gemini.api_key=$GOOGLE_API_KEYGenerate a default spec: auto_label default_specs results_dir=/results module_name=auto_label, then set autolabel_type: "image_referring_expression". All fields support Hydra dot-notation overrides on the command line.
See references/configuration.md for the full YAML structure, all parameters, model/endpoint setup, and error patterns.
Recommended pilot workflow
1. Run on 5-10 images with all four steps. 2. Inspect step_0_region_expr/annotations.jsonl — are object types, colors, and discriminating phrases accurate? 3. Inspect step_2_grounding_expr/annotations.jsonl — are objects grouped sensibly, and do bbox coordinates match the described groups? 4. Inspect step_3_double_check/annotations.jsonl — were mismatched bboxes removed or tightened? Are any new errors introduced (rare)? 5. If quality is insufficient, switch the VLM to a stronger model (e.g. gemini-2.5-pro or a larger Qwen3-VL endpoint), raise media_resolution / max_output_tokens, then re-run with workflow.force_reprocess=true. 6. Scale to the full dataset once satisfied.
Configuration
Key configuration fields (full reference in references/configuration.md):
| Field | Default | Description |
|---|---|---|
workflow.steps | ["0","1","2","3"] | Which steps to execute (0=region_expr, 1=image_caption, 2=grounding_expr, 3=double_check) |
workflow.max_workers | 4 | Parallel threads per step (watch API rate limits) |
workflow.force_reprocess | false | Ignore cached per-step outputs and reprocess from scratch |
workflow.output_format | "jsonl" (set to "both" in the default spec) | "jsonl", "legacy", or "both" |
vlm.backend | "gemini" | "gemini" or "openai" (OpenAI-compatible endpoint) |
data.image_dir | required | Directory of input images (.jpg / .jpeg / .png) |
data.kitti_label_dir | required (unless resuming) | Directory of KITTI-format .txt label files |
data.input_annotations_jsonl | "" | Optional pre-seeded annotations.jsonl (skips KITTI seeding) |
Inputs
Two ways to seed the pipeline:
1. Image directory + KITTI labels (default). Set data.image_dir and data.kitti_label_dir. The orchestrator walks the image directory, reads the matching <stem>.txt KITTI file, parses bboxes (fields 0 + 4-7), reads each image's width/height via PIL, and writes a seed_annotations.jsonl to results_dir/. 2. Pre-seeded annotations JSONL (resume / pre-computed regions). Set data.input_annotations_jsonl to a file with one {"image_id", "image_path", "width", "height", "kitti_bboxes": [...]} object per line.
Outputs
All outputs go to results_dir/:
seed_annotations.jsonl— initial per-image records (unlessinput_annotations_jsonlwas supplied).step_0_region_expr/annotations.jsonl— addsregions[](each withbbox/bbox_2d,type,color,description).step_1_image_caption/annotations.jsonl— addscaption(string).step_2_grounding_expr/annotations.jsonl— addsexpressions[](each{text, instances: [{bbox: [x1,y1,x2,y2]}]}).step_3_double_check/annotations.jsonl— same shape as Step 2, with bboxes removed/updated.results_dir/annotations.jsonl— copy of the last completed step's output.- When
workflow.output_formatis"legacy"or"both", each step also writes byte-compatiblestep_<N>_*/labels/<stem>.txt.stepNfiles for the original 2d-data-engine tooling.
Prerequisites
- Container:
nvcr.io/nvidia/tao/tao-toolkit:6.26.3-pyt - API access: At least one VLM endpoint (Gemini API key or OpenAI-compatible endpoint capable of image input)
- PIL / Pillow: Required to read image dimensions during seeding (already present in the TAO container)
Evaluation Report
Evaluation of the tao-generate-referring-expressions skill before publication through NVSkills-Eval.
This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.
Evaluation Summary
- Skill:
tao-generate-referring-expressions - Evaluation date: 2026-06-06
- NVSkills-Eval profile:
external - Environment:
astra-sandbox - Dataset: 1 evaluation tasks
- Attempts per task: 2
- Pass threshold: 50%
- Overall verdict: FAIL
The skill should be reviewed before NVSkills-Eval publication. Skill owners should address the applicable findings below and rerun NVSkills-Eval to refresh this benchmark.
Agents Used
claude-codecodex
Metrics Used
Reported benchmark dimensions:
- Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
- Correctness: checks whether the agent follows the expected workflow and produces the correct final output.
- Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
- Effectiveness: checks whether the agent performs measurably better with the skill than without it.
- Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.
Underlying evaluation signals used in this run:
security(Security): checks for unsafe operations, secret leakage, and unauthorized access.skill_execution(Skill Execution): verifies that the agent loaded the expected skill and workflow.skill_efficiency(Efficiency): checks routing quality, decoy avoidance, and redundant tool usage.accuracy(Accuracy): grades final-answer correctness against the reference answer.goal_accuracy(Goal Accuracy): checks whether the overall user task completed successfully.behavior_check(Behavior Check): verifies expected behavior steps, including safety expectations.token_efficiency(Token Efficiency): compares token usage with and without the skill.
Test Tasks
The benchmark dataset contained 1 evaluation tasks:
- Positive tasks: 1 tasks where the skill was expected to activate.
- Negative tasks: 0 tasks where no skill was expected.
- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred.
Task composition is derived from the evaluation dataset when possible. Entries with expected_skill set are treated as positive skill-activation cases, while entries with expected_skill: null are treated as negative activation cases.
Results
| Dimension | Num | claude-code | codex |
|---|---|---|---|
| Security | 2 | 100% (+0%) | 100% (+0%) |
| Correctness | 2 | 75% (+65%) | 73% (+73%) |
| Discoverability | 2 | 40% (+40%) | 48% (+48%) |
| Effectiveness | 2 | 94% (+61%) | 84% (+70%) |
| Efficiency | 2 | 43% (+17%) | 62% (+34%) |
Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.
Tier 1: Static Validation Summary
Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 13 total findings.
Top findings:
- MEDIUM SCHEMA/folder_hierarchy: Unexpected nesting depth for general skill (
skills/data/tao-generate-referring-expressions) - MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (
skills/data/tao-generate-referring-expressions/SKILL.md) - MEDIUM SECURITY/Unknown (SQP-2): The skill collects API keys (GOOGLE_API_KEY, OpenAI-compatible keys) and transmits them as part of VLM endpoint configur (
SKILL.md:57) - MEDIUM SECURITY/Unknown (SQP-2): The configuration reference shows API keys being stored in plaintext YAML files (e.g.,
vlm.gemini.api_key,vlm.openai (references/configuration.md:14`) - MEDIUM SECURITY/Unknown (SQP-2): The skill configures Gemini as the default VLM backend and will transmit user-provided images (and potentially sensitive (
references/skill_info.yaml:30)
Tier 2: Deduplication Summary
Tier 2 validation reported findings. NVSkills-Eval ran 2 checks and found 1 total findings.
Top findings:
- HIGH DUPLICATE/duplicate: Duplicate content found within SKILL.md:
"# Image Referring Expression Pipeline" in SKILL.md (lines 1-4) vs "## Purpose" in SKILL.md (lines 5-8) (SKILL.md:1)
[
{
"id": "tao-generate-referring-expressions-basic",
"question": "A user request: \"Generate referring-expression annotations from images with KITTI labels.\" Identify which TAO skill applies and, reading only that skill's documentation, outline the steps it prescribes. Do NOT run any commands, scripts, web searches, or other tools \u2014 describe the plan only.",
"expected_skill": "tao-generate-referring-expressions",
"expected_script": null,
"ground_truth": "Identify tao-generate-referring-expressions as the applicable skill and summarize its documented workflow from SKILL.md without executing anything.",
"expected_behavior": [
"Identifies tao-generate-referring-expressions as the relevant skill",
"Outlines the documented workflow steps from SKILL.md",
"Does not run commands, scripts, or web searches"
]
}
]
Image Referring Expression — Full Configuration Reference
Complete YAML Structure
Generate a default experiment spec with auto_label default_specs results_dir=/results module_name=auto_label, then set autolabel_type: "image_referring_expression".
results_dir: ??? # Required — output directory
autolabel_type: "image_referring_expression"
image_referring_expression:
# --- VLM (vision-language model, used for all four steps) ---
vlm:
backend: "gemini" # "gemini" or "openai"
gemini:
api_key: "" # Or set GOOGLE_API_KEY env var
model: "gemini-3.1-flash-lite-preview"
media_resolution: "MEDIA_RESOLUTION_HIGH" # LOW / MEDIUM / HIGH
temperature: 0.3
max_output_tokens: 8192
timeout: 120
openai: # For OpenAI-compatible endpoints (NIM, vLLM, etc.)
api_key: ""
base_url: "" # e.g. "https://inference-api.nvidia.com/v1" — no /chat/completions
model_name: "" # e.g. "Qwen/Qwen3-VL-235B-A22B-Instruct"
temperature: 0.7
max_tokens: 4096
timeout: 60
# --- Workflow ---
workflow:
steps: ["0", "1", "2", "3"] # 0=region_expr, 1=image_caption, 2=grounding_expr, 3=double_check
max_workers: 4 # Parallel threads per step
force_reprocess: false # Ignore cached step outputs
output_format: "both" # "jsonl", "legacy", or "both"
# --- Input data ---
data:
image_dir: ??? # Directory of input images (.jpg / .jpeg / .png)
kitti_label_dir: ??? # Directory of KITTI-format .txt label files
input_annotations_jsonl: "" # Optional: pre-seeded annotations.jsonl to resume fromKey Configuration Decisions
| Decision | Config field | Guidance |
|---|---|---|
| Which steps to run | workflow.steps | Start with all (["0","1","2","3"]). Drop "1" to skip the holistic caption (Step 2 falls back to image-only context). Drop "3" for fast iteration without verification. Run ["0"] first when tuning region-description prompts |
| Caption vs no caption | include / exclude "1" | When Step 1 is included, Step 2 receives the holistic caption as extra context. When omitted, Step 2 still runs using the image and Step 0 region descriptions alone |
| VLM provider | vlm.backend | "gemini" for Google Gemini models, "openai" for any OpenAI-compatible endpoint (NIM, vLLM, etc.) |
| Parallelism | workflow.max_workers | Higher = faster but watch API rate limits. Start with 4, drop to 1-2 if you hit 429s. Steps 0 and 1 also run in parallel with each other when both are enabled — this can double the API load on a single endpoint |
| Resume vs restart | workflow.force_reprocess | false reuses each step's existing annotations.jsonl (and labels/ legacy files). Set true to regenerate everything |
| Resume from prior run | data.input_annotations_jsonl | Point at an existing unified annotations.jsonl to skip the KITTI seeding pass |
| Output format | workflow.output_format | "jsonl" for the unified schema only; "legacy" for byte-compatible 2d-data-engine .txt.stepN files only; "both" (recommended) emits both |
| Image resolution | vlm.gemini.media_resolution | Use MEDIA_RESOLUTION_HIGH for accurate bbox-to-object matching in Steps 0/2/3. Lower resolutions are cheaper but degrade localization |
| Output truncation | vlm.gemini.max_output_tokens / vlm.openai.max_tokens | Step 0 (one entry per object) and Step 2 (one line per group) can be long; raise this if you see parse failures |
Model / Endpoint Configuration
Gemini (default)
Set the API key via environment variable or config:
export GOOGLE_API_KEY=your_key_hereOr in the YAML: image_referring_expression.vlm.gemini.api_key: "your_key".
Recommended model assignments:
- For all steps:
gemini-2.5-flash(fast, good enough for most images) orgemini-2.5-pro(better at small / cluttered objects and at the Step 3 verification pass).
Temperature guidance:
- Region & grounding (Steps 0, 2, 3): 0.2-0.3 for stable, factual output.
- Caption (Step 1): 0.3-0.5 for slightly more natural phrasing.
OpenAI-compatible endpoints
For NVIDIA Inference API, vLLM-served Qwen3-VL, NIM endpoints, etc.:
image_referring_expression:
vlm:
backend: "openai"
openai:
base_url: "https://inference-api.nvidia.com/v1" # no /chat/completions
model_name: "gcp/google/gemini-3-flash-preview"
api_key: "your_key"
temperature: 0.3
max_tokens: 8192For self-hosted models, the pipeline accepts any endpoint that speaks the OpenAI chat-completions API. Two common ways to provision one:
1. `skills/applications/tao-run-inference-service` skill — workflow for standing up a TAO inference microservice locally. Should support Cosmos, Qwen, and Gemma. Check that skill's references/service.yaml valid_network_arch_config_basenames for the current model list. 2. Bring-your-own deployment — vLLM, NIM, or any other OpenAI-compatible server.
image_referring_expression:
vlm:
backend: "openai"
openai:
base_url: "http://localhost:8000/v1"
model_name: "Qwen/Qwen3-VL-8B-Instruct" # must match vLLM --served-model-name
api_key: "EMPTY" # vLLM ignores it but the SDK requires non-null
temperature: 0.3
max_tokens: 4096
timeout: 300All Parameters
| Parameter | Default | Description |
|---|---|---|
workflow.steps | ["0","1","2","3"] | Pipeline steps to execute (0=region_expr, 1=image_caption, 2=grounding_expr, 3=double_check) |
workflow.max_workers | 4 | Thread pool size for parallel API calls within each step |
workflow.force_reprocess | false | Ignore cached step outputs and reprocess from scratch |
workflow.output_format | "jsonl" (set to "both" in the default spec) | Output format: "jsonl", "legacy", or "both" |
vlm.backend | "gemini" | VLM backend: "gemini" or "openai" |
vlm.gemini.api_key | "" | Gemini API key (or set GOOGLE_API_KEY env var) |
vlm.gemini.model | "gemini-3.1-flash-lite-preview" | Gemini model name |
vlm.gemini.media_resolution | "MEDIA_RESOLUTION_HIGH" | Image resolution sent to Gemini (LOW/MEDIUM/HIGH) |
vlm.gemini.temperature | 0.3 | VLM sampling temperature |
vlm.gemini.max_output_tokens | 8192 | Maximum tokens in Gemini response |
vlm.gemini.timeout | 120 | Request timeout in seconds |
vlm.openai.api_key | "" | API key for OpenAI-compatible endpoint |
vlm.openai.base_url | "" | Base URL of the OpenAI-compatible endpoint (no /chat/completions suffix) |
vlm.openai.model_name | "" | Model name to send in the OpenAI request |
vlm.openai.temperature | 0.7 | OpenAI-compatible sampling temperature |
vlm.openai.max_tokens | 4096 | Maximum tokens in the OpenAI-compatible response |
vlm.openai.timeout | 60 | Request timeout in seconds |
data.image_dir | (required) | Directory of input images (.jpg / .jpeg / .png) |
data.kitti_label_dir | (required unless resuming) | Directory of KITTI-format .txt label files (one per image, matched by stem) |
data.input_annotations_jsonl | "" | Optional unified annotations.jsonl to seed the pipeline; bypasses KITTI seeding |
Input KITTI Label Schema
One line per object, space-separated, with at least 8 fields. Lines with fewer than 8 fields are silently skipped.
<type> <truncated> <occluded> <alpha> <bbox_left> <bbox_top> <bbox_right> <bbox_bottom> [<height> <width> <length> <x> <y> <z> <rotation_y> <score>]<type>(field 0) — string class name (car,pedestrian,truck, ...). Used by Step 0 prompts and Step 2 grouping.<bbox_left> <bbox_top> <bbox_right> <bbox_bottom>(fields 4-7) — pixel-space[x1, y1, x2, y2].- All remaining fields are accepted but ignored.
Bboxes are normalized to a 0-1000 coordinate scale before being sent to the VLM in Step 0 and Step 2, then converted back to pixel coordinates in the output records.
Resume Schema (data.input_annotations_jsonl)
When supplied, each line must have at least:
{
"image_id": "stem-of-image-filename",
"image_path": "/abs/or/relative/path/to/image.jpg",
"width": 1920,
"height": 1080,
"kitti_bboxes": [[x1, y1, x2, y2, "type"], ...],
"source": "image_referring_expression",
"pipeline_steps": []
}This is exactly the format produced by the default seeding pass (results_dir/seed_annotations.jsonl).
Output Layout
results_dir/
├── seed_annotations.jsonl # initial per-image records (skipped if resuming)
├── annotations.jsonl # copy of the last completed step's output
├── step_0_region_expr/
│ ├── annotations.jsonl # adds regions[] to each record
│ └── labels/<stem>.txt.step0 # legacy 2d-data-engine format (when output_format != "jsonl")
├── step_1_image_caption/
│ ├── annotations.jsonl # adds caption to each record
│ └── labels/<stem>.txt.step1
├── step_2_grounding_expr/
│ ├── annotations.jsonl # adds expressions[] to each record
│ └── labels/<stem>.txt.step2
└── step_3_double_check/
├── annotations.jsonl # expressions[] with bboxes removed/updated
└── labels/<stem>.txt.step3Each output record carries:
image_id,image_path,width,height— preserved from the seed.kitti_bboxes— original parsed KITTI rows ([x1, y1, x2, y2, type]).regions[](after Step 0) —{bbox: [x1,y1,x2,y2], bbox_2d: [..], type, color, description}per object.caption(after Step 1) — holistic, location-agnostic scene caption.expressions[](after Step 2 and updated by Step 3) —{text, instances: [{bbox: [x1,y1,x2,y2]}, ...]}.pipeline_steps[]— list of step names that have processed this record.source— set to"image_referring_expression".
Error Patterns
| Error | Cause | Fix |
|---|---|---|
GOOGLE_API_KEY not set | Gemini API key missing | export GOOGLE_API_KEY=your_key or set image_referring_expression.vlm.gemini.api_key in the YAML |
| 429 / rate limit errors | Too many parallel API calls (made worse when Steps 0+1 run in parallel against the same endpoint) | Reduce workflow.max_workers, or use different endpoints for Steps 0 and 1 |
image_referring_expression: no input records | data.image_dir is empty / not a directory, and no input_annotations_jsonl was supplied | Confirm data.image_dir exists and contains .jpg / .jpeg / .png files |
Step 0 produces empty regions[] for every image | KITTI label files missing or malformed (each line needs at least 8 space-separated fields) | Verify data.kitti_label_dir and that each <stem>.txt matches the image stem; check the first few label lines |
failed to build query: 'type' warning in Step 2 | KITTI line missing the type field (field 0) | Fix the offending label file; lines with fewer than 8 fields are silently skipped |
| Truncated / unparseable VLM output (Step 0 or Step 2) | Response cut off before the end of the array / before all group lines were emitted | Raise vlm.gemini.max_output_tokens / vlm.openai.max_tokens; lower temperature; for very large images split into smaller batches |
| Step 2 grouping looks wrong even though Step 0 was good | VLM cannot localize at the requested resolution | Raise media_resolution to MEDIA_RESOLUTION_HIGH; consider a stronger model |
| Step 3 introduces new errors | Verification model is too aggressive | Disable Step 3 (drop "3" from workflow.steps) or switch to a stronger model |
| Re-runs skip everything | Each step's annotations.jsonl already exists | Set image_referring_expression.workflow.force_reprocess=true to regenerate |
Legacy .txt.stepN files missing | workflow.output_format is "jsonl" | Set workflow.output_format=both (or legacy) |
Unknown autolabel_type | YAML missing or wrong autolabel_type | Set autolabel_type: "image_referring_expression" at the top of the spec |
network_arch: tao-generate-referring-expressions
type: data
container_image: tao_toolkit.pyt
gpu_spec_key: null
required_credentials: []
actions:
generate:
command: auto_label generate -e {config_path}
config_format: yaml
mode: args
inputs:
image-dir:
type: folder
kitti-label-dir:
type: folder
input-annotations-jsonl:
type: file
outputs:
results-dir:
type: folder
args:
results_dir: '{results_dir}'
image_referring_expression.data.image_dir: '{image_dir}'
image_referring_expression.data.kitti_label_dir: '{kitti_label_dir}'
image_referring_expression.data.input_annotations_jsonl: '{input_annotations_jsonl}'
image_referring_expression.vlm.backend: '{vlm_backend}'
image_referring_expression.workflow.steps: '{steps}'
image_referring_expression.workflow.output_format: '{output_format}'
defaults:
vlm_backend: gemini
steps: '["0","1","2","3"]'
output_format: both
input_annotations_jsonl: ''
tags:
- image
- referring-expression
- grounding
- kitti
- bounding-boxes
- phrase-grounding
- vlm
- auto-label
description: Four-step image referring-expression pipeline that turns images plus
KITTI bounding-box labels into region descriptions, scene captions, grouped grounding
expressions, and (optionally) verified expressions via VLM distillation.
vLLM Server Setup for Vision-Language Models
This guide walks the user through standing up a self-hosted vLLM server that exposes an OpenAI-compatible /v1/chat/completions endpoint for vision-language models (VLMs). Once the server is running, point the pipeline at it with vlm.backend: "openai" and the matching base_url / model_name / api_key values.
1. Prerequisites
- NVIDIA GPU(s) with enough VRAM for the chosen VLM (≥24 GB for 7-8B-class models, ≥80 GB for 32B+ models; tensor-parallel across multiple GPUs is supported via
--tensor-parallel-size). - A recent NVIDIA driver and
nvidia-container-toolkit(for the recommended Docker path). - Docker (recommended). Python ≥ 3.10 is only needed for the optional host install path.
- For gated HuggingFace repos: a
HF_TOKENwith access to the model.
2. Install vLLM
Default: Docker (Option A). Use it unless you have a specific reason to install on the host — Docker pins a known-good CUDA/PyTorch/vLLM combination and avoids local environment drift. The pip path (Option B) is provided for advanced users who need to patch vLLM, debug locally, or run on a system where Docker is not available.
Option A — Docker (default, recommended)
docker pull vllm/vllm-openai:latestOption B — pip (host install, advanced)
pip install --upgrade "vllm>=0.7.0"3. Pick a VLM checkpoint
vLLM supports many vision models; the pipelines just need an OpenAI-compatible chat-completions endpoint that accepts image inputs. Common picks:
| Model | HuggingFace repo | Notes |
|---|---|---|
| Qwen3-VL-8B-Instruct | Qwen/Qwen3-VL-8B-Instruct | Newer Qwen3 family |
| Qwen3-VL-235B-A22B-Instruct | Qwen/Qwen3-VL-235B-A22B-Instruct | MoE; requires serious hardware |
If the chosen repo is gated, accept its license on the HuggingFace web UI first, then export HF_TOKEN=<your_token> before launching.
4. Launch the server
Use the launcher that matches the install path. Both commands listen on 0.0.0.0:8000 and serve an OpenAI-compatible API at /v1. Prefer Option A (Docker) unless you installed via pip in Section 2.
Option A — Docker (default, recommended)
docker run --runtime nvidia --gpus all \
-p 8000:8000 \
-e HF_TOKEN=<your_hf_token> \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model Qwen/Qwen3-VL-8B-Instruct \
--served-model-name Qwen/Qwen3-VL-8B-Instruct \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.9 \
--media-io-kwargs '{"video": {"num_frames": -1, "fps": -1}}'Option B — pip install (host launch, advanced)
export HF_TOKEN=<your_hf_token> # only required for gated models
vllm serve Qwen/Qwen3-VL-8B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--dtype bfloat16 \
--served-model-name Qwen/Qwen3-VL-8B-Instruct \
--tensor-parallel-size 1 \
--max-model-len 32768 \
--gpu-memory-utilization 0.9 \
--media-io-kwargs '{"video": {"num_frames": -1, "fps": -1}}'Key flags:
--served-model-name <NAME>— value to use later forvlm.openai.model_name. Defaults to the full HF repo path if omitted.--tensor-parallel-size <N>— number of GPUs to shard the model across.--max-model-len <N>— context window; image tokens count against this, so leave headroom for multi-image prompts.--limit-mm-per-prompt image=<N>— max images per request. Bump it if a prompt sends multiple images.--gpu-memory-utilization <0..1>— lower (e.g.0.85) if you hit OOM at load time.
The first launch downloads weights to ~/.cache/huggingface; expect several minutes for 7B+ models.
5. Verify the endpoint
Wait for the log line Uvicorn running on http://0.0.0.0:8000, then sanity-check the server.
List served models:
curl http://localhost:8000/v1/modelsSend a minimal vision request:
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen3-VL-8B-Instruct",
"messages": [{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/640px-Cat03.jpg"}},
{"type": "text", "text": "Describe this image in one sentence."}
]
}],
"max_tokens": 128
}'A non-empty choices[0].message.content in the response confirms the server is ready.
6. Wire the server into the pipeline
Once verified, collect these three values from the running server and pass them to the pipeline spec:
base_url— e.g.http://localhost:8000/v1(no/chat/completionssuffix). If vLLM runs on another host, usehttp://<host_or_ip>:8000/v1and make sure the port is reachable.model_name— must match--served-model-nameexactly.api_key— vLLM ignores it but the OpenAI SDK requires a non-null string; use"EMPTY"if no auth is configured.
YAML snippet:
vlm:
backend: "openai"
openai:
base_url: "http://localhost:8000/v1"
model_name: "Qwen3-VL-8B-Instruct"
api_key: "EMPTY"
temperature: 0.3
max_tokens: 4096
timeout: 300Common issues
| Symptom | Fix |
|---|---|
CUDA out of memory on startup | Lower --max-model-len, drop --gpu-memory-utilization (e.g. 0.85), pick a smaller model, or raise --tensor-parallel-size to spread across more GPUs |
Model architectures ['…'] are not supported | Upgrade vLLM (pip install -U vllm) or use a newer Docker tag — VLM support changes per release |
| 401 / 403 during HuggingFace download | Set HF_TOKEN in the launch env and accept the model's license on the HuggingFace web UI |
| First request hangs for minutes | The model is still warming up — wait for the Uvicorn running log line and a successful GET /v1/models |
image is too large / token overflow | Pre-resize images before sending, or raise --max-model-len |
| Empty / truncated responses | Raise vlm.openai.max_tokens in the pipeline spec; lower temperature for more deterministic output |
Description: <br>
Four-step image referring-expression pipeline: turns images plus KITTI bounding-box labels into region descriptions, scene captions, grounded referring expressions, and (optionally) verified expressions via VLM distillation. <br>
This skill is ready for commercial/non-commercial use. <br>
Owner
NVIDIA <br>
License/Terms of Use: <br>
Apache 2.0 <br>
Use Case: <br>
Developers and engineers who need to generate rich, grounded referring-expression annotations from images with KITTI-format bounding-box labels for training or evaluating object-grounding and referring-expression models. <br>
Deployment Geography for Use: <br>
Global <br>
Known Risks and Mitigations: <br>
Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. <br> Mitigation: Review and scan skill before deployment. <br>
Reference(s): <br>
- Configuration Reference <br>
- vLLM Server Setup <br>
- Skill Info <br>
Skill Output: <br>
Output Type(s): [Files, Shell commands, Configuration instructions] <br> Output Format: [JSONL annotations with per-step outputs and optional legacy text format] <br> Output Parameters: [1D] <br> Other Properties Related to Output: [Per-step outputs cached and resumable; supports jsonl, legacy, or both output formats] <br>
Evaluation Agents Used: <br>
- Claude Code (
claude-code) <br> - Codex (
codex) <br>
Evaluation Tasks: <br>
Evaluated against 1 evaluation task (1 positive skill-activation case) with 2 attempts per task in the NVSkills-Eval external profile. <br>
Evaluation Metrics Used: <br>
Reported benchmark dimensions: <br>
- Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. <br>
- Correctness: Checks whether the agent follows the expected workflow and produces the correct final output. <br>
- Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant. <br>
- Effectiveness: Checks whether the agent performs measurably better with the skill than without it. <br>
- Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work. <br>
Underlying evaluation signals used in this run: <br>
security: Checks for unsafe operations, secret leakage, and unauthorized access. <br>skill_execution: Verifies that the agent loaded the expected skill and workflow. <br>skill_efficiency: Checks routing quality, decoy avoidance, and redundant tool usage. <br>accuracy: Grades final-answer correctness against the reference answer. <br>goal_accuracy: Checks whether the overall user task completed successfully. <br>behavior_check: Verifies expected behavior steps, including safety expectations. <br>token_efficiency: Compares token usage with and without the skill. <br>
Evaluation Results: <br>
| Dimension | Num | claude-code | codex |
|---|---|---|---|
| Security | 2 | 100% (+0%) | 100% (+0%) |
| Correctness | 2 | 75% (+65%) | 73% (+73%) |
| Discoverability | 2 | 40% (+40%) | 48% (+48%) |
| Effectiveness | 2 | 94% (+61%) | 84% (+70%) |
| Efficiency | 2 | 43% (+17%) | 62% (+34%) |
Skill Version(s): <br>
0.1.0 (source: frontmatter) <br>
Ethical Considerations: <br>
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>
(For Release on NVIDIA Platforms Only) <br> Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns here. <br>
{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAidGFvLWdlbmVyYXRlLXJlZmVycmluZy1leHByZXNzaW9ucyIsCiAgICAgICJkaWdlc3QiOiB7CiAgICAgICAgInNoYTI1NiI6ICJmZjU5MmY1ODlkNTE4OTEzZGYxZDM4ZWUyNjlmYmUyNGZjOTI2OWEzZDNlOGJkZmM5NWU4MjQyZWZkMTA1MzU3IgogICAgICB9CiAgICB9CiAgXSwKICAicHJlZGljYXRlVHlwZSI6ICJodHRwczovL21vZGVsX3NpZ25pbmcvc2lnbmF0dXJlL3YxLjAiLAogICJwcmVkaWNhdGUiOiB7CiAgICAicmVzb3VyY2VzIjogWwogICAgICB7CiAgICAgICAgIm5hbWUiOiAiQkVOQ0hNQVJLLm1kIiwKICAgICAgICAiZGlnZXN0IjogIjUxMzkzZWVjOTM4ODY0NmM4Y2ZlOWEyNmIzYWJlNGIyOTNkZTdlNTRiN2MzMGQzYTUwNjc5ZTQ0ZTAyYWJlM2YiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiU0tJTEwubWQiLAogICAgICAgICJkaWdlc3QiOiAiYjMxMDM3MDNmMjQyNTYxOWMyNTlkYmQxNGM1MjljNmViNmY4MWJjNmE4MmEyMjViNzdmODhkMTgzODdlZDVmNCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJldmFscy9ldmFscy5qc29uIiwKICAgICAgICAiZGlnZXN0IjogImU0OGY3NGJhYWY3ZWNjOGVjOGI0Y2I4N2JlOWU0N2MzOGRlZDQ2ZDYwYTdhYTVlNjJjMWUzNzBmN2ViMTI2ZmUiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy9jb25maWd1cmF0aW9uLm1kIiwKICAgICAgICAiZGlnZXN0IjogIjE5YThkNDVlY2QyNDcyNmYwZDI0ZTlhZGRmOTIxMzk0NTA5ZWQ3MTI0MDRkZmRmYWJkOGFiNWY2ZTdiMTIxMTYiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy9za2lsbF9pbmZvLnlhbWwiLAogICAgICAgICJkaWdlc3QiOiAiODg3MzY4ZjU3NDE3ZjNlZjc4NzZhYTJlN2I2NzBiYzQ5NTViZjUwNmU3MzI0MDgxOGM0ZjJjNTZkNTVkZTUxYSIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJyZWZlcmVuY2VzL3ZsbG1fc2VydmVyLm1kIiwKICAgICAgICAiZGlnZXN0IjogImNjMmNhY2E5N2Y3YWJkYzJiMzgxNDRmNDUxMTg0ZTE1OWJiNjQ4ZDkyMDlmMTM3YWM3MzMzMWYxYzVjNzU4OWMiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAic2tpbGwtY2FyZC5tZCIsCiAgICAgICAgImRpZ2VzdCI6ICJjN2JmYWJjM2M3ZDM0OTFjMjcxNzQ2YTU4NTQzNDI1ZjMyYTg5NDZkY2Q5YjgzNzNiZGJjYjUwNTY0MzQzOTU0IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfQogICAgXSwKICAgICJzZXJpYWxpemF0aW9uIjogewogICAgICAiaGFzaF90eXBlIjogInNoYTI1NiIsCiAgICAgICJpZ25vcmVfcGF0aHMiOiBbCiAgICAgICAgIi5naXQiLAogICAgICAgICIuZ2l0aHViIiwKICAgICAgICAiLmdpdGlnbm9yZSIsCiAgICAgICAgIi5naXRhdHRyaWJ1dGVzIgogICAgICBdLAogICAgICAibWV0aG9kIjogImZpbGVzIiwKICAgICAgImFsbG93X3N5bWxpbmtzIjogZmFsc2UKICAgIH0KICB9Cn0=","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMBU05nwB77yfRRtxANmirreEFfo9gmHMrbhB0pCtA0Ie2+Gf0kC1V1j5gvNw2sMJ4AIxAJrZdiWCGahcnlNjTL8q8nIck4TNP1KYj6bM7UnUVxuS2abYswZN+le61Af+8Motcw==","keyid":""}]}}Related skills
How it compares
Pick tao-generate-referring-expressions over tao-generate-image-grounding when the task is phrase-level referring expressions rather than general image grounding annotations.
FAQ
What does tao-generate-referring-expressions produce?
tao-generate-referring-expressions builds referring-expression datasets and TAO workflows that link natural language descriptions to image regions, preparing inputs for vision-language grounding fine-tunes.
Which models use tao-generate-referring-expressions output?
tao-generate-referring-expressions feeds NVIDIA TAO vision-language grounding pipelines that map phrases to bounding regions or masks for detection, segmentation, or agent perception features.
How is tao-generate-referring-expressions invoked?
tao-generate-referring-expressions is invoked through a coding agent that reads its SKILL.md contract and dispatches dataset generation jobs via the NVIDIA TAO Execution SDK.