Vss Deploy Detection Tracking 2d

Purpose

Deploy, debug, and operate the RTVI-CV detection / tracking 2D microservice and drive its REST API.

Prerequisites

• Active VSS deployment reachable on $HOST_IP (see vss-deploy-profile and references/).
• NGC credentials in $NGC_CLI_API_KEY and $NVIDIA_API_KEY for any image pulls.
• curl, jq, and Docker available on the caller.

Instructions

Follow the routing tables and step-by-step workflows below. Each section that ends in workflow, quick start, or flow is intended to be executed top-to-bottom. Detailed reference material lives in references/ and helper scripts live in scripts/ — call them via run_script when the skill points to a script by name.

Examples

Worked end-to-end examples are kept under evals/ (each *.json manifest contains a runnable scenario) and inline in the per-workflow curl blocks below. Run a Tier-3 evaluation with nv-base validate <this-skill-dir> --agent-eval to replay them.

Limitations

• Requires the matching VSS profile / microservice to be deployed and reachable from the caller.
• NGC-hosted models and NIMs may be subject to rate-limits, GPU memory requirements, and license restrictions.
• Concurrency, GPU memory, and storage limits depend on the host hardware and the profile's compose file.

Troubleshooting

• Error: REST call returns connection refused. Cause: target microservice not running. Solution: probe /docs or /health; redeploy via vss-deploy-profile or the matching vss-deploy-* skill.
• Error: HTTP 401/403 from NGC pulls. Cause: missing/expired NGC_CLI_API_KEY. Solution: docker login nvcr.io and re-export the key before retrying.
• Error: container OOM or model fails to load. Cause: insufficient GPU memory for the selected profile. Solution: switch to a smaller variant or free GPUs via docker compose down.

RTVI-CV — Detection & Tracking (Unified Skill)

Unified skill for the Real Time Video Intelligence CV (RTVI-CV) microservice. Two action surfaces in one skill:

• Deploy / operate / debug / tear down the RTVI-CV container locally → see `references/deploy-vss-detection-tracking-2d.md`
• Call the RTVI-CV REST API (streams, health, metrics, embeddings) on a running instance → see `references/usage-vss-detection-tracking-2d.md`

Service: rtvi-cv (metropolis_perception_app)

Image: nvcr.io/<org>/<repo>:<tag> — user-supplied at deploy time

REST port: 9000 (/api/v1 — /live, /ready, /startup, /metrics, /stream/add, /stream/remove, embeddings)

Hardware: x86/aarch64 dGPU (T4, A100, L40, H100, B200, RTX), SBSA (Spark, Grace-Hopper), Jetson (Thor, Orin, Xavier)

---

Action routing — pick once per invocation

Selection rule: match the user's phrasing against the table above and immediately load the corresponding reference file. Do not mix the flows — DEPLOY assumes no running container yet; API USAGE assumes the container is already running on http://<host>:9000.

If intent is genuinely ambiguous (e.g., the user says just "I want to use rtvi-cv"), ask one AskQuestion: deploy a new instance, or call an already-running one?

---

What lives where

vss-deploy-detection-tracking-2d/
├── SKILL.md          # this file (routing + contracts)
├── assets/           # data files (deploy-defaults.yml — single source of truth for tags / refs / paths / GPU)
├── evals/            # Tier-3 eval manifests (deploy-evals.json, usage-evals.json)
├── scripts/          # 23 bash + python helpers (see `scripts/` for the full inventory)
└── references/       # workflow runbooks (deploy / api-usage / teardown / troubleshooting / …)

For the full per-file inventory and what each reference covers, see `references/workflow-reference.md`.

All scripts are invoked from the skill root via $SKILL_DIR/scripts/<name> — paths inside the deploy reference doc are preserved verbatim and resolve correctly when the agent runs from skill root.

---

Available Scripts

Helpers live in scripts/ and are invoked from the skill root by name — call each via run_script("scripts/<name>") so the agent records a proper tool invocation.

For the full inventory of helpers (cache, GPU checks, setup) browse scripts/; each script's --help describes its arguments.

How to use this skill

1. Read this file first. It only routes — it does not contain workflows. 2. Match the user's intent against the routing table above. 3. Load exactly one reference doc (DEPLOY or API USAGE). Don't preload both — each reference is large and contains its own full contract. 4. Follow the loaded reference exactly. The reference docs are the byte-for-byte preserved contracts from the predecessor skills vss-deploy-detection-tracking-2d (deploy/teardown/debug) and rtvicv-api (REST API) — every step ordering invariant, bash-batching rule, box-rendering rule, and AskQuestion contract is retained. 5. For DEPLOY, the reference doc enforces its own startup contract: one-line acknowledgement → planning-tool call (TodoWrite array of 5 todos, OR 5 successive TaskCreate calls on newer Claude Code) → Step 1 question. Do not narrate, do not pre-flight, and never print "loading TodoWrite/TaskCreate" or any deferred-tool resolution prose — the planning tool is loaded silently.

---

Output contract — DEPLOY flow

When running the DEPLOY / TEARDOWN / DEBUG flow, the agent MUST honour all four items below on every successful deploy. These are the user's only feedback channel between steps; skipping any of them is a behaviour regression.

1. Render every step's exit in a fixed-width box — Step 1 Deploy targets, Step 2 Pipeline configuration, Step 3 Container, Step 4 Apply configuration, Step 5 Plan + Results. Not just the final summary. The box is the user's step receipt. Geometry is fixed (see § "Universal box format" below). Per-step content rules (what rows go inside each box) live in `references/deploy-vss-detection-tracking-2d.md` under "Step N box content rule". 2. After the Step 5 Results box, issue the Step 6 `AskUserQuestion` from `references/next-steps.md` § "11.c" — never replace it with a free-form Next steps bullet list. The menu is the deploy's exit handle: it lets the user run metrics, manage streams, tail logs, or tear down with one click instead of having to remember curl URLs. 3. After the user picks a Step 6 bucket, issue the follow-up `AskUserQuestion` from `references/next-steps.md` § "11.d" — never substitute prose + ready-to-copy curl examples + a free-text "want me to run X?" question. Each bucket has its own menu of concrete actions; the user picks the action, then the skill emits the API box and runs the curl. Per-bucket follow-ups:

• Manage streams → Add / Remove / List. **Remove builds its

options dynamically from /stream/get-stream-info** — one option per active stream labelled <camera_id> · <camera_url> plus "Remove ALL" when ACTIVE > 1 (full spec: § "remove_streams sub-flow").

• Stop the deployment → Stop app / Stop container / Full teardown.
• Check metrics & FPS → no follow-up; run collect_metrics.sh

directly after printing the /api/v1/metrics API box.

• Check liveness / readiness → no follow-up; probe all three

health endpoints after printing their API boxes. 4. Render the FULL per-step content, not an overview row — rendering the box is necessary but not sufficient. Each step has a row composition spec in `references/deploy-vss-detection-tracking-2d.md` under "Step N box content rule". Step 4 (Apply configuration) is where the agent collapses most often — its canonical per-use-case key list lives in `references/apply-config.md` § "Per-use-case complete edit list", and the agent MUST emit one ✔ [section] key=value — annotation row per key in that table for the active use case + settings. A section with 5 keys → 5 rows; a section with 6 keys → 6 rows. Never one overview row per section.

Forbidden (these are the shortcuts the agent falls back to under pressure, and they break the user's UX):

• ❌ Internal tool-loading narration. Never print "I need to load

TodoWrite (a deferred tool the skill calls for the task widget)", "Loading TaskCreate…", "Calling ToolSearch for the planning tool…", or any other text about resolving / loading / fetching deferred tools. The agent loads tools silently. The user only ever sees the ✔ <pinned-values> summary line followed by the widget — never any scaffolding around tool resolution.

• ❌ **Collapsing all 5 deploy steps into a single TaskCreate's

description field. When `TaskCreate` is the available planning tool, issue 5 separate TaskCreate calls** back-to-back (one per step). See references/task-list.md § "Initial TaskCreate calls" for the verbatim template. Same rule for TodoWrite — one call with all 5 todos in the todos:[…] array; never one todo whose content is a multi-line list.

• ❌ Silently choosing `dynamic` stream-mode. The skill default is

stream_mode=static — the agent bakes auto-discovered file:// URLs into the DS main config's [source-list] block before app start. Switch to dynamic only when the user explicitly asks ("add streams later via REST", "use dynamic stream mode") OR when they pick dynamic in the Step 2 AskQuestion. Picking dynamic for a generic "deploy rtvi-cv with N streams" query breaks the deploy rubric and the user's /metrics expectations. See `references/pipeline-config.md` § "Defaults — the skill is static-mode by default" for the full rationale.

• ❌ A one-line ✔ App ready in Ns, N streams, fps total Y in place of

the Step 5 Results box.

• ❌ ASCII box-drawing chars (+, -, =, *) instead of light

box-drawing chars (┌ ─ ┐ │ └ ┘).

• ❌ Skipping Step 6 on the assumption "the user knows what to do next".
• ❌ After Step 6, dumping a markdown wall of prose + multiple curl

blocks + a closing "want me to run any of these?" — that's the shape the agent falls back to and it bypasses both the 11.d menu and the per-API-call box. The user picks from a menu; the skill shows the resolved API box; the skill runs it. No free-text Q.

• ❌ Step 4 overview collapses — these are explicitly banned by the

deploy doc's Step 4 content rule:

• ✔ Batch size 3 (tile grid: 1×3) → required: 5 separate rows

([streammux] batch-size=3, [primary-gie] batch-size=3, [source-list] max-batch-size=3, [tiled-display] rows=1, [tiled-display] columns=3).

• ✔ Output sink eglsink → required: one row per sink key

(4 keys for eglsink, e.g. [sink0] enable=1, type=2, sync=0, qos=0 — read apply-config.md for the exact list).

• ✔ Sources static (3 streams, http-port=9000) → required: six

annotated [source-list] rows.

• ✔ Tile grid 1 row × 3 cols (single row) → required: two

rows, [tiled-display] rows=1 and [tiled-display] columns=3.

Universal box format

The geometry contract for every step-exit box (Step 1 through Step 5 Results). The same shape across every box; only the title and the body rows change per step.

• Width: 128 chars corner-to-corner — ┌ at column 1, ┐ at

column 128. Wider terminals leave the box flush-left; do not stretch it. Inner content area is 124 chars (with one space margin on each side inside the │ borders).

• Light box-drawing chars only: ┌ ─ ┐ │ └ ┘. No +, -, =,

* ASCII fallbacks.

• Top border — title CENTERED: ┌ + N₁ dashes + ␣ + title + ␣

+ N₂ dashes + ┐, where N₁ + N₂ + len(title) + 2 = 126. Distribute the pad: N₁ = floor((126 − len(title) − 2) / 2), N₂ = 126 − len(title) − 2 − N₁. N₁ and N₂ differ by at most 1.

• Body: one │ <content padded to inner-content 124> │ per fact.

Each fact line uses the ✔ <key-padded-to-13> <value> form (two spaces in, glyph, key right-padded to 13, two spaces, value).

• Blank lines between groups: render │ <124 spaces> │ between

logical groups (e.g. Identity / Model / Videos in Step 1) so the user can scan the box at a glance.

• Bottom border: └ + 126 dashes + ┘ — solid border, no title.

Standard step titles (used at the top of each step's box):

┌─────────────────────────────────────────────────────── Deploy targets ───────────────────────────────────────────────────────┐
┌─────────────────────────────────────────────────── Pipeline configuration ───────────────────────────────────────────────────┐
┌───────────────────────────────────────────────────────── Container ──────────────────────────────────────────────────────────┐
┌──────────────────────────────────────────────────── Apply configuration ─────────────────────────────────────────────────────┐
┌──────────────────────────────────────────────── Perception Application — Plan ───────────────────────────────────────────────┐
┌────────────────────────────────────────────── Perception Application — Results ──────────────────────────────────────────────┐

Per-step content rules (which rows go in which box, mode-aware row hiding, the apply-config sectioned layout, the Step 5 PLAN-then-RESULT pattern, the Step 3 docker run synthesis requirement) live in `references/deploy-vss-detection-tracking-2d.md` under "Step N box content rule" — read those when rendering the corresponding step.

Quick triggers (mnemonic)

bump:1

Evaluation Report

Evaluation of the vss-deploy-detection-tracking-2d skill before publication through NVSkills-Eval.

This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.

Evaluation Summary

• Skill: vss-deploy-detection-tracking-2d
• Evaluation date: 2026-06-08
• NVSkills-Eval profile: external
• Environment: astra-sandbox
• Dataset: 2 evaluation tasks
• Attempts per task: 2
• Pass threshold: 50%
• Overall verdict: FAIL

The skill should be reviewed before NVSkills-Eval publication. Skill owners should address the applicable findings below and rerun NVSkills-Eval to refresh this benchmark.

Agents Used

• claude-code
• codex

Metrics Used

Reported benchmark dimensions:

• Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
• Correctness: checks whether the agent follows the expected workflow and produces the correct final output.
• Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
• Effectiveness: checks whether the agent performs measurably better with the skill than without it.
• Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.

Underlying evaluation signals used in this run:

• security (Security): checks for unsafe operations, secret leakage, and unauthorized access.
• skill_execution (Skill Execution): verifies that the agent loaded the expected skill and workflow.
• skill_efficiency (Efficiency): checks routing quality, decoy avoidance, and redundant tool usage.
• accuracy (Accuracy): grades final-answer correctness against the reference answer.
• goal_accuracy (Goal Accuracy): checks whether the overall user task completed successfully.
• behavior_check (Behavior Check): verifies expected behavior steps, including safety expectations.
• token_efficiency (Token Efficiency): compares token usage with and without the skill.

Test Tasks

The benchmark dataset contained 2 evaluation tasks:

• Positive tasks: 1 tasks where the skill was expected to activate.
• Negative tasks: 1 tasks where no skill was expected.
• Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred.

Task composition is derived from the evaluation dataset when possible. Entries with expected_skill set are treated as positive skill-activation cases, while entries with expected_skill: null are treated as negative activation cases.

Results

Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.

Tier 1: Static Validation Summary

Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 5 total findings.

Top findings:

• MEDIUM QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.author' (skills/vss-deploy-detection-tracking-2d/SKILL.md)
• MEDIUM QUALITY/quality_discoverability: Description contains vague words (skills/vss-deploy-detection-tracking-2d/SKILL.md)
• MEDIUM SCHEMA/author_missing: Author not specified in metadata (skills/vss-deploy-detection-tracking-2d/SKILL.md)
• LOW QUALITY/quality_discoverability: Description very long (366 chars, recommend 50-150) (skills/vss-deploy-detection-tracking-2d/SKILL.md)
• LOW SCRIPT_LINT/magic_numbers: calibration_manager.py contains magic numbers (skills/vss-deploy-detection-tracking-2d/scripts/calibration_manager.py)

Tier 2: Deduplication Summary

Tier 2 validation reported findings. NVSkills-Eval ran 2 checks and found 2 total findings.

Top findings:

• HIGH DUPLICATE/duplicate: Duplicate content found across references/deploy-vss-detection-tracking-2d.md and references/start-app.md and references/ux-conventions.md:

"### Universal box format (every step exit)" in references/deploy-vss-detection-tracking-2d.md (lines 622-627) vs "### Pre-rendered top + bottom borders — COPY VERBATIM" in references/deploy-vss-detection-tracking-2d.md (lines 817-821) vs "### Worked example — warehouse-2d (eglsink + dynamic + cache hit, batch=3)" in references/start-app.md (lines 318-322) vs "## Final deploy receipt — the "Perception Application — Results" box" in references/ux-conventions.md (lines 180-189) (references/deploy-vss-detection-tracking-2d.md:622)

• HIGH DUPLICATE/duplicate: Duplicate content found across references/next-steps.md and references/troubleshooting.md:

"### Bonus quick-checks (liveness / readiness / startup — shown only when explicitly asked)" in references/next-steps.md (lines 305-311) vs "# Readiness — pipeline is ready (after streams attached)" in references/troubleshooting.md (lines 14-15) (references/next-steps.md:305)

RTVI-CV API Reference

Complete endpoint reference for the Real Time Video Intelligence CV (RTVI-CV) microservice REST API.

Base URL: http://<host>:9000 | All endpoints prefixed with /api/v1

---

Endpoints

POST /api/v1/stream/add — Add a new video stream

Request body:

{
  "key": "sensor",
  "value": {
    "camera_id": "<string, required — unique stream identifier>",
    "camera_url": "<string, required — video source URL>",
    "change": "camera_add",
    "camera_name": "<string, optional — display name, defaults to camera_id>",
    "creation_time": "<ISO 8601, optional — only for http/https URLs>",
    "metadata": {
      "resolution": "<string, optional — default '1920 x1080'>",
      "codec": "<string, optional — default 'h264'>",
      "framerate": "<integer, optional — default 30>"
    }
  },
  "headers": {
    "source": "<string, optional — source system>",
    "created_at": "<ISO 8601, optional>"
  }
}

Responses:

curl template:

curl -s -X POST "${BASE_URL}/api/v1/stream/add" \
  -H "Content-Type: application/json" \
  -d '{
    "key": "sensor",
    "value": {
      "camera_id": "${CAMERA_ID}",
      "camera_name": "${CAMERA_NAME}",
      "camera_url": "${CAMERA_URL}",
      "change": "camera_add",
      "metadata": { "resolution": "1920 x1080", "codec": "h264", "framerate": 30 }
    }
  }'

---

POST /api/v1/stream/remove — Remove an existing video stream

Request body:

{
  "key": "sensor",
  "value": {
    "camera_id": "<string, required — must match existing stream>",
    "camera_url": "<string, required — must match URL used when adding>",
    "change": "camera_remove",
    "camera_name": "<string, optional>"
  }
}

Responses:

curl template:

curl -s -X POST "${BASE_URL}/api/v1/stream/remove" \
  -H "Content-Type: application/json" \
  -d '{
    "key": "sensor",
    "value": {
      "camera_id": "${CAMERA_ID}",
      "camera_name": "${CAMERA_NAME}",
      "camera_url": "${CAMERA_URL}",
      "change": "camera_remove"
    }
  }'

---

GET /api/v1/stream/get-stream-info — List active streams

Headers: Accept: application/json (default) or Accept: text/plain (Prometheus)

JSON response shape:

{
  "status": "HTTP/1.1 200 OK",
  "reason": "Stream info retrieved successfully",
  "stream-info": {
    "stream-count": 2,
    "stream-list": [
      {
        "camera_id": "camera_001",
        "camera_name": "Front Door Camera",
        "camera_url": "rtsp://192.168.1.100:554/stream1",
        "source_id": 0,
        "sensor_id": "sensor_0"
      }
    ]
  }
}

curl:

curl -s "${BASE_URL}/api/v1/stream/get-stream-info" -H "Accept: application/json"

---

GET /api/v1/live — Liveness probe

JSON response:

{
  "status": "HTTP/1.1 200 OK",
  "reason": "Application is alive",
  "live-info": { "ds-liveness": "YES" }
}

curl:

curl -s "${BASE_URL}/api/v1/live" -H "Accept: application/json"

---

GET /api/v1/ready — Readiness probe

JSON response:

{
  "status": "HTTP/1.1 200 OK",
  "reason": "Application is ready",
  "ready-info": { "ds-ready": "YES" }
}

curl:

curl -s "${BASE_URL}/api/v1/ready" -H "Accept: application/json"

---

GET /api/v1/startup — Startup probe

JSON response:

{
  "status": "HTTP/1.1 200 OK",
  "reason": "Application has started",
  "startup-info": { "ds-startup": "YES" }
}

curl:

curl -s "${BASE_URL}/api/v1/startup" -H "Accept: application/json"

---

GET /api/v1/metrics — Performance metrics

Headers:

JSON response shape:

{
  "status": "HTTP/1.1 200 OK",
  "reason": "Metrics retrieved successfully",
  "metrics-info": {
    "stream-count": 2,
    "stream-stats": [
      {
        "sensor_id": "sensor_0",
        "sensor_name": "camera_001",
        "source_id": 0,
        "fps": 29.97,
        "frame_number": 1234,
        "latency_ms": 45.2
      }
    ],
    "system-stats": {
      "GPU_gb": 4.5,
      "RAM_gb": 8.2,
      "cpu_util": 45.3,
      "gpu_util": 78.9
    }
  }
}

Prometheus format example:

# HELP fps_metrics FPS metrics from ds
# TYPE fps_metrics gauge
fps_metrics{app_name="ds",metric_name="stream_fps",sensor_id="1",source_id="0"} 29.80
# HELP latency_metrics Latency metrics from ds
# TYPE latency_metrics gauge
latency_metrics{app_name="ds",metric_name="stream_latency_ms",sensor_id="1",source_id="0"} 402.39
# HELP memory_metrics Memory metrics from ds
# TYPE memory_metrics gauge
memory_metrics{app_name="ds",metric_name="system_ram_memory_gb"} 8.40
memory_metrics{app_name="ds",metric_name="system_gpu_memory_gb"} 1.34
# HELP utilization_metrics Utilization metrics from ds
# TYPE utilization_metrics gauge
utilization_metrics{app_name="ds",metric_name="system_gpu_utilization"} 6
utilization_metrics{app_name="ds",metric_name="system_cpu_utilization"} 7.5
# HELP stream_count Stream count from ds
# TYPE stream_count gauge
stream_count{app_name="ds",metric_name="stream_count"} 2

curl (JSON):

curl -s "${BASE_URL}/api/v1/metrics" -H "Accept: application/json"

curl (Prometheus):

curl -s "${BASE_URL}/api/v1/metrics" -H "Accept: text/plain"

curl (with OpenTelemetry):

curl -s "${BASE_URL}/api/v1/metrics" \
  -H "Accept: application/json" \
  -H "X-Refresh-Period: 5000" \
  -H "X-OTLP-URL: http://otel-collector:4318"

---

GET /api/v1/metadata — Service metadata

JSON response:

{
  "version": "1.0.0",
  "sub_version": "a3f5c8d",
  "licenseInfo": {
    "name": "NVIDIA-Proprietary",
    "path": "/opt/mm/LICENSE",
    "url": "file:///opt/mm/LICENSE"
  }
}

curl:

curl -s "${BASE_URL}/api/v1/metadata"

---

POST /api/v1/generate_text_embeddings — Generate text embeddings

Request body:

{
  "text_input": "<string, required — text to embed>",
  "model": "<string, required — e.g. 'cosmos-embed1-448p'>"
}

Responses:

curl:

curl -s -X POST "${BASE_URL}/api/v1/generate_text_embeddings" \
  -H "Content-Type: application/json" \
  -d '{ "text_input": "${TEXT}", "model": "cosmos-embed1-448p" }'

---

Supported Video Protocols

Supported Codecs

h264, h265, hevc, vp8, vp9, av1

Python Helper (stdlib only)

import json, urllib.request

def call_rtvi_api(base_url, method, path, body=None):
    url = f"{base_url}{path}"
    data = json.dumps(body).encode() if body else None
    req = urllib.request.Request(url, data=data, method=method)
    req.add_header("Content-Type", "application/json")
    req.add_header("Accept", "application/json")
    with urllib.request.urlopen(req, timeout=10) as resp:
        return json.loads(resp.read())

Apply Configuration Inside the Container

Detailed bash for Step 4 of the workflow.

ONE-CALL FAST PATH — use this (single permission prompt for all of Step 4)

Refresh scripts THEN call `apply_config.sh` in a single chained bash call. This collapses script copy + chmod + 6 sub-step exec calls into ONE permission prompt:

SKILL_DIR="$HOME/.claude/skills/vss-deploy-detection-tracking-2d"
CONTAINER="<CONTAINER_NAME>"

docker exec "$CONTAINER" rm -rf /tmp/scripts && \
docker cp   "$SKILL_DIR/scripts" "$CONTAINER:/tmp/" && \
docker exec "$CONTAINER" chmod -R +x /tmp/scripts/ && \
docker exec "$CONTAINER" /tmp/scripts/apply_config.sh \
    --usecase  "<usecase>" \
    --batch    "<N>" \
    --sink     "<fakesink|eglsink|filedump>" \
    --stream-mode "<dynamic|static>" \
    [--onnx    "<container-onnx-path>"]    # pass if already resolved in Step 1.g — skips 4.a re-scan
    [--videos  "<container-videos-dir>"]   # pass if already resolved in Step 1.g — skips 4.a re-scan
    [--force-rebuild]                      # bypass engine cache

The script-refresh pattern matters. Use `docker cp scripts

<container>:/tmp/ (no trailing . or /), preceded by rm -rf

/tmp/scripts. The trailing-. form (docker cp scripts/.

<container>:/tmp/scripts/`) nests the files into

/tmp/scripts/scripts/ when /tmp/scripts/ already exists from a

prior session, leaving chmod /tmp/scripts/*.sh matching nothing.

The rm -rf upfront makes the cp deterministic regardless of prior

state.

Output markers to parse:

• RESOLVE_OK: <label>=<path> — 4.a found the asset
• RESOLVE_AMBIGUOUS: <label> count=<N> — ambiguity → the skill must drive an AskQuestion, then re-run with --onnx / --videos flag
• ENGINE_PRELAUNCH: HIT_EXACT|HIT_COMPAT|MISS — 4.f result
• CONFIG_APPLY_OK usecase=<uc> batch=<N> sink=<sink> — all sub-steps done

Auto-co-location for warehouse-3d. When --onnx <path> is supplied and --labels / --anchor are not, the script defaults LABELS and ANCHOR to siblings of the ONNX (labels.txt and the first *.npy in the ONNX's parent dir). This is structurally safe — every warehouse NGC resource ships these three files in the same directory (vss-warehouse-app-data/models/sparse4d/ov/) — and it prevents RESOLVE_AMBIGUOUS: labels count=2 when prior smartcity-rtdetr resources (which also contain a labels.txt) are still cached under /opt/storage/resources/. The canonical Step 4 call from SKILL.md (--onnx ... --videos ...) therefore works for warehouse-3d as-is — explicit --labels / --anchor are only needed for non-NGC layouts.

Parallelism inside the script:

• 4.a (discovery) runs first (path dependency for 4.b-4.e)
• 4.f (engine cache lookup) starts immediately after 4.a and runs in the background — it is read-only and never touches the config files
• 4.b → 4.c → 4.d → 4.e run sequentially (they all write to overlapping files — ds-main-config.txt in particular — so concurrent writes would corrupt the file)
• The script waits for the 4.f background job before printing CONFIG_APPLY_OK

Only fall back to the per-sub-step flow below when debugging a specific sub-step failure, or when RESOLVE_AMBIGUOUS requires the skill to ask the user and retry with an explicit path.

---

Step 4 exit box — required sectioned format

When apply_config.sh returns CONFIG_APPLY_OK …, the agent renders the Step 4 exit box using the sectioned layout below — NOT a flat ✔ list. Each section maps to one sub-step:

• Model ← 4.b (path substitution into PGIE / config.yaml)
• Batch size ← 4.c (update_batch_size.sh)
• Output sink ← 4.d (update_output_sink.sh)
• Stream sources ← 4.e (update_stream_sources.sh)
• Engine cache ← 4.f (prelaunch_nvinfer_engine.sh / setup_gdino.sh / setup_sparse4d.sh)
• Backups ← side-effect of all the above

Use the universal box geometry from SKILL.md § "Universal box format" (128 chars wide, centered title, blank-line separators between sections).

The box is constructed dynamically — the agent reads the actual sub-step output + the per-use-case key table below + the user's chosen settings, then emits one `✔` row per concrete `<section> <key>=<value>` edit, with a plain-English annotation explaining what that key does. Rows are grouped by filename: the basename is a sub-header within each section, then the ✔ rows for that file follow, indented.

Required row form

   <basename>
       ✔ <[section]> <key>=<value>          — short plain-English annotation

The word Edited is never printed — every row inside the box is an edit; the prefix is redundant. The — separator + annotation tells the user what the key actually does (e.g. [sink0] type=2 — turn on EGL display).

Forbidden patterns (what the agent slips into)

Counting rule

Number of ✔ rows in each section MUST equal the row count in the per-use-case table below, given the user's chosen settings.

(Sink count assumes the warehouse-2d / smartcity table below where [sink0] enable + type are folded into one row when both are written together, and [sink0] nvdslogger=1 is rendered as its own row since it's a perf-measurement signal, not part of the sink-mode triple. Either form is fine — the agent picks one row per logical edit.)

If the agent's box doesn't have the exact row count, it collapsed — re-render with one row per key.

---

Per-use-case complete edit list

These tables are the source of truth for what the agent renders in each section. Every row corresponds to one ✔ Edited line in the box.

warehouse-2d

Each row below = one ✔ line in the box. The Annotation column is the canonical plain-English text the agent prints after the — on that row.

Model section (4.b):

Batch size section (4.c):

Output sink section (4.d) — base rows for any sink:

For filedump ALSO (6 extra rows):

Stream sources section (4.e):

warehouse-3d

Model section (4.b) — Sparse4D uses videotemplate, so the model config lives in config.yaml (no PGIE):

Batch size section (4.c) — note: no [primary-gie] (videotemplate), plus two extra files unique to warehouse-3d:

Output sink section (4.d) — same [sink0] [sink2] [tiled-display] [osd] keys as warehouse-2d. Plus, for eglsink ONLY:

Stream sources section (4.e) — same six [source-list] keys as warehouse-2d.

smartcity-rtdetr

Model section (4.b):

Batch size section (4.c):

Output sink section (4.d) — same five keys as warehouse-2d but on run_config-api-rtdetr-protobuf.txt instead of ds-main-config.txt.

Stream sources section (4.e) — same six [source-list] keys but on run_config-api-rtdetr-protobuf.txt.

smartcity-gdino

Model section (4.b) — GDINO uses Triton/nvinferserver, so the model flow goes through setup_gdino.sh (file copy + engine build):

Batch size section (4.c):

Output sink section (4.d) — same five keys as warehouse-2d but on run_config-api-rtdetr-protobuf.txt (the GDINO main config).

Stream sources section (4.e) — same six [source-list] keys on run_config-api-rtdetr-protobuf.txt.

---

Worked example — warehouse-2d (eglsink + static streams + cache hit, batch=3)

┌──────────────────────────────────────────────────── Apply configuration ─────────────────────────────────────────────────────┐
│                                                                                                                              │
│  Model                                                                                                                       │
│     ds-ppl-analytics-pgie-config.yml                                                                                         │
│         ✔ onnx-file = <resolved abs path>                  — pin RT-DETR ONNX                                                │
│                                                                                                                              │
│  Batch size  (value=3, tile grid 1×3)                                                                                        │
│     ds-main-config.txt                                                                                                       │
│         ✔ [streammux] batch-size=3                         — muxer input batch                                               │
│         ✔ [primary-gie] batch-size=3                       — PGIE inference batch                                            │
│         ✔ [source-list] max-batch-size=3                   — source-list capacity                                            │
│         ✔ [tiled-display] rows=1                           — tile grid rows                                                  │
│         ✔ [tiled-display] columns=3                        — tile grid cols                                                  │
│     ds-ppl-analytics-pgie-config.yml                                                                                         │
│         ✔ engine-filename → _b3_                           — engine name follows new batch                                   │
│                                                                                                                              │
│  Output sink  (eglsink — display)                                                                                            │
│     ds-main-config.txt                                                                                                       │
│         ✔ [sink0]  enable=1   type=2                       — turn on EGL display sink                                        │
│         ✔ [sink0]  nvdslogger=1                            — emit per-stream FPS to /api/v1/metrics                          │
│         ✔ [sink2]  enable=0                                — disable file-dump sink                                          │
│         ✔ [tiled-display] enable=1                         — show tile grid (composite)                                      │
│         ✔ [osd]    enable=1                                — draw bbox / labels                                              │
│                                                                                                                              │
│  Stream sources  (static, 3)                                                                                                 │
│     ds-main-config.txt                                                                                                       │
│         ✔ [source-list] num-source-bins=3                  — bake 3 sources into pipeline                                    │
│         ✔ [source-list] list=<3 file:// URLs>              — exact URLs (Camera_01..03)                                      │
│         ✔ [source-list] sensor-id-list=…                   — Camera_01;Camera_02;Camera_03                                   │
│         ✔ [source-list] sensor-name-list=…                 — same as ids                                                     │
│         ✔ [source-list] http-port=9000                     — REST listen port                                                │
│         ✔ [tests] file-loop=1                              — loop videos (eglsink/fakesink)                                  │
│                                                                                                                              │
│  Engine cache                                                                                                                │
│         ✔ HIT_SYMLINK b3 → b4 base   (no rebuild — saved ~3 min)                                                             │
│         ✔ bound model-engine-file = _b3_                                                                                     │
│                                                                                                                              │
│  Backups                                                                                                                     │
│         ✔ *.bak preserved on first edit  (mode 0600)                                                                         │
│                                                                                                                              │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

For warehouse-3d the Batch section grows by:

• config.yaml num_sensors=N — number of cameras for Sparse4D BEV
• ds-mtmc-preprocess-config.txt network-input-shape=N;3;540;960 — preprocess tensor leading dim

And for warehouse-3d + eglsink the Sink section grows by:

• config.yaml generate_3d_bbox=True — render 3D BEV bounding boxes
• $SPARSE4D_REPO/configs/config.yaml generate_3d_bbox=True — same flag for staged copy

For smartcity-gdino the Batch section grows by 5 extra rows:

• config_triton_nvinferserver_gdino.txt max_batch_size=N — Triton nvinferserver batch
• <dir>/config.pbtxt max_batch_size=N — Triton ensemble/<dir> batch (×4 Triton dirs)

If a section has no edits to report (e.g. cache MISS — Engine cache shows will build during launch (~3-5 min) instead of the HIT row), still render the section with one row stating that.

---

Path Setup (for manual sub-step debugging only)

DO NOT stage configs to `/opt/storage/configs/`. Every script in scripts/ (via common.sh's CONFIGS default) edits the configs IN-PLACE at the canonical reference-configs path below. Copying configs into /opt/storage/configs/ and editing them there is dead work — the scripts won't read them, and the app loads from the canonical path. The metropolis_perception_app -c <path> command should always point at the canonical path, not a staged copy.

Every command below assumes these are exported:

export CONFIGS=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/metropolis_perception_app/reference-configs
export SPARSE4D_REPO=/opt/nvidia/deepstream/deepstream/sources/sparse4d
export TRITON_REPO=/opt/nvidia/deepstream/deepstream/sources/TritonGdino/triton_model_repo
export RESOURCES=/opt/storage/resources

Mount the skill's scripts into the container (or docker cp them).

See § ONE-CALL FAST PATH above for the single-permission-prompt variant and the canonical docker cp src /tmp/scripts nesting-gotcha note (always rm -rf /tmp/scripts first to avoid nested /tmp/scripts/scripts/).

---

4.a — Discover NGC Resource Paths

NGC directory names change per version — discover them at runtime. The one-liners below use | head -n1 for brevity, but the agent MUST NOT use that in production — if two NGC resource versions are unpacked on the host at the same time, head -n1 silently picks one. Use the resolve_or_ask helper below (or call the shared resolve_unique_path function from common.sh, which already emits RESOLVE_OK / RESOLVE_AMBIGUOUS markers on stderr). For every resolved path, print a visible Using … line so the user can see the model / video dir choice on the terminal.

Recommended pattern

# resolve_or_ask <label> <find-expression...>  -> prints the chosen path on stdout;
# drives an AskQuestion (via the agent) on ambiguity.
resolve_or_ask() {
    local label="$1"; shift
    mapfile -t CANDS < <(find "$@" 2>/dev/null | sort)
    case ${#CANDS[@]} in
        0)  echo "ERROR: no match for $label under '$*'" >&2; return 2 ;;
        1)  echo "Using $label: $(basename "${CANDS[0]}") (${CANDS[0]})" >&2
            printf '%s\n' "${CANDS[0]}" ;;
        *)  # Agent should replace this branch with an AskQuestion covering CANDS[@]
            echo "AMBIGUOUS: $label — $(printf '%d candidates' "${#CANDS[@]}")" >&2
            printf '  [%d] %s\n' "${!CANDS[@]}" "${CANDS[@]}" >&2
            return 3 ;;
    esac
}

Concrete lookups (layout-agnostic — no hardcoded NGC subdirectory names)

Rule: each find is constrained only by extension or context-independent filename (*.onnx, labels.txt, *.npy, calibration.json, *.mp4). Never include directory filters like -path '*/mtmc/*', -name 'nv-warehouse-4cams', -name 'vss-warehouse-app-data*' — those assume a specific NGC resource layout and will silently fail when the resource is restructured. The 0/1/>1 dispatch in resolve_or_ask handles the multi-candidate case by asking the user.

Skip this step entirely if the var is already set by Step 1.g. The

resource-plan scan (resource-plan.md § 7.d) commits $WAREHOUSE_2D_ONNX,

$WAREHOUSE_2D_VIDEOS, $SPARSE4D_ONNX, $SMC_VIDEOS, etc. directly —

either from a single-candidate scan, from a hint match

(MODEL_NAME_HINT / VIDEOS_DIR_HINT), or from a user picker. When the

var is already populated, skip the resolve_or_ask call for that asset

— don't re-ask the user the same disambiguation twice. Wrap each call:

>

```bash

: "${WAREHOUSE_2D_ONNX:=$(resolve_or_ask 'warehouse-2d ONNX' "$RESOURCES" -type f -name '*.onnx')}"

```

>

The := default-assignment only fires when the var is unset/empty.

# Helper: find directories under $1 that contain at least one *.mp4 or *.mkv
find_video_dirs() {
    find "$1" -type d -exec sh -c '
        for d; do
            ls "$d"/*.mp4 "$d"/*.mkv 2>/dev/null | head -n1 | grep -q . && echo "$d"
        done
    ' _ {} +
}

# ---------- warehouse-2d ----------
WAREHOUSE_2D_ONNX=$(resolve_or_ask 'warehouse-2d ONNX' \
    "$RESOURCES" -type f -name '*.onnx')
WAREHOUSE_2D_VIDEOS=$(resolve_or_ask 'warehouse-2d videos dir' \
    <(find_video_dirs "$RESOURCES"))

# ---------- warehouse-3d ----------
SPARSE4D_ONNX=$(resolve_or_ask 'warehouse-3d (sparse4d) ONNX' \
    "$RESOURCES" -type f -name '*.onnx')
SPARSE4D_LABELS=$(resolve_or_ask 'sparse4d labels' \
    "$RESOURCES" -type f -name 'labels.txt')
SPARSE4D_ANCHOR=$(resolve_or_ask 'sparse4d anchor' \
    "$RESOURCES" -type f -name '*.npy')
# calibration.json: prefer NGC-resource-shipped, fall back to the repo copy
SPARSE4D_CALIB=$(resolve_or_ask 'sparse4d calibration' \
    "$RESOURCES" -type f -name 'calibration.json') \
  || SPARSE4D_CALIB="$CONFIGS/warehouse-3d/calibration.json"
WAREHOUSE_3D_VIDEOS=$(resolve_or_ask 'warehouse-3d videos dir' \
    <(find_video_dirs "$RESOURCES"))

# ---------- smartcity-rtdetr / smartcity-gdino ----------
# Pass the ONNX from Step 5's NGC-resource reference, or fall back to a bare *.onnx scan.
RTDETR_ONNX=$(resolve_or_ask 'smartcity-rtdetr ONNX' \
    "$RESOURCES" -type f -name '*.onnx')
GDINO_ONNX=$(resolve_or_ask 'smartcity-gdino ONNX' \
    "$RESOURCES" -type f -name '*.onnx')
SMC_VIDEOS=$(resolve_or_ask 'smartcity videos dir' \
    <(find_video_dirs "$RESOURCES"))

If the same use case needs to disambiguate multiple ONNXs (e.g. both RT-DETR and GDINO models live under $RESOURCES because both NGC models were pulled), the user's pick in the AskQuestion drives which ONNX the skill uses. Print the chosen basename + path on one line, the decision landmark on the next — the terminal output is the contract that a user can audit after the fact.

Ambiguity handling (non-negotiable)

If any resolve_or_ask call returns 3 (multiple candidates), the agent MUST pause and drive an AskQuestion:

{
  "questions": [
    {
      "id": "pick_<label>",
      "prompt": "Multiple <label> candidates found under $RESOURCES. Which one should I use?",
      "options": [
        {"id": "0", "label": "<basename-0> — <full-path-0>"},
        {"id": "1", "label": "<basename-1> — <full-path-1>"}
      ]
    }
  ]
}

Then set the variable from the chosen candidate and print Using <label>: <basename> (<full-path>) so the decision is visible on the terminal.

---

4.b — Substitute Discovered Paths Into Config Placeholders

The shipped configs now use generic <PATH_TO_*> tokens (see reference-configs/README.md § Placeholders). update_yaml_flat / update_ds_config from common.sh find the key and rewrite its value, so they work whether the current value is the placeholder or a previously-substituted path. Each helper verifies the write and fails loud if the edit didn't land.

source /tmp/scripts/common.sh

# ---------- warehouse-2d ----------
# Only onnx-file is required — model-engine-file is commented out in the shipped
# config; DeepStream auto-builds the engine next to the ONNX on first run.
update_yaml_flat $CONFIGS/warehouse-2d/ds-ppl-analytics-pgie-config.yml \
    onnx-file "$WAREHOUSE_2D_ONNX"

# ---------- warehouse-3d ----------
# All four Sparse4D keys MUST be set. engine_file must point at the persistent
# cache directory so sparse4d_setup.sh's build output is reused next deploy.
ONNX_BASE=$(basename "$SPARSE4D_ONNX")
update_yaml_flat $CONFIGS/warehouse-3d/config.yaml onnx_file    "$SPARSE4D_ONNX"
update_yaml_flat $CONFIGS/warehouse-3d/config.yaml engine_file  "$ENGINE_CACHE_DIR/${ONNX_BASE}_b${BATCH}.engine"
update_yaml_flat $CONFIGS/warehouse-3d/config.yaml labels_file  "$SPARSE4D_LABELS"
update_yaml_flat $CONFIGS/warehouse-3d/config.yaml anchor       "$SPARSE4D_ANCHOR"
# Calibration: if the NGC resource supplied one, copy it over the shipped default.
[[ -n "$SPARSE4D_CALIB" && "$SPARSE4D_CALIB" != "$CONFIGS/warehouse-3d/calibration.json" ]] && \
    cp "$SPARSE4D_CALIB" "$CONFIGS/warehouse-3d/calibration.json"

# ---------- smartcity-rtdetr ----------
# Same as warehouse-2d — only onnx-file; model-engine-file stays commented.
update_ds_config $CONFIGS/smartcities/rt-detr/rtdetr-960x544.txt \
    "[property]" onnx-file "$RTDETR_ONNX"

Why no `model-engine-file` substitution for warehouse-2d / smartcity-rtdetr? In both shipped configs that line is commented out because DeepStream auto-builds the engine next to the ONNX on first run (suffix _b<N>_gpu<G>_fp<P>.engine) and reuses it on every subsequent run. The post-launch hook cache_nvinfer_engine.sh (invoked by run_app_and_wait.sh — see start-app.md § 5.e and § 4.g of this file) symlinks the auto-built engine into $ENGINE_CACHE_DIR so future deploys can reuse it via the tiered cache lookup. Writing an explicit model-engine-file here would override that auto-build and pin the engine to a path we no longer control.

---

4.c — Update Batch Size (one command covers every file)

/tmp/scripts/update_batch_size.sh <usecase> <N>

This handles every batch-size touch point for the use case (see usecases.md).

---

4.d — Update Output Sink

Use the dedicated script — don't do it inline. The script is idempotent, updates all sink-related keys in one place, and verifies each key landed before returning:

docker exec <CONTAINER_NAME> /tmp/scripts/update_output_sink.sh <usecase> <sink_mode>
# Optional (filedump only):
#   --output-file /opt/storage/output/my_run.mp4   (override the default filename)
#   --container 1                                  (force true MP4 bytes; default is 2=MKV muxer
#                                                   for on-kill recoverability even with .mp4 filename)

Expected stdout on success: SINK_UPDATE_OK <usecase> <sink_mode>.

What it writes

Filedump defaults — output path /opt/storage/output/<usecase>_output.mp4 (standard .mp4 extension) + container muxer 2 (MKV). The extension and the muxer are decoupled by design: the .mp4 filename is the user-facing standard while the bytes on disk are written by the MKV muxer for on-kill recoverability (MP4's moov atom is only finalized on a clean exit; MKV streams stay playable up to the last written frame). VLC/ffmpeg/mpv detect by content, not filename, so the file plays cleanly. Override filename with --output-file <path>, or force true MP4 bytes with --container 1 (e.g. for a downstream tool that parses the moov atom).

Why a script (not inline edits)

Sink configuration spans four config sections ([sink0], [sink2], [tiled-display], [osd]) that must be set as a coherent group. A single script makes that atomic and verifiable:

1. Applies ALL keys in one logical unit (no partial state) 2. Verifies each key by re-reading the config after editing — fails loudly if any didn't land 3. Handles warehouse-3d's generate_3d_bbox toggle automatically 4. Pre-creates the filedump output directory and cleans stale files

Note on [tiled-display] enable

DeepStream's nvmultistreamtiler recognizes three meaningful values:

The skill writes one of these three values explicitly so the config is readable and predictable. Some shipped reference-configs default to 3; that happens to work for display too (DS treats any non-zero as "enabled"), but it makes the config ambiguous about intent — the explicit 1 for display vs 3 for perf-only path makes the agent's output sink choice legible at a glance.

Tile grid (rows × columns)

[tiled-display] rows and [tiled-display] columns are written by update_batch_size.sh (Step 4.c) using the closest-to-square formula ROW=floor(sqrt(N)), COL=ceil(N/ROW). Examples: N=1→1×1, N=4→2×2, N=6→2×3, N=8→2×4, N=9→3×3, N=16→4×4.

---

4.e — Configure Stream Sources

Dynamic mode (default): no config edit needed — use-nvmultiurisrcbin=1 starts with zero streams, and users add them via the REST API at http://localhost:9000.

Static mode: pre-populate the source list. Use the video directory discovered in Step 4.a for the current use case:

No hardcoded directory names — whichever directory the Step 4.a video-dir scan landed on (after user confirmation if multiple candidates) is the one used here.

CRITICAL — camera_id MUST match calibration.json for warehouse-3d

For warehouse-3d the camera_id (dynamic REST camera_id field, or static sensor-id-list / sensor-name-list) MUST exactly match the `id` of a sensor entry in `calibration.json`. If it doesn't, Sparse4D cannot find the camera's projection matrix, silently falls back to identity, and the BEV bounding boxes will be wrong. The log spams:

Warning: No projection matrix found for camera <name>. Using identity matrix.

Always discover the valid camera IDs before adding streams:

python3 -c 'import json; d=json.load(open("/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/metropolis_perception_app/reference-configs/warehouse-3d/calibration.json")); [print(s["id"]) for s in d["sensors"]]'

For the default warehouse-3d resource this prints Camera, Camera_01, Camera_02, Camera_03 — matching the .mp4 filename stems in $WAREHOUSE_3D_VIDEOS. Do NOT invent names like cam1/cam2/cam3/cam4 for warehouse-3d.

Safe rule of thumb (warehouse-3d): reuse the video filename stem as the camera_id (e.g. Camera_01.mp4 → camera_id=Camera_01). Those were calibrated together.

warehouse-2d and smartcity use cases do NOT have this constraint — their camera_ids are opaque identifiers.

Static mode example (4 streams)

source /tmp/scripts/common.sh

case "<usecase>" in
    warehouse-2d)      VIDEOS=$WAREHOUSE_2D_VIDEOS ;;
    warehouse-3d)      VIDEOS=$WAREHOUSE_3D_VIDEOS ;;
    smartcity-rtdetr|smartcity-gdino) VIDEOS=$SMC_VIDEOS ;;
esac

URLS="file://$VIDEOS/Camera.mp4;file://$VIDEOS/Camera_01.mp4;file://$VIDEOS/Camera_02.mp4;file://$VIDEOS/Camera_03.mp4"

# For warehouse-3d: NAMES MUST match calibration.json sensor ids (video stems work).
# For warehouse-2d / smartcity: any unique names.
NAMES="Camera;Camera_01;Camera_02;Camera_03"
N=4

update_ds_config "$MAIN" "[source-list]" num-source-bins   "$N"
update_ds_config "$MAIN" "[source-list]" list              "$URLS"
update_ds_config "$MAIN" "[source-list]" sensor-id-list    "$NAMES"
update_ds_config "$MAIN" "[source-list]" sensor-name-list  "$NAMES"
update_ds_config "$MAIN" "[source-list]" max-batch-size    "$N"

Important (warehouse-3d): Sparse4D expects the camera extrinsics in calibration.json to match the video viewpoints. If the NGC resource contains multiple video directories, Step 4.a asks the user which one to use — pick the directory whose .mp4 stems appear as sensors[].id entries in calibration.json. Re-using 2D videos with a 3D calibration file will produce garbage BEV boxes.

For RTSP in static mode, replace URLS with the rtsp://... list the user provided, and set each index of NAMES to the calibration entry that corresponds to that RTSP feed.

Dynamic mode example (REST API — warehouse-3d, 4 streams)

# --network=host → reach the app at localhost:9000
VIDEOS=/opt/storage/resources/.../videos/warehouse-4cams-20mx20m-synthetic   # or $WAREHOUSE_3D_VIDEOS inside container
for NAME in Camera Camera_01 Camera_02 Camera_03; do
  curl -s -X POST http://localhost:9000/api/v1/stream/add \
    -H 'Content-Type: application/json' \
    -d "{\"key\":\"sensor\",\"value\":{\"camera_id\":\"$NAME\",\"camera_name\":\"$NAME\",\"camera_url\":\"file://$VIDEOS/${NAME}.mp4\",\"change\":\"camera_add\",\"metadata\":{}}}"
done

REST /stream/remove requirements

• Remove requires both `camera_id` AND `camera_url` in the payload. A remove with only camera_id returns STREAM_REMOVE_FAIL, Source url empty.
• The camera_id on remove must EXACTLY match what was used at add time (case-sensitive).
• To rename a stream, remove it first (with the correct url), then re-add with the new id. Do NOT re-add the same url with a different id while the old one is still active (max-batch-size reject).
• For warehouse-3d, do NOT live-fix wrong camera_ids by remove+add while traffic is flowing — Sparse4D can crash with std::logic_error: basic_string: construction from null is not valid mid-remove. Safer path: stop the app, correct the IDs, restart.

---

4.f.1 — Sink-specific dependency install (filedump only) — now automatic

The DeepStream container ships without the software video encoder needed for [sink2] type=3 (File sink / MP4 / MKV mux). Previously this required a separate manual step in the agent flow; it is now performed atomically inside `update_output_sink.sh filedump` and is no longer a discrete workflow step.

Skip entirely for fakesink and eglsink — they don't need the encoder.

What the script does (automatic)

Before editing [sink2], update_output_sink.sh runs its ensure_encoder_deps function:

1. Validate via plugin registry (not marker): gst-inspect-1.0 x264enc — if the plugin is registered, skip the install. This is the real success signal; a marker file alone is not trusted. 2. Stale marker? If /opt/storage/.user_additional_install.done exists but x264enc is missing (partial install, volume copied from another host, etc.), the marker is removed and the install is retried. 3. Install: cd /opt/nvidia/deepstream/deepstream && ./user_additional_install.sh — installs libx264-dev, libx265-dev, libmp3lame-dev, and the GStreamer "ugly" plugins (mp4mux, h264parse, matroskamux, etc.). Output is streamed to /tmp/ds_user_install.log in case the apt-get under the hood fails. 4. Re-verify: gst-inspect-1.0 x264enc again. If still missing after install, the script aborts Step 4.d (no config edit is made), so the agent doesn't end up with a half-applied filedump sink that crashes at pipeline build. 5. On success: writes /opt/storage/.user_additional_install.done so future calls short-circuit at step 1.

Why validation, not marker-only

A stale marker can exist when:

• A previous install ran but was partially interrupted (e.g. agent retried before apt finished).
• The host volume was copied from a different machine.
• Someone ran touch /opt/storage/.user_additional_install.done manually.

With marker-only checks, these cases produce a silent Failed to create sink_sub_bin_encoder1 at pipeline build — long after the config edit has landed. With gst-inspect validation, the problem is caught and fixed during Step 4.d itself.

Overriding

Pass --skip-encoder-install to update_output_sink.sh if you plan to flip [sink2] enc-type=0 (hardware encoder via nvv4l2h264enc) yourself afterwards, or if you're working offline and need to defer the install.

Agent status reporting

Relay ENCODER_DEPS: lines from update_output_sink.sh stdout:

Disk usage

user_additional_install.sh adds ~250 MB of packages to the container. On a --rm container the packages are discarded at teardown, but the marker on the host means the next deploy detects the missing plugins (via gst-inspect) and re-runs automatically.

4.f — Use-case-specific setup

All 4 use cases now use the same tiered engine cache lookup (exact → compatible larger-batch → miss). The script names differ but the strategy is uniform. Override with FORCE_ENGINE_REBUILD=1 / --force / --exact-only.

How the cache avoids re-builds

$ENGINE_CACHE_DIR defaults to /opt/storage/engines/ which is the host-mounted ~/rtvicv-storage/engines/, so built engines survive container restarts.

Cache filenames use the ONNX basename (with `.onnx`) as the stem plus a `_b<N>` batch suffix, so every entry is version-scoped to the exact model it came from. Bumping the ONNX version produces a new cache name automatically — no stale-engine risk.

All setup scripts call engine_cache_hit <onnx-basename> <batch> <ext>, which returns:

1. Exact match (<onnx-basename>_b<N>.<ext> exists) — best TRT performance, always preferred 2. Compatible match (smallest cached engine for the same ONNX with batch ≥ N) — reused via TRT dynamic shapes, skips the rebuild 3. Miss — rebuild, then call cache_engine to save for next time

Set FORCE_ENGINE_REBUILD=1 in the environment (or pass --force to either setup script) to bypass the cache and rebuild from scratch.

warehouse-2d / smartcity-rtdetr pre-launch (nvinfer tiered lookup)

# The ONNX paths were already resolved in Step 4.a (resolve_or_ask, with
# AskQuestion fallback on multi-candidate). Just reuse the variables —
# do NOT re-scan with `find ... | head -n1` (that silently picks one
# when the user has multiple NGC resource versions unpacked).

# warehouse-2d
/tmp/scripts/prelaunch_nvinfer_engine.sh --onnx "$WAREHOUSE_2D_ONNX" --batch <N>

# smartcity-rtdetr
/tmp/scripts/prelaunch_nvinfer_engine.sh --onnx "$RTDETR_ONNX" --batch <N>

What it does:

1. Computes the target path: <ONNX>_b<N>_gpu0_fp16.engine 2. Exact match — if that file exists (not a stale symlink), exit 0 (DS will deserialize it directly on launch). 3. Compatible match — if missing, scans (a) the ONNX directory and (b) $ENGINE_CACHE_DIR for any _b<M>_gpu0_fp16.engine with M ≥ N. Picks the smallest M that fits. 4. If a compatible engine is found → creates a symlink at the target path so DS sees the engine at its expected location. TRT dynamic shapes let the larger engine serve the smaller batch natively. 5. If nothing suitable → miss; DS will build from ONNX during launch (~3-5 min).

Example: user deployed batch=4 yesterday, wants batch=3 today

Requested: <ONNX>_b3_gpu0_fp16.engine (doesn't exist)
Scanning:  <ONNX>_b2_gpu0_fp16.engine — skip, batch too small
           <ONNX>_b4_gpu0_fp16.engine — MATCH (batch 4 >= 3)
Result:    symlink <ONNX>_b3_gpu0_fp16.engine -> <ONNX>_b4_gpu0_fp16.engine
           DS loads the b4 engine, serves batch=3 natively via dynamic shapes
           → 3-5 min build skipped.

Flags:

• --exact-only or env ENGINE_EXACT_MATCH_ONLY=1 — disables the compatible-batch fallback
• --gpu <N> — GPU index in the filename (default 0)
• --precision fp16 / fp32 — precision suffix (default fp16)

warehouse-3d extras

export LD_PRELOAD=$SPARSE4D_REPO/libmsda_fp16.so
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$SPARSE4D_REPO:/usr/local/lib/python3/dist-packages/torch/lib
/tmp/scripts/setup_sparse4d.sh --batch <N>

The script reads the Sparse4D ONNX path from config.yaml's onnx_file: key (which Step 4.b already substituted to $SPARSE4D_ONNX) and uses its basename as the cache stem. It then updates config.yaml's engine_file: to point at $ENGINE_CACHE_DIR/<sparse4d-onnx-basename>_b<N>.engine and, on a cache miss, runs sparse4d_setup.sh which builds directly into the cache. On cache hit the setup is skipped entirely. If onnx_file: still holds the literal <PATH_TO_ONNX_MODEL> placeholder (Step 4.b was skipped), the script errors out rather than falling back to a path-guess.

smartcity-gdino extras

/tmp/scripts/setup_gdino.sh --batch <N>

Copies the ONNX into $TRITON_REPO/gdino_trt/1/model.onnx, then:

• Cache hit → symlinks Triton's fixed model.plan at the cached engine, skips trtexec
• Cache miss → runs trtexec → saves to $ENGINE_CACHE_DIR/<ONNX-basename>_b<N>.plan → symlinks model.plan to the cached file

4.g — Cache the DS-auto-built engine (warehouse-2d, smartcity-rtdetr only — post-launch reference, invoked from start-app.md § 5.e)

Run AFTER the app has started and the engine build has completed (signal: the REST server replies on :9000, or the app log shows the pipeline running).

DeepStream's nvinfer ignores the model-engine-file path for writes — it always saves the built engine next to the ONNX as <onnx-name>_b<N>_gpu<G>_fp<P>.engine. This script symlinks that auto-built engine into $ENGINE_CACHE_DIR/<ONNX-basename>_b<N>.engine so the PGIE config's model-engine-file path resolves correctly on the next deploy — avoiding a 3-5 min rebuild. Using the ONNX basename as the cache stem keeps entries version-scoped.

warehouse-2d

# $WAREHOUSE_2D_ONNX was resolved in Step 4.a (with user-confirm on ambiguity).
docker exec <CONTAINER_NAME> /tmp/scripts/cache_nvinfer_engine.sh \
    --onnx "$WAREHOUSE_2D_ONNX" --batch <N>

smartcity-rtdetr

# $RTDETR_ONNX was resolved in Step 4.a.
docker exec <CONTAINER_NAME> /tmp/scripts/cache_nvinfer_engine.sh \
    --onnx "$RTDETR_ONNX" --batch <N>

Effect

Skipping conditions

cache_nvinfer_engine.sh is safe to run at any time — it's idempotent and exits cleanly if the engine hasn't been built yet (it just logs ENGINE_CACHE: LINK_SKIP). If the expected auto-built engine isn't found, the deploy still works (DS handles its own cache), but the symlink won't be created — re-run after the engine is built.

Why not run this for warehouse-3d / smartcity-gdino?

Those use custom setup scripts (setup_sparse4d.sh, setup_gdino.sh) that build directly into the cache, so no post-build linking is needed. cache_nvinfer_engine.sh is only for nvinfer-based models that use DeepStream's auto-build path.

4.h — Deployment log (every deploy creates one — owned by start-app.md § 5.a)

Every rtvicv-deploy run MUST produce a persistent log file under $STORAGE/logs/<usecase-and-model>_<timestamp>.txt (persisted to ~/rtvicv-storage/logs/ on the host). The log is initialized by scripts/write_deployment_log.sh before the app starts and captures the full deployment context in one file.

What goes into the log (in order)

1. Header — timestamp, host, user 2. Deployment Settings — use case, batch size, sink, platform, stream mode, input type, videos dir, docker image, NGC resource 3. Docker Run Command — the exact multi-line docker run ... used to start the container 4. App Launch Command — the metropolis_perception_app -c <cfg> command about to run 5. Config file dumps — full content of every config file this use case touches:

• warehouse-2d: ds-main-config.txt, ds-ppl-analytics-pgie-config.yml, ds-nvdcf-accuracy-tracker-config.yml, ds-detector-labels.txt
• warehouse-3d: ds-main-config.txt, config.yaml, calibration.json, ds-mtmc-preprocess-config.txt, ds-mtmc-videotemplate_custom_lib_config.txt
• smartcity-rtdetr: run_config-api-rtdetr-protobuf.txt, rtdetr-960x544.txt, rtdetr-960x544-labels.txt
• smartcity-gdino: run_config-api-rtdetr-protobuf.txt, config_triton_nvinferserver_gdino.txt

6. Runtime log — the app's stdout/stderr appended after launch

Invocation (wire into Step 5)

LOG=$(docker exec <CONTAINER_NAME> /tmp/scripts/write_deployment_log.sh \
    --usecase "$USECASE" --batch "$BATCH" --sink "$SINK" \
    --platform "$PLATFORM" --stream-mode "$STREAM_MODE" --input-type "$INPUT_TYPE" \
    --videos "$VIDEOS_DIR" --image "$RTVI_CV_IMAGE" --ngc "$NGC_REF" \
    --docker-cmd "$DOCKER_RUN_CMD" --app-cmd "$APP_CMD")

# $LOG now points at /opt/storage/logs/deployment_YYYYMMDD_HHMMSS.txt
# Start the app and APPEND its output to the same file:
docker exec -d <CONTAINER_NAME> bash -c "$APP_CMD >> \"$LOG\" 2>&1"

Why this matters

• Debug later — every deploy captures the exact config state that was used, even after the container exits (--rm cleanup)
• Reproducibility — share the log file with a colleague to reproduce a specific run
• Rebuild-free config diffing — compare two deployments by just diffing their log files
• Engine build traces — if the engine build fails mid-run, the full trtexec/TRT output is preserved
• Persistence — $STORAGE/logs/ is the host-mounted ~/rtvicv-storage/logs/ so logs survive container teardown

Log file location

Users can tail -f ~/rtvicv-storage/logs/<usecase-and-model>_<ts>.txt from any shell to watch the build + runtime progress in real time.

Environment, Secrets, Mounts & GPU Selection

Reference for everything the host must provide before docker run: credentials, storage layout, environment variables, GPU selection, and port mapping.

---

Required Secrets & Credentials

The skill writes ~/.ngc/config with permissions 0600. The container itself never receives a ~/.ngc mount — all NGC downloads run on the host via scripts/fetch_resources.sh and the resulting files are staged into ~/rtvicv-storage/resources/.

---

Required Volume Mounts

Create the storage tree before docker run:

mkdir -p ~/rtvicv-storage/resources \
         ~/rtvicv-storage/engines \
         ~/rtvicv-storage/logs

---

Required Environment Variables

No runtime env vars are required inside the container — all configuration is applied to INI/YAML files under /opt/nvidia/deepstream/.../reference-configs/<use-case>/.

---

Optional / Feature-Flag Environment Variables

---

GPU Selection & Hardware

# Default — pin to GPU 0 (single-GPU systems and the common case on
# multi-GPU hosts where the user wants a deterministic device).
docker run --gpus '"device=0"' ...

# Specific GPU by index (multi-GPU host, pick a non-default device)
docker run --gpus '"device=1"' ...

# Multiple specific GPUs
docker run --gpus '"device=0,1"' ...

# Specific GPU by UUID (most precise — survives index changes after
# host reboot or driver reload)
docker run --gpus '"device=GPU-<uuid>"' ...

# All GPUs — only when you genuinely need every device on the host
docker run --gpus all ...

# Jetson / SBSA — use --runtime nvidia, then --gpus picks visibility
docker run --runtime nvidia --gpus '"device=0"' ...

Default for the vss-deploy-detection-tracking-2d skill: `--gpus '"device=$DEFAULT_GPU_ID"'`. DEFAULT_GPU_ID is emitted by scripts/load_defaults.sh from assets/deploy-defaults.yml > runtime.gpu_id (ships at 0). Pinning a specific device avoids accidentally claiming every GPU on a multi-GPU host (a common surprise during smoke-testing on a shared workstation). The agent uses the YAML value unless the user explicitly asks for a different device (e.g. "run on gpu 1") or for all. Per-deploy overrides do NOT mutate the YAML.

Verify the image's CUDA architecture support against your GPU:

nvidia-smi --query-gpu=compute_cap --format=csv,noheader

Images are built against CUDA 12.x and target SM 7.5+ (Turing and newer).

---

Port Conflict Map

For parallel deploys, give each container its own http-port and a different container name — see references/container-reuse.md.

---

Dry Run / Pre-flight

# Verify image exists and matches platform arch
docker manifest inspect "$RTVI_CV_IMAGE" 2>/dev/null | \
  python3 -c "import sys,json; d=json.load(sys.stdin); \
              print([m['platform']['architecture'] for m in d.get('manifests',[])])"

# Test NGC auth before downloading.
# IMPORTANT: pipe the API key via stdin (--password-stdin). Passing the
# token as `-p "$NGC_API_KEY"` would expose it in `ps aux` and shell
# history — never use that form even in examples.
printf '%s' "$NGC_API_KEY" | docker login nvcr.io -u '$oauthtoken' --password-stdin \
  && echo "auth OK"
ngc config current && echo "NGC config OK"

To preview the full docker run command without launching it, pass --dry-run in your skill query.