Omniverse Realtime Viewer

<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> <!-- SPDX-License-Identifier: Apache-2.0 -->

Omniverse Realtime Viewer

This is the top-level entry point for the Omniverse Realtime Viewer skill package. It is self-contained: all required routing, conventions, and validation guidance live in the selected references.

Use the focused reference documents as implementation recipes. This file chooses the right recipes and preserves the architectural rules that must hold across all generated viewer apps.

Instructions

Start by classifying the requested viewer, then read only the references needed for that delivery path and feature set. Implement the render path first, layer interaction and UI behavior on top of it, and finish by capturing validation evidence from references/validation.md.

Read Order

1. Read references/routing.md to choose the delivery path and focused references. 2. Read references/conventions.md before implementing camera, input, selection, viewport, streaming protocol, scene loading, or environment behavior. 3. For broad viewer requests, read references/usd-viewer-app/README.md. 4. If the delivery path is unclear, read references/streaming-vs-local/README.md. 5. If the prompt includes layout, panels, controls, inspectors, status, or UX, read references/viewer-ux-workflow/README.md and then the focused viewer UI references. This applies to React/WebRTC, Tauri, Electron, ovui, ovwidgets, and Dear ImGui apps; "frontend" means user-facing UI, not only browser UI. 6. For viewport interaction, read references/viewer-input-routing/README.md before references/camera-controls/README.md, references/native-picking-selection/README.md, or references/object-selection/README.md. 7. Read only the focused capability references needed for the requested app. 8. Use references/validation.md to capture review evidence before handoff.

Non-Negotiables

• Use ovrtx for all USD and 3D rendering.
• Browser apps display an ovstream WebRTC video stream plus UI. The browser

does not render USD geometry.

• Do not substitute WebGL, Three.js, Babylon.js, PlayCanvas, A-Frame,

model-viewer, react-three-fiber, glTF browser viewers, or other client-side 3D renderers.

• If local validation cannot run because the GPU/runtime environment is absent,

scaffold the ovrtx path and document the runtime requirement. Do not add a browser-renderer fallback.

• Keep user USD files unmodified. Viewer cameras, render products, render vars,

settings, selection metadata, and runtime state belong in session/composite layers or app state.

• Keep one owner for renderer.step(), stage mutation, native picking,

selection writes, and live attribute writes.

• Keep dependency acquisition in references/dependencies/README.md and deployment choices in

references/cloud-deployment/README.md; do not duplicate package locations or deployment setup.

Focused Reference Families

• Entry points and recipes: references/usd-viewer-app/README.md, references/streaming-viewer-recipe/README.md,

references/ovui-local-viewer-recipe/README.md, references/streaming-vs-local/README.md, references/electron-shm-viewer/README.md, references/ovwidgets-editor-shell/README.md.

• Rendering and stage: references/ovrtx-rendering/README.md, references/stage-loading/README.md, references/stage-management/README.md,

references/render-settings/README.md, references/aov-switching/README.md, references/stage-hierarchy/README.md, references/stage-queries/README.md, references/stage-attribute-reads/README.md, references/prim-transform-safety/README.md, references/usd-sample-data/README.md.

• Delivery and runtime: references/streaming-server/README.md, references/streaming-client/README.md,

references/streaming-messages/README.md, references/streaming-lifecycle/README.md, references/local-viewer/README.md, references/tauri-local-viewer/README.md, references/cpp-native-viewer/README.md, references/headless-shm-cli/README.md, references/viewer-backend-interface/README.md, references/webgl-shm-transport/README.md.

• Viewer UI/UX: references/viewer-ux-workflow/README.md, references/viewer-layout-patterns/README.md,

references/viewer-control-patterns/README.md, references/viewer-data-view-patterns/README.md, references/viewer-feedback-status/README.md.

• Interaction: references/viewer-input-routing/README.md, references/camera-controls/README.md,

references/object-selection/README.md, references/native-picking-selection/README.md, references/selection-feedback/README.md, references/selection-animation/README.md, references/transform-manipulator/README.md, references/gl-viewport-overlay/README.md, references/ovui-library/README.md, references/prim-pick-effects/README.md, references/prim-info-display/README.md, references/viewport-overlays/README.md.

• Infrastructure: references/dependencies/README.md, references/windows-native-setup/README.md, references/cloud-assets/README.md,

references/cloud-deployment/README.md, references/troubleshooting/README.md.

Build Workflow

1. Classify the prompt by delivery path, target user, required capabilities, runtime environment, validation needs, and explicit constraints. 2. Select a small reference set. Start with the recipe or routing reference, then add focused capabilities such as camera, picking, hierarchy, properties, render settings, transform tools, cloud assets, or deployment. 3. Read selected references before writing app code. Follow their build order, import order, data-channel contracts, and renderer ownership rules. 4. Implement the core render path first, then input routing and camera, then selection and data panels, then scene/settings features, then packaging or deployment. 5. Treat the selected references as the behavior contract for API shape, compatibility, and generated project structure. 6. Capture validation evidence before calling the viewer ready.

Examples

• For a browser viewer request, use the streaming recipe references plus camera,

picking, hierarchy, properties, render settings, and stream-status references.

• For a local workstation viewer request, use the local or native delivery

references plus renderer setup, stage loading, viewport input, and validation.

Completion Checklist

• Selected references match the user's intent and delivery path.
• No code path uses a browser-side 3D renderer for USD.
• The generated app has one clear owner for render stepping and stage mutation.
• User USD files remain untouched by viewer-owned session data.
• Camera, input, selection, scene loading, and stream behavior follow

references/conventions.md.

• Setup/build/run results and visual interaction evidence are captured with

references/validation.md.

Evaluation Report

Evaluation of the omniverse-realtime-viewer skill before publication through NVSkills-Eval.

This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.

Evaluation Summary

• Skill: omniverse-realtime-viewer
• Evaluation date: 2026-05-28
• NVSkills-Eval profile: external
• Overall verdict: FAIL
• Tier 3 live agent evaluation: not available in this report

Agents Used

• Tier 3 agent details were not available in this report.

Metrics Used

Reported benchmark dimensions:

• Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
• Correctness: checks whether the agent follows the expected workflow and produces the correct final output.
• Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
• Effectiveness: checks whether the agent performs measurably better with the skill than without it.
• Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.

Underlying evaluation signals used in this run:

• No Tier 3 evaluation signal details were available in this report.

Test Tasks

Tier 3 evaluation task details were not available in this report.

Results

Tier 3 dimension rollup was not available in this report.

Tier 1: Static Validation Summary

Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 32 total findings.

Top findings:

• MEDIUM PII/gps_coordinates: GPS coordinates (location information) (references/stage-hierarchy/fallback-worker-protocol.md:55)
• MEDIUM PII/gps_coordinates: GPS coordinates (location information) (references/stage-hierarchy/fallback-worker-protocol.md:56)
• MEDIUM PII/gps_coordinates: GPS coordinates (location information) (references/headless-shm-cli/README.md:156)
• MEDIUM PII/gps_coordinates: GPS coordinates (location information) (references/headless-shm-cli/README.md:169)
• MEDIUM PII/gps_coordinates: GPS coordinates (location information) (references/cpp-native-viewer/interaction-features.md:20)

Tier 2: Deduplication Summary

Tier 2 validation reported findings. NVSkills-Eval ran 2 checks and found 21 total findings.

Top findings:

• HIGH DUPLICATE/duplicate: Duplicate content found across references/stage-hierarchy/README.md and references/stage-queries/README.md:

"### prim_list_handle Use" in references/stage-hierarchy/README.md (lines 91-102) vs "## prim_list_handle" in references/stage-queries/README.md (lines 124-129) (references/stage-hierarchy/README.md:91)

• HIGH DUPLICATE/duplicate: Duplicate content found across SKILL.md and references/conventions.md and references/routing.md and references/stage-hierarchy/fallback-worker-protocol.md and references/streaming-messages/server-handler-map.md and references/streaming-server/frame-loop-and-continuity.md and references/troubleshooting/scenario-playbooks.md and references/validation.md:

"(preamble)" in SKILL.md (lines 1-3) vs "(preamble)" in references/conventions.md (lines 1-3) vs "(preamble)" in references/routing.md (lines 1-3) vs "(preamble)" in references/stage-hierarchy/fallback-worker-protocol.md (lines 1-3) vs "(preamble)" in references/streaming-messages/server-handler-map.md (lines 1-3) vs "(preamble)" in references/streaming-server/frame-loop-and-continuity.md (lines 1-3) vs "(preamble)" in references/troubleshooting/scenario-playbooks.md (lines 1-3) vs "(preamble)" in references/validation.md (lines 1-3) (SKILL.md:1)

• HIGH DUPLICATE/duplicate: Duplicate content found across references/ovrtx-rendering/README.md and references/stage-loading/README.md and references/stage-management/README.md:

"## Stage Composition APIs" in references/ovrtx-rendering/README.md (lines 36-48) vs "## ovrtx 0.3 Stage Composition APIs" in references/stage-loading/README.md (lines 13-25) vs "## Stage Composition Policy" in references/stage-management/README.md (lines 32-40) (references/ovrtx-rendering/README.md:36)

• HIGH DUPLICATE/duplicate: Duplicate content found across references/conventions.md and references/electron-shm-viewer/protocol-interaction-lifecycle.md and references/ovui-local-viewer-recipe/setup-shell-renderer.md and references/stage-management/README.md and references/streaming-viewer-recipe/server-runtime.md:

"## Scene Loading" in references/conventions.md (lines 83-94) vs "## Scene Loading, Queries, And Settings" in references/electron-shm-viewer/protocol-interaction-lifecycle.md (lines 88-116) vs "## 5. Implement Scene Loading" in references/ovui-local-viewer-recipe/setup-shell-renderer.md (lines 125-159) vs "## Adding This To An Existing Omniverse Realtime Viewer" in references/stage-management/README.md (lines 172-183) vs "## 5. Implement Scene Loading" in references/streaming-viewer-recipe/server-runtime.md (lines 164-198) (references/conventions.md:83)

• HIGH DUPLICATE/duplicate: Duplicate content found across references/stage-hierarchy/README.md and references/stage-queries/README.md:

"### AND / OR / NOT Filters" in references/stage-hierarchy/README.md (lines 43-71) vs "## Filter Construction" in references/stage-queries/README.md (lines 35-70) (references/stage-hierarchy/README.md:43)

Publication Recommendation

The skill should be reviewed before NVSkills-Eval publication. Skill owners should address the findings above and rerun NVSkills-Eval to refresh this benchmark.

AOV Switching

Triggers

Use this skill for requests mentioning AOV, render var, changeAOVRequest, activeAOVState, availableAOVsResult, HdrColor, NormalSD, segmentation view, or display render output.

Use this when the Omniverse Realtime Viewer needs to stream something other than LdrColor, such as HDR color, normals, instance segmentation, or semantic segmentation.

Keep one WebRTC video stream. AOV selection changes which ovrtx render var is copied into a persistent CUDA BGRA8 stream buffer before calling ovstream.stream_video().

For ovrtx AOV, RenderVar tensor, mapping, or release-specific behavior not covered here, read references/dependencies for acquisition guidance and supplemental dependency documentation.

Architecture

composite USDA orderedVars
    -> ovrtx frame_output.render_vars
    -> runtime displayable-AOV discovery
    -> selected AOV maps on CUDA
    -> Warp converts named tensor dtype/shape to BGRA8
    -> ovstream VideoFrame.from_cuda_array
    -> React dropdown state from data-channel events

Do not create a separate stream per AOV. The browser receives the same video track; only the server-side source render var changes.

Server State

Keep display state on the server, not only in React. The server is authoritative because it knows which render vars ovrtx actually produced on recent frames.

# Displayable AOVs requested by the composite stage, in preferred display order.
# Only AOVs that ovrtx actually produces full-resolution data for are included.
DISPLAY_AOVS = (
    "LdrColor",                 # uint8  RGBA [H,W,4]
    "HdrColor",                 # uint16 RGBA [H,W,4], fp16 packed as uint16
    "NormalSD",                 # uint32 RGBA [H,W,4], packed float bits
    "InstanceSegmentationSD",   # uint32 [H,W,1], display/debug instance IDs
    "SemanticSegmentationSD",   # uint32 [H,W,1], display/debug semantic IDs
    "DepthSD",                  # uint32 [H,W,1], float32 bits packed as uint32
    "DiffuseAlbedoSD",          # uint8  RGBA [H,W,4]
)

self._active_aov: str = "LdrColor"
self._available_aovs: Set[str] = {"LdrColor"}
self._aov_error: Optional[str] = None

Runtime discovery should filter frame_output.render_vars through DISPLAY_AOVS. Do not expose every reported key; many requested render vars currently map to empty tensors or fail when mapped.

def _update_available_aovs(self, render_vars: Any, notify: bool = False) -> None:
    names = set(render_vars.keys()) if hasattr(render_vars, "keys") else set(render_vars)
    available = {name for name in DISPLAY_AOVS if name in names}
    if not available:
        available = {"LdrColor"}

    changed = available != self._available_aovs
    self._available_aovs = available

    if self._active_aov not in self._available_aovs:
        self._active_aov = "LdrColor"
        changed = True

    if notify and changed and self._stream_server:
        available_payload = self.get_available_aovs()
        self._message_handler.send_message(
            "availableAOVsResult",
            {"aovs": available_payload, "available": available_payload},
        )
        self._message_handler.send_message("activeAOVState", self.get_active_aov_state())

Composite Stage

Request all candidate render vars in the composite stage so future ovrtx support can surface without changing the stage wrapper again. The UI should still expose only DISPLAY_AOVS.

def RenderProduct "ViewportTexture0"
{
    rel camera = </OVCamera>
    rel orderedVars = [
        </Render/Vars/LdrColor>,
        </Render/Vars/HdrColor>,
        </Render/Vars/Depth>,
        </Render/Vars/Normal>,
        </Render/Vars/InstanceSeg>,
        </Render/Vars/SemanticSeg>,
        </Render/Vars/Metallic>,
        </Render/Vars/Roughness>,
        </Render/Vars/Emissive>,
        </Render/Vars/Diffuse>,
        </Render/Vars/Specular>,
        </Render/Vars/AO>,
        </Render/Vars/DirectDiffuse>,
        </Render/Vars/DirectSpecular>,
        </Render/Vars/IndirectDiffuse>,
        </Render/Vars/IndirectSpecular>,
        </Render/Vars/MotionVectors>,
    ]
}

def RenderVar "Normal"
{
    uniform string sourceName = "NormalSD"
}
def RenderVar "InstanceSeg"
{
    uniform string sourceName = "InstanceSegmentationSD"
}

The keys in frame_output.render_vars are source names such as NormalSD, not necessarily the RenderVar prim names such as Normal.

InstanceSegmentationSD is a display/debug AOV in this skill. Do not use it as the required picking path for 0.3 viewers; use ovrtx pick queries and resolve pick-hit path IDs through the renderer path dictionary.

Message Protocol

Use the standard data-channel envelope:

{"event_type":"changeAOVRequest","payload":{"aov":"NormalSD"}}

The server sends both aovs and available in availableAOVsResult for compatibility. Frontends should accept either field.

self._handlers = {
    "changeAOVRequest": self._handle_change_aov,
    "getAvailableAOVs": self._handle_get_available_aovs,
    "toggleSegView": self._handle_toggle_seg_view,
}

def _handle_change_aov(self, payload: Dict[str, Any]) -> None:
    requested = payload.get("aov") or payload.get("name")
    if not isinstance(requested, str) or not requested:
        self._send_aov_state({"result": "error", "reason": "Missing AOV name"})
        return

    previous = getattr(self.server, "_active_aov", "LdrColor")
    if self.server.set_active_aov(requested):
        self._send_aov_state({"result": "success", "previous": previous})
        return

    self._send_aov_state({
        "result": "error",
        "requested": requested,
        "reason": "AOV is not available for the current render product",
    })

Conversion Pipeline

Allocate one long-lived BGRA8 CUDA buffer and copy/convert each selected AOV into it. This keeps ovstream frame handoff stable even when the selected AOV has a different dtype.

def _ensure_stream_buffer(self, height: int, width: int) -> bool:
    if self._stream_buf is None:
        self._stream_buf = wp.zeros((height, width, 4), dtype=wp.uint8, device="cuda:0")
        return True
    return self._stream_buf.shape[0] == height and self._stream_buf.shape[1] == width

Map the selected render var on CUDA, choose the tensor to display, wrap it with Warp via DLPack, and dispatch by dtype and shape. Most display AOVs are single-tensor outputs, so the mapped render var itself is the DLPack producer. Multi-tensor render vars must be addressed by tensor name; do not use older single-tensor convenience access in new code.

def _display_tensor(mapped: Any, preferred: tuple[str, ...] = ("Color", "color", "data")) -> Any:
    try:
        return wp.from_dlpack(mapped)
    except TypeError:
        for name in preferred:
            try:
                return wp.from_dlpack(mapped[name])
            except (KeyError, TypeError):
                pass
        raise

with fout.render_vars[aov_name].map(device=Device.CUDA) as rv:
    src = _display_tensor(rv)
    shape = tuple(int(dim) for dim in src.shape)
    height, width = shape[0], shape[1]
    channels = shape[2] if len(shape) >= 3 else 1
    dtype = src.dtype
    dim = (width, height)

    if dtype == wp.uint8 and len(shape) == 3 and channels == 4:
        wp.copy(self._stream_buf, src)
        wp.launch(_swap_rb, dim=dim, inputs=[self._stream_buf], device="cuda:0")
        return True

    if dtype == wp.uint32 and len(shape) == 3 and channels == 1:
        wp.launch(_colorize_seg_3d, dim=dim, inputs=[src, self._stream_buf], device="cuda:0")
        return True

Always fall back to LdrColor if the active AOV cannot be copied. If that also fails, keep streaming the last good buffer instead of sending an invalid frame.

copied = self._copy_aov_to_stream_buffer(fout, self._active_aov)
if not copied and self._active_aov != "LdrColor":
    copied = self._copy_aov_to_stream_buffer(fout, "LdrColor")

Production Display Conversion Rules

Before calling stream_video(), the selected AOV must be visualization-ready BGRA8 in the server-owned CUDA stream buffer. Use these conversions:

Dispatch by AOV name, dtype, shape, and channel count. Image outputs are channel-last [H,W,C]; scalar AOVs are expected as [H,W,1]. Do not assume that all uint32 [H,W,1] values are segmentation; DepthSD uses the same shape but needs depth visualization.

Warp Kernels

Warp does not provide a simple bit-cast path in these kernels. The current HdrColor and NormalSD conversions are visualization approximations, not numerically exact decoders.

Frontend Wiring

React keeps local UI state, but the server event stream corrects it whenever discovery changes or a requested AOV is rejected.

case 'activeAOVState': {
  const payload = event.payload as ActiveAOVStatePayload;
  if (Array.isArray(payload.available) && payload.available.length > 0) {
    setAvailableAOVs(payload.available);
  }
  setActiveAOV(payload.active || 'LdrColor');
  break;
}
case 'availableAOVsResult': {
  const payload = event.payload as AvailableAOVsResultPayload;
  const names = payload.aovs || payload.available || [];
  if (names.length > 0) {
    setAvailableAOVs(names);
  }
  break;
}

sendMessage({
  event_type: 'changeAOVRequest',
  payload: { aov: selectedAOV },
});

ovrtx Findings

The composite stage requests 17 render vars. In this implementation, ovrtx reports them, but only the listed render vars currently produce useful full-resolution data in the streaming path:

Enabling Additional AOVs via Path-Tracing Flags

Many AOVs produce empty tensors by default because the RTX path-tracing AOV passes are disabled. To unlock DepthSD, DiffuseAlbedoSD, and potentially more:

1. Add API schemas to the RenderProduct

def RenderProduct "ViewportTexture0" (
    prepend apiSchemas = ["OmniRtxSettingsCommonAdvancedAPI_1", "OmniRtxSettingsPtAdvancedAPI_1", "OmniRtxSettingsRtAdvancedAPI_1"]
)
{
    token omni:rtx:rendermode = "RealTimePathTracing"
    ...
}

2. Enable PT AOV flags

bool omni:rtx:pt:diAOV = 1
bool omni:rtx:pt:giAOV = 1
bool omni:rtx:pt:diffuseFilterAOV = 1
bool omni:rtx:pt:reflectionsAOV = 1
bool omni:rtx:pt:refractionFilterAOV = 1
bool omni:rtx:pt:refractionsAOV = 1
bool omni:rtx:pt:selfIllumAOV = 1
bool omni:rtx:pt:volumesAOV = 1
bool omni:rtx:pt:worldNormalsAOV = 1
bool omni:rtx:pt:worldPosAOV = 1
bool omni:rtx:pt:zDepthAOV = 1
bool omni:rtx:pt:denoising:optix:denoiseAOVs = 1
float omni:rtx:pt:zDepthMin = 0.1
float omni:rtx:pt:zDepthMax = 10000

3. Use correct source names

Some AOV source names differ from intuitive guesses:

Using the wrong sourceName causes map() failures or empty tensors even when the render pass is enabled.

Current status with PT flags

• Working (7): LdrColor, HdrColor, NormalSD, InstanceSegmentationSD, SemanticSegmentationSD, DepthSD, DiffuseAlbedoSD
• Still empty (needs investigation): DirectDiffuse, DirectSpecular, IndirectDiffuse, IndirectSpecular, Emissive, Specular, AmbientOcclusion, Metallic, MotionVectors
• Still fails to map: Roughness

The lighting decomposition AOVs may need more PT convergence samples or a different configuration. See docs/ovrtx_aov_deep_dive.md for the full investigation.

Gotchas

• Keep picking independent of display AOV. Use ovrtx pick queries for selection and treat InstanceSegmentationSD as a visualization/debug output.
• Reset _active_aov and _available_aovs on stage load. AOV availability is render-product/runtime state, not global app state.
• Send AOV state after initial client connection. A browser can connect after startup and miss the stage-open response.
• Do not trust a render var just because it appears in fout.render_vars; mapping can still fail or yield empty data.
• HdrColor is half-float data exposed as uint16; the current conversion is for display only.
• NormalSD is float bit-pattern data exposed as uint32; exact decoding needs a real bit-cast path.
• ovstream expects BGRA8. Every displayable AOV must end in a uint8 [H,W,4] buffer.
• Scalar render outputs are channel-last [H,W,1]. Keep old [H,W] kernel paths only as compatibility fallbacks if supporting pre-0.3 builds.

See also: ovrtx-rendering, streaming-server, streaming-messages, render-settings, object-selection.

Camera Auto-Select

Purpose

When a user says "here's my stage" the viewer should open to a meaningful camera on the very first frame — not to an arbitrary default position. This skill inspects the stage for authored cameras and picks the best one, or computes a fit-all fallback.

Triggers

Use when:

• Loading a user-provided USD stage for the first time.
• Building a viewer application that must "just work" with arbitrary stages.
• The default camera position is wrong or unhelpful for a given scene.
• A build pipeline needs to determine the hero camera before deployment.

Priority Heuristic

Evaluate cameras in this order. Stop at the first match.

Implementation

Stage Introspection (Python / pxr)

import math
from pxr import Usd, UsdGeom, Gf

HERO_NAME_PATTERNS = ["main", "hero", "default", "persp", "perspective"]


def find_best_camera(stage: Usd.Stage) -> str | None:
    """Return the prim path of the best camera, or None if bbox-fit needed."""

    # Priority 1: explicit defaultCamera in layer metadata
    root_layer = stage.GetRootLayer()
    default_cam = root_layer.customLayerData.get("defaultCamera")
    if default_cam:
        prim = stage.GetPrimAtPath(default_cam)
        if prim and prim.IsA(UsdGeom.Camera):
            return str(prim.GetPath())

    # Collect all cameras
    cameras = []
    for prim in stage.Traverse():
        if prim.IsA(UsdGeom.Camera):
            cameras.append(prim)

    if not cameras:
        return None  # caller should use bbox-fit

    # Priority 2: name matching
    for cam in cameras:
        name_lower = cam.GetName().lower()
        for pattern in HERO_NAME_PATTERNS:
            if pattern in name_lower:
                return str(cam.GetPath())

    # Priority 3: single camera
    if len(cameras) == 1:
        return str(cameras[0].GetPath())

    # Priority 4: widest FOV, skip top-down
    best_cam = None
    lowest_focal = float("inf")
    for cam in cameras:
        focal = cam.GetAttribute("focalLength").Get() or 50.0
        if _is_top_down(stage, cam):
            continue
        if focal < lowest_focal:
            lowest_focal = focal
            best_cam = cam

    if best_cam:
        return str(best_cam.GetPath())

    # Priority 5: first in traversal
    return str(cameras[0].GetPath())


def _is_top_down(stage: Usd.Stage, cam_prim) -> bool:
    """Heuristic: camera looking straight down (X rotation ~90°)."""
    xformable = UsdGeom.Xformable(cam_prim)
    xform = xformable.ComputeLocalToWorldTransform(Usd.TimeCode.Default())
    # Extract the forward vector (negative Z in camera space)
    forward = xform.TransformDir(Gf.Vec3d(0, 0, -1))
    up_axis = UsdGeom.GetStageUpAxis(stage)
    if up_axis == UsdGeom.Tokens.z:
        world_down = Gf.Vec3d(0, 0, -1)
    else:
        world_down = Gf.Vec3d(0, -1, 0)
    # If forward is within ~10° of straight down, it's top-down
    dot = Gf.Dot(forward.GetNormalized(), world_down)
    return dot > math.cos(math.radians(10))

Bbox-Fit Fallback

When no authored camera exists, compute a fit-all orbit:

def compute_bbox_fit_camera(stage: Usd.Stage, fov_deg: float = 60.0):
    """Return (target, distance, elevation, azimuth) for an OrbitCamera."""
    bbox_cache = UsdGeom.BBoxCache(Usd.TimeCode.Default(), ["default", "render"])
    world_bbox = bbox_cache.ComputeWorldBound(stage.GetPseudoRoot())
    bbox_range = world_bbox.ComputeAlignedBox()

    center = (bbox_range.GetMin() + bbox_range.GetMax()) / 2.0
    size = bbox_range.GetMax() - bbox_range.GetMin()
    max_dim = max(size[0], size[1], size[2])

    # Distance to fit the bounding sphere in view
    half_fov = math.radians(fov_deg / 2.0)
    distance = (max_dim / 2.0) / math.tan(half_fov) * 1.2  # 20% padding

    # Default orbit angles: slight elevation, 3/4 azimuth
    elevation = math.radians(25.0)
    azimuth = math.radians(-45.0)

    return {
        "target": [center[0], center[1], center[2]],
        "distance": distance,
        "elevation": elevation,
        "azimuth": azimuth,
    }

Emitting camera_config.json

During app build or stage load, write a config the frontend can consume:

import json

def emit_camera_config(stage: Usd.Stage, output_path: str = "camera_config.json"):
    """Write camera config for the frontend."""
    cameras = []
    for prim in stage.Traverse():
        if prim.IsA(UsdGeom.Camera):
            cam = UsdGeom.Camera(prim)
            cameras.append({
                "path": str(prim.GetPath()),
                "name": prim.GetName(),
                "focalLength": cam.GetFocalLengthAttr().Get() or 50.0,
            })

    best = find_best_camera(stage)
    config = {
        "cameras": cameras,
        "defaultCamera": best,
        "hasBboxFallback": best is None,
    }

    if best is None:
        config["bboxFit"] = compute_bbox_fit_camera(stage)

    with open(output_path, "w") as f:
        json.dump(config, f, indent=2)

    return config

Integration Points

Server (stage-loading)

Call find_best_camera() immediately after Usd.Stage.Open(). If a camera is found, set it as the active render camera for the first frame:

stage = Usd.Stage.Open(stage_path)
best_camera = find_best_camera(stage)
if best_camera:
    # Point the render product at the authored camera
    renderer.set_active_camera(best_camera)
else:
    # Use bbox-fit orbit as the session camera
    fit = compute_bbox_fit_camera(stage)
    orbit_camera.target = fit["target"]
    orbit_camera.distance = fit["distance"]
    orbit_camera.elevation = fit["elevation"]
    orbit_camera.azimuth = fit["azimuth"]

Frontend (streaming-client)

On stage load response, read the camera list and set the initial view. If the app includes a camera picker (see camera-picker skill), populate it from the same data.

Build Pipeline

For pre-built/deployed apps where the stage is known at build time, run emit_camera_config() during the build step and bundle the JSON with the app assets. The frontend reads it at startup without needing a round-trip to the server.

Gotchas

• defaultCamera in layer metadata is a custom field — not all stages set it.

The heuristic handles this gracefully.

• Some stages define cameras inside referenced assets (props with internal

cameras). Filter to cameras under /World/Cameras or at the root level to avoid picking internal asset cameras.

• Top-down cameras are useful for plan views but make poor defaults for first

impressions. The heuristic deprioritizes them.

• For multi-GPU or multi-viewport setups, each viewport can have its own

camera. This skill picks the initial default only.

See Also

• camera-controls — orbit, pan, zoom, and fly input handling.
• camera-picker — UI dropdown for switching between stage cameras.
• stage-loading — stage open and session setup.
• stage-hierarchy — traversal and bbox computation.

Camera Controls

Triggers

Use this skill for requests mentioning orbit camera, pan, zoom, camera controls, viewport navigation, fit to scene, camera aspect, letterbox coordinates, camera gizmos, row-major camera matrices, or cameras inside geometry.

ovrtx does not provide native camera input handling. The camera is a USD prim, and the app updates its omni:xform every frame or after input changes.

Read viewer-input-routing first when the task involves WebRTC/SHM input callbacks, ovui button ids, viewport input gating, wheel events, or click-vs-drag dispatch. This skill owns camera state and camera math.

Input Mapping

This section is a camera-facing summary. viewer-input-routing is the primary source for transport normalization and input ownership.

For local ovui callbacks, button ids differ from the OrbitCamera helper:

• ovui: 0=left, 1=right, 2=middle
• OrbitCamera: 0=left, 1=middle, 2=right
• Local ovui maps exactly as 0 -> left/orbit, 2 -> middle/pan, and 1 -> right/dolly.

def camera_button_from_ovui(button: int) -> int | None:
    return {0: 0, 2: 1, 1: 2}.get(button)

For WebRTC ovstream.InputEvent callbacks, do not treat raw button integers as browser DOM button ids. ovstream.MouseButton uses NONE=0, LEFT=1, MIDDLE=2, RIGHT=3. Normalize to the shared camera helper convention before calling camera or pick code:

def camera_button_from_ovstream(raw_button) -> int | None:
    try:
        button = raw_button if isinstance(raw_button, ovstream.MouseButton) else ovstream.MouseButton(raw_button)
    except Exception:
        return None
    if button == ovstream.MouseButton.LEFT:
        return 0
    if button == ovstream.MouseButton.MIDDLE:
        return 1
    if button == ovstream.MouseButton.RIGHT:
        return 2
    return None

Use left drag for orbit, middle drag for pan, right drag for dolly/zoom, and wheel for zoom. For desktop apps with modifier keys, use Alt+LMB for orbit, Alt+MMB for pan, Alt+RMB for dolly. Optionally support RMB+WASD fly mode for free camera movement. Left-click selection should fire only on release when movement stayed below the drag threshold.

Render Aspect

When creating or explicitly reconfiguring the render product resolution, update camera viewport dimensions and projection aspect in the same operation. For USD cameras, keep horizontal aperture stable and derive vertical aperture from the render size:

def update_camera_aspect(stage, camera_path: str, width: int, height: int) -> None:
    cam = stage.GetPrimAtPath(camera_path)
    if not cam or not cam.IsValid() or width <= 0 or height <= 0:
        return
    h_attr = cam.GetAttribute("horizontalAperture")
    v_attr = cam.GetAttribute("verticalAperture")
    h_aperture = float(h_attr.Get() or 20.955)
    v_attr.Set(h_aperture * float(height) / float(width))

Browser streaming should keep a fixed server render resolution, display the video with object-fit: contain, and avoid sending resize messages for CSS layout changes. NVST handles letterbox coordinate mapping for WebRTC input carried as binary InputEvent structs; app-owned DOM math should still use the visible image rectangle before orbit, pan, zoom, or pick calculations.

Input transport rules:

• WebRTC: use the NVST native input channel and handle InputEvent structs from ovstream callbacks.
• SHM: use ovstream.ShmClient.send_input_event() from Python, or ovstream_shm_client_send_input_event() from C, with InputEvent structs; do not send JSON mouseInput.
• In-process: call camera controller methods directly from the Python/C++ UI event loop.

For browser-streamed React apps, gate native input with an app-level viewport ownership flag. UI panels should send setViewportInputActive {active:false}; the viewport sends active:true on pointer entry/down and active:false on pointer leave. The server should ignore native input while inactive and cancel any drag state:

def set_viewport_input_active(self, active: bool) -> None:
    self._viewport_input_active = bool(active)
    if not self._viewport_input_active:
        self.camera.cancel_interaction()

def handle_input(self, event):
    if not self._viewport_input_active:
        self.camera.cancel_interaction()
        return
    # Normal orbit, pan, zoom, and click-to-pick handling.

This prevents sidebar, tree, top-bar, and inspector interactions from reaching the orbit controller as stale WebRTC mouse input.

For WebRTC servers, initialize the viewport input gate to active when the only native input source is the stream surface. Otherwise the first mouse-down of a click can arrive before the React setViewportInputActive {active:true} data channel message, and the release will not be recognized as a click. DOM panels should still send active:false on pointer enter/down to disable camera and picking while the user interacts with UI chrome.

Drag Threshold (Click vs Drag Discrimination)

A short press-and-release should be treated as a click (selection, context menu), not a drag (orbit, pan, dolly). Track movement from press to release and compare against a threshold. The default desktop threshold is a 1 px delta: any move event with abs(dx) > 1.0 or abs(dy) > 1.0 turns the gesture into a drag.

DRAG_THRESHOLD_PX = 1.0  # pixels of movement before press becomes drag

class InputState:
    def __init__(self):
        self.last_x: float = 0.0
        self.last_y: float = 0.0
        self.exceeded_threshold: bool = False

    def on_press(self, x: float, y: float):
        self.last_x = x
        self.last_y = y
        self.exceeded_threshold = False

    def on_move(self, x: float, y: float) -> bool:
        """Returns True if this motion exceeds the drag threshold."""
        dx = x - self.last_x
        dy = y - self.last_y
        self.last_x = x
        self.last_y = y
        if not self.exceeded_threshold:
            if abs(dx) > DRAG_THRESHOLD_PX or abs(dy) > DRAG_THRESHOLD_PX:
                self.exceeded_threshold = True
        return self.exceeded_threshold

    def was_click(self) -> bool:
        """Call on release — True means the gesture was a click, not a drag."""
        return not self.exceeded_threshold

Usage rules:

• LMB: if was_click() → fire selection pick at release position. If threshold exceeded → it was an orbit drag, do not select.
• RMB: if was_click() → show context menu (see local-viewer). If threshold exceeded → it was a look/dolly, suppress menu.
• MMB: always pan (no click action on middle button).
• Transform gizmo: if the press begins on or near a selected transform

handle/pivot, enter transform-drag mode for the whole mouse-down and suppress orbit and click-pick on release.

• Use the same coordinate space passed to the camera helper. Local and Tauri

pointer events should be mapped through the letterboxed image rect first, so the camera sees render-pixel coordinates.

• Use a 1 px threshold for precise desktop input. Increase to 8–10 only for

touch-first input.

• For browser-streamed React apps, a 4–6 px threshold is often more tolerant of

WebRTC/browser pointer jitter around click selection.

Gizmo Hit Testing And Input Ownership

For lightweight local viewers that combine a SceneView overlay with app-owned mouse callbacks, keep a single input owner for each mouse-down. Project the selected prim pivot into the visible rendered image rectangle and treat a press near that point as transform intent; otherwise route the press through normal camera/pick behavior.

def project_world_to_viewport(point, view, proj, image_rect, widget_origin):
    p = np.array([point[0], point[1], point[2], 1.0], dtype=np.float64)
    clip = proj @ (view @ p)
    if abs(float(clip[3])) < 1e-8:
        return None
    ndc = clip[:3] / clip[3]
    if not np.isfinite(ndc).all() or ndc[2] < -1.0 or ndc[2] > 1.0:
        return None
    off_x, off_y, draw_w, draw_h = image_rect
    x = widget_origin[0] + off_x + (ndc[0] * 0.5 + 0.5) * draw_w
    y = widget_origin[1] + off_y + (1.0 - (ndc[1] * 0.5 + 0.5)) * draw_h
    return float(x), float(y)

def pointer_is_near_selected_gizmo(screen_x, screen_y, selected_pivot):
    projected = project_world_to_viewport(selected_pivot, view, proj, image_rect, widget_origin)
    if projected is None:
        return False
    dx = screen_x - projected[0]
    dy = screen_y - projected[1]
    return dx * dx + dy * dy <= 160.0 * 160.0

This fallback does not replace a real axis-handle manipulator when the shell has one. It ensures the viewer still satisfies direct manipulation when a standalone ovui build displays the gizmo but does not deliver lower-level handle drag events into the app's transform model.

Sanitize State

NaN camera state poisons projection, picking, overlays, and ovrtx writes.

MIN_DISTANCE = 0.01
MAX_ELEVATION = math.pi / 2 - 0.01

def sanitize_camera(camera) -> None:
    if not math.isfinite(float(camera.azimuth)):
        camera.azimuth = -1.5708
    if not math.isfinite(float(camera.elevation)):
        camera.elevation = 0.0
    camera.elevation = max(-MAX_ELEVATION, min(MAX_ELEVATION, camera.elevation))
    try:
        camera.distance = max(MIN_DISTANCE, float(camera.distance))
    except Exception:
        camera.distance = MIN_DISTANCE
    if not math.isfinite(camera.distance):
        camera.distance = MIN_DISTANCE
    target = np.asarray(camera.target, dtype=np.float64)
    if target.shape != (3,) or not np.isfinite(target).all():
        target = np.array([-74.5, 103.0, -22.5], dtype=np.float64)
    camera.target = target

Call this before handling input and before generating matrices.

Row-Major ovrtx Camera Matrix

ovrtx consumes USD GfMatrix4d row-vector layout:

M = np.eye(4, dtype=np.float64)
M[0, :3] = right       # X basis
M[1, :3] = up          # Y basis
M[2, :3] = -forward    # camera local -Z looks forward
M[3, :3] = eye         # translation

For Y-up scenes:

forward = target - eye
forward /= np.linalg.norm(forward)
world_up = np.array([0.0, 1.0, 0.0])
right = np.cross(forward, world_up); right /= np.linalg.norm(right)
up = np.cross(right, forward)

Use world_up = [0, 0, 1] for Z-up scenes. The common mistake is putting axes in columns, which puts the camera inside or under geometry.

If your camera helper returns a GL view matrix, convert it:

world_matrix = np.ascontiguousarray(np.linalg.inv(view_matrix).T, dtype=np.float64)

Write To ovrtx

xform = np.ascontiguousarray(camera.get_camera_xform(), dtype=np.float64)
if xform.shape == (4, 4) and np.isfinite(xform).all():
    renderer.write_attribute(
        prim_paths=["/Session/Cameras/Main"],
        attribute_name="omni:xform",
        tensor=xform.reshape(1, 4, 4),
        semantic=ovrtx.Semantic.XFORM_MAT4x4,
        prim_mode=ovrtx.PrimMode.CREATE_NEW,
    )

Use the actual inline session camera path from stage-loading.

Fit Camera To Stage

Search for authored UsdGeom.Camera prims first. If the app policy allows stage cameras, copy the selected authored camera's focal length, apertures, clipping range, projection, and transform into the viewer camera before falling back to bounds fitting.

If no authored camera exists, compute a world bbox via stage-hierarchy, then set target to the bbox center and distance from max dimension and focal length/field of view. Choose the initial view for the kind of stage:

• For general object/prop scenes, a three-quarter orbit view is usually safe.
• For Z-up exterior or architectural scenes, avoid a steep roof-down first view.

Prefer a lower elevation overview so walls, entrances, windows, racks, and scene context are visible.

• For very wide or flat scenes, increase distance and lower elevation rather

than aiming straight down.

When the first view matters, render 4-6 candidate camera poses, build a small contact sheet, and choose the least occluded view. Candidate sets should vary azimuth, elevation, and distance while keeping the same bbox target.

Inline Local Camera Gizmo

For local ovui, build the gizmo directly in the viewport ZStack with omni.ui_scene.SceneView; do not use the streaming server's headless overlay compositor.

class OverlayCamera(sc.AbstractManipulatorModel):
    def get_as_floats(self, item):
        if item == self.get_item("projection"):
            f = 1.0 / math.tan(math.radians(30.0))
            return [f,0,0,0, 0,f,0,0, 0,0,-1.002,-1, 0,0,-0.2002,0]
        if item == self.get_item("view"):
            return [1,0,0,0, 0,1,0,0, 0,0,1,0, 0,0,-4,1]
        return []

class OrbitRingManipulator(sc.Manipulator):
    def __init__(self, on_orbit_delta, pixel_scale: float, **kwargs):
        super().__init__(**kwargs)
        self._on_orbit_delta = on_orbit_delta
        self._pixel_scale = float(pixel_scale)
        self._drag = sc.DragGesture(on_changed_fn=self._on_changed, on_began_fn=lambda _s: self.invalidate(), on_ended_fn=lambda _s: self.invalidate())
        self._drag.mouse_button = 0
    def on_build(self):
        for axis, color in enumerate((0xD134BCFF, 0x9EFF6BFF, 0x94C7FFFF)):
            pts = [([math.cos(i*math.tau/72), math.sin(i*math.tau/72), 0], [math.cos(i*math.tau/72), 0, math.sin(i*math.tau/72)], [0, math.cos(i*math.tau/72), math.sin(i*math.tau/72)])[axis] for i in range(73)]
            for a, b in zip(pts, pts[1:]):
                sc.Line(a, b, color=color, thickness=3.0, intersection_thickness=18.0, gesture=self._drag)
        sc.Screen(gesture=self._drag)
    def _on_changed(self, sender):
        payload = getattr(sender, "gesture_payload", None)
        if payload is not None:
            dx_ndc, dy_ndc = payload.mouse_moved
            self._on_orbit_delta(float(dx_ndc), float(-dy_ndc), self._pixel_scale)

Toggle the gizmo from a header button. DragGesture instances must be created once and reused.

Gotchas

• Use omni:xform, not authored USD xformOp:*, for live ovrtx camera updates.
• Use Semantic.XFORM_MAT4x4 and PrimMode.CREATE_NEW.
• Skip writes if the 4x4 matrix is non-finite.
• Clamp local mouse coordinates through the visible rendered image rect so letterboxing does not skew orbit/pick math.

Alt+Modifier Input Mapping (Desktop Apps)

For Qt or native windowing with modifier keys:

def on_mouse_press(event):
    if event.modifiers() & Alt:
        if event.button() == LeftButton:
            mode = "orbit"
        elif event.button() == MiddleButton:
            mode = "pan"
        elif event.button() == RightButton:
            mode = "dolly"
    elif event.button() == RightButton:
        mode = "fly_look"  # enter RMB+WASD fly mode
    elif event.button() == LeftButton:
        mode = "select"  # click-to-select (fire on release if no drag)

WASD Fly Mode

When the right mouse button is held, enable keyboard-driven fly movement:

class FlyState:
    def __init__(self):
        self.keys_held: set[str] = set()
        self.speed = 2.0  # units/second, adjustable via scroll wheel while RMB held

    def update(self, camera, dt: float):
        if not self.keys_held:
            return
        forward = camera.forward_vector()
        right = camera.right_vector()
        up = camera.world_up  # [0,0,1] for Z-up, [0,1,0] for Y-up
        move = np.zeros(3, dtype=np.float64)
        if "w" in self.keys_held: move += forward
        if "s" in self.keys_held: move -= forward
        if "d" in self.keys_held: move += right
        if "a" in self.keys_held: move -= right
        if "e" in self.keys_held: move += up
        if "q" in self.keys_held: move -= up
        norm = np.linalg.norm(move)
        if norm > 1e-6:
            move = move / norm * self.speed * dt
        camera.target += move
        # eye moves with target (no orbit change)

While in fly mode, mouse movement rotates the camera view (adjust azimuth/elevation without changing distance). Scroll wheel adjusts fly speed.

Generated Module Checklist - camera.py

• [ ] OrbitCamera.__init__(width: int, height: int)
• [ ] OrbitCamera.on_mouse_button_down(x: float, y: float, button: int) -> None
• [ ] OrbitCamera.on_mouse_button_up(x: float, y: float, button: int) -> bool
• [ ] OrbitCamera.on_mouse_move(x: float, y: float) -> None
• [ ] OrbitCamera.orbit_delta(dx: float, dy: float, scale: float = 1.0) -> None
• [ ] OrbitCamera.on_scroll(delta: float) -> None
• [ ] OrbitCamera.get_camera_xform() -> np.ndarray
• [ ] OrbitCamera.get_view_matrix() -> np.ndarray
• [ ] OrbitCamera.get_projection_matrix(aspect_ratio=None) -> np.ndarray
• [ ] OrbitCamera._sanitize_state() -> None
• [ ] Press/release state distinguishes click from drag using the 1 px threshold.
• [ ] Matrix rows are right, up, negative-forward, translation.

Generated Module Checklist - server input routing

• [ ] MessageHandler.on_input(event) -> None
• [ ] Mouse move calls camera.on_mouse_move(x, y).
• [ ] Left-button release calls camera.on_mouse_button_up(..., 0) and picks only when it returns True.
• [ ] Middle-button input maps to camera button 1.
• [ ] Right-button input maps to camera button 2.
• [ ] Wheel input calls camera.on_scroll(delta).
• [ ] Browser pointer events are not duplicated as JSON messages.

See also: viewer-input-routing, local-viewer, stage-loading, stage-hierarchy, prim-info-display, viewport-overlays.

Camera Picker

Purpose

Stages with multiple authored cameras (e.g., a warehouse with top-down, perspective, dock-level, and aisle views) should expose those cameras to the user via a simple dropdown. This skill implements the full round-trip: server enumerates cameras → frontend renders dropdown → user selects → server switches active camera → stream updates.

Triggers

Use when:

• The stage contains two or more UsdGeom.Camera prims.
• A user asks for a camera selector, camera picker, view switcher, or viewport

dropdown.

• Building a viewer that must support multiple viewpoints.
• The camera-auto-select skill detected multiple cameras and the app should

let users explore them.

Message Protocol

Server → Client: camera_list

Sent once after stage load (or stage switch) alongside or after push_initial_state:

{
  "event_type": "camera_list",
  "payload": {
    "cameras": [
      {
        "path": "/World/Cameras/Cam_Persp",
        "name": "Cam_Persp",
        "focalLength": 35.0
      },
      {
        "path": "/World/Cameras/Cam_TopDown",
        "name": "Cam_TopDown",
        "focalLength": 50.0
      },
      {
        "path": "/World/Cameras/Cam_DockLevel",
        "name": "Cam_DockLevel",
        "focalLength": 35.0
      },
      {
        "path": "/World/Cameras/Cam_Aisle",
        "name": "Cam_Aisle",
        "focalLength": 28.0
      }
    ],
    "activeCamera": "/World/Cameras/Cam_Persp"
  }
}

Client → Server: set_camera

User selects a different camera:

{
  "event_type": "set_camera",
  "payload": {
    "path": "/World/Cameras/Cam_TopDown"
  }
}

Server → Client: camera_changed

Confirms the switch (allows UI to sync if multiple clients are connected):

{
  "event_type": "camera_changed",
  "payload": {
    "activeCamera": "/World/Cameras/Cam_TopDown"
  }
}

Server Implementation

Enumerating Cameras

from pxr import Usd, UsdGeom


def get_camera_list(stage: Usd.Stage) -> list[dict]:
    """Return all authored cameras suitable for the picker."""
    cameras = []
    for prim in stage.Traverse():
        if prim.IsA(UsdGeom.Camera):
            cam = UsdGeom.Camera(prim)
            cameras.append({
                "path": str(prim.GetPath()),
                "name": prim.GetName(),
                "focalLength": cam.GetFocalLengthAttr().Get() or 50.0,
            })
    return cameras

Handling set_camera

When the server receives set_camera:

def handle_set_camera(self, payload: dict) -> None:
    camera_path = payload.get("path", "")
    prim = self.stage.GetPrimAtPath(camera_path)
    if not prim or not prim.IsA(UsdGeom.Camera):
        self.send_error(f"Invalid camera path: {camera_path}")
        return

    cam = UsdGeom.Camera(prim)
    xformable = UsdGeom.Xformable(prim)
    xform = xformable.ComputeLocalToWorldTransform(Usd.TimeCode.Default())

    # Option A: Copy authored camera transform to the session camera
    # This preserves orbit controls centered on where the camera looks.
    self._apply_camera_xform(xform, cam)

    # Option B: Switch the render product to point at the authored prim
    # self.renderer.set_active_camera(camera_path)

    self.active_camera = camera_path
    self.broadcast({
        "event_type": "camera_changed",
        "payload": {"activeCamera": camera_path},
    })


def _apply_camera_xform(self, xform, cam_schema) -> None:
    """Apply an authored camera's transform and lens to the session camera."""
    import numpy as np
    from pxr import Gf

    # Extract position and orientation
    eye = xform.ExtractTranslation()
    forward = xform.TransformDir(Gf.Vec3d(0, 0, -1)).GetNormalized()

    # Compute orbit parameters from the authored camera
    focal_length = cam_schema.GetFocalLengthAttr().Get() or 50.0

    # Set orbit camera to look from this position in the authored direction
    # Use a reasonable target distance based on focal length
    target_distance = focal_length * 0.5  # heuristic: longer lens = farther target
    target = eye + forward * target_distance

    self.orbit_camera.target = np.array([target[0], target[1], target[2]])
    self.orbit_camera.distance = target_distance
    # Recompute azimuth/elevation from the authored transform
    self.orbit_camera.set_from_eye_and_target(
        eye=np.array([eye[0], eye[1], eye[2]]),
        target=np.array([target[0], target[1], target[2]]),
    )

    # Update focal length on the render camera
    self.orbit_camera.focal_length = focal_length

Sending Camera List on Stage Load

In the stage load handler, after push_initial_state:

def on_stage_loaded(self, stage: Usd.Stage) -> None:
    # ... existing push_initial_state logic ...

    cameras = get_camera_list(stage)
    if cameras:
        from camera_auto_select import find_best_camera
        active = find_best_camera(stage) or cameras[0]["path"]
        self.active_camera = active
        self.broadcast({
            "event_type": "camera_list",
            "payload": {
                "cameras": cameras,
                "activeCamera": active,
            },
        })

Frontend Implementation (React)

CameraPicker Component

import React from "react";

interface CameraInfo {
  path: string;
  name: string;
  focalLength: number;
}

interface CameraPickerProps {
  cameras: CameraInfo[];
  activeCamera: string;
  onSelect: (path: string) => void;
}

export function CameraPicker({ cameras, activeCamera, onSelect }: CameraPickerProps) {
  if (cameras.length < 2) return null; // No picker needed for 0-1 cameras

  return (
    <div className="camera-picker">
      <label htmlFor="camera-select">Camera</label>
      <select
        id="camera-select"
        value={activeCamera}
        onChange={(e) => onSelect(e.target.value)}
      >
        {cameras.map((cam) => (
          <option key={cam.path} value={cam.path}>
            {formatCameraName(cam.name)} ({cam.focalLength}mm)
          </option>
        ))}
      </select>
    </div>
  );
}

function formatCameraName(name: string): string {
  // "Cam_TopDown" -> "Top Down", "Cam_Persp" -> "Persp"
  return name
    .replace(/^Cam_?/i, "")
    .replace(/([a-z])([A-Z])/g, "$1 $2")
    .replace(/_/g, " ")
    .trim() || name;
}

Wiring Into the App

function ViewerApp() {
  const [cameras, setCameras] = useState<CameraInfo[]>([]);
  const [activeCamera, setActiveCamera] = useState("");

  useEffect(() => {
    // Listen for camera_list from server
    stream.on("camera_list", (payload) => {
      setCameras(payload.cameras);
      setActiveCamera(payload.activeCamera);
    });

    stream.on("camera_changed", (payload) => {
      setActiveCamera(payload.activeCamera);
    });
  }, []);

  const handleCameraSelect = (path: string) => {
    stream.send({ event_type: "set_camera", payload: { path } });
  };

  return (
    <div className="viewer">
      <header className="toolbar">
        <CameraPicker
          cameras={cameras}
          activeCamera={activeCamera}
          onSelect={handleCameraSelect}
        />
      </header>
      <VideoViewport />
    </div>
  );
}

Styling

Place the picker in the toolbar/header bar alongside other controls (render settings, scene tree toggle, etc.). Keep it compact:

.camera-picker {
  display: flex;
  align-items: center;
  gap: 8px;
}

.camera-picker select {
  padding: 4px 8px;
  border-radius: 4px;
  background: var(--surface-2);
  color: var(--text-primary);
  border: 1px solid var(--border);
  font-size: 13px;
}

.camera-picker label {
  font-size: 12px;
  color: var(--text-secondary);
  text-transform: uppercase;
  letter-spacing: 0.5px;
}

Behavior Rules

1. Hide the picker when ≤1 camera exists. If the stage has zero or one camera, the dropdown adds no value. The camera-auto-select skill handles the default; no UI needed.

2. Show the picker when ≥2 cameras exist. Even if auto-select picked a good default, the user should be able to explore other views.

3. Include an "Orbit (free)" entry when the viewer supports free orbit mode. Selecting it returns to the user-controlled orbit camera without snapping to any authored camera:

   <option value="__orbit__">Free Orbit</option>

4. Preserve orbit state on switch. When the user selects an authored camera, apply its transform to the orbit controller. The user can then orbit from that starting point. Switching cameras does not lock the viewport.

5. Re-emit `camera_list` on stage switch. If the user loads a different stage, the old camera list is stale. Treat it like a fresh load.

6. SHM/Electron path. Same message protocol over the SHM JSON channel. The Electron renderer handles camera_list and camera_changed identically to WebRTC.

Keyboard Shortcuts (Optional)

For power users, bind number keys to cameras:

Only activate when the viewport has focus (not when typing in a text field).

Gotchas

• Authored cameras may have different aspect ratios or clipping planes.

When switching, update the render product resolution or adjust vertical aperture to match the stream aspect (see camera-controls skill).

• Some stages nest cameras inside referenced assets (props). Filter to

cameras that are direct children of a Cameras Xform or at the scene root to avoid showing internal asset cameras.

• The orbit controller's set_from_eye_and_target must handle both Y-up and

Z-up stages. Check UsdGeom.GetStageUpAxis().

See Also

• camera-auto-select — picks the initial camera; picker shows the alternatives.
• camera-controls — orbit, pan, zoom after a camera is chosen.
• streaming-messages — message protocol patterns.
• stage-loading — stage open lifecycle.

Cloud Assets

Triggers

Use this skill for requests mentioning S3 assets, MinIO, cloud assets, buckets, remote USD files, ovstorage, asset caches, object storage, asset catalogs, thumbnail grids, or S3 browsing.

ovrtx requires local filesystem paths. Cloud stages must be synced to a local cache that preserves relative directory structure for textures, materials, sublayers, and referenced USD files.

Architecture

Browser -> ov_web_viewer_server
  -> StorageManager
  -> ovstorage S3 client
  -> S3/MinIO bucket samples_data/
  -> local cache /tmp/ov-stage-cache/samples_data/
  -> renderer.open_usd(local_path) or open_usd_from_string(inline root)

The manager connects to S3, syncs the full tree on first load, validates cache by size on later loads, and resolves requested filenames to local cached paths.

MinIO Setup

curl -sSL https://dl.min.io/server/minio/release/linux-amd64/minio -o /tmp/minio && chmod +x /tmp/minio
curl -sSL https://dl.min.io/client/mc/release/linux-amd64/mc -o /tmp/mc && chmod +x /tmp/mc
mkdir -p /tmp/minio-data
MINIO_ROOT_USER=minioadmin MINIO_ROOT_PASSWORD=minioadmin /tmp/minio server /tmp/minio-data --address :9000 --console-address :9001 &
/tmp/mc alias set local http://localhost:9000 minioadmin minioadmin
/tmp/mc mb local/ov-viewer-samples
/tmp/mc cp --recursive samples/samples_data/ local/ov-viewer-samples/samples_data/

Dependencies

pip install ovstorage
pip install "boto3>=1.34"  # optional for generated direct-S3 helpers

Use ovstorage for the primary authenticated object-storage path. Generate the viewer-local StorageManager wrapper in the app server.

Config

@dataclass(frozen=True)
class StorageConfig:
    enabled: bool = False
    bucket_url: str = "s3://ov-viewer-samples"
    endpoint_url: str = "http://localhost:9000"
    region: str = "us-east-1"
    addressing_style: str = "path"  # MinIO uses path; AWS often uses virtual
    cache_dir: str = "/tmp/ov-stage-cache"
    prefix: str = "samples_data"

    @classmethod
    def from_env(cls):
        return cls(
            enabled=os.environ.get("OVSTORAGE_ENABLED", "0") == "1",
            bucket_url=os.environ.get("OVSTORAGE_BUCKET_URL", "s3://ov-viewer-samples"),
            endpoint_url=os.environ.get("OVSTORAGE_ENDPOINT_URL", "http://localhost:9000"),
            region=os.environ.get("OVSTORAGE_REGION", "us-east-1"),
            addressing_style=os.environ.get("OVSTORAGE_ADDRESSING", "path"),
            cache_dir=os.environ.get("OVSTORAGE_CACHE_DIR", "/tmp/ov-stage-cache"),
            prefix=os.environ.get("OVSTORAGE_PREFIX", "samples_data"),
        )

Manager Behaviors

class StorageManager:
    def __init__(self, config):
        self.config = config
        self.enabled = config.enabled
        if self.enabled:
            import ovstorage
            self._client = ovstorage.open(config.bucket_url, config=ovstorage.Config(
                s3_endpoint_url=config.endpoint_url,
                s3_region=config.region,
                s3_addressing_style=config.addressing_style,
            ))

    def sync_all(self) -> bool:
        entries = self._client.walk(f"{self.config.prefix}/", max_depth=10)
        files = [e for e in entries if e.kind.value == "file"]
        for entry in files:
            data = self._client.read(entry.relative_path)
            dest = Path(self.config.cache_dir) / entry.relative_path
            if dest.exists() and dest.stat().st_size == len(data):
                continue
            dest.parent.mkdir(parents=True, exist_ok=True)
            dest.write_bytes(data)
        return True

    def resolve_or_passthrough(self, path: str) -> str:
        if not self.enabled:
            return path
        self.sync_all()
        return self.resolve_stage(os.path.basename(path))

Server Integration

• Add sibling imports for storage_config and storage_manager.
• If using dynamic _import_sibling, register sys.modules[name] = mod before exec_module; Python dataclasses need __module__ resolvable.
• Initialize StorageManager(StorageConfig.from_env()) in the server.
• In _load_stage(), first line should resolve: url = self._storage.resolve_or_passthrough(url).
• Add --storage CLI flag and set OVSTORAGE_ENABLED=1 when present.

Environment

Run:

AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin python3 server/ov_web_viewer_server.py --storage --port 49100

Expected logs include enabled bucket, sync start, sync counts, and loading from /tmp/ov-stage-cache/....

Troubleshooting

Public S3 Asset Browsing

For browsing NVIDIA content buckets that need no credentials, use direct HTTPS listing:

Bucket Discovery

List objects under a prefix using the S3 REST XML API:

import xml.etree.ElementTree as ET
import urllib.request
import urllib.parse

S3_BUCKET = "omniverse-content-production"
S3_BASE_URL = f"https://{S3_BUCKET}.s3.us-west-2.amazonaws.com"

def list_objects(prefix: str, delimiter: str = "/", max_keys: int = 1000) -> tuple[list[str], list[str]]:
    """List object keys and common prefixes (subdirectories) under a prefix."""
    params = urllib.parse.urlencode({
        "list-type": "2",
        "prefix": prefix,
        "delimiter": delimiter,
        "max-keys": str(max_keys),
    })
    url = f"{S3_BASE_URL}?{params}"
    with urllib.request.urlopen(url, timeout=15) as resp:
        tree = ET.fromstring(resp.read())
    ns = {"s3": "http://s3.amazonaws.com/doc/2006-03-01/"}
    keys = [c.text for c in tree.findall(".//s3:Key", ns) if c.text]
    prefixes = [c.text for c in tree.findall(".//s3:CommonPrefixes/s3:Prefix", ns) if c.text]
    return keys, prefixes

Asset Catalog & Manifest

Many NVIDIA content buckets include manifest files (manifest.json, index.json, catalog.json) at category roots. Check for these first — they list assets with metadata (name, path, tags, description) without needing full prefix enumeration:

MANIFEST_NAMES = ["manifest.json", "index.json", "catalog.json", "assets.json"]

def try_load_manifest(prefix: str) -> Optional[list[dict]]:
    for name in MANIFEST_NAMES:
        url = f"{S3_BASE_URL}/{urllib.parse.quote(prefix + name, safe='/')}"
        try:
            with urllib.request.urlopen(url, timeout=10) as resp:
                return json.loads(resp.read())
        except Exception:
            continue
    return None

Fall back to prefix listing when no manifest exists.

Thumbnail Loading

Thumbnails live alongside USD files (commonly .png or .jpg with matching stem, or in a thumbnails/ subdirectory). Load them lazily in background threads and cache locally:

from pathlib import Path
from concurrent.futures import ThreadPoolExecutor

THUMBNAIL_EXTS = [".png", ".jpg", ".jpeg", ".webp"]

def find_thumbnail_key(usd_key: str, available_keys: list[str]) -> Optional[str]:
    stem = PurePosixPath(usd_key).stem
    parent = str(PurePosixPath(usd_key).parent)
    candidates = [
        f"{parent}/{stem}{ext}" for ext in THUMBNAIL_EXTS
    ] + [
        f"{parent}/thumbnails/{stem}{ext}" for ext in THUMBNAIL_EXTS
    ]
    for c in candidates:
        if c in available_keys:
            return c
    return None

def download_thumbnail(url: str, cache_path: Path, timeout: float = 10.0) -> Optional[Path]:
    if cache_path.exists():
        return cache_path
    cache_path.parent.mkdir(parents=True, exist_ok=True)
    try:
        urllib.request.urlretrieve(url, str(cache_path))
        return cache_path
    except Exception:
        return None

Use a thread pool (4-8 workers) for concurrent thumbnail downloads. Decode and scale images off the UI thread.

Category Structure

Organize assets by S3 prefix hierarchy. Common prefix = category name:

from pathlib import PurePosixPath

def categorize_assets(keys: list[str], base_prefix: str) -> dict[str, list[str]]:
    categories: dict[str, list[str]] = {}
    for key in keys:
        if not any(key.endswith(ext) for ext in [".usd", ".usda", ".usdc"]):
            continue
        rel = PurePosixPath(key).relative_to(PurePosixPath(base_prefix))
        cat = str(rel.parent) if rel.parent != PurePosixPath(".") else "General"
        categories.setdefault(cat, []).append(key)
    return categories

Local Cache Strategy

Cache downloaded assets preserving the S3 key structure so relative USD references (textures, sublayers) resolve correctly:

def cache_path_for_key(key: str, cache_root: Path) -> Path:
    return cache_root / key

def download_asset_tree(usd_key: str, cache_root: Path) -> Path:
    """Download the USD file and its directory siblings (textures, materials)."""
    parent_prefix = str(PurePosixPath(usd_key).parent) + "/"
    sibling_keys, _ = list_objects(parent_prefix, delimiter="")
    for k in sibling_keys:
        dest = cache_path_for_key(k, cache_root)
        if not dest.exists():
            url = f"{S3_BASE_URL}/{urllib.parse.quote(k, safe='/')}"
            download_thumbnail(url, dest)  # reuse download helper
    return cache_path_for_key(usd_key, cache_root)

ovrtx file loads require a local filesystem path, so always resolve through the cache before calling renderer.open_usd() or composing an inline root with open_usd_from_string().

When To Use Direct HTTPS vs S3 API

See also: stage-management, stage-loading, cloud-deployment.

Cloud Deployment

Triggers

Use this skill for requests mentioning deploy, OKAS 1, cloud deployment, session APIs, Docker, health checks, Brev, launchables, or remote deployment.

Brev Launchable Deployment (Recommended for Demo)

Use the permanent omniverse-realtime-viewer launchable: <https://brev.nvidia.com/launchable/deploy?launchableID=env-3EHjQXkUNYv2pOa3idjBeJOauvH>

Instance type: g5.xlarge (A10G, 23 GB VRAM).

Prerequisites

• Brev account with an omniverse-realtime-viewer-launchable instance (A10G or better)
• TCP/UDP Port Rules open: 80, 1024, 47998, 49100
• SSH access configured (uses Brev SSH key, typically ~/.brev/brev.pem; run brev refresh to regenerate)

Architecture

Two access modes are supported:

Option A: HTTPS via Brev domain (port 80 + Caddy WSS proxy)

Brev exposes port 80 as https://frontend-<id>.brevlab.com with TLS termination at their edge. Browsers enforce secure WebSocket (wss://) — plain ws:// is blocked as mixed-content. Caddy on port 80 serves both the frontend AND proxies the internal @nvidia/ov-web-rtc Direct signaling endpoint used by standalone ovstream:

Browser → https://frontend-<id>.brevlab.com (Brev TLS edge)
  └── port 80 → Caddy
        ├── /sign_in* → reverse_proxy localhost:49100 (ovstream WebSocket signaling)
        ├── /*        → file_server (pre-built frontend)
        └── UDP media → <PUBLIC_IP>:47998 (direct, no proxy)

The frontend is built with VITE_SIGNALING_PORT=443 so @nvidia/ov-web-rtc Direct mode connects to wss://frontend-<id>.brevlab.com:443/sign_in (same origin as the page). Brev routes this → port 80 → Caddy → localhost:49100.

This is the exposed route for standalone ovstream Direct signaling. The deployment layer may provide auth, launch, routing, and lifecycle management, but the browser WebRTC config still uses @nvidia/ov-web-rtc Direct mode with the exposed signaling endpoint. Do not replace it with a Kit, OVC, NVCF, or GFN client connection profile.

Option B: Direct IP access (port 1024 + nginx)

For internal testing where TLS isn't needed, access the viewer directly via http://<PUBLIC_IP>:1024. nginx on port 1024 proxies both the frontend and signaling:

Browser → http://<PUBLIC_IP>:1024/
  ├── nginx (port 1024) → /            → Vite dev server or static files (port 5173/3000)
  │                     → /sign_in     → ovstream signaling server (port 49100)
  └── UDP media → <PUBLIC_IP>:47998

No TLS, no mixed-content issues (both page and WebSocket are plain HTTP). Frontend uses default VITE_SIGNALING_PORT=1024 (same port as page).

Note: Do not use Brev's Cloudflare secure link (https://) with Option B.

NVST extracts the client IP from getpeername() on the TCP socket — Cloudflare

in the middle causes NAT hole-punch failure.

Critical Configuration

Option A: Caddyfile (port 80 — frontend + WSS proxy)

{
    auto_https off
}

:80 {
    handle /sign_in* {
        reverse_proxy localhost:49100
    }
    handle {
        root * /opt/ov-viewer/clients/webrtc-browser/dist
        file_server
    }
}

Install Caddy:

curl -o /tmp/caddy.tar.gz -sL "https://github.com/caddyserver/caddy/releases/download/v2.8.4/caddy_2.8.4_linux_amd64.tar.gz"
tar -xzf /tmp/caddy.tar.gz -C /tmp caddy
sudo mv /tmp/caddy /usr/local/bin/caddy

Option B: nginx (port 1024 — direct access)

server {
    listen 1024;

    location / {
        proxy_pass http://localhost:3000/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
    }

    location /sign_in {
        proxy_pass http://localhost:49100/sign_in;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
    }
}

Server (ov_web_viewer_server.py)

The server binds signaling on port 49100 and media on port 47998:

python3 ov_web_viewer_server.py --port 49100 --public-ip "$PUBLIC_IP" \
    --stage /opt/ov-viewer/samples_data/stage01.usd

• --public-ip sets the ICE candidate IP in SDP. Required for NAT traversal.
• Media defaults to UDP :47998 — must match Brev port rule.

Frontend build (bake signaling port)

The frontend reads VITE_SIGNALING_PORT at build time:

cd clients/webrtc-browser

# Option A (HTTPS via Brev domain):
VITE_SIGNALING_PORT=443 npx vite build

# Option B (direct IP on port 1024):
VITE_SIGNALING_PORT=1024 npx vite build

This makes the SDK connect to the matching port for signaling. With Option A, Caddy proxies the WebSocket transparently. With Option B, nginx proxies it on the same port.

Deployment Steps

1. Build frontend locally:

   export VIEWER_ROOT=/path/to/generated-viewer
   cd "$VIEWER_ROOT/clients/webrtc-browser"
   npm install --ignore-scripts
   # Choose one:
   VITE_SIGNALING_PORT=443 npx vite build   # Option A (HTTPS)
   VITE_SIGNALING_PORT=1024 npx vite build  # Option B (direct)

2. Refresh SSH and rsync payload:

   brev refresh
   INSTANCE="omniverse-realtime-viewer-launchable-XXXXXX"

   # Wheels (~2.5GB ovrtx + ovstream)
   rsync -az "$VIEWER_ROOT/deps/wheels/" $INSTANCE:/tmp/ov-deploy/

   # Server + samples + frontend
   rsync -az "$VIEWER_ROOT/server" $INSTANCE:/tmp/ov-deploy/
   rsync -az "$VIEWER_ROOT/samples_data" $INSTANCE:/tmp/ov-deploy/
   rsync -az "$VIEWER_ROOT/clients/webrtc-browser/dist" $INSTANCE:/tmp/ov-deploy/frontend-dist

3. Remote setup (SSH into instance):

   # System deps
   sudo apt-get update -qq
   sudo apt-get install -y -qq python3-pip python3-venv python3-dev \
       libgomp1 libatomic1 libgl1 libglx0 libx11-6 libxau6 libxdmcp6 \
       libxcb1 libbsd0 libmd0 libegl1 libglib2.0-0

   # Deploy directory
   sudo mkdir -p /opt/ov-viewer && sudo chown $(whoami) /opt/ov-viewer
   cp -r /tmp/ov-deploy/server /opt/ov-viewer/
   cp -r /tmp/ov-deploy/samples_data /opt/ov-viewer/
   mkdir -p /opt/ov-viewer/clients/webrtc-browser
   cp -r /tmp/ov-deploy/frontend-dist /opt/ov-viewer/clients/webrtc-browser/dist

   # Python venv + wheels
   cd /opt/ov-viewer/server
   python3 -m venv .venv
   source .venv/bin/activate
   pip install --upgrade pip -q
   pip install /tmp/ov-deploy/*.whl -q
   pip install numpy warp-lang "usd-core==24.11" -q
   python3 -c "import ovstream, ovrtx; print('OK')"
   deactivate

   # Install Caddy (Option A only)
   curl -o /tmp/caddy.tar.gz -sL "https://github.com/caddyserver/caddy/releases/download/v2.8.4/caddy_2.8.4_linux_amd64.tar.gz"
   tar -xzf /tmp/caddy.tar.gz -C /tmp caddy && sudo mv /tmp/caddy /usr/local/bin/caddy

4. Start server:

   cd /opt/ov-viewer/server
   source .venv/bin/activate
   export OVRTX_SKIP_USD_CHECK=1
   OVSTREAM_DIR=$(python3 -c "import ovstream, os; print(os.path.dirname(ovstream.__file__))")
   OVRTX_BIN=$(python3 -c "import ovrtx, os; print(os.path.join(os.path.dirname(ovrtx.__file__), 'bin'))")
   export LD_LIBRARY_PATH="${OVSTREAM_DIR}:${OVRTX_BIN}:${LD_LIBRARY_PATH:-}"
   PUBLIC_IP=$(curl -sf ifconfig.me)

   nohup python3 ov_web_viewer_server.py --port 49100 --public-ip "$PUBLIC_IP" \
       --stage /opt/ov-viewer/samples_data/stage01.usd > /tmp/server.log 2>&1 &

5. Wait for shader warmup (~5-10 min cold on A10G):

   # Monitor: port 49100 appears when ready
   watch -n5 'ss -tlnp | grep 49100 && echo READY || echo WAITING'

6. Start reverse proxy:

   # Option A: Caddy on port 80
   sudo caddy run --config /path/to/Caddyfile &

   # Option B: nginx on port 1024 (install + configure per above)
   sudo apt-get install -y nginx
   # Add server block to /etc/nginx/sites-enabled/default, then:
   sudo nginx -s reload

7. Access:

• Option A: https://frontend-<id>.brevlab.com/
• Option B: http://<PUBLIC_IP>:1024/

Gotchas & Troubleshooting

Why Two Options

Option A (Caddy + HTTPS) is required for demos and external access. Brev's HTTPS proxy terminates TLS at the edge and forwards to port 80. Browsers enforce wss:// from HTTPS pages — Caddy solves this by serving both frontend and signaling on the same origin.

Option B (nginx + direct IP) is simpler for internal dev/testing. No TLS means no mixed-content issues. Access via http://<PUBLIC_IP>:1024 bypasses Brev's Cloudflare tunnel entirely. However, NVST cannot determine the client IP through Cloudflare, so never use the Brev https:// URL with Option B.

Port 1024 and 80 are opened via Brev's "TCP/UDP Port Rules" (actual AWS security group entries), NOT via "Secure Links" (which only proxy TCP through Cloudflare).

---

Docker Container Deployment

For containerized deployments without a full orchestrator, build a standalone Docker image that bundles the ovrtx server, ovstream, sample data, and a pre-built frontend.

Base Image And System Dependencies

Use nvidia/cuda:12.6.3-base-ubuntu22.04 as the base. This provides CUDA runtime libraries without the full toolkit overhead.

Required apt packages for ovrtx rendering and ovstream:

# syntax=docker/dockerfile:1
FROM nvidia/cuda:12.6.3-base-ubuntu22.04

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
    python3 python3-pip python3-dev \
    libgomp1 libatomic1 \
    libgl1 libglx0 libegl1 libopengl0 \
    libx11-6 libxau6 libxdmcp6 libxcb1 libbsd0 libmd0 \
    libglib2.0-0 \
    curl ca-certificates \
  && rm -rf /var/lib/apt/lists/*

The # syntax=docker/dockerfile:1 directive at the top of the Dockerfile is required for BuildKit features like bind mounts.

Installing Large Python Wheels Efficiently

ovrtx and ovstream wheels can be hundreds of megabytes. Using a regular COPY + pip install doubles image size because layers retain both the wheel and the installed package. Use BuildKit bind mounts instead:

# Place wheels in deps/wheels/ relative to build context
RUN --mount=type=bind,source=deps/wheels,target=/tmp/wheels \
    pip install --no-cache-dir /tmp/wheels/*.whl

This mounts the wheels at build time without copying them into a layer. The final image only contains the installed packages.

.dockerignore

A .dockerignore file is critical. Without it, COPY . /app sends node_modules/ to the Docker daemon and can inject platform-incompatible native binaries into the image:

**/node_modules
**/.git
**/__pycache__
*.egg-info

Runtime Requirements

The container must be started with GPU access and X11 display forwarding for ovrtx headless rendering:

docker run --gpus all \
  -e DISPLAY=:99 \
  -e PUBLIC_IP=<reachable-ip> \
  -v /tmp/.X11-unix:/tmp/.X11-unix \
  -p 49100:49100 \
  -p 47998:47998/udp \
  -p 8081:8081 \
  ovrtx-viewer:latest

Shader Compilation Cold Start

After a fresh container start, the first scene load triggers GPU shader compilation. Expected times:

Do not connect clients or mark the service as ready until /healthz returns 200. The health endpoint gates on the first successfully rendered and converted frame, which occurs after shader compilation completes; it should not require an attached browser client.

Entrypoint Pattern

COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

#!/bin/bash
set -e

# Start Xvfb if no display is available
if ! xdpyinfo -display "${DISPLAY:-:99}" >/dev/null 2>&1; then
  Xvfb ${DISPLAY:-:99} -screen 0 1920x1080x24 &
  sleep 1
fi

export OVRTX_SKIP_USD_CHECK=1
exec python3 /app/server/ov_web_viewer_server.py \
  --port "${PORT:-49100}" \
  --health-port "${HEALTH_PORT:-8081}" \
  --public-ip "${PUBLIC_IP:-$(curl -s ifconfig.me)}" \
  --stage "${STAGE_PATH:-/app/samples_data/stage01.usd}"

Sample Dockerfile (Complete)

# syntax=docker/dockerfile:1
FROM nvidia/cuda:12.6.3-base-ubuntu22.04

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y --no-install-recommends \
    python3 python3-pip python3-dev \
    libgomp1 libatomic1 \
    libgl1 libglx0 libegl1 libopengl0 \
    libx11-6 libxau6 libxdmcp6 libxcb1 libbsd0 libmd0 \
    libglib2.0-0 \
    xvfb \
    curl ca-certificates \
  && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Install Python wheels without layer bloat
RUN --mount=type=bind,source=deps/wheels,target=/tmp/wheels \
    pip install --no-cache-dir /tmp/wheels/*.whl

# Copy server code and sample data
COPY server/ /app/server/
COPY samples_data/ /app/samples_data/

# Copy pre-built frontend (optional, for self-contained image)
COPY frontend/dist/ /app/frontend/dist/

COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

EXPOSE 49100 47998/udp 8081
ENTRYPOINT ["/entrypoint.sh"]

Build And Run

# Build (requires BuildKit)
DOCKER_BUILDKIT=1 docker build -t ovrtx-viewer:latest .

# Run
docker run --gpus all \
  -e PUBLIC_IP=$(curl -s ifconfig.me) \
  -p 49100:49100 \
  -p 47998:47998/udp \
  -p 8081:8081 \
  ovrtx-viewer:latest

If the host already has an X11 display or Xvfb running, mount the socket and set DISPLAY accordingly. Otherwise the entrypoint starts its own Xvfb instance.

---

OKAS 1 / Generic Session Orchestration

For production-style deployments, keep the Omniverse Realtime Viewer contract portable. Use OKAS 1 or a generic container/session orchestrator that can start one GPU container per Omniverse Realtime Viewer session, expose WebRTC signaling and media ports, route the browser to the frontend, and terminate the container when the session ends.

OKAS is orchestration/session management, not a different WebRTC client profile. It may allocate GPU resources, start the container, inject environment/config, publish routes, and manage session lifecycle. After OKAS resolves a session endpoint, the frontend uses standalone ovstream Direct config: server and signalingPort point at the exposed signaling endpoint, while media remains negotiated by WebRTC.

Registration Contract

Register the Omniverse Realtime Viewer with portable metadata:

{
  "id": "ovrtx-viewer",
  "name": "Omniverse Realtime Viewer",
  "image": "ovrtx-viewer:0.2.0",
  "description": "Omniverse Realtime Viewer using ovrtx rendering with ovstream WebRTC delivery",
  "gpuRequired": true,
  "ports": {
    "signaling": 49100,
    "mediaUdp": 47998,
    "health": 8081
  }
}

Keep deployment recipes portable. Do not bind generated apps to app registries, session-manager paths, sidecars, or caching services unless the selected deployment target explicitly provides them.

Launch Contract

A session launcher should run the same server command the Brev path uses:

export OVRTX_SKIP_USD_CHECK=1
python3 server/ov_web_viewer_server.py \
  --port "${PORT:-49100}" \
  --health-port "${HEALTH_PORT:-8081}" \
  --public-ip "${PUBLIC_IP}" \
  --stage "${STAGE_PATH:-samples/samples_data/stage01.usd}"

Health And Ports

Docker

cd deploy
docker build -t ovrtx-viewer:0.2.0 .

deploy/entrypoint.sh launches the server with $PORT and $STAGE_PATH.

Session Lifecycle

POST /sessions {application:"ovrtx-viewer"}
  → spawn one GPU process/container
  → poll GET :8081/healthz
  → mark ready
  → browser connects to frontend and internal Direct signaling
  → WebRTC media flows
  → DELETE /sessions/{id}
  → SIGTERM

Related

• streaming-server for ovstream ServerConfig details and frame handling.
• streaming-client for frontend WebRTC SDK usage.
• streaming-lifecycle for connection/reconnection behavior.
• cloud-assets when deployed sessions load stages from S3/MinIO.
• OKAS 1 or your orchestrator documentation for portal/session APIs.

<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> <!-- SPDX-License-Identifier: Apache-2.0 -->

Omniverse Realtime Viewer Conventions

These conventions are the shared behavior contract for all focused references in this skill package. If a focused reference needs one of these values, use this file instead of inventing a local rule.

Architecture

• All USD and 3D rendering uses ovrtx.
• Browser-streamed apps display an ovstream video stream in a video element.

The browser does not render USD geometry.

• Desktop apps display frames rendered by ovrtx in-process or through local

pixel transport.

Mouse And Input

• Left mouse button drag: orbit.
• Middle mouse button drag: pan.
• Right mouse button drag: dolly/zoom.
• Scroll wheel: zoom; scroll up zooms in.
• Left click selection fires on mouse release, not press.
• A press becomes a drag when either axis moves by more than 1.0 pixel.
• Local ovui button IDs are remapped before calling shared camera code:

0 -> left/orbit, 2 -> middle/pan, 1 -> right/dolly.

• WebRTC input uses the NVST native input channel. The browser streaming

library forwards binary InputEvent structs to ovstream; React does not implement client-side 3D camera math or send JSON camera input.

• SHM input uses ovstream.ShmClient.send_input_event() from Python or

ovstream_shm_client_send_input_event() from C with InputEvent structs. Do not send JSON mouseInput for SHM camera control.

• In-process transports call the Python/C++ camera, selection, and settings

APIs directly.

Selection

• Default viewer behavior is single-select. Selecting a new prim replaces the

previous selection; clicking empty space clears selection. If multi-select is requested, every subscriber must explicitly support mixed values and multiple highlighted prims.

• Viewport selection should use the selected delivery path's native picking

route first, then a documented fallback when native picking cannot resolve a selectable prim.

• Selection state is keyed by stable USD prim paths and synchronized across the

viewport, tree, property panel, and any status or info surfaces.

• Selection feedback should be renderer-visible and work for arbitrary valid

USD scenes. Prefer native outlines or selection groups when the selected renderer path supports them.

• Material-driven glow, visibility changes, or shader-parameter effects are

optional pick effects. Use them only when the active stage exposes compatible targets or the user explicitly asks for that behavior.

• Selection animation is optional and product-specific. If requested, keep it

parameterized, reversible, and safe for the stage's units and coordinate system; do not assume a fixed lift direction, duration, or asset scale.

Viewport And Rendering

• Choose render size from the delivery skill and product requirements. Keep it

fixed for a session unless the viewport-resize/README.md skill is explicitly selected.

• UI resize scales the displayed image; it does not dynamically resize the

render product unless the viewport-resize/README.md skill is explicitly selected.

• Browser video uses object-fit: contain.
• NVST maps pointer coordinates between the contained video and intrinsic stream

resolution. Local apps use the visible image content rect for the same letterbox mapping.

• Every stream frame handed to ovstream is BGRA8. Convert or colorize AOVs on

the server before stream_video().

• Shader bake/compile can take time on first load. Complete warmup before

accepting client connections when startup latency matters.

Camera

• The camera is a USD prim, updated by writing omni:xform.
• Camera matrices are row-major: row 0 = right, row 1 = up, row 2 = -forward,

row 3 = eye/translation.

• Fit the camera to the stage on initial load unless the app restores an

explicit saved camera state.

• Camera gizmos are ovui overlays: local apps draw them in the viewport UI;

streaming apps composite server-side ovui output into the BGRA stream.

Scene Loading

• User USD files are not modified by viewer setup.
• Viewer camera, render product, render vars, settings, and selection metadata

live in a session layer or composite wrapper.

• Clear selection, hover, temporary effects, and any viewer-authored runtime

overrides on every stage load.

• Do not call renderer.step() while reset_stage(), add_usd(), or a

session/composite rebuild is mutating the renderer.

• Detect the scene root dynamically on the server and pass root_prim_path to

clients instead of hardcoding /World.

Streaming Protocol

• WebRTC signaling and media ports are selected by the streaming/deployment

skills. Do not hardcode deployment-specific ports unless the selected reference or hosting environment requires them.

• App messages travel over the data channel as JSON with

{ "event_type": "...", "payload": {...} }.

• AppStreamer client messages may be wrapped by the streaming library; unwrap

before dispatching on the server.

Environment

• Set OVRTX_SKIP_USD_CHECK=1 before importing or constructing ovrtx.
• Import/setup order for streaming servers is:

environment variables -> ovrtx/renderer -> streaming helpers -> pxr only behind the chosen isolation boundary.

• Keep renderer.step() ownership on one render thread or UI loop.

C++ Interaction Features

Orbit Camera Control

Use an orbit camera with azimuth, elevation, distance, and target. Mouse input updates camera state; the render loop writes the camera transform to omni:xform before ovrtx_step().

#include <algorithm>
#include <array>
#include <cmath>

struct OrbitCamera {
    using Mat4 = std::array<double, 16>;

    double azimuth = -1.5707963267948966;
    double elevation = 0.2912652529540066;
    double distance = 500.0;
    std::array<double, 3> target = {-74.5, 103.0, -22.5};
    double lastX = 0.0;
    double lastY = 0.0;
    int dragButton = -1;  // 0 orbit, 1 pan, 2 dolly

    void beginDrag(int button, double x, double y)
    {
        dragButton = button;
        lastX = x;
        lastY = y;
    }

    void drag(double x, double y)
    {
        const double dx = x - lastX;
        const double dy = y - lastY;
        lastX = x;
        lastY = y;

        if (dragButton == 0) {
            azimuth += dx * 0.006;
            elevation = std::clamp(elevation + dy * 0.006, -1.45, 1.45);
        } else if (dragButton == 2) {
            distance = std::max(1.0, distance * std::exp(dy * 0.01));
        }
    }

    void scroll(double yoffset)
    {
        distance = std::max(1.0, distance * std::exp(-yoffset * 0.08));
    }

    Mat4 cameraToWorld() const
    {
        const double ce = std::cos(elevation);
        const std::array<double, 3> eye = {
            target[0] + distance * ce * std::cos(azimuth),
            target[1] + distance * std::sin(elevation),
            target[2] + distance * ce * std::sin(azimuth),
        };

        auto normalize = [](std::array<double, 3> v) {
            const double len = std::sqrt(v[0] * v[0] + v[1] * v[1] + v[2] * v[2]);
            return std::array<double, 3>{v[0] / len, v[1] / len, v[2] / len};
        };
        auto cross = [](std::array<double, 3> a, std::array<double, 3> b) {
            return std::array<double, 3>{
                a[1] * b[2] - a[2] * b[1],
                a[2] * b[0] - a[0] * b[2],
                a[0] * b[1] - a[1] * b[0],
            };
        };

        const std::array<double, 3> forward = normalize({
            target[0] - eye[0], target[1] - eye[1], target[2] - eye[2]});
        const std::array<double, 3> worldUp = {0.0, 1.0, 0.0};
        const std::array<double, 3> right = normalize(cross(forward, worldUp));
        const std::array<double, 3> up = cross(right, forward);

        return {
            right[0], right[1], right[2], 0.0,
            up[0], up[1], up[2], 0.0,
            -forward[0], -forward[1], -forward[2], 0.0,
            eye[0], eye[1], eye[2], 1.0,
        };
    }
};

Write the camera matrix with OVRTX_SEMANTIC_XFORM_MAT4x4:

static bool writeMat4Attribute(
    ovrtx_renderer_t* renderer,
    const std::string& primPath,
    const char* attributeName,
    const OrbitCamera::Mat4& matrix)
{
    int64_t shape[] = {1};
    int64_t strides[] = {1};

    DLTensor tensor = {};
    tensor.data = const_cast<double*>(matrix.data());
    tensor.device = {kDLCPU, 0};
    tensor.ndim = 1;
    tensor.dtype = {static_cast<std::uint8_t>(kDLFloat), 64, 16};
    tensor.shape = shape;
    tensor.strides = strides;

    ovrtx_input_buffer_t input = {};
    input.tensors = &tensor;
    input.tensor_count = 1;

    const ovx_string_t path = toOvxString(primPath);
    const ovx_string_t paths[] = {path};

    ovrtx_binding_desc_or_handle_t binding = {};
    binding.binding_desc.prim_list = {paths, 1};
    binding.binding_desc.attribute_name = {0, literal_to_ovx_string(attributeName)};
    binding.binding_desc.attribute_type = {
        {static_cast<std::uint8_t>(kDLFloat), 64, 16},
        false,
        OVRTX_SEMANTIC_XFORM_MAT4x4,
    };
    binding.binding_desc.prim_mode = OVRTX_BINDING_PRIM_MODE_CREATE_NEW;

    const ovrtx_enqueue_result_t write =
        ovrtx_write_attribute(renderer, &binding, &input, OVRTX_DATA_ACCESS_SYNC);
    return ok(write);
}

writeMat4Attribute(renderer, "/OVCamera", "omni:xform", camera.cameraToWorld());

Do not write xformOp:transform for live camera movement. OVRTX consumes the Fabric omni:xform attribute for live transforms.

Native Picking

For click selection, enqueue a native pick query before the step that should produce the pick result. Read OVRTX_RENDER_VAR_PICK_HIT after that same step.

struct PendingPick {
    bool pending = false;
    int left = 0;
    int top = 0;
    int right = 0;
    int bottom = 0;
};

static bool enqueuePick(
    ovrtx_renderer_t* renderer,
    const std::string& renderProductPath,
    PendingPick& pendingPick)
{
    if (!pendingPick.pending) return false;

    const ovrtx_pick_query_desc_t desc = {
        toOvxString(renderProductPath),
        pendingPick.left,
        pendingPick.top,
        pendingPick.right,
        pendingPick.bottom,
        0,
    };
    pendingPick = {};

    const ovrtx_enqueue_result_t pick = ovrtx_enqueue_pick_query(renderer, &desc);
    if (!ok(pick)) {
        printLastOvrtxError("Failed to enqueue pick query");
        return false;
    }
    return true;
}

Decode the pick-hit output by checking params and resolving primPath IDs through the path dictionary.

#include <ovx/path_dictionary/path_dictionary.h>
#include <ovx/path_dictionary/path_dictionary_helper.h>
#include <ovx/path_dictionary/path_dictionary_utils.h>

#include <algorithm>
#include <cstring>
#include <vector>

static const DLTensor* findTensor(const ovrtx_render_var_output_t& output, const char* name)
{
    for (size_t i = 0; output.tensors && i < output.num_tensors; ++i) {
        if (output.tensors[i].name &&
            sameString(*output.tensors[i].name, name)) {
            return output.tensors[i].dl;
        }
    }
    return nullptr;
}

static const DLTensor* findParam(const ovrtx_render_var_output_t& output, const char* name)
{
    for (size_t i = 0; output.params && i < output.num_params; ++i) {
        if (sameString(output.params[i].name, name)) return &output.params[i].dl;
    }
    return nullptr;
}

static bool readU64(const DLTensor& tensor, size_t index, std::uint64_t& value)
{
    if (!tensor.data || tensor.dtype.lanes != 1) return false;
    const auto* base = static_cast<const std::uint8_t*>(tensor.data) + tensor.byte_offset;
    const size_t bytes = tensor.dtype.bits / 8;
    const int64_t stride = tensor.strides ? tensor.strides[0] : 1;
    const auto* ptr = base + index * static_cast<size_t>(stride) * bytes;

    value = 0;
    if (tensor.dtype.code == static_cast<std::uint8_t>(kDLUInt)) {
        std::memcpy(&value, ptr, std::min(bytes, sizeof(value)));
        return true;
    }
    return false;
}

static std::string resolvePrimPathId(ovrtx_renderer_t* renderer, ovx_primpath_t pathId)
{
    path_dictionary_instance_t dictionary = {};
    if (!ok(ovrtx_get_path_dictionary(renderer, &dictionary))) {
        printLastOvrtxError("Failed to get OVRTX path dictionary");
        return {};
    }

    std::vector<ovx_token_t> tokenBuffer(256);
    ovx_token_t* tokensPerPath[] = {nullptr};
    size_t tokenCounts[] = {0};
    size_t pathsProcessed = 0;

    ovx_api_result_t tokenResult = path_dictionary_get_tokens_from_paths(
        &dictionary,
        &pathId,
        1,
        tokenBuffer.data(),
        tokenBuffer.size(),
        tokensPerPath,
        tokenCounts,
        &pathsProcessed);
    if (tokenResult.status != OVX_API_SUCCESS || pathsProcessed != 1 || !tokensPerPath[0]) {
        return {};
    }

    std::vector<ovx_string_t> tokenStrings(tokenCounts[0]);
    ovx_api_result_t stringResult = path_dictionary_get_strings_from_tokens(
        &dictionary,
        tokensPerPath[0],
        tokenCounts[0],
        tokenStrings.data());
    if (stringResult.status != OVX_API_SUCCESS) {
        return {};
    }

    std::string path;
    for (ovx_string_t token : tokenStrings) {
        std::string segment = fromOvxString(token);
        if (segment.empty()) continue;
        if (segment.front() != '/') path.push_back('/');
        path += segment;
    }
    return path.empty() ? "/" : path;
}

static std::vector<std::string> decodePickPaths(
    ovrtx_renderer_t* renderer,
    const ovrtx_render_var_output_t& pickOutput)
{
    std::uint64_t magic = 0;
    std::uint64_t version = 0;
    std::uint64_t hitCount = 0;
    const DLTensor* magicParam = findParam(pickOutput, "magic");
    const DLTensor* versionParam = findParam(pickOutput, "version");
    const DLTensor* hitCountParam = findParam(pickOutput, "hitCount");
    if (!magicParam || !versionParam || !hitCountParam ||
        !readU64(*magicParam, 0, magic) ||
        !readU64(*versionParam, 0, version) ||
        !readU64(*hitCountParam, 0, hitCount)) {
        return {};
    }
    if (magic != OVRTX_PICK_HIT_MAGIC || version != OVRTX_PICK_HIT_VERSION) {
        return {};
    }

    const DLTensor* primPathTensor = findTensor(pickOutput, "primPath");
    if (!primPathTensor || !primPathTensor->shape) return {};

    const size_t count =
        std::min(static_cast<size_t>(hitCount), static_cast<size_t>(primPathTensor->shape[0]));

    std::vector<std::string> paths;
    for (size_t i = 0; i < count; ++i) {
        std::uint64_t id = 0;
        if (!readU64(*primPathTensor, i, id) || id == 0) continue;
        std::string path = resolvePrimPathId(renderer, static_cast<ovx_primpath_t>(id));
        if (!path.empty() && std::find(paths.begin(), paths.end(), path) == paths.end()) {
            paths.push_back(path);
        }
    }
    return paths;
}

In the render loop, handle pick output beside LdrColor:

const bool pickQueued = enqueuePick(renderer, renderProductPath, pendingPick);

// After ovrtx_step() and ovrtx_fetch_results():
if (pickQueued && sameString(var.render_var_name, OVRTX_RENDER_VAR_PICK_HIT)) {
    ovrtx_render_var_output_t mapped = {};
    if (ok(ovrtx_map_render_var_output(renderer, var.output_handle,
            ovrtx_timeout_infinite, &mapped)) &&
        mapped.status == OVRTX_EVENT_COMPLETED) {
        std::vector<std::string> picked = decodePickPaths(renderer, mapped);
        setSelectedPrim(picked.empty() ? std::string{} : picked.front());
    }
    if (mapped.map_handle != OVRTX_INVALID_HANDLE) {
        ovrtx_unmap_render_var_output(renderer, mapped.map_handle, {});
    }
}

Treat left and top as inclusive, right and bottom as exclusive. A single click is a 1x1 rectangle: {x, y, x + 1, y + 1}.

Selection Outline

Enable outlines in renderer config, then write omni:selectionOutlineGroup/OVRTX_ATTR_NAME_SELECTION_OUTLINE_GROUP on selected prims. Group 0 clears the outline; group 1 is primary selection.

static bool writeU8Attribute(
    ovrtx_renderer_t* renderer,
    const std::string& primPath,
    const char* attributeName,
    std::uint8_t value)
{
    int64_t shape[] = {1};
    int64_t strides[] = {1};

    DLTensor tensor = {};
    tensor.data = &value;
    tensor.device = {kDLCPU, 0};
    tensor.ndim = 1;
    tensor.dtype = {static_cast<std::uint8_t>(kDLUInt), 8, 1};
    tensor.shape = shape;
    tensor.strides = strides;

    ovrtx_input_buffer_t input = {};
    input.tensors = &tensor;
    input.tensor_count = 1;

    const ovx_string_t path = toOvxString(primPath);
    const ovx_string_t paths[] = {path};

    ovrtx_binding_desc_or_handle_t binding = {};
    binding.binding_desc.prim_list = {paths, 1};
    binding.binding_desc.attribute_name = {0, literal_to_ovx_string(attributeName)};
    binding.binding_desc.attribute_type = {
        {static_cast<std::uint8_t>(kDLUInt), 8, 1},
        false,
        OVRTX_SEMANTIC_NONE,
    };
    binding.binding_desc.prim_mode = OVRTX_BINDING_PRIM_MODE_CREATE_NEW;

    const ovrtx_enqueue_result_t write =
        ovrtx_write_attribute(renderer, &binding, &input, OVRTX_DATA_ACCESS_SYNC);
    return ok(write);
}

static void setSelectionOutline(
    ovrtx_renderer_t* renderer,
    const std::string& previousPath,
    const std::string& nextPath)
{
    if (!previousPath.empty()) {
        writeU8Attribute(renderer, previousPath,
            OVRTX_ATTR_NAME_SELECTION_OUTLINE_GROUP, 0);
    }
    if (!nextPath.empty()) {
        writeU8Attribute(renderer, nextPath,
            OVRTX_ATTR_NAME_SELECTION_OUTLINE_GROUP, 1);
    }
}

If the installed SDK provides ovrtx_set_selection_outline_group(), prefer that helper for bulk updates. The attribute write above is the explicit fallback and is useful when combining outline state with other per-prim writes.

EffectLayer Prim-Pick Effects

EffectLayer faders are optional material effects, not the baseline selection signal. Keep native outlines enabled for all selected prims, then write inputs:Fader only for known material EffectLayer targets.

static std::string effectLayerPathForPrim(const std::string& primPath)
{
    if (primPath == "/World/Cone") {
        return "/World/Misc/Looks/Steel_Stainless/EffectLayer";
    }
    if (primPath == "/World/Cube") {
        return "/World/Misc/Looks/Concrete_Rough/EffectLayer";
    }
    if (primPath == "/World/Sphere") {
        return "/World/Misc/Looks/MetallicGreen_OmniPbr/EffectLayer";
    }
    return {};
}

static bool writeFloatAttribute(
    ovrtx_renderer_t* renderer,
    const std::string& primPath,
    const char* attributeName,
    float value,
    ovrtx_binding_prim_mode_t primMode)
{
    int64_t shape[] = {1};
    int64_t strides[] = {1};

    DLTensor tensor = {};
    tensor.data = &value;
    tensor.device = {kDLCPU, 0};
    tensor.ndim = 1;
    tensor.dtype = {static_cast<std::uint8_t>(kDLFloat), 32, 1};
    tensor.shape = shape;
    tensor.strides = strides;

    ovrtx_input_buffer_t input = {};
    input.tensors = &tensor;
    input.tensor_count = 1;

    const ovx_string_t path = toOvxString(primPath);
    const ovx_string_t paths[] = {path};

    ovrtx_binding_desc_or_handle_t binding = {};
    binding.binding_desc.prim_list = {paths, 1};
    binding.binding_desc.attribute_name = {0, literal_to_ovx_string(attributeName)};
    binding.binding_desc.attribute_type = {
        {static_cast<std::uint8_t>(kDLFloat), 32, 1},
        false,
        OVRTX_SEMANTIC_NONE,
    };
    binding.binding_desc.prim_mode = primMode;

    const ovrtx_enqueue_result_t write =
        ovrtx_write_attribute(renderer, &binding, &input, OVRTX_DATA_ACCESS_SYNC);
    return ok(write);
}

static void setEffectLayerFader(
    ovrtx_renderer_t* renderer,
    const std::string& primPath,
    float fader)
{
    const std::string effectPath = effectLayerPathForPrim(primPath);
    if (effectPath.empty()) return;

    writeFloatAttribute(renderer, effectPath, "inputs:Fader", fader,
        OVRTX_BINDING_PRIM_MODE_EXISTING_ONLY);
}

For shared materials, compute active EffectLayer targets from the complete selected set. Do not turn off a shared material fader just because one of several selected prims was deselected.

Author neutral startup values in the session layer when a sample material defaults to visible glow:

over "World"
{
    over "Misc"
    {
        over "Looks"
        {
            over "Concrete_Rough"
            {
                over "EffectLayer"
                {
                    float inputs:Fader = 0
                }
            }
        }
    }
}

Use CREATE_NEW for load-time resets authored by the viewer. Use EXISTING_ONLY for runtime toggles when the target shader input must already exist.

Selection Animation

Selection animation is just another live omni:xform write. Store the selected prim's base transform, then write an app-defined reversible offset every frame before ovrtx_step(). Choose the motion direction, magnitude, and timing from the product brief, stage units, asset scale, and coordinate system.

enum class AnimationPhase {
    Idle,
    Rising,
    Hovering,
    Falling,
};

struct PrimAnimation {
    std::string path;
    OrbitCamera::Mat4 baseTransform;
    AnimationPhase phase = AnimationPhase::Idle;
    double t = 0.0;
    double offset = 0.0;
    double hoverTime = 0.0;
    double fallStartOffset = 0.0;
};

static double clamp01(double value)
{
    return std::clamp(value, 0.0, 1.0);
}

static double easeOutQuint(double t)
{
    const double inv = 1.0 - clamp01(t);
    return 1.0 - inv * inv * inv * inv * inv;
}

static OrbitCamera::Mat4 offsetTransform(
    const PrimAnimation& animation,
    int translationIndex)
{
    OrbitCamera::Mat4 transform = animation.baseTransform;
    transform[translationIndex] += animation.offset;
    return transform;
}

static void updateSelectionAnimation(
    ovrtx_renderer_t* renderer,
    std::vector<PrimAnimation>& animations,
    double deltaSeconds)
{
    constexpr int kTranslationIndex = 13;       // app-defined axis in row-major matrix
    constexpr double kBaseOffset = 0.05;        // stage units; choose from asset scale
    constexpr double kRiseDuration = 0.25;
    constexpr double kFallDuration = 0.25;
    constexpr double kHoverAmplitude = 0.0;     // optional additional stage-unit offset
    constexpr double kHoverFrequency = 1.5;
    constexpr double kPi = 3.14159265358979323846;

    deltaSeconds = std::clamp(deltaSeconds, 1.0 / 240.0, 0.1);

    for (PrimAnimation& animation : animations) {
        if (animation.phase == AnimationPhase::Idle) continue;

        if (animation.phase == AnimationPhase::Rising) {
            animation.t += deltaSeconds / kRiseDuration;
            animation.offset = kBaseOffset * easeOutQuint(animation.t);
            if (animation.t >= 1.0) {
                animation.phase = AnimationPhase::Hovering;
                animation.hoverTime = 0.0;
            }
        } else if (animation.phase == AnimationPhase::Hovering) {
            animation.hoverTime += deltaSeconds;
            animation.offset = kBaseOffset +
                kHoverAmplitude * std::sin(2.0 * kPi * kHoverFrequency * animation.hoverTime);
        } else if (animation.phase == AnimationPhase::Falling) {
            animation.t += deltaSeconds / kFallDuration;
            const double s = clamp01(animation.t);
            animation.offset = animation.fallStartOffset * (1.0 - s);
            if (s >= 1.0) {
                animation.offset = 0.0;
                animation.phase = AnimationPhase::Idle;
            }
        }

        writeMat4Attribute(renderer, animation.path, "omni:xform",
            offsetTransform(animation, kTranslationIndex));
    }
}

On selection change:

static void selectPrim(
    ovrtx_renderer_t* renderer,
    std::string& selectedPath,
    std::vector<PrimAnimation>& animations,
    const std::string& nextPath)
{
    if (selectedPath == nextPath) return;

    constexpr double kBaseOffset = 0.05;        // keep in sync with animation config

    for (PrimAnimation& animation : animations) {
        if (animation.path == selectedPath) {
            animation.phase = AnimationPhase::Falling;
            animation.t = 0.0;
            animation.fallStartOffset = animation.offset;
        }
        if (animation.path == nextPath) {
            animation.phase = AnimationPhase::Rising;
            animation.t = animation.offset > 0.0 ? clamp01(animation.offset / kBaseOffset) : 0.0;
        }
    }

    setSelectionOutline(renderer, selectedPath, nextPath);
    // Update optional material effects in a separate manager when enabled.
    selectedPath = nextPath;
}

Only animate prims whose base transforms are known. For arbitrary scenes, query or initialize base omni:xform values first; do not overwrite unknown authored transforms with identity.

C++ Native ImGui OVRTX Viewer

Triggers

Use this skill for C++ viewer, ImGui viewer, native viewer, C++ OVRTX, GLFW viewer, native Dear ImGui viewport, native executable, no-Python local desktop viewer, or requests that use the OVRTX C API directly.

Use this for a focused native binary: GLFW window, OpenGL pixel presentation, Dear ImGui controls, inline USDA session layer, OVRTX C API renderer, CPU-mapped LdrColor, and direct camera/picking/selection logic.

For ovrtx C API behavior, native viewer behavior, renderer lifecycle guidance, or release-specific behavior not covered here, read references/dependencies for acquisition guidance and supplemental dependency documentation.

Read Order

Critical Rules

• Do not use Three.js, WebGL scene rendering, glTF viewers, or browser-native rendering for USD.
• OpenGL is only the pixel presentation path for frames already rendered by OVRTX.
• Keep renderer creation, stage load/reset, pick query enqueue, result mapping, and ovrtx_write_attribute() calls on one owner thread.
• Use the selected OVRTX C API and helper contracts from the references; do not mix Python renderer assumptions into this path.
• Apply viewer-control-patterns to Dear ImGui UI: choose controls by user intent first, pair approximate sliders with numeric inputs when exact values matter, clamp values before sending them to OVRTX, and surface the effective backend value when it differs.
• Choose C++/ImGui only when the app should run as a native executable on the GPU workstation and does not need web UI reuse.

See also: ovrtx-rendering, stage-loading, viewer-input-routing, viewer-control-patterns, camera-controls, native-picking-selection, selection-feedback, selection-animation, prim-transform-safety, streaming-vs-local, ovui-local-viewer-recipe, tauri-local-viewer, and electron-shm-viewer.

Electron SHM Dependencies

Electron + SHM Dependencies

Purpose: local separate-process Electron viewers where a Python ovrtx server renders frames and Electron presents already-rendered pixels through a SharedArrayBuffer/WebGL transport. Electron does not render USD or 3D scene content.

Read nvidia-runtime.md for the current ovrtx and ovstream acquisition sources before setting up Electron SHM.

Required components:

• Node.js 18+ for SharedArrayBuffer support and N-API native addons.
• Electron 28+ for COOP/COEP-compatible BrowserWindow configuration and contextBridge isolation.
• node-gyp plus build-essential for N-API native addon compilation.
• libovstream_shm_client.so from the ovstream package, available to the native addon at runtime.
• /dev/shm mounted with sufficient size. Defaults such as 64 MB may be too small; use at least 512 MB for a 1080p ring buffer.
• Python 3.10+ for the ovrtx server process.

Minimal verification:

node --version
npm --version
python3 --version
df -h /dev/shm

Common Electron + SHM dependency failures:

• Native addon build fails: install node-gyp, compiler toolchains, and headers matching the active Node/Electron ABI.
• Runtime cannot load libovstream_shm_client.so: install the matching ovstream package and expose the native library path to Electron.
• Shared memory attach fails or frames drop under load: enlarge /dev/shm, especially in containers.
• SharedArrayBuffer is unavailable: use Electron 28+ and configure COOP/COEP-compatible BrowserWindow settings.