Pmndrs Xr

PMNDRS XR

Use this skill for WebXR experiences built with React Three Fiber and the React Three XR/uikit/handle packages.

Core Workflow

1. Inspect the app structure and package manager before editing. 2. If the workspace is empty, scaffold a minimal runnable vertical slice first: package scripts, src/main, src/App, styles, and vitexec/play. Get that slice building before expanding the scene. 3. Use the React Three Fiber architecture: @react-three/xr plus <Canvas> and <XR store={store}>. 4. Enter AR/VR from a real user gesture such as a DOM button calling store.enterAR() or store.enterVR(). 5. Put immersive status, score, checklist, report, and controls inside the 3D/WebXR scene. DOM is acceptable for the initial session-entry button and optional desktop fallback, not as headset UI. 6. Build interactions through real spatial input: pointer events when appropriate, documented handle components, controller/hand pose, near interaction, controller select events, hit-test, or anchors. 7. For editor/configurator/assembly tasks, use visible handles/gizmos or grabbable affordances and prove drag/translate/rotate/snap behavior through object transform changes. 8. Validate with a live XR walkthrough when the task is an XR experience. The passing path should enter an emulated XR session, move the headset/viewer, aim controllers, interact with visible in-scene targets, record video, and assert view composition.

Reference Routing

Read the smallest relevant reference first:

• Architecture and package choice: references/architecture.md
• In-scene UI and visual quality: references/ui-and-visual-quality.md
• Domain fit for games, training, simulation, commerce, and tools: references/use-case-patterns.md
• Spatial interaction patterns: references/interactions.md
• XR validation with vitexec/IWER: references/validation.md
• Dependency/version pitfalls: references/dependencies.md

Generated upstream docs are also bundled:

• Getting started: references/docs/getting-started/introduction.md
• Store/session options: references/docs/tutorials/store.md
• Interactions: references/docs/tutorials/interactions.md
• Origin/teleport: references/docs/tutorials/origin.md, references/docs/tutorials/teleport.md
• AR features: references/docs/tutorials/anchors.md, dom-overlay.md, hit-test.md, object-detection.md
• Handles: references/docs/handles/introduction.md
• Full index: references/index.md

If unsure which page applies, search references:

rg -n "createXRStore|TeleportTarget|XRHitTest|pointerEventsType|XROrigin|DefaultXRController|Handle|uikit" path/to/skill/references

Quality Bar

• Build the actual experience as the first screen, not a landing page.
• For complex XR apps, avoid long up-front design or broad node_modules archaeology. Read only the specific bundled reference needed for the immediate API choice, then implement and validate the smallest real path before adding polish.
• Avoid placeholder-cube demos. Use domain-specific environment context, recognizable objects, lighting/material variation, spatial affordances, and visible feedback.
• Keep desktop fallback useful, but do not let it become the proof for XR behavior.
• Do not weaken validation to pass. If controller input misses, fix target geometry, ray pose, readiness, occlusion, or event handling.
• Use patch/file-edit operations for generated JSX/TS/HTML/vitexec files. Do not use shell heredocs for code files.

Before Finishing

Confirm the result has:

• createXRStore and user-facing session entry.
• <XR store={store}> around scene content.
• In-scene XR UI for immersive status/score/report/controls.
• Real pointer/controller/handle/hit-test interaction, not validation-only state mutation.
• For manipulation tasks, before/after transform assertions for dragged, grabbed, rotated, scaled, or snapped objects.
• A build plus browser/XR validation covering the main path.
• Recording or screenshots plus logs proving active XR, controller input, viewer movement, and readable target composition.

Architecture

Architecture

• Import XR and createXRStore from @react-three/xr.
• Create the store once, outside React render loops where practical.
• Enter AR/VR from a real user gesture: store.enterAR() or store.enterVR().
• Render scene content inside <XR store={store}>.
• Configure default controllers, hands, gaze, screen input, teleport pointers, and feature flags through createXRStore before replacing low-level implementations.
• Check the installed XRStoreOptions types before adding store options. In current @react-three/xr/@pmndrs/xr, fields such as originReferenceSpace and raw optionalFeatures are not valid store options; use the typed feature/session-init options such as hitTest, planeDetection, anchors, domOverlay, or customSessionInit when they exist.
• For apps validated with a vitexec-installed IWER runtime, set createXRStore({ emulate: false, offerSession: false, ... }) unless you intentionally need the package's localhost emulator. Do not let an app auto-offer a session while the walkthrough is also requesting/granting one.
• Do not use deprecated XRButton, ARButton, VRButton, Interactive, RayGrab, Controllers, or Hands for new work unless the task is explicitly migration/compatibility.
• Guard non-XR camera controls during immersive sessions. WebXR owns the camera transform.

AR

• Use hit-test, anchors, planes, or meshes according to the requested real-world behavior.
• XRDomOverlay is only appropriate for handheld AR that explicitly targets DOM overlay.
• For immersive VR or non-handheld AR, use in-scene UI instead of DOM overlays.

Existing Apps

• Inspect package versions before adding dependencies.
• Follow local build tooling and source layout.
• Keep desktop/non-XR interaction functional unless the user asks for XR-only.

Dependencies And Versions

Package Compatibility

• Use versions that resolve in the selected package manager/registry.
• If install fails on an assumed version, query available versions and pin a published one before continuing.
• Keep Vite and @vitejs/plugin-react peers aligned. Current @vitejs/plugin-react 6.x expects Vite 8.x; do not pair it with Vite 7.x.
• Current @react-three/drei 10.x expects React/React DOM 19.x and @react-three/fiber 9.x. Do not combine drei 10 with React 18 or fiber 8.
• If a demo does not need drei helpers, omit @react-three/drei.
• @react-three/xr 6.x accepts @react-three/fiber >=8, React >=18, and Three *, but companion packages may have stricter peers.
• Strict TypeScript apps that import three or React DOM directly should include matching type packages such as @types/three, @types/react, and @types/react-dom.
• For WebXR/IWER validation, use a published iwer 2.x release. Registry checks during eval work found iwer@2.2.1 available and iwer@1.9.x unavailable.
• vitexec must be available locally when package scripts call it.

Code Generation

• Do not generate JSX, TypeScript, HTML, or vitexec files through shell heredocs such as cat <<EOF, cat > file <<EOF, or tee <<EOF.
• Use patch/file-edit operations; heredocs routinely corrupt nested quotes, JSX self-closing tags, HTML doctype syntax, and JavaScript template strings.

Generated Scratch Apps

• Do not run git commands inside generated scratch apps unless you confirm they are git repositories.
• Do not leave a Vite/dev preview server running after validation.
• Prefer npm run build or the project’s focused validation scripts before final summary.

Performance

All performance optimizations for non-immersive 3D web applications are also applicable for immersive XR web applications. Relevant guides on the topic of performance for 3D web applications are the R3F performance guide, R3F performance pitfalls, and the Threejs tips and tricks. In general, it is good to check if your web application's performance is GPU- or CPU-bound to select the correct optimization techniques.

When it comes to immersive XR web applications, there are a few other options to improve GPU performance. We also recommend taking a look at this WebXR Performance Optimization guide.

Frame Buffer Scaling

The frame buffer scaling factor allows to control the size of the frame buffer your web application draws to. Decreasing this number can improve performance but reduces the resolution. A frame buffer scaling factor of 1.0 sets the frame buffer resolution to the native display resolution.

Foveation

Foveation allows rendering with a lower resolution at the edges of the eye's viewport to improve GPU performance. If your application is optimized and the performance is still GPU bound, consider increasing the foveation to improve performance.

Convert to XR

In case your app is already an R3F app build with natuerlich or @react-three/xr v5. Check out the migration guides for natuerlich and @react-three/xr v5.

The first step is to install the latest version of react-three/xr.

npm install @react-three/xr@latest

Next, we import the createXRStore and create a xr store.

import { createXRStore, XR } from '@react-three/xr'

const store = createXRStore()

Using the store, we can set up a way for the user to enter the XR experience. For this example, we will just add 2 HTML buttons above your canvas.

<>
    <button onClick={() => store.enterVR()}>Enter VR</button>
    <button onClick={() => store.enterAR()}>Enter AR</button>
    <Canvas>
 ...your scene
    </Canvas>
</>

Lastly, use the store to setup the XR component to wrap your scene.

<>
    <button onClick={() => store.enterVR()}>Enter VR</button>
    <button onClick={() => store.enterAR()}>Enter AR</button>
    <Canvas>
        <XR store={store}>
 ...your scene
        </XR>
    </Canvas>
</>

Your application is now useable with an AR or VR headset.

If something did not work as expected, check out the FAQ, create an issue on github, or message us on Discord.

With this basic XR setup, you can start expanding the features of your XR application. The following questions might help you in integrating those features.

How do I move around in my scene?

↳ Checkout out the tutorial about [XROrigin](../tutorials/origin.md) or [Teleportation](../tutorials/teleport.md).

How can I customize the way my hands/controllers/... feel, look, or interact with the scene?

↳ Check out the tutorial about [Custom Hands/Controllers/...](../tutorials/custom-inputs.md).

How do interactions work in XR, and how can I build more advanced interactions?

↳ Check out the tutorial about [Interactions](../tutorials/interactions.md)

How can I leverage the mixed reality features of my headset, such as Plane Detection?

↳ Check out the tutorial about [Object Detection](../tutorials/object-detection.md)

What else can I do? I need inspiration.

↳ Check out the [examples](./examples.md).

Development Setup

Building WebXR experiences has all the advantages of building for the web, with a lot of tool chains, debugging tools, and resources available. Because there are so many tools available, it is often hard to choose a development setup. Therefore, we would like to recommend you a development setup for developing with react-three/xr.

1. Build tool: vite

Vite is easy to set up and has all the features we need. We recommend using the packages @vitejs/plugin-react to enable hot module reloading and @vitejs/plugin-basic-ssl to enable https, which is required for any WebXR experience.

A basic vite.config.ts would look like this:

import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'
import basicSsl from '@vitejs/plugin-basic-ssl'

export default defineConfig({
  plugins: [react(), basicSsl()],
})

Now with vite and ssl setup, we can host out application by starting executing vite --host. The --host argument makes sure that the server is exposed to your local network, so you can access the app from your headset or smartphone. When starting the server, vite also shows you your local IP. Now type in this IP including the port number into your smartphone's or headset's browser (e.g. https://192.168.0.43:5173).

2. emulator: iwer/devui

Developing WebXR experiences often requires testing WebXR-specific features, which either require an actual device or an emulator. An emulator allows testing without a specific device and without continuously switching a headset on and off.

react-three/xr includes the iwer/devui emulator out of the box, which is in its 0.x status and very much experimental in its current stage. The emulator builds on IWER by Meta Quest and adds a easy to use overlay on top of your application. The emulator is automatically injected if the emulate config has inject set to true, or if the app is served on localhost and if no WebXR support is detected. In case the emulator was not injected automatically, the user can inject it by pressing Window/Command + Alt/Option + E. After the emulator is injected, the user can press the button to enter the XR experience provided by the application.

!iwer/devui

The existing Immersive Web Emulator is a extension that provides similar emulation but its current version (1.x) is not supported because it does not comply to the WebXR spec. Starting with IWE 2.0, iwer/devui will be integrated into IWE, making it compatible with react-three/xr. If you have IWE 1.x installed and active, please turn them off, as it will prevent activiting the integrated iwer/devui.

Another supported alternative is the Apple Vision Pro Simulator.

3. ADB

Sometimes, a WebXR experience works flawlessly in the emulator but fails on the actual device. In this case we recommend using Android Device Bridge (ADB) to debug your experience from another device. For example, to remotely debug a WebXR experience on a Meta Quest 3, the Meta Quest 3 must be in developer mode, and remote debugging must be enabled. Then, the Meta Quest 3 must be connected to a PC via a USB cable with ADB installed. With this setup, the Chrome browser can now be used to access chrome://inspect to inspect specific tabs on the device remotely.

If you are using the Apple Vision Pro, there are similar features to remotely debug a WebXR experience on the Apple Vision Pro from a Safari browser on another device.

Examples

<Grid cols={3}> <li>  </li> <li>  </li> <li>  </li> <li>  </li> <li>  </li> <li>  </li> <li>  </li> <li>  </li> <li>  </li> <li>  by Sung Powley </li> <li>  </li> <li>  </li> </Grid>

FAQ

How can I read the camera position or rotation in XR?

The current global camera transformation can be accessed through getWorldPosition or getWorldQuaternionn This works inside of XR, as well as, outside of XR.

useFrame((state) => console.log(state.camera.getWorldPosition(new Vector3())))

How can I change the camera position in XR?

In contrast to non-immersive 3D applications, the camera transformation in MR/VR/AR applications should never be directly controlled by the developer since the user's head movement must control the camera's transformation. Therefore, pmndrs/xr provides the XROrigin component, which allows to control where the session's origin is placed inside the 3D scene. The session origin is at the users' feet once they recenter their session. This allows to implicitly control the camera position but prevents the user from getting motion sick when their movement is not reflected in the camera's movement.

##

Having problems accessing the camera position or rotation.

Check if you have OrbitControls, CameraControls from @react-three/drei, or other controls in your scene and make sure to place an <IfInSessionMode deny={['immersive-ar', 'immersive-vr']}> guard around them when in XR or replace them with OrbitHandles or MapHandles from @react-three/handle. This prevents overwriting the camera transformation which is controlled through WebXR when inside an immersive session and allows to access the correct transformation.

import { OrbitHandles } from '@react-three/handle'
import { noEvents, PointerEvents } from '@react-three/xr'

<Canvas events={noEvents}>
  <PointerEvents />
  <OrbitHandles />
</Canvas>

I cannot enter the XR session!

1. Missing Https If you are trying to enter the AR or VR modus and nothing is happening, make sure that you are accessing the website using https://. In case you are using vite, we recommend using the @vitejs/plugin-basic-ssl to try out your vite application on your device while developing.

2. Missing XR component If you made sure that the website is accessed using https:// and still nothing happens when executing enterAR or enterVR, it is likely that the <XR> component is missing. Be sure to add the <XR> component directly into the <Canvas> and make sure both the <Canvas> and the <XR> component are present when the button is pressed.

3. Entering while loading content If you cannot enter the VR or AR experience, there might be assets in your scene that are loading. Make sure to place a suspense boundary around your scene. With this setup, the <XR> component stays mounted while your scene loads.

<Canvas>
  <XR>
    <Suspense>... your scene</Suspense>
  </XR>
</Canvas>

How can I exit an XR session?

store.getState().session?.end()

Is WebGPU supported?

WebGPU is finding its way to more and more devices. However, AR and VR devices do not yet implement WebGPU for WebXR, which requires the WebXR-WebGPU-Binding. Therefore, WebGPU is not yet usable for WebXR in general.

How can I put HTML in my XR scene?

If you are targeting only handheld AR experiences (e.g., for smartphones), you can use dom overlay. Here's a tutorial for using XRDomOverlays in your react-three/xr experience.

For non-handheld VR and AR experiences, you can use react-three/uikit, which renders user interfaces directly inside the 3D scene and is aligned with HTML and CSS concepts.

Does it work on iOS?

WebXR for VR experiences is supported on Safari for Apple Vision Pro. WebXR is not supported on iOS Safari yet. The alternative is to use products such as Variant Launch, which allow to build WebXR experiences for iOS.

XRSpace

If you are placing <XRSpace> components outside of the <XROrigin> while changing the transformation of the <XROrigin> (e.g. by setting <XROrigin position={[0,1,0]} />), the elements rendered inside of the <XRSpace> will not be transformed with the origin. If the transformations of the origin should be applied to the <XRSpace>, make sure to place those components inside the <XROrigin>. Not placing <XRSpace> components into the <XROrigin> can be useful in scenarios where you want to move the <XROrigin> independently from the <XRSpace>. For instance, building a virtual elevator where your actual room is duplicated into the x-axis so that you can use the elevator to travel between multiple instances of your room.

onClick does not play video or allow file uploading (in certain browsers)

As a performance optimization the react-three/xr event system batches html user events per frame. This only applies if you are using PointerEvents, forwardHtmlEvents, or forwardObjectEvents. This can cause issue when executing functions that require a user action. For instance, uploading a file through a input element in a safari can only be triggered manually when immediately caused by a user input. For these use cases, please disable the event batching performance optimization through the options by setting batchEvents to false.

Introduction

npm install three @react-three/fiber @react-three/xr@latest

What does it look like?

import { Canvas } from '@react-three/fiber'
import { XR, createXRStore } from '@react-three/xr'
import { useState } from 'react'

const store = createXRStore()

export function App() {
  const [red, setRed] = useState(false)
  return <>
    <button onClick={() => store.enterAR()}>Enter AR</button>
    <Canvas>
      <XR store={store}>
        <mesh pointerEventsType={{ deny: 'grab' }} onClick={() => setRed(!red)} position={[0, 1, -1]}>
          <boxGeometry />
          <meshBasicMaterial color={red ? 'red' : 'blue'} />
        </mesh>
      </XR>
    </Canvas>
  </>
}

Turn any @react-three/fiber app into an XR experience

1. const store = createXRStore() create a xr store 2. store.enterAR() call enter AR when clicking on a button 3. <XR>...</XR> wrap your content with the XR component

... or read this guide for converting a react-three/fiber app to XR.

Tutorials

• 💾 Store
• 👆 Interactions
• 👌 Handles
• 🧊 Object Detection
• ✴ Origin
• 🪄 Teleport
• 🕹️ Gamepad
• ➕ Secondary Input Sources
• 📺 Layers
• 🎮 Custom Controller/Hands/...
• ⚓️ Anchors
• 📱 Dom Overlay
• 🎯 Hit Test
• ⛨ Guards

External Tutorials

• 🥇 **WebXR First Steps React** by Meta Quest

Roadmap

• 🤳 XR Gestures
• 🕺 Tracked Body

Migration guides

• from @react-three/xr v5
• from natuerlich

Sponsors

This project is supported by a few companies and individuals building cutting-edge 3D Web & XR experiences. Check them out!

Showcases

<Grid cols={2}> <li> !volu.dev Spatial development hub </li> </Grid>

Are we missing your public product? Please message us via Twitter or Discord.

Handle Component

The Handle component is the core component of the @react-three/handle library, which is built on the HandleStore. This store provides developers with more control over the current state and user interactions.

Handle Store

The handle store manages the handle's state and translates user interactions into state modifications.

Exposed Functions

getState Allows retrieval of the state of the current interaction. If no interaction is currently happening, getState returns undefined.

capture Normally, a pointer is captured when the user interacts with an object. The capture method allows programmatic initiation of an interaction without requiring user input.

save Allows saving the current state so that modifications to the handle options do not affect modifications from previous interactions.

cancel Normally, an interaction is canceled when the user releases the button that started the interaction. The cancel function allows programmatic cancellation of the interaction.

Handle Component

Example

<Handle translate="as-scale" scale={{ uniform: true }}>
    <mesh>
        <boxGeometry />
    </mesh>
</Handle>

Allows scaling of the cube by dragging it outward from the cube's center.

Properties

handleRef Allows overriding to pass a custom handle object.

targetRef Allows to pass in a ref to an object that should be the target of the handle. Alternatively targetRef can be set to "from-context" use the target provided by a surrounding HandleTarget component.

getHandleOptions Allows passing a function that dynamically generates options to override the current handle options.

bind Allows disabling automatic binding of the event listeners to the provided handle, which can be necessary when capturing pointers manually.

apply The apply function is used to apply a state modification that originates from a user interaction to the state. This property allows overriding the default apply function, giving the developer complete control over how modifications affect the state. For instance, instead of applying the modification directly, the developer can apply it to their own state management solution. The state management solution can then apply the modification to the handle target.

projectRays Allows to configure whether rays from input devices should be projected onto the interaction space (3D plane or 3D Line).

alwaysUpdate In situations where the handle target is placed inside a constantly changing group, the alwaysUpdate flag ensures that the handle target's transformation is updated every frame to reflect the current state of the handle.

multitouch By default, handles can be interacted with using multiple input devices. By setting multitouch to false, only the first input device will be used.

filter Allows to filter interactions based on the event. Return false to ignore the event.

stopPropagation By default, events that occur on handles are not propagated upwards and therefore do not reach their ancestors. Setting stopPropagation to false will re-enable event propagation for events that occur on the handle.

rotate The rotate property allows configuring if and how the user can rotate the target. Setting rotate to false disables rotation. Setting rotate to x restricts rotation to the x-axis. Setting rotate to { x: false, y: [0, Math.PI] } disables rotation on the x-axis and restricts rotation on the y-axis to be between 0 and 180°, while rotation on the z-axis is enabled.

scale The scale property allows configuring if and how the user can scale the target. Setting scale to false disables scaling. Setting scale to x restricts scaling to the x-axis. Setting scale to { x: false, y: [1, 2] } disables scaling on the x-axis and restricts the scaling factor on the y-axis to be between 1 and 2, while scaling on the z-axis is enabled.

translate The translate property allows configuring if and how the user can translate the target. Setting translate to false disables translation. Setting translate to x restricts translation to the x-axis. Setting translate to { x: false, y: [-1, 1] } disables translation on the x-axis and restricts translation on the y-axis to be between -1 and 1, while translation on the z-axis is enabled. Furthermore, the translate property can be configured to transform translations into rotations and/or scalings using translate="as-scale", allowing the user to scale the target by grabbing and moving the handle.

ref Allows retrieval of a reference to the internal handle store (<Handle ref={handleStoreRef}>).

Handle Target Component

The HandleTarget component allows declaratively specifying a handle target that is hierarchically above the Handle component. To prevent accidentally providing a different target to a handle, using the target from the context requires setting targetRef="from-context" on the Handle component.

Example

<HandleTarget>
    <group>
        <mesh>
            <boxGeometry />
            <Handle targetRef="from-context">
                <mesh position-x={2}>  
                    <boxGeometry />
                </mesh>
            </Handle>
        </mesh>
    </group>
</HandleTarget>

Handles

Handles are everywhere, from scrollbar thumbs to window bars to door handles.


Handles make it possible to move, rotate, and scale 2D or 3D objects using intuitive gestures. @react-three/handle makes it super simple to build these intuitive 3D handles for XR and non-XR devices with only a few lines of code. Let's showcase that while building a 3D door.

 _Door model from witnessk_

To build our door, we have two components: the door handle and the door body, which the handle should rotate when grabbed. Therefore, the door body is the handle target, on which the transformations from the handle should be applied.

We can build this door by downloading the GLTF model from Sketchfab and transforming the model into its individual parts using gltfjsx. This separates the door frame, the door body, and the door handle, allowing us to wrap the door handle with the <Handle> component and the door body with the <HandleTarget> component.

```tsx {6,10,14-15} export function Door() { const { nodes, materials } = useGLTF('/door.glb') return ( <group rotation={[-Math.PI / 2, 0, 0]} scale={100}> <group position={[-0.435, -0.101, 0.249]}> <HandleTarget> <mesh geometry={nodes.Plane001_Glossy_0.geometry} material={materials.Glossy} /> <mesh geometry={nodes.Plane001_Door_0.geometry} material={materials.Door} /> <mesh geometry={nodes.Plane003_Door_0.geometry} material={materials.Door} position={[0.852, 0.017, 0.782]} /> <Handle targetRef="from-context" translate="as-rotate" rotate={{ x: false, y: false, z: [-Math.PI, 0] }}> <group position={[0.81, 0.043, 0.803]}> <mesh geometry={nodes.Circle002_Glossy_0.geometry} material={materials.Glossy} /> </group> </Handle> </HandleTarget> </group> <mesh geometry={nodes.Plane002_Glossy_0.geometry} material={materials.Glossy} /> <mesh geometry={nodes.Plane002_Door_0.geometry} material={materials.Door} /> </group> ) }

Next, we need to configure the handle to rotate the door when grabbed. We instruct it to use the target from the context using `targetRef="from-context"`, making sure the transformations are applied to the door body. Additionally, we ensure that moving the handle is translated into a rotation using the `translate="as-rotate"` property. Lastly, we disable rotations on all axes except for the `z` axis and limit the rotation between -180° and 0°.

*Learn more about all the available properties for the handle component [here](./handle-component.md).*

The final result looks like this (I added an additional handle around the door handle that allows it to rotate on its own y-axis).

![](./door-handle.gif)

### Editor Example

Handles are made for all kinds of use cases, from games to professional applications, which we emphasized by building the following editor demo that uses over 40 handles for moving the elements on the screen, resizing the virtual screen, and moving the virtual camera. The nice part is that it works on all devices, ranging from mouse-driven PCs to eye-driven mixed-reality headsets (Apple Vision Pro).

![](./editor.gif)

You can check it out [here](https://pmndrs.github.io/xr/examples/editor/) and read the source code (700 LOC) [here](https://github.com/pmndrs/xr/tree/main/examples/editor/app.tsx).

### Screen Handles

A specific type of handle is the screen handle, where not an individual object is the handle, but the whole screen. Therefore, we build screen handles, which allow panning, zooming, and rotating the camera using swipe, drag, and scroll interactions. These are built on the ideas of `OrbitControls` and `MapControls` from Three.js, but respect the event system (e.g., you don't need to disable them when you drag an object) and are automatically forwarded to virtual screens, as shown in the editor demo.

Learn more about the available screen handles [here](./screen-handle-components.md).

### Prebuilt Handles

For many use cases, such as 3D editors, handles often come in specific forms, like the `TransformControls` available in Three.js. These opinionated pre-built handles have proven to be very useful, which is why `@react-three/handle` ships with implementations for `TransformHandles` and `PivotHandles`.

![](./prebuild-handles.gif)

Learn more about the available prebuilt handles [here](./screen-handle-components.md).

## Sponsors

This project is supported by a few companies and individuals building cutting-edge 3D Web & XR experiences. Check them out!

![Sponsors Overview](https://bbohlender.github.io/sponsors/screenshot.png)

Prebuilt Handles for Editor Use Cases

The Handle component allows for the orchestration of multiple handles together to achieve interactions typically found in 3D editors, such as the TransformControls offered by Three.js. Since these prebuilt handles are commonly used, @react-three/handles includes two of them: TransformHandles (based on TransformControls from Three.js) and PivotHandles (based on PivotControls from @react-three/drei). These prebuilt handles can be used in XR and non-XR environments, are highly configurable for use cases including multi-user editing, can be utilized through virtual screens, and respect the event system of the scene, preventing accidental interactions with multiple objects at once.

Using the TransformHandles and PivotHandles in React Three Fiber requires disabling the built-in event system and adding the event system of @react-three/xr.

import { TransformHandles } from '@react-three/handle'
import { noEvents, PointerEvents } from '@react-three/xr'

<Canvas events={noEvents}>
  <PointerEvents />
  <TransformHandles>
    <mesh>
      <boxGeometry />
    </mesh>
  </TransformHandles>
</Canvas>

Transform Handles

Alias for `TransformControls`


Properties

alwaysUpdate In situations where the transform handles are placed inside a constantly changing group, the alwaysUpdate flag ensures that the transform handles' transformation is updated every frame to reflect the current state of the handle.

apply Allows overriding the default apply function, giving the developer complete control over how modifications affect the state. For instance, instead of applying the modification directly, the developer can apply it to their own state management solution. The state management solution can then apply the modification to the handle target.

stopPropagation Setting stopPropagation to false will re-enable event propagation for events that occur on the handles.

space Allows configuring whether the transformations should happen in "local" or "world" space. This property has no effect when the mode property is set to scale, as scaling must occur on the local axis.

mode Allows configuring whether the transformation should be "translate", "rotate", or "scale", which also changes the visualization of the transform handles.

x Allows configuring the transformation on the x-axis. Setting x to false disables transformations on the x-axis and also hides the respective user interface.

y Allows configuring the transformation on the y-axis. Setting y to false disables transformations on the y-axis and also hides the respective user interface.

z Allows configuring the transformation on the z-axis. Setting z to false disables transformations on the z-axis and also hides the respective user interface.

e The e axis represents the axis for rotating the transform handles in screen space, which is only available when mode is set to rotation. Setting e to false disables rotation in screen space and also hides the respective user interface.

enabled Setting enabled to false momentarily disables the transform handles.

fixed By default, the transform handles have a fixed size independent of their distance from the camera, which means they scale up when they move away from the camera. Setting fixed to false will make them appear smaller when further away from the camera.

size The size property allows configuring the size of the transform handles, which has no effect on their contents.

Pivot Handles

Alias for `PivotControls`

In contrast to the transform handles, the pivot handles only operate in local space but allow rotation, scaling, and translation transformations simultaneously.


Properties

scale The scale property allows configuring if and how the user can scale the pivot handles. Setting scale to false disables scaling. Setting scale to x restricts scaling to the x-axis and only shows the user interface elements for scaling on the x-axis. Similarly, setting scale to { x: false } hides the user interface elements for scaling on the x-axis and only allows scaling on the y- and z-axes.

translation The translation property allows configuring if and how the user can translate the pivot handles. Setting translation to false disables translation. Setting translation to x restricts translation to the x-axis and only shows the user interface elements for translation on the x-axis. Similarly, setting translation to { x: false } hides the user interface elements for translation on the x-axis and only allows translation on the y- and z-axes.

rotation The rotation property allows configuring if and how the user can rotate the pivot handles. Setting rotation to false disables rotation. Setting rotation to x restricts rotation to the x-axis and only shows the user interface elements for rotating on the x-axis. Similarly, setting rotation to { x: false } hides the user interface elements for rotating on the x-axis and only allows rotation on the y- and z-axes.

alwaysUpdate In situations where the pivot handles are placed inside a constantly changing group, the alwaysUpdate flag ensures that the pivot handles' transformation is updated every frame to reflect the current state of the handle.

apply Allows overriding the default apply function, giving the developer complete control over how modifications affect the state. For instance, instead of applying the modification directly, the developer can apply it to their own state management solution. The state management solution can then apply the modification to the handle target.

stopPropagation Setting stopPropagation to false will re-enable event propagation for events that occur on the handles.

enabled Setting enabled to false momentarily disables the pivot handles.

fixed By default, the pivot handles have a fixed size independent of their distance from the camera, which means they scale up when they move away from the camera. Setting fixed to false will make them appear smaller when further away from the camera.

size The size property allows configuring the size of the pivot handles, which has no effect on their contents.

Screen Handles

Screen handles are available for screen-based devices like smartphones and PCs and allow users to move the camera by dragging, swiping, and scrolling on the screen. Three.js directly offers OrbitControls and MapControls, which are built for this purpose, and we are building on top of their success. The main difference is that the OrbitHandles and MapHandles we provide use the event system of the scene, which means that interactions with objects in the scene prevent dragging the camera. Furthermore, the event system can forward the screen inputs on a virtual screen to a virtual camera inside a virtual scene, which is showcased in the editor example.

Using the OrbitHandles and MapHandles in React Three Fiber requires disabling the built-in event system and adding the event system of @react-three/xr.

import { OrbitHandles } from '@react-three/handle'
import { noEvents, PointerEvents } from '@react-three/xr'

<Canvas events={noEvents}>
  <PointerEvents />
  <OrbitHandles />
</Canvas>

Orbit Handles

alias for `OrbitControls`

Orbit handles allow the user to orbit around a center, which can be moved by panning with a right click or two fingers touching the screen.

Properties

store The screen camera store contains the current state of the camera, including the position of the origin, the yaw and pitch of the camera, and the distance of the camera to the origin. By passing an external store to the Orbit Handles, you can take control of the current state of the camera from multiple sources.

apply The apply function applies the modifications to the screen camera state that occurred based on user interactions to the screen camera store. Overriding this function allows you to manually modify the screen camera store and prevent or restrict certain modifications, such as the maximum pitch angle.

enabled Allows you to disable the orbit handles momentarily by setting enabled to false.

damping Allows you to configure how much the camera movement should be smoothed. Set to true for the default damping and false to disable damping.

camera Allows you to configure the camera on which the transformation should be applied. Not setting the camera will apply the transformations to the default camera of the scene.

rotate Allows you to configure the rotation speed and a filter function to disable rotation on certain input devices. Setting rotate to false will disable rotation entirely.

zoom Allows you to configure the zoom speed and a filter function to disable zooming on certain input devices. Setting zoom to false will disable zooming entirely.

pan Allows you to configure the pan speed and a filter function to disable panning on certain input devices. Setting pan to false will disable panning entirely.

Map Handles

alias for `MapControls`

Map handles have the same properties and functionality as the orbit handles but move the camera's transform origin only in the X and Z plane, which is perfect for building applications with a flat map.

from Natuerlich

@react-three/xr is inspired by natuerlich, and therefore, many things are similar, especially the way interactions are handled. However, a few things have been changed and renamed.

• use XROrigin instead of ImmersiveSessionOrigin
• use <Canvas><XR>...</XR></Canvas> instead of XRCanvas
• configure settings such as foveation through createXRStore
• use store.enterXR instead of useEnterXR
• use DragControls TBD instead of Grabbale
• don't add hands and controllers yourself, and configure them through the createXRStore options. Click here for more info regarding controller/hand/... customization.
• use teleport as described here

from @react-three/xr v5

The goal of @react-three/xr v6 is to align this library closer to the react-three ecosystem. We, therefore, focussed on supporting the react-three/fiber event handlers. Another focus of v6 is to reduce boilerplate and provide more defaults while also giving developers more access to the lower-level WebXR primitives. In combination, these changes allow developers to build XR experiences that interoperate with the whole react-three ecosystem using only a few lines of code.

For everybody that is transitioning from v5 to v6, we have created a small compatibility layer that includes XRButton, ARButton, VRButton, useInteraction, useXREvent, Interactive, and RayGrab. However, we recommend transitioning away from the compatibility layer as the new recommended way of building with @react-three/xr is more aligned with the whole react-three ecosystem.

For the Controllers and Hands components there are not correspondances in @react-three/xr v6 since input methods such as controllers, hands, but also transient-pointers are added by default. Users can configure the default implementation of those input methods as described here. The teleportation feature of @react-three/xr v5 has also slightly changed. The new API is explained here.

Anchors

Anchors allow to anchor virtual objects into the physical world in AR experiences. react-three/xr offers a multitude of ways to create and manage anchors. A simple solution is useXRAnchor, which works similarly to useState as it returns the current anchor and a function to request a new anchor as a tuple.

const [anchor, requestAnchor] = useXRAnchor()

With the requestAnchor function, we can request an anchor relative to the "world", a "space", or a "hitTestResult"

requestAnchor({ relativeTo: "space", space: ... })

Once the anchor is created, the useXRAnchor hook exposes it as anchor. We can now use this anchor to put content into it using the <XRSpace> component.

<XRSpace space={anchor.anchorSpace}>...your content</XRSpace>

The following example shows a Anchor component that uses the useXRAnchor hook and the XRSpace component to anchor a Box to the position of the right hand or controller when the respective hand or controller is selected (pinch/trigger).

export function Anchor() {
  const [anchor, requestAnchor] = useXRAnchor()
  const controllerState = useXRInputSourceState('controller', 'right')
  const handState = useXRInputSourceState('hand', 'right')
  const inputSource = controllerState?.inputSource ?? handState?.inputSource
  useXRInputSourceEvent(
    inputSource,
    'select',
    async () => {
      if (inputSource == null) {
        return
      }
      requestAnchor({ relativeTo: 'space', space: inputSource.targetRaySpace })
    },
    [requestAnchor, inputSource],
  )
  if (anchor == null) {
    return null
  }
  return (
    <XRSpace space={anchor.anchorSpace}>
      <mesh scale={0.1}>
        <boxGeometry />
      </mesh>
    </XRSpace>
  )
}

Custom Hands/Controllers/...

@react-three/xr provides a set of default hands, controllers, transient pointers, gazes, and screen input that can be configured and completely exchanged with your own implementation. The following example shows how to configure the ray color of the ray pointer in the users hand.

const store = createXRStore({ hand: { rayPointer: { rayModel: { color: 'red' } } } })

In some cases, the default hand/controller/... implementations are not enough. The following code sample shows how to provide your own custom hand implementation though the xr store options.

const store = createXRStore({ hand: CustomHand })

Custom Hand

Let's build our own hand implementation which renders the normal hand model but only has a touch interaction which works using the middle finger.

First we're getting the state of the hand, creating a reference to the position of the middle finger, and creating a touch pointer.

const state = useXRInputSourceStateContext('hand')
const middleFingerRef = useRef<Object3D>(null)
const pointer = useTouchPointer(middleFingerRef, state)

Next, we use the state to place an XRSpace for setting up the middleFingerRef and add an XRHandModel and PointerCursorModel to render the hand and a cursor visualization.

<XRSpace ref={middleFingerRef} space={state.inputSource.hand.get('middle-finger-tip')!}/>
<Suspense>
  <XRHandModel />
</Suspense>
<PointerCursorModel pointer={pointer} opacity={defaultTouchPointerOpacity} />

<details> <summary>Full Code</summary>

export function CustomHand() {
  const state = useXRInputSourceStateContext('hand')
  const middleFingerRef = useRef<Object3D>(null)
  const pointer = useTouchPointer(middleFingerRef, state)
  return (
    <>
      <XRSpace ref={middleFingerRef} space={state.inputSource.hand.get('middle-finger-tip')!} />
      <Suspense>
        <XRHandModel />
      </Suspense>
      <PointerCursorModel pointer={pointer} opacity={defaultTouchPointerOpacity} />
    </>
  )
}

</details>

This tutorial also applies to building custom controllers, transient pointers, gaze, and screen input implementations.

Dom Overlay

For hand-held AR experiences, such as those using a Smartphone, WebXR offers the dom overlay capability, allowing developers to use HTML code overlayed over the experience. In case scene 3D overlays or overlays in non-handheld AR/VR experiences are needed, check out pmndrs/uikit.

We can add dom overlay content to an experience using the XRDomOverlay component, which allows to write html code inside it. This HTML code will be overlayed over the hand-held AR experience.

<XRDomOverlay
  style={{ width: '100%', height: '100%', display: 'flex', alignItems: 'center', justifyContent: 'center' }}
>
  <div style={{ backgroundColor: 'red', padding: '1rem 2rem' }}>Hello World</div>
</XRDomOverlay>

The following shows the complete code for a simple AR experience with a Hello World button that can toggle its color when clicked on.

const store = createXRStore()

export function App() {
  const [bool, setBool] = useState(false)
  return (
    <>
      <button onClick={() => store.enterAR()}>Enter AR</button>
      <Canvas>
        <XR store={store}>
          <ambientLight />
          <XRDomOverlay
            style={{ width: '100%', height: '100%', display: 'flex', alignItems: 'center', justifyContent: 'center' }}
          >
            <div
              style={{ backgroundColor: bool ? 'red' : 'green', padding: '1rem 2rem' }}
              onClick={() => setBool((b) => !b)}
            >
 Hello World
            </div>
          </XRDomOverlay>
        </XR>
      </Canvas>
    </>
 )
}

Gamepad

All XR controllers are part of the state inside the xr store. The existing controllers can be read using the useXR hook. Alternatively, a specific xr controller can be retrived using useXRInputSourceState("controller", "left").

The xr controller state contains the gamepad state. Based on the name of one specific component, its state can be polled every frame. The following example shows how to read the thumbstick of the right controller to implement locomotion in combination with the XROrigin.

To keep the code snipped focussed on the gamepad API, the locomotion does not happen relative to the users head rotation. This can be added by retrieving the users head position from reading the camera world quaternion and extracting the users rotation on the y axis. (There is also a useXRControllerLocomotion hook available to help avoid implementing locomotion from scratch if that is the end goal).

const store = createXRStore()

export function App() {
  return (
    <>
      <button onClick={() => store.enterVR()}>Enter VR</button>
      <Canvas>
        <XR store={store}>
          <ambientLight />
          <Locomotion />
          <mesh scale={[10, 1, 10]} position={[0, -0.5, 0]}>
            <boxGeometry />
            <meshBasicMaterial color="green" />
          </mesh>
        </XR>
      </Canvas>
    </>
  )
}

function Locomotion() {
  const controller = useXRInputSourceState('controller', 'right')
  const ref = useRef<Group>(null)
  useFrame((_, delta) => {
    if (ref.current == null || controller == null) {
      return
    }
    const thumstickState = controller.gamepad['xr-standard-thumbstick']
    if (thumstickState == null) {
      return
    }
    ref.current.position.x += (thumstickState.xAxis ?? 0) * delta
    ref.current.position.z += (thumstickState.yAxis ?? 0) * delta
  })
  return <XROrigin ref={ref} />
}

Guards

Guards allow to conditionally display or include content. For instance, the IfInSessionMode guard allows only displaying a background when the session is not an AR session. The IfInSessionMode can receive either a list of allow session modes or a list of deny session modes.

import { Canvas } from '@react-three/fiber'
import { IfInSessionMode, XR, createXRStore } from '@react-three/xr'

const store = createXRStore()

export function App() {
  return (
    <>
      <button onClick={() => store.enterAR()}>Enter AR</button>
      <Canvas>
        <XR store={store}>
          <IfInSessionMode deny="immersive-ar">
            <color args={['red']} attach="background" />
          </IfInSessionMode>
        </XR>
      </Canvas>
    </>
 )
}

Hit Test

Hit testing is a technique that allows developers to check for intersections with real-world surfaces in AR experiences. @react-three/xr provides hooks and components for setting up hit testing. This tutorial covers all the hit testing hooks available in React Three XR and demonstrates how to use them effectively.

Overview of Hit Testing Components

React Three XR provides three hooks for hit testing:

• `useXRHitTest` - Provides continuous hit testing with automatic frame updates
• `useXRHitTestSource` - Lower-level hook for creating and managing hit test sources
• `useXRRequestHitTest` - One-time hit test requests on demand

Additionally, React Three XR provides the XRHitTest component, which is a convenience wrapper for using the useXRHitTest hook to perform continuous hit testing.

All rays cast by these components originate from the source's position and are cast in the direction that the source object is oriented (quaternion; typically -z).

useXRHitTest Hook

The useXRHitTest hook is the most commonly used hook for hit testing in the library. It automatically performs hit tests every frame and calls your callback function with the results.

What it does:

Sets up continuous hit testing that runs every frame, providing real-time intersection data with the real world.

When to use it:

Use this when you need continuous tracking of where a ray intersects with real-world surfaces, such as for cursor positioning, object placement previews, or interactive AR elements.

Parameters:

• fn - Callback function that receives hit test results and a function to retrieve the world matrix
• relativeTo - The object, XR space, or reference space to cast rays from. This reference must be static in your scene.
• trackableType - Optional parameter specifying what types of surfaces to hit test against

const matrixHelper = new Matrix4()
const hitTestPosition = new Vector3()

function ContinuousHitTest() {
  const previewRef = useRef<Mesh>(null)

  useXRHitTest(
    (results, getWorldMatrix) => {
      if (results.length === 0) return

      getWorldMatrix(matrixHelper, results[0])
      hitTestPosition.setFromMatrixPosition(matrixHelper)
    },
    'viewer', // Cast rays from the viewer reference space. This will typically be either the camera or where the user is looking
    'plane' // Only hit test against detected planes
  )

  useFrame(() => {
    if (hitTestPosition && previewRef.current) {
      previewRef.current.position.copy(hitTestPosition)
    }
  })

  return (
      {/* Renders a sphere where the hit test intersects with the plane */}
      <mesh ref={previewRef} position={hitPosition}>
        <sphereGeometry args={[0.05]} />
        <meshBasicMaterial color="red" />
      </mesh>
  )
}

XRHitTest

XRHitTest is a component that wraps the useXRHitTest hook. This makes it easier to add hit testing anywhere within your component tree.

const matrixHelper = new Matrix4()
const hitTestPosition = new Vector3()

const store = createXRStore({
  hand: () => {
    const inputSourceState = useXRInputSourceStateContext()

    return (
      <>
        <DefaultXRHand />
        <XRHitTest
          space={inputSourceState.inputSource.targetRaySpace}
          onResults={(results, getWorldMatrix) => {
            if (results.length === 0) return
            getWorldMatrix(matrixHelper, results[0])
            hitTestPosition.setFromMatrixPosition(matrixHelper)
          }}
        />
      </>
    )
  },
})

XRHitTest has all of the same functionality as the useXRHitTest hook, just that it's built as a component.

useXRHitTestSource Hook

The useXRHitTestSource hook provides lower-level access to hit test sources, giving you more control over when and how hit tests are performed. It is the same as the useXRHitTest hook, the only difference being that you have to manually check for hit test results; typically every frame, or every few frames.

What it does:

Does the same thing as the useXRHitTest hook, but does not automatically hit test every frame.

When to use it:

In most cases you should use either useXRHitTest or useXRRequestHitTest, but you can use this hook when you have a static hit test source that you only want to occasionally perform constant hit tests from. Or if you want to recreate the useXRHitTest behavior manually.

Parameters:

• relativeTo - The object, XR space, or reference space to cast rays from
• trackableType - Optional parameter specifying what types of surfaces to hit test against

Returns:

A hit test source object that you can use with frame.getHitTestResults()

function ManualHitTest() {
  const meshRef = useRef<Mesh>(null)
  const hitTestSource = useXRHitTestSource(meshRef)
  const [someCondition, setSomeCondition] = useState(false)
  const [hitResults, setHitResults] = useState<XRHitTestResult[]>([])

  useFrame((_, __, frame: XRFrame | undefined) => {
    // Only perform hit testing when certain conditions are met
    if (frame && hitTestSource && someCondition) {
      const results = frame.getHitTestResults(hitTestSource.source)
      setHitResults(results)
    }
  })

  return (
    <mesh ref={meshRef}>
      {/* Render hit test results. This will put spheres everywhere the hit test succeeds. In a real app don't use index as the key */}
      {hitResults.map((result, index) => {
        const matrix = new Matrix4()
        hitTestSource?.getWorldMatrix(matrix, result)
        const position = new Vector3().setFromMatrixPosition(matrix)
        return (
          <mesh key={index} position={position}>
            <sphereGeometry args={[0.05]} />
            <meshBasicMaterial color="red" />
          </mesh>
        )
      })}
    </mesh>
  )
}

useXRRequestHitTest Hook

The useXRRequestHitTest hook provides a function for one-time hit test requests. Useful for event-driven hit testing. Cannot be called in the useFrame hook.

What it does:

Returns a function that can perform a single hit test request when called.

When to use it:

Use this for event-driven hit testing, such as when a user taps the screen, clicks a button, or performs a gesture. It's ideal for placing objects or checking intersections at specific moments.

Returns:

A function that takes the same parameters as other hit test hooks and returns a promise with hit test results

const matrixHelper = new Matrix4()
function EventDrivenHitTest() {
  const requestHitTest = useXRRequestHitTest()
  const [placedObjects, setPlacedObjects] = useState<Vector3[]>([])

  const handleTap = async () => {
    const hitTestResult = await requestHitTest('viewer', ['plane', 'mesh'])
    const { results, getWorldMatrix } = hitTestResult
    if (results?.length > 0) {
      getWorldMatrix(matrixHelper, results[0])
      const position = new Vector3().setFromMatrixPosition(matrixHelper)
      setPlacedObjects((prev) => [...prev, position])
    }
  }

  return (
    <>
      <IfInSessionMode allow={'immersive-ar'}>
        <XRDomOverlay>
          <button onClick={handleTap}>Place Object</button>
        </XRDomOverlay>
      </IfInSessionMode>

      {/* Render placed objects */}
      {placedObjects.map((position, index) => (
        <mesh key={index} position={position}>
          <sphereGeometry args={[0.1]} />
          <meshBasicMaterial color="blue" />
        </mesh>
      ))}
    </>
  )
}

Trackable Types

All hit testing hooks support specifying trackable types to control what surfaces the hit tests should target:

• 'plane' - Hit test against detected planes (floors, walls, tables)
• 'point' - Hit test against feature points in the environment
• 'mesh' - Hit test against detected meshes (requires mesh detection support)

You can specify a single type or an array of types:

// Single type
useXRHitTest(callback, spaceRef, 'plane')

// Multiple types
useXRHitTest(callback, spaceRef, ['plane', 'mesh'])

Practical Example: Object Placement

Here's a complete example combining multiple hooks for a robust object placement system:

const matrixHelper = new Matrix4()
const hitTestPositionHelper = new Vector3()

function ObjectPlacement() {
  const [placedObjects, setPlacedObjects] = useState<Vector3[]>([])
  const [previewPosition, setPreviewPosition] = useState<Vector3 | null>(null)
  const controllerRef = useRef<Group>(null)

  // Continuous hit testing for preview
  useXRHitTest(
    (results, getWorldMatrix) => {
      if (results.length > 0) {
        getWorldMatrix(matrixHelper, results[0])
        const position = hitTestPositionHelper.setFromMatrixPosition(matrixHelper)
        setPreviewPosition(position)
      } else {
        setPreviewPosition(null)
      }
    },
    'viewer', // Use viewer space for screen-based hit testing
  )

  const placeObject = async () => {
    if (previewPosition) {
      setPlacedObjects((prev) => [...prev, previewPosition.clone()])
    }
  }

  return (
    <>
      {/* Preview object at hit test position */}
      {previewPosition && (
        <mesh position={previewPosition}>
          <sphereGeometry args={[0.05]} />
          <meshBasicMaterial color="yellow" transparent opacity={0.7} />
        </mesh>
      )}

      {/* Placed objects */}
      {placedObjects.map((position, index) => (
        <mesh key={index} position={position}>
          <sphereGeometry args={[0.05]} />
          <meshBasicMaterial color="green" />
        </mesh>
      ))}

      {/* Placement trigger */}
      <IfInSessionMode allow={'immersive-ar'}>
        <XRDomOverlay>
          <button onClick={placeObject}>Place Object</button>
        </XRDomOverlay>
      </IfInSessionMode>
    </>
  )
}

Alternatively, for devices that provide mesh detection -- such as newer Meta Quest devices -- you can also add normal pointer event listeners to an XR Mesh to achieve the same behavior. Check out this tutorial for more information about mesh detection.

Interactions

On this page, you can learn the basics behind pointer events and interactions in react-three/xr. From experience, we found that many people are interested in more high level interactions, which can be build with the concept of handles. Check out the handles pages to learn more about the concept and the library we built for it.

@react-three/xr uses the same pointer events as @react-three/fiber, which allows building interactions that work on non-XR devices as well as XR devices. So, just like you'd expect from @react-three/fiber and everywhere else in react, interactions are built using

• onPointerMove
• onPointerCancel
• onPointerDown
• onPointerUp
• onPointerEnter
• onPointerLeave
• onPointerOver
• onPointerOut
• onClick
• onDblClick
• onContextMenu
• onWheel

The following example shows how to bind an onClick handler to a mesh that gets executed when the mesh is clicked. This interaction will work in non-XR devices as well as in XR devices using @react-three/xr.

<mesh onClick={(event) => console.log("I've been clicked", event)}>
  <boxGeometry />
</mesh>

The event object provided to the onClick handler contains useful information, such as the intersection point in world space.

The way pointer events are handled can be configured using the pointerEvents, pointerEventsType, and pointerEventsOrder properties, which are available on all threejs objects.

The pointerEvents property corresponds to the pointerEvents property of CSS, which allows to completely disable pointer events for an element and its children. However, children can also re-enable pointer events by setting pointerEvents="auto".

The pointerEventsType property allows to blacklist or whitelist pointer events for specific pointer types. For instance, setting pointerEventsType={{ deny: "grab" }} prevents triggering pointer events from grabbing the object or any of its children.

The pointerEventsOrder allows to overwrite the sorting order, similar to how renderOrder allows to overwrite the rendering order in threejs. The default pointer events order is 0. Setting it to a value greater than 0 will ensure it is intersected before anything with a lower pointer events order. Setting pointerEventsOrder is helpful for building an interactive x-ray object that is always rendered above anything else and should, therefore, always be interacted with first. For instance, this can be used to build controls that are overlayed over the object that they control.

Pointer Capture

Another concept that @react-three/fiber leverages from the web is pointer captures. Pointer captures allow to force all consecutive events to be emitted to a specific object, even if that object is not intersected. This is useful for building dragging interactions without a complex global state. Typically, a pointer capture is set using object.setPointerCapture in the event handler of onPointerDown with the pointerId of the pointer that pressed on the object.

.The following example illustrates how pointer events can be built to create a simple dragging implementation (that only works if the mesh is not inside a transformed group).

function DraggableCube() {
  const isDraggingRef = useRef(false)
  const meshRef = useRef<Mesh>(null)

  return (
    <mesh
      ref={meshRef}
      onPointerDown={(e) => {
        if (isDraggingRef.current) {
          return
 }
        isDraggingRef.current = true
        meshRef.position.copy(e.point)
 }}
      onPointerMove={(e) => {
        if (!isDraggingRef.current) {
          return
 }
        meshRef.position.copy(e.point)
 }}
      onPointerUp={(e) => (isDraggingRef.current = false)}
    >
      <boxGeometry />
    </mesh>
 )
}

Layers

Layers allow to render videos, images, and complete scenes with higher performance and higher quality while preserving battery life and latency for quad, cylinder, and equirect shapes using the WebXR Layer API. Layers are perfect for use cases that display flat, high-quality content, such as videos, images, and user interfaces. The following example illustrates how to create a layer that renders a video.

First, we create a layer at 0, 1.5, -0.5 with a scale of 0.5 that displays a video assigned to src and starts that video when clicked.

<XRLayer position={[0, 1.5, -0.5]} onClick={() => video.play()} scale={0.5} src={video} />

The assigned video is an HTML video element that is loaded from test.mp4.

const video = useMemo(() => {
  const result = document.createElement('video')
  result.src = 'test.mp4'
  return result
}, [])

Combined, the final app looks like this

export function App() {
  const video = useMemo(() => {
    const result = document.createElement('video')
    result.src = 'test.mp4'
    return result
  }, [])
  return (
    <Canvas>
      <XR store={store}>
        <XRLayer position={[0, 1.5, -0.5]} onClick={() => video.play()} scale={0.5} src={video} />
      </XR>
    </Canvas>
  )
}

Instead of images and videos, Layers can also be used to display dynamically rendered content. The following example illustrates how to render a red cube onto the layer. This scene will be re-rendered every frame, allowing for fully dynamic content.

<XRLayer position={[0, 1.5, -0.5]} scale={0.5}>
  <mesh>
    <boxGeometry />
    <meshBasicMaterial color="red" />
  </mesh>
</XRLayer>

Object Detection

@react-three/xr allows to use the devices mesh and plane detection functionality to detect the meshes and planes in the environment to modify the rendering, allow physics interactions with the environment, and more.

Detected Planes

The detected planes are accessible through the useXRPlanes hook or directly from useXR(xr => xr.detectPlanes) and manually go through the returned array. To render the planes in the correct place, the planes' space must provided to the XRSpace component. The following example shows how to render the red planes for all detected walls.

function RedWalls() {
  const wallPlanes = useXRPlanes('wall')
  return (
    <>
      {wallPlanes.map((plane) => (
        <XRSpace space={plane.planeSpace}>
          <XRPlaneModel plane={plane}>
            <meshBasicMaterial color="red" />
          </XRPlaneModel>
        </XRSpace>
 ))}
    </>
 )
}

Detected Meshes

Mesh detection provides access to the geometry of the environment. Similarly to xr planes, @react-three/fiber allows to retrieve detected meshes using useXRMeshes and offers the XRMeshModel to render the individual meshes.

Origin

The origin of an XR session is a 3D transformation that represents the position of the feet when the user recenters the session. Therefore, we recommend treating the session origin as the position of the feet. @react-three/xr provides the XROrigin component to control this transformation and place it anywhere inside the scene.

Roller coaster example

Since the XROrigin is a react component, it can be placed anywhere, including moving objects. In the following example, we'll create an XR roller coaster.

const store = createXRStore({
  controller: false,
})

export function App() {
  return (
    <>
      <button onClick={() => store.enterVR()}>Enter VR</button>
      <Canvas>
        <directionalLight position={[1, 1, 1]} />
        <ambientLight />
        <OrbitControls />
        <XR store={store}>
          <RollerCoaster />
        </XR>
      </Canvas>
    </>
  )
}

function RollerCoaster() {
  const gltf = useGLTF('rollercoaster.glb')

  const mixer = useMemo(() => new AnimationMixer(gltf.scene), [])
  useEffect(() => {
    for (const animation of gltf.animations) {
      mixer.clipAction(animation).play()
    }
  }, [gltf, mixer])
  useFrame((state, delta) => mixer.update(delta))
  return (
    <>
      <primitive object={gltf.scene} />
      {createPortal(
        <group rotation-y={-Math.PI / 2} rotation-x={Math.PI / 2}>
          <XROrigin scale={0.24} position-y={-0.1} />
        </group>,
        gltf.scene.getObjectByName('Sessel')!,
      )}
    </>
  )
}

Resizing example

Transforming the XROrigin is not limited to position but can also include rotation and scale. The following example shows how the XROrigin can be used to achieve a resizing interaction.

const store = createXRStore()

function App() {
  const [miniature, setMinitature] = useState(false)
  return (
    <>
      <button onClick={() => store.enterAR()}>Enter VR</button>
      <Canvas>
        <XR store={store}>
          <Gltf src="model.glb" />
          <XROrigin scale={miniature ? 0.01 : 1} position-y={miniature ? -1 : 0} />
        </XR>
      </Canvas>
    </>
  )
}

Secondary Input Sources

Most standalone XR headsets support hand and controller tracking. While typical XR experiences often support both input methods, they only use the primary inputs, which refers to one input per hand and limits the inputs to 2. However, the headset often also tracks the secondary input sources. By enabling the secondaryInputSources flag when creating an xr store, we can access the secondary input sources and use them to track real-world objects, for example.

createXRStore({ secondaryInputSources: true })

Secondary input sources are exposed to the developer just like primary input sources, with the exception of the isPrimary flat, which is false. The following example illustrates how to show the primary input controllers using the default controller components while rendering the secondary input controllers as simple cubes.

createXRStore({
  secondaryInputSources: true,
  controller: () => {
    const { isPrimary } = useXRInputSourceStateContext('controller')
    if (isPrimary) {
      return <DefaultXRController />
    }
    return (
      <mesh>
        <boxGeometry />
      </mesh>
    )
  },
})

Store

The xr store is the central part of all @react-three/xr experiences and allows to configure those experiences using a large set of options, control the experience using various functions, and provide access to the current state of the xr experience.

Options

When creating a xr store, there are a lot of options that can be used for configuration.

For instance, we can use these options to disable the default controller on the left hand.

createXRStore({
  controller: { left: false },
})

The following tables show all the available options.

Functions

The xr store provides a large set of functions to modify and control the xr store. For instance, key functions are the store.enterAR and store.enterVR functions. The following table gives an overview of the complete set of functions that the xr store provides.

State

Alongside a set of functions, the xr store also provides the state of the current experience. For instance, the state of the xr store contains the current XRSession inside state.session.

The following table provides a list of properties available in the state of the xr store.

useXR

The useXR hook allows to retrieve the state from the xr store. For instance, const session = useXR(xr => xr.session) allows us to always get the current session from any component that is placed inside the <XR> component.

Teleportation

First we need to enable the teleport pointer inside the hands and controllers.

const store = createXRStore({
  hand: { teleportPointer: true },
  controller: { teleportPointer: true },
})

Secondly, we need to control out own state because that gives us the complete control when teleportation should be mixed with more complex behaviors. Therefore we're using useState to store the current position. But we can also use any other local or global state manager.

const [position, setPosition] = useState(new Vector3())

Next, we need to use the position to control the user's origin.

<XROrigin position={position}>

Lastly, we need to add a teleport target to our scene. In this case, we're using a simple 10x10 meter green box. We need to bind our setPosition function to the onTeleport handler of the TeleportTarget to update the user's position whenever the user teleports.

<TeleportTarget onTeleport={setPosition}>
    <mesh scale={[10, 1, 10]} position={[0, -0.5, 0]}>
        <boxGeometry />
        <meshBasicMaterial color="green" />
    </mesh>
</TeleportTarget>

Combined, this example looks like this.

const store = createXRStore({
  hand: { teleportPointer: true },
  controller: { teleportPointer: true },
})

export function App() {
  const [position, setPosition] = useState(new Vector3())
  return (
    <>
      <button onClick={() => store.enterVR()}>Enter VR</button>
      <Canvas>
        <XR store={store}>
          <ambientLight />
          <XROrigin position={position} />
          <TeleportTarget onTeleport={setPosition}>
            <mesh scale={[10, 1, 10]} position={[0, -0.5, 0]}>
              <boxGeometry />
              <meshBasicMaterial color="green" />
            </mesh>
          </TeleportTarget>
        </XR>
      </Canvas>
    </>
 )
}

!Recording of teleport example

Reference Index

Read the smallest file that matches the task.

docs/advanced

• Performance

docs/API

• useRequestXRAnchor
• useXRAnchor
• useXRControllerLocomotion
• useLoadXRControllerLayout
• useLoadXRControllerModel
• useXRControllerButtonEvent
• XRControllerComponent
• XRControllerModel
• DefaultXRController
• DefaultXRControllerGrabPointer
• DefaultXRGaze
• DefaultXRHand
• DefaultXRHandGrabPointer
• DefaultXRHandTouchPointer
• DefaultXRInputSourceRayPointer
• DefaultXRInputSourceTeleportPointer
• DefaultXRScreenInput
• DefaultXRTransientPointer
• XRDomOverlay
• IfFacingCamera
• ShowIfFacingCamera
• IfSessionVisible
• ShowIfSessionVisible
• IfInSessionMode
• ShowIfInSessionMode
• IfSessionModeSupported
• ShowIfSessionModeSupported
• XRHandModel
• useXRHitTest
• useXRHitTestSource
• useXRRequestHitTest
• XRHitTest
• useHover
• useInitRoomCapture
• useXRSessionFeatureEnabled
• useXRSessionModeSupported
• useXRSessionVisibilityState
• useXRInputSourceEvent
• XRLayer
• useXRMeshes
• useXRMeshGeometry
• XRMeshModel
• XROrigin
• useXRPlaneGeometry
• useXRPlanes
• XRPlaneModel
• CombinedPointer
• PointerCursorModel
• PointerRayModel
• useGrabPointer
• useLinesPointer
• usePointerXRInputSourceEvents
• useRayPointer
• useTouchPointer
• useApplyXRSpaceMatrix
• useGetXRSpaceMatrix
• useXRSpace
• XRSpace
• XRSpaceType
• TeleportTarget
• createXRStore
• NotInXR
• UNSAFE_useXRStore
• useXR
• useXRStore
• XR

docs/getting-started

• Convert to XR
• Development Setup
• Examples
• FAQ
• Introduction
• Showcases

docs/handles

• Handle Component
• Handles
• Prebuilt Handles for Editor Use Cases
• Screen Handles

docs/migration

• from Natuerlich
• from @react-three/xr v5

docs/tutorials

• Anchors
• Custom Hands/Controllers/...
• Dom Overlay
• Gamepad
• Guards
• Hit Test
• Interactions
• Layers
• Object Detection
• Origin
• Secondary Input Sources
• Store
• Teleportation

packages/handle

• @pmndrs/handle

packages/pointer-events

• @pmndrs/pointer-events

packages/react

• @react-three/handle

packages/xr

• @pmndrs/xr

Spatial Interactions

Match The Input To The Experience

• Do not default every XR experience to a controller laser pointer.
• UI panels, menus, inspection checklists, and configurators may use ray/pointer selection because the user is selecting controls at a distance.
• Boxing, fitness, grabbing, sorting, and repair tasks should use near interaction: hand/controller pose, glove or tool models, grab/touch pointers, collision volumes, or distance/contact checks.
• Shooting games should attach a visible blaster/gun/tool to the controller and fire from the muzzle. Use the weapon transform for raycasts/projectiles, and show recoil, tracers, muzzle flashes, or projectile travel.
• Vehicle, drone, machinery, and process simulations should expose domain controls and telemetry. Drive them through controller/hand input, handles, or spatial waypoints, and show visible motion or state progression over time.
• Commerce and AR placement should use surface reticles, hit-test/plane/anchor affordances, footprints, snap/fit indicators, and variant/material controls instead of a detached preview-only model.
• Validation should prove the same metaphor the user sees. For boxing, move the glove/fist into the pad. For shooting, aim the gun model and fire from the muzzle. Avoid tests that only move an invisible pointer.
• For controller-mounted tools, keep the active tool/muzzle ref tied to the in-headset object, not a desktop fallback. Before deriving a world-space ray from a moving XR-space child, force world matrices current with updateWorldMatrix or an equivalent scene update.
• For controller selection rays, prefer XRFrame.getPose(inputSource.targetRaySpace, referenceSpace) when available. Rendered controller or grip objects may be oriented for visuals and can point a different direction than the WebXR target ray; use them as a fallback, not the primary source of truth.

Pointer Events

• R3F pointer events work across screen, ray, grab, touch, and gaze pointers.
• Prefer onPointerDown or onPointerUp for XR selection-critical targets. onClick alone can be unreliable with emulated controller select pulses.
• Avoid binding both pointerdown and click to the same non-idempotent action; trigger pulses can produce both.
• For larger hit areas, use transparent or visible active geometry. Hidden meshes can be unreliable raycast targets.

Sequential Flows

• Completed/locked panels should visibly remain complete but stop receiving ray hits when they sit in front of later targets.
• Keep active panels and targets in clear line of sight from intended headset poses.
• If a visible object can receive the ray before the intended hitbox, attach the same handler to that user-facing geometry or move the target.
• For games with timed, popping, moving, or cooldown targets, derive selectable state from actual visible/active state in the frame loop. Validation should wait for a target face that is truly raised/visible before aiming, and assertions should require hits to increase, not merely shots.
• Thin discs and tiny faces are easy for controller rays to slip past. Give important game targets visible hit depth or a generous production hit plate that matches the intended affordance.

Handles And Editors

• Use @react-three/handle for editor-style move, rotate, scale, orbit, and map interactions.
• Components to check include Handle, HandleTarget, TransformHandles, PivotHandles, OrbitHandles, and MapHandles.
• A handles/editor experience should expose visible manipulation affordances, not only indirect panel buttons. Users should be able to point/select a gizmo or grabbable object, hold input, move the controller, and see an object translate, rotate, scale, orbit, or snap.
• Configurators, robot teaching tools, assembly benches, and layout editors should prove at least one controller-driven drag/translate and one controller-driven rotate/orientation edit when those operations are part of the task.
• Validate handles by recording before/after object transforms and asserting real deltas. Avoid passing tests that only click "move" or "rotate" buttons without moving a spatial handle or object.
• Use the handle library's mode names exactly: translate, rotate, and scale. Do not invent aliases like move.
• Check the installed @react-three/handle .d.ts files before passing handle props. For example, common handle visibility props are disabled/hidden; do not assume an enabled prop exists.
• Rotation rings, tiny torus handles, and small swatches are easy to miss with emulated controller rays. Add visible, generous hit plates or grabbable surfaces that match the user-facing affordance, and attach the same handler/ref to that surface.
• Guard desktop camera controls during immersive sessions.

XR Select Bridges

• In complex editor/control-panel flows, if R3F pointer events from emulated XR select are unreliable, add a production-quality XR select bridge.
• useXRInputSourceEvent(inputSource, "selectstart", handler, deps) is a valid React Three XR controller-trigger path for direct actions such as firing a controller-held tool or confirming a held object interaction.
• useXRControllerButtonEvent(controller, "xr-standard-trigger", handler) is also valid for tool-like trigger input such as blasters, sprayers, scanners, or machinery controls.
• The bridge should raycast from WebXR targetRaySpace to the same visible in-scene hit plates and dispatch the same user-facing control actions.
• Keep it tied to real XR input and visible targets. Do not mutate workflow state directly from validation scripts.
• Avoid double-counting actions when a custom XR select bridge and mesh onPointerDown/desktop fallback both point at the same state transition. In immersive sessions, route controller select through one path and keep desktop pointer handlers gated to non-XR fallback.

@pmndrs/handle

@pmndrs/handle

framework agnostic handle implementation for threejs

How to use

//lets create a handle to translate the object on the x axis (the target and the handle are both the object)
const store = new HandleStore(object, () => ({ scale: false, rotate: false, translate: "x" }))
store.bind(object)

@pmndrs/pointer-events

pointer-events

_framework agnostic pointer-events implementation for three.js_

based on 🎯 Designing Pointer-events for 3D & XR

How to use

import * as THREE from 'three'
import { forwardHtmlEvents } from '@pmndrs/pointer-events'

const canvas = document.getElementById('canvas')
const scene = new THREE.Scene()
const camera = new THREE.PerspectiveCamera(70, width / height, 0.01, 10)
camera.position.z = 1
const { update } = forwardHtmlEvents(canvas, () => camera, scene)

const width = window.innerWidth,
  height = window.innerHeight

const geometry = new THREE.BoxGeometry(0.2, 0.2, 0.2)
const material = new THREE.MeshBasicMaterial({ color: new THREE.Color('red') })
const mesh = new THREE.Mesh(geometry, material)
scene.add(mesh)

mesh.addEventListener('pointerover', () => material.color.set('blue'))
mesh.addEventListener('pointerout', () => material.color.set('red'))

const renderer = new THREE.WebGLRenderer({ antialias: true })
renderer.setSize(width, height)
renderer.setAnimationLoop(() => {
  update()
  renderer.render(scene, camera)
})

Filtering

Based on the css pointer-events property, the behavior of pointer events can be configured with the values none, listener, or auto.

object.pointerEvents = 'none'

The values none and auto correspond to the css properties, where none means that an object is not directly targetted and auto means the object is always targetted for events. The additional value listener, which is the default value, expresses that the object is only targetted by events if the object has any listeners. In 3D scenes this default is more reasonable than auto, which is the default in the web, because 3D scenes often contain semi-transparent content, such as particles, that should not catch pointer events by default.

In addition to the pointerEvents property, each 3D object can also filter events based on the pointerType with the pointerEventsType property. This property defaults to the value all, which expresses that pointer events from pointers of all types should be accepted. To filter specific pointer types, such as screen-mouse, which represents a normal mouse used through a 2D screen, pointerEventsType can be set to { allow: "screen-mouse" } or { deny: "screen-touch" }. pointerEventsType's allow and deny accept strings and array of strings. In case more custom logic is needed, pointerEventsType also accepts a function. In general the pointer types screen-touch, screen-pen, ray, grab, and touch are used by default. For pointer events that were forwarded through a portal using forwardObjectEvents, their pointerType is prefixed with forward-, while events forwarded from the dom to the scene are prefixed with screen-.

But wait ... there's more

Create your own Pointer that can represent a WebXR controller or something else. These Pointer can use a normal Ray for intersection, or a set of Lines, or even a Sphere, for grab and touch events.

Performance

In some cases multi-modal interactivity requires multiple pointers at the same time. Executing pointer.move, such as in the following example, can lead to performance issues because the scene graph will be traversed several times.

leftGrabPointer.move()
leftTouchPointer.move()
leftRayPointer.move()
rightGrabPointer.move()
rightTouchPointer.move()
rightRayPointer.move()

In this case, performance can be improved by combining the pointer using CombinedPointer, which will traverse the scene graph once per combined pointer, calculating the intersections for each pointer on each object.

const leftPointer = new CombinedPointer()
const rightPointer = new CombinedPointer()
leftPointer.register(leftGrabPointer)
leftPointer.register(leftTouchPointer)
leftPointer.register(leftRayPointer)
rightPointer.register(rightGrabPointer)
rightPointer.register(rightTouchPointer)
rightPointer.register(rightRayPointer)

leftPointer.move()
rightPointer.move()

Pitfalls

The pointerEvents attribute of any Mesh/Object3D/... will not be cloned when cloning the object.

@react-three/handle

@react-three/handle

The one to handle them all 👌 <br/>

npm install three @react-three/fiber @react-three/handle@latest @react-three/xr@latest

import { Canvas } from '@react-three/fiber'
import { noEvents, PointerEvents } from '@react-three/xr'
import { Handle } from '@react-three/handle'

export function App() {
  return (
    <Canvas events={noEvents}>
      <PointerEvents />
      <Environment preset="city" />
      <Handle>
        <mesh position-z={-1}>
          <boxGeometry />
          <meshStandardMaterial color="red" />
        </mesh>
      </Handle>
    </Canvas>
  )
}

Documentation

Sponsors

This project is supported by a few companies and individuals building cutting-edge 3D Web & XR experiences. Check them out!

@pmndrs/xr

xr

Turn any threejs app into an interactive immersive experience. <br/>

npm install three @pmndrs/xr

What does it look like?

import { createXRStore } from '@pmndrs/xr'
import { BoxGeometry, Mesh, MeshBasicMaterial, PerspectiveCamera, Scene, WebGLRenderer } from 'three'
import { GLTFLoader } from 'three/examples/jsm/loaders/GLTFLoader.js'

const camera = new PerspectiveCamera(70, 1, 0.01, 100)
const scene = new Scene()
const canvas = document.getElementById('root') as HTMLCanvasElement
const renderer = new WebGLRenderer({ antialias: true, canvas, alpha: true })

const boxMaterial = new MeshBasicMaterial({ color: 'red' })
const box = new Mesh(new BoxGeometry(), boxMaterial)
box.pointerEventsType = { deny: 'grab' }
let red = false
box.addEventListener('click', () => {
  red = !red
  boxMaterial.color.set(red ? 'red' : 'blue')
})
scene.add(box)

const store = createXRStore(canvas, scene, camera, renderer.xr)
document.getElementById('enter-ar')?.addEventListener('click', () => store.enterAR())

let prevTime: undefined | number

renderer.setAnimationLoop((time, frame) => {
  const delta = prevTime == null ? 0 : time - prevTime
  prevTime = time
  store.update(frame, delta)
  renderer.render(scene, camera)
})

function updateSize() {
  renderer.setSize(window.innerWidth, window.innerHeight)
  renderer.setPixelRatio(window.devicePixelRatio)
  camera.aspect = window.innerWidth / window.innerHeight
  camera.updateProjectionMatrix()
}

updateSize()
window.addEventListener('resize', updateSize)

How to enable XR for your threejs app?

1. const store = createXRStore(canvas, scene, camera, renderer.xr) create a xr store 2. store.enterAR() call enter AR when clicking on a button

Tutorials

_The following tutorials contain code for react-three/fiber but all informations are also applicable for @pmndrs/xr._

• 👌 Interactions
• 🔧 Options
• 🧊 Object Detection
• ✴ Origin
• 🪄 Teleport
• 🕹️ Gamepad
• 🎮 Custom Controller/Hands/...
• ⛨ Guards

Roadmap

• 🤳 XR Gestures
• ➕ Multimodal
• ⚓️ Anchors
• 📺 Layers
• 📱 Dom Overlays
• 🕺 Tracked Body
• 🎯 Hit Test
• ↕ pmndrs/controls

Sponsors

This project is supported by a few companies and individuals building cutting-edge 3D Web & XR experiences. Check them out!

In-Scene UI And Visual Quality

In-Scene UI

• Immersive VR and non-handheld AR cannot rely on DOM/HTML for the primary status panel, checklist, score, report, or controls.
• Use @react-three/uikit for React Three Fiber panels.
• DOM buttons remain appropriate for the initial browser user gesture and optional laptop fallback.
• Do not use @react-three/drei <Html> or XRDomOverlay as immersive VR/non-handheld UI.
• In-scene UI should not cover the whole headset view. Keep panels compact and place them beside, above, below, or behind the active target so the central recording still shows the world, affordances, and task object.
• Camera-following HUDs are acceptable when compact. They must leave the center of the viewport open for the scene and should not become the only visible content in the recording.
• For eval-friendly uikit apps, persistent UI should stay well below roughly one third of the headset view during normal play. The harness samples uikit roots over time, so a large temporary report is fine only if the walkthrough returns to a world-visible pose afterward.
• Prefer small persistent panels over dashboard-sized panels in the forward view. For typical 500-600px wide uikit panels, use a small world scale or pixel size and place the panel laterally so the projected panel normally stays closer to 10-20% of the view, leaving headroom for temporary callouts and reports.
• Treat the whole uikit root as the measured footprint. A single wide root with several rows of controls can still fail even if individual buttons are small; split optional controls into smaller roots or compress status text when the world object must remain visible.
• Prefer sizing uikit roots with pixelSize on the Container. Avoid shrinking a parent group and also using tiny uikit layout units, because double scaling can make the panel technically present but unreadable in the headset recording.
• Treat camera orientation as part of visual quality. After interacting with a panel, handle, or product control, turn or move back so the next recorded seconds frame the active world object and not an empty wall, blank AR background, or white page area.

Package Notes

• React uikit companion packages are commonly @react-three/uikit, @react-three/uikit-default, and @react-three/uikit-lucide.
• During eval work, 1.0.67 was observed as a published version for those uikit packages.

Visual Quality

• Build a domain-specific scene, not a placeholder-cube demo.
• Include recognizable environment context: floor, walls, lanes, studio, warehouse, showroom, range, workstation, or similar.
• Use multiple object types and readable scale cues.
• Add lighting and material variation.
• Make interactive affordances obvious from plausible headset/controller poses.
• Keep the primary game/training/tool object visible while UI is displayed. If the user looks at a menu or report, return the viewer to the active world object or frame both the panel and object before continuing.
• Prefer plain 6-digit color hex plus explicit opacity/alpha props for Three/uikit colors. Some Three color paths warn on 8-digit CSS hex such as #111820ee.
• Show feedback for hover, selection, hit, miss, completion, errors, score changes, wave changes, or configuration changes.
• In games, show motion, progression, score, failure states, and final results.
• In tools/training apps, show task state, current target, completion criteria, and final report/summary.
• In simulations, show telemetry, constraints, warnings, recovery, and mission/process outcome.
• In commerce/configuration, show selected product or part, variants/materials, dimensions/fit, price/readiness, and final configuration/cart summary.

Common Pitfalls

• UI text exists only in DOM while the headset sees empty 3D space.
• A large uikit panel or HUD fills the entire recorded viewport, hiding pads, products, handles, robots, or vehicles.
• The viewer turns toward an empty white, grey, or transparent AR background after a successful action, so the video proves logs but not the experience.
• Controls are hidden behind walls, rails, old panels, or decorative meshes.
• Targets are too tiny, too close, or lack active hit geometry.
• Repeated cards/panels look polished on desktop but are unreachable by XR rays.

Use-Case Patterns

Use the task domain to choose interaction, evidence, and UI. Do not copy these as fixed templates; treat them as pressure tests for whether the experience feels real.

Games

• Show changing game state: score, waves/rounds, ammo/energy, accuracy, timer, health/fail state, and final result.
• Match the main input to the fantasy: weapon from a muzzle, glove/fist contact, grabbing, throwing, steering, rhythm timing, or gaze only when it is truly the mechanic.
• Validate visible play: hits, misses, motion, transitions, and end state should come from controller/headset input.

Training

• Model an ordered task with orientation, current instruction, progress, mistakes/hazards, corrective action, and pass/fail report.
• Let the learner inspect real-feeling stations, tools, parts, or objects from plausible headset positions.
• Validate each critical step through visible spatial interaction and view checks, not by directly advancing task state.

Simulation

• Represent the simulated system with telemetry, constraints, warning/recovery state, and visible movement over time.
• Drive the system from user input: controller pose/buttons, hand interaction, steering/throttle controls, handles, or spatial waypoints.
• Validate before/after simulated state such as position, orientation, velocity, waypoint progress, warning state, or process completion.

Commerce And Configuration

• Make the product inspectable at useful scale with variants/materials, dimensions, fit/readiness, price or summary, and reset/confirm paths.
• For AR placement, use hit-test, planes, anchors, reticles, footprints, or equivalent surface affordances. Show placement and fit feedback in the XR scene even if handheld AR also uses DOM overlay.
• For spatial configuration, prove actual transforms or variant changes: place, drag, rotate, scale, snap, swap, and summarize the configured result.

Professional Tools

• Prefer visible handles, gizmos, grabbable affordances, annotations, and clear selected/locked/error states.
• Keep controls dense enough for repeated work but readable in headset. Primary status and manipulation feedback should stay in-scene.
• Validate real edits with transform/state deltas and a final reviewable summary.