Realitykit

RealityKit

Build AR experiences on iOS using RealityKit for rendering and ARKit for world tracking. Covers RealityView, entity management, raycasting, scene understanding, and gesture-based interactions. Targets Swift 6.3 / iOS 26+.

Contents

• Setup
• RealityView Basics
• Loading and Creating Entities
• Anchoring and Placement
• Raycasting
• Gestures and Interaction
• Scene Understanding
• Common Mistakes
• Review Checklist
• References

Setup

Project Configuration

1. Add NSCameraUsageDescription to Info.plist 2. On iOS, RealityViewCameraContent displays an AR camera view by default (iOS 18+, macOS 15+); use .virtual camera mode for explicit non-AR fallback 3. No entitlement is required for basic AR. If AR is core to the app, add the arkit required-device capability; otherwise gate AR UI with isSupported.

Device Requirements

AR features require devices with an A9 chip or later. Always check ARWorldTrackingConfiguration.isSupported before presenting AR UI.

import ARKit

guard ARWorldTrackingConfiguration.isSupported else {
    showUnsupportedDeviceMessage()
    return
}

Key Types

RealityView Basics

RealityView is the SwiftUI entry point for RealityKit. RealityViewCameraContent is the iOS/macOS content type. On iOS, it uses an AR camera view by default and can use content.camera = .virtual for non-AR mode when requested or when AR/camera access is unavailable.

import ARKit
import SwiftUI
import RealityKit

struct ARExperienceView: View {
    var body: some View {
        RealityView { (content: RealityViewCameraContent) in
            if !ARWorldTrackingConfiguration.isSupported {
                content.camera = .virtual
            }

            let sphere = ModelEntity(
                mesh: .generateSphere(radius: 0.05),
                materials: [SimpleMaterial(
                    color: .blue,
                    isMetallic: true
                )]
            )
            sphere.position = [0, 0, -0.5]  // 50cm in front of camera
            content.add(sphere)
        }
    }
}

Make and Update Pattern

Use the update closure to respond to SwiftUI state changes:

struct PlacementView: View {
    @State private var modelColor: UIColor = .red

    var body: some View {
        RealityView { content in
            let box = ModelEntity(
                mesh: .generateBox(size: 0.1),
                materials: [SimpleMaterial(
                    color: .red,
                    isMetallic: false
                )]
            )
            box.name = "colorBox"
            box.position = [0, 0, -0.5]
            content.add(box)
        } update: { content in
            if let box = content.entities.first(
                where: { $0.name == "colorBox" }
            ) as? ModelEntity {
                box.model?.materials = [SimpleMaterial(
                    color: modelColor,
                    isMetallic: false
                )]
            }
        }

        Button("Change Color") {
            modelColor = modelColor == .red ? .green : .red
        }
    }
}

Loading and Creating Entities

Loading from USDZ Files

Load 3D models asynchronously to avoid blocking the main thread:

RealityView { content in
    if let robot = try? await ModelEntity(named: "robot") {
        robot.position = [0, -0.2, -0.8]
        robot.scale = [0.01, 0.01, 0.01]
        content.add(robot)
    }
}

Adding Components

Entities use an ECS (Entity Component System) architecture. Add components to give entities behavior:

let box = ModelEntity(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .red, isMetallic: false)]
)

// Make it respond to physics
box.components.set(PhysicsBodyComponent(
    massProperties: .default,
    material: .default,
    mode: .dynamic
))

// Add collision shape for interaction
box.components.set(CollisionComponent(
    shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
))

// Enable input targeting for gestures
box.components.set(InputTargetComponent())

Anchoring and Placement

AnchorEntity

Use AnchorEntity to anchor content to detected surfaces or world positions:

RealityView { content in
    // Anchor to a horizontal surface
    let floorAnchor = AnchorEntity(.plane(
        .horizontal,
        classification: .floor,
        minimumBounds: [0.2, 0.2]
    ))

    let model = ModelEntity(
        mesh: .generateBox(size: 0.1),
        materials: [SimpleMaterial(color: .orange, isMetallic: false)]
    )
    floorAnchor.addChild(model)
    content.add(floorAnchor)
}

Anchor Targets

Raycasting

Keep RealityKit scene queries separate from ARKit real-world raycasts:

• RealityViewCameraContent.ray(through:in:to:) returns a camera ray in

RealityKit coordinate spaces. It projects a screen point into the virtual scene; it is not proof of a detected physical surface.

• RealityViewCameraContent.hitTest(point:in:query:mask:) hits virtual

entities made hittable by CollisionComponent shapes. Use those shapes for entity picking and targeted gestures, not ARKit plane detection.

• Use AnchorEntity(.plane(...)) for simple placement on detected planes.
• Use ARKit ARRaycastQuery plus ARSession.raycast(_:) when the task needs

a one-shot intersection with real-world surfaces, then anchor with AnchorEntity(raycastResult:).

let results = session.raycast(query)
if let result = results.first {
    let anchor = AnchorEntity(raycastResult: result)
    anchor.addChild(model)
    content.add(anchor)
}

Do not treat entity hit tests as substitutes for ARKit surface raycasts.

Gestures and Interaction

For gesture-based entity interaction, add CollisionComponent for the hittable shape and InputTargetComponent for input targeting. Use AccessibilityComponent for entity labels/actions. Hand detailed SwiftUI gesture composition and VoiceOver/Switch Control policy to sibling skills.

Drag Gesture on Entities

struct DraggableARView: View {
    var body: some View {
        RealityView { content in
            let box = ModelEntity(
                mesh: .generateBox(size: 0.1),
                materials: [SimpleMaterial(color: .blue, isMetallic: true)]
            )
            box.position = [0, 0, -0.5]
            box.components.set(CollisionComponent(
                shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
            ))
            box.components.set(InputTargetComponent())
            box.name = "draggable"
            content.add(box)
        }
        .gesture(
            DragGesture()
                .targetedToAnyEntity()
                .onChanged { value in
                    let entity = value.entity
                    guard let parent = entity.parent else { return }
                    entity.position = value.convert(
                        value.location3D,
                        from: .local,
                        to: parent
                    )
                }
        )
    }
}

For selection, CollisionComponent is the mechanism that makes an entity hittable by RealityViewCameraContent.hitTest, SpatialTapGesture, or targetedToAnyEntity(). Pair it with InputTargetComponent; this enables virtual entity picking, not ARKit surface detection.

Scene Understanding

Per-Frame Updates

Subscribe to scene update events for continuous processing:

RealityView { content in
    let entity = ModelEntity(
        mesh: .generateSphere(radius: 0.05),
        materials: [SimpleMaterial(color: .yellow, isMetallic: false)]
    )
    entity.position = [0, 0, -0.5]
    content.add(entity)

    _ = content.subscribe(to: SceneEvents.Update.self) { event in
        let time = Float(event.deltaTime)
        entity.position.y += sin(Float(Date().timeIntervalSince1970)) * time * 0.1
    }
}

Platform Boundaries

On visionOS, ARKit provides a different API surface with ARKitSession, WorldTrackingProvider, and PlaneDetectionProvider. These visionOS-specific types are not available on iOS. On iOS, RealityKit handles world tracking automatically through RealityViewCameraContent.

For iOS architecture or migration notes, explicitly name the iOS RealityKit path and handoffs:

• Gate AR with ARWorldTrackingConfiguration.isSupported.
• Host content with RealityViewCameraContent.
• Build scenes from Entity/ModelEntity and place with AnchorEntity.
• Include a compact Handoffs line in architecture/review notes:

CollisionComponent + InputTargetComponent handle RealityKit interaction; AccessibilityComponent handles RealityKit entity accessibility metadata; detailed SwiftUI gestures and VoiceOver/Switch Control policy belong to siblings.

Treat existing SCNView/SCNNode work as either a separate SceneKit path or an explicit migration to RealityKit, not a mixed scene graph.

Common Mistakes

DON'T: Skip AR capability checks

Not all devices support AR. Showing a black camera view with no feedback confuses users.

// WRONG -- no device check
struct MyARView: View {
    var body: some View {
        RealityView { content in
            // Fails silently on unsupported devices
        }
    }
}

// CORRECT -- check support and show fallback
struct MyARView: View {
    var body: some View {
        if ARWorldTrackingConfiguration.isSupported {
            RealityView { content in
                // AR content
            }
        } else {
            ContentUnavailableView(
                "AR Not Supported",
                systemImage: "arkit",
                description: Text("This device does not support AR.")
            )
        }
    }
}

DON'T: Load heavy models synchronously

Loading large USDZ files on the main thread causes frame drops and hangs. The make closure of RealityView is async -- use it.

// WRONG -- synchronous load blocks the main thread
RealityView { content in
    let model = try! Entity.load(named: "large-scene")
    content.add(model)
}

// CORRECT -- async load
RealityView { content in
    if let model = try? await ModelEntity(named: "large-scene") {
        content.add(model)
    }
}

DON'T: Forget collision and input target components for interactive entities

Gestures only work on entities that have both CollisionComponent and InputTargetComponent. Without them, taps and drags pass through.

// WRONG -- entity ignores gestures
let box = ModelEntity(mesh: .generateBox(size: 0.1))
content.add(box)

// CORRECT -- add collision and input components
let box = ModelEntity(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .red, isMetallic: false)]
)
box.components.set(CollisionComponent(
    shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
))
box.components.set(InputTargetComponent())
content.add(box)

DON'T: Create new entities in the update closure

The update closure runs on every SwiftUI state change. Creating entities there duplicates content on each render pass.

// WRONG -- duplicates entities on every state change
RealityView { content in
    // empty
} update: { content in
    let sphere = ModelEntity(mesh: .generateSphere(radius: 0.05))
    content.add(sphere)  // Added again on every update
}

// CORRECT -- create in make, modify in update
RealityView { content in
    let sphere = ModelEntity(mesh: .generateSphere(radius: 0.05))
    sphere.name = "mySphere"
    content.add(sphere)
} update: { content in
    if let sphere = content.entities.first(
        where: { $0.name == "mySphere" }
    ) as? ModelEntity {
        // Modify existing entity
        sphere.position.y = newYPosition
    }
}

DON'T: Ignore camera permission

RealityKit on iOS needs camera access. If the user denies permission, the view shows a black screen with no explanation.

// WRONG -- no permission handling
RealityView { content in
    // Black screen if camera denied
}

// CORRECT -- check and request permission
struct ARContainerView: View {
    @State private var cameraAuthorized = false

    var body: some View {
        Group {
            if cameraAuthorized {
                RealityView { content in
                    // AR content
                }
            } else {
                ContentUnavailableView(
                    "Camera Access Required",
                    systemImage: "camera.fill",
                    description: Text("Enable camera in Settings to use AR.")
                )
            }
        }
        .task {
            let status = AVCaptureDevice.authorizationStatus(for: .video)
            if status == .authorized {
                cameraAuthorized = true
            } else if status == .notDetermined {
                cameraAuthorized = await AVCaptureDevice
                    .requestAccess(for: .video)
            }
        }
    }
}

Review Checklist

• [ ] NSCameraUsageDescription set in Info.plist
• [ ] AR device capability checked before presenting AR views
• [ ] Camera permission requested and denial handled with a fallback UI
• [ ] arkit required-device capability added when AR is the app's core purpose
• [ ] 3D models loaded asynchronously in the make closure
• [ ] Entities created in make, modified in update (not created in update)
• [ ] Interaction notes say CollisionComponent makes entities hittable/pickable and pairs with InputTargetComponent
• [ ] Boundary/review notes include a Handoffs line naming AccessibilityComponent and routing detailed SwiftUI/accessibility policy to siblings
• [ ] Entity hit tests, RealityViewCameraContent.ray(...), and ARKit real-world surface raycasts are not conflated
• [ ] ARKit surface raycast placements use ARSession.raycast(_:) and AnchorEntity(raycastResult:)
• [ ] SceneEvents.Update subscriptions used for per-frame logic (not SwiftUI timers)
• [ ] Large scenes use ModelEntity(named:) async loading, not Entity.load(named:)
• [ ] Anchor entities target appropriate surface types for the use case
• [ ] Entity names set for lookup in the update closure

References

• Read references/realitykit-patterns.md for physics, animations, lighting, ECS, accessibility, and performance patterns.
• RealityKit framework
• RealityView
• RealityViewCameraContent
• RealityViewCamera
• Entity
• ModelEntity
• AnchorEntity
• ARKit framework
• ARKit in iOS
• Verifying Device Support and User Permission
• ARWorldTrackingConfiguration
• ARRaycastQuery
• ARSession.raycast(_:))
• Loading entities from a file

RealityKit + ARKit Extended Patterns

Overflow reference for the realitykit skill. Contains advanced patterns that exceed the main skill file's scope.

Contents

• Physics Simulation
• Entity Animations
• Custom Materials and Lighting
• Entity Component System
• Occlusion and Environment
• RealityKit + SwiftUI Integration
• Performance Tips

Physics Simulation

Rigid Body Physics

import RealityKit

func createPhysicsScene(content: RealityViewCameraContent) {
    // Ground plane (static body)
    let ground = ModelEntity(
        mesh: .generatePlane(width: 2, depth: 2),
        materials: [SimpleMaterial(color: .gray, isMetallic: false)]
    )
    ground.components.set(PhysicsBodyComponent(
        massProperties: .default,
        material: .default,
        mode: .static
    ))
    ground.components.set(CollisionComponent(
        shapes: [.generateBox(size: [2, 0.01, 2])]
    ))
    ground.position = [0, -0.5, -1]
    content.add(ground)

    // Falling box (dynamic body)
    let box = ModelEntity(
        mesh: .generateBox(size: 0.1),
        materials: [SimpleMaterial(color: .red, isMetallic: true)]
    )
    box.components.set(PhysicsBodyComponent(
        massProperties: .init(mass: 1.0),
        material: PhysicsMaterialResource.generate(
            staticFriction: 0.5,
            dynamicFriction: 0.5,
            restitution: 0.7
        ),
        mode: .dynamic
    ))
    box.components.set(CollisionComponent(
        shapes: [.generateBox(size: [0.1, 0.1, 0.1])]
    ))
    box.position = [0, 0.5, -1]
    content.add(box)
}

Applying Forces and Impulses

// Apply a continuous force (in Newtons)
if var physics = entity.components[PhysicsBodyComponent.self] {
    // Forces are applied per-frame via the physics simulation
}

// Apply an instantaneous impulse
if let physicsEntity = entity as? HasPhysicsBody {
    physicsEntity.addForce([0, 10, 0], relativeTo: nil)
    physicsEntity.applyLinearImpulse([5, 0, 0], relativeTo: nil)
}

Collision Detection

RealityView { content in
    // Setup entities...

    _ = content.subscribe(to: CollisionEvents.Began.self) { event in
        print("Collision between \(event.entityA.name) and \(event.entityB.name)")
    }

    _ = content.subscribe(to: CollisionEvents.Ended.self) { event in
        print("Collision ended between \(event.entityA.name) and \(event.entityB.name)")
    }
}

Entity Animations

Transform Animation

func animateEntity(_ entity: Entity) {
    // Move to a new position over 2 seconds
    var transform = entity.transform
    transform.translation = [0.5, 0, -0.5]

    entity.move(
        to: transform,
        relativeTo: entity.parent,
        duration: 2.0,
        timingFunction: .easeInOut
    )
}

Orbit Animation

func orbitAnimation(
    entity: Entity,
    content: RealityViewCameraContent
) {
    var angle: Float = 0
    let radius: Float = 0.3
    let center = SIMD3<Float>(0, 0, -0.8)

    _ = content.subscribe(to: SceneEvents.Update.self) { event in
        angle += Float(event.deltaTime) * 1.5
        entity.position = center + SIMD3<Float>(
            cos(angle) * radius,
            0,
            sin(angle) * radius
        )
    }
}

Playing USDZ Animations

RealityView { content in
    if let character = try? await ModelEntity(named: "character") {
        content.add(character)

        // Play all available animations
        if let animation = character.availableAnimations.first {
            character.playAnimation(
                animation.repeat(duration: .infinity),
                transitionDuration: 0.5,
                startsPaused: false
            )
        }
    }
}

Custom Materials and Lighting

SimpleMaterial Variations

// Metallic material
let metallic = SimpleMaterial(
    color: .init(tint: .gray, texture: nil),
    roughness: .float(0.1),
    isMetallic: true
)

// Transparent material
var transparent = SimpleMaterial()
transparent.color = .init(
    tint: UIColor.blue.withAlphaComponent(0.5),
    texture: nil
)
transparent.blending = .transparent(opacity: 0.5)

// Textured material
if let texture = try? await TextureResource(named: "wood") {
    var textured = SimpleMaterial()
    textured.color = .init(
        tint: .white,
        texture: .init(texture)
    )
}

Environment Lighting

RealityView { content in
    // Use image-based lighting
    if let resource = try? await EnvironmentResource(named: "studio") {
        content.environment.lighting.resource = resource
        content.environment.lighting.intensityExponent = 1.0
    }
}

Entity Component System

Custom Components

Define custom components to attach data to entities:

import RealityKit

struct HealthComponent: Component {
    var currentHealth: Float
    var maxHealth: Float

    var healthPercentage: Float {
        currentHealth / maxHealth
    }
}

// Register the component (once, at app launch)
HealthComponent.registerComponent()

// Attach to an entity
let enemy = ModelEntity(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .red, isMetallic: false)]
)
enemy.components.set(HealthComponent(
    currentHealth: 100,
    maxHealth: 100
))

Custom Systems

Systems process entities with matching components each frame:

struct HealthColorSystem: System {
    static let query = EntityQuery(where: .has(HealthComponent.self))

    init(scene: Scene) { }

    func update(context: SceneUpdateContext) {
        for entity in context.entities(
            matching: Self.query,
            updatingSystemWhen: .rendering
        ) {
            guard let health = entity.components[HealthComponent.self],
                  var model = entity.components[ModelComponent.self]
            else { continue }

            let ratio = health.healthPercentage
            let color = UIColor(
                red: CGFloat(1 - ratio),
                green: CGFloat(ratio),
                blue: 0,
                alpha: 1
            )
            model.materials = [SimpleMaterial(
                color: color,
                isMetallic: false
            )]
            entity.components.set(model)
        }
    }
}

// Register the system
HealthColorSystem.registerSystem()

Occlusion and Environment

Occlusion Material

Use occlusion materials to hide virtual content behind real-world surfaces:

let occlusionPlane = ModelEntity(
    mesh: .generatePlane(width: 1, depth: 1),
    materials: [OcclusionMaterial()]
)
occlusionPlane.position = [0, -0.5, -1]
content.add(occlusionPlane)

This makes virtual objects appear to go behind real-world surfaces.

Environment Configuration

RealityView { content in
    // Configure the background
    content.environment.background = .color(.clear)  // AR camera passthrough

    // Adjust lighting intensity
    content.environment.lighting.intensityExponent = 1.2
}

RealityKit + SwiftUI Integration

AR Model Viewer with Controls

import SwiftUI
import RealityKit
import AVFoundation

@Observable
@MainActor
final class ARViewModel {
    var selectedModelName: String = "robot"
    var modelScale: Float = 0.01
    var isPlaced = false

    private var currentEntity: ModelEntity?

    func loadModel(into content: RealityViewCameraContent) async {
        // Remove existing model
        currentEntity?.removeFromParent()

        guard let model = try? await ModelEntity(
            named: selectedModelName
        ) else { return }

        model.scale = [modelScale, modelScale, modelScale]
        model.position = [0, -0.2, -0.8]
        model.name = "currentModel"

        // Enable interaction
        model.generateCollisionShapes(recursive: true)
        model.components.set(InputTargetComponent())

        content.add(model)
        currentEntity = model
        isPlaced = true
    }
}

struct ARModelViewer: View {
    @State private var viewModel = ARViewModel()
    @State private var cameraAuthorized = false

    var body: some View {
        ZStack {
            if cameraAuthorized {
                RealityView { content in
                    await viewModel.loadModel(into: content)
                } update: { content in
                    if let model = content.entities.first(
                        where: { $0.name == "currentModel" }
                    ) {
                        let s = viewModel.modelScale
                        model.scale = [s, s, s]
                    }
                }
                .gesture(
                    DragGesture()
                        .targetedToAnyEntity()
                        .onChanged { value in
                            value.entity.position = value.convert(
                                value.location3D,
                                from: .local,
                                to: value.entity.parent!
                            )
                        }
                )
            } else {
                ContentUnavailableView(
                    "Camera Required",
                    systemImage: "camera.fill",
                    description: Text("Allow camera access for AR.")
                )
            }

            VStack {
                Spacer()
                controlsPanel
            }
        }
        .task {
            let status = AVCaptureDevice.authorizationStatus(for: .video)
            if status == .authorized {
                cameraAuthorized = true
            } else if status == .notDetermined {
                cameraAuthorized = await AVCaptureDevice
                    .requestAccess(for: .video)
            }
        }
    }

    private var controlsPanel: some View {
        VStack {
            Picker("Model", selection: $viewModel.selectedModelName) {
                Text("Robot").tag("robot")
                Text("Chair").tag("chair")
                Text("Lamp").tag("lamp")
            }
            .pickerStyle(.segmented)

            Slider(
                value: $viewModel.modelScale,
                in: 0.001...0.05,
                step: 0.001
            ) {
                Text("Scale")
            }
        }
        .padding()
        .background(.ultraThinMaterial)
        .clipShape(.rect(cornerRadius: 16))
        .padding()
    }
}

Entity Accessibility

Make AR content accessible to assistive technologies:

let model = ModelEntity(
    mesh: .generateBox(size: 0.1),
    materials: [SimpleMaterial(color: .blue, isMetallic: false)]
)
var accessibility = AccessibilityComponent()
accessibility.isAccessibilityElement = true
accessibility.label = "Blue cube"
accessibility.value = "Interactive 3D object"
accessibility.traits = [.button]
accessibility.systemActions = [.activate]
model.components.set(accessibility)

Performance Tips

Entity Pooling

Reuse entities instead of creating and destroying them repeatedly:

@MainActor
final class EntityPool {
    private var available: [ModelEntity] = []
    private let mesh: MeshResource
    private let materials: [any Material]

    init(mesh: MeshResource, materials: [any Material], initialCount: Int) {
        self.mesh = mesh
        self.materials = materials
        for _ in 0..<initialCount {
            let entity = ModelEntity(mesh: mesh, materials: materials)
            entity.isEnabled = false
            available.append(entity)
        }
    }

    func acquire() -> ModelEntity {
        if let entity = available.popLast() {
            entity.isEnabled = true
            return entity
        }
        return ModelEntity(mesh: mesh, materials: materials)
    }

    func release(_ entity: ModelEntity) {
        entity.isEnabled = false
        entity.removeFromParent()
        available.append(entity)
    }
}

Reducing Draw Calls

• Merge static entities into a single entity when possible
• Use instancing for repeated identical geometry
• Limit the number of unique materials in a scene
• Use LOD (level of detail) models for objects at varying distances

Memory Management

• Unload entities that are no longer visible
• Use ModelEntity(named:in:) with a specific bundle to control resource loading
• Monitor memory with Instruments (RealityKit Trace template)
• Set entity.isEnabled = false instead of removing entities you will reuse