Axiom Media

Media

You MUST use this skill for ANY camera, photo, audio, haptic, or media playback work.

Quick Reference

Decision Tree

digraph media {
    start [label="Media task" shape=ellipse];
    what [label="Which media feature?" shape=diamond];

    start -> what;
    what -> "skills/camera-capture.md" [label="camera capture"];
    what -> "skills/photo-library.md" [label="photo pickers\n/ library"];
    what -> "skills/avfoundation-ref.md" [label="audio / AVFoundation"];
    what -> "skills/avfoundation-video-ref.md" [label="video write/export\n/ sample-buffer (OS27)"];
    what -> "skills/shazamkit.md" [label="ShazamKit\n/ audio recognition"];
    what -> "skills/music-understanding.md" [label="music analysis\n(key/tempo/structure)"];
    what -> "skills/haptics.md" [label="haptic feedback"];
    what -> "skills/now-playing.md" [label="Now Playing\n/ remote commands"];
    what -> "skills/carplay-hig.md" [label="CarPlay app design\n/ categories / entitlements"];
    what -> "skills/dockkit.md" [label="DockKit stands\n/ gimbals / tracking"];
}

1. Camera capture? → skills/camera-capture.md (patterns), skills/camera-capture-ref.md (API), skills/camera-capture-diag.md (debugging) 2. Photo pickers / library? → skills/photo-library.md, skills/photo-library-ref.md 3. Audio / AVFoundation (audio)? → skills/avfoundation-ref.md; video write/export/playback, sample-buffer engine, resumable export, iOS 27 deprecations? → skills/avfoundation-video-ref.md (OS27) 4. ShazamKit / audio recognition? → skills/shazamkit.md, skills/shazamkit-ref.md 5. On-device music analysis (key, tempo, structure, pace, instruments, loudness)? → skills/music-understanding.md (OS27) 6. Haptics? → skills/haptics.md 7. Now Playing / remote commands? → skills/now-playing.md, skills/now-playing-carplay.md, skills/now-playing-musickit.md 8. CarPlay app design, category selection, entitlement request? → skills/carplay-hig.md (start here for any CarPlay work) 9. DockKit motorized stands / gimbals, subject tracking, custom motor control? → skills/dockkit.md 10. Want camera code audit? → Launch camera-auditor agent (detects deprecated APIs and architectural gaps: missing interruption handlers, runtime-error recovery, audio session deactivation, permission-denied UX, RotationCoordinator on iOS 17+; scores RELIABLE / FRAGILE / BROKEN)

Cross-Domain Routing

Camera + permissions (camera access denied, Info.plist missing):

• Camera code → stay here (camera-capture)
• Privacy manifest / Info.plist → invoke axiom-integration (privacy-ux reference)
• Build/entitlement errors → invoke axiom-build

ShazamKit + microphone permissions:

• Microphone NSMicrophoneUsageDescription → invoke axiom-integration (privacy-ux reference)
• ShazamKit API and matching → stay here (shazamkit)

Now Playing + background audio:

• Now Playing metadata/controls → stay here (now-playing)
• Background audio mode / BGTaskScheduler → invoke axiom-integration (background-processing reference)

Photo library + privacy:

• Photo picker (PHPicker, PhotosPicker) → stay here (photo-library) — no permissions needed
• Full PHPhotoLibrary access → stay here (photo-library-ref) — limited access model
• Privacy manifest for photo usage → invoke axiom-integration (privacy-ux reference)

DockKit + camera / custom inference:

• DockKit stand control, framing, motor, tracking states → stay here (dockkit)
• Underlying AVCaptureSession setup → stay here (camera-capture)
• Custom Vision / Core ML inference feeding observations → invoke axiom-vision
• Camera permission (NSCameraUsageDescription) → invoke axiom-integration (privacy-ux reference)

Anti-Rationalization

Example Invocations

User: "How do I set up a camera preview?" → Read: skills/camera-capture.md

User: "My camera app takes a second before preview appears" → Read: skills/camera-capture.md (Pattern 8, deferred start)

User: "Support the Center Stage front camera" / "Capture 48MP photos" → Read: skills/camera-capture-ref.md

User: "Camera freezes when I get a phone call" → Read: skills/camera-capture-diag.md

User: "How do I let users pick photos in SwiftUI?" → Read: skills/photo-library.md

User: "Implement haptic feedback for button taps" → Read: skills/haptics.md

User: "Now Playing info doesn't appear on Lock Screen" → Read: skills/now-playing.md

User: "How do I identify songs with ShazamKit?" → Read: skills/shazamkit.md

User: "How do I detect a song's tempo / key / beat grid on-device?" / "Analyze audio loudness or structure" → Read: skills/music-understanding.md

User: "Track a subject with a motorized stand" / "Control a DockKit gimbal" → Read: skills/dockkit.md

User: "Check my camera code for issues" → Launch: camera-auditor agent

AVFoundation Audio Reference

Quick Reference

// AUDIO SESSION SETUP
import AVFoundation

try AVAudioSession.sharedInstance().setCategory(
    .playback,                              // or .playAndRecord, .ambient
    mode: .default,                         // or .voiceChat, .measurement
    options: [.mixWithOthers, .allowBluetoothHFP]
)
try AVAudioSession.sharedInstance().setActive(true)

// AUDIO ENGINE PIPELINE
let engine = AVAudioEngine()
let player = AVAudioPlayerNode()
engine.attach(player)
engine.connect(player, to: engine.mainMixerNode, format: nil)
try engine.start()
player.scheduleFile(audioFile, at: nil)
player.play()

// INPUT PICKER (iOS 26+)
import AVKit
let picker = AVInputPickerInteraction()
picker.delegate = self
myButton.addInteraction(picker)
// In button action: picker.present()

// AIRPODS HIGH QUALITY (iOS 26+)
try AVAudioSession.sharedInstance().setCategory(
    .playAndRecord,
    options: [.bluetoothHighQualityRecording, .allowBluetoothA2DP]
)

---

AVAudioSession

Categories

Modes

Options

// Mixing
.mixWithOthers          // Play with other apps
.duckOthers             // Lower other audio while playing
.interruptSpokenAudioAndMixWithOthers  // Pause podcasts, mix music

// Bluetooth
.allowBluetoothHFP      // HFP (calls); replaces deprecated .allowBluetooth
.allowBluetoothA2DP     // High quality stereo
.bluetoothHighQualityRecording  // iOS 26+ AirPods recording

// Routing
.defaultToSpeaker       // Route to speaker (not receiver)
.allowAirPlay           // Enable AirPlay

Interruption Handling

NotificationCenter.default.addObserver(
    forName: AVAudioSession.interruptionNotification,
    object: nil,
    queue: .main
) { notification in
    guard let userInfo = notification.userInfo,
          let typeValue = userInfo[AVAudioSessionInterruptionTypeKey] as? UInt,
          let type = AVAudioSession.InterruptionType(rawValue: typeValue) else {
        return
    }

    switch type {
    case .began:
        // Pause playback
        player.pause()

    case .ended:
        guard let optionsValue = userInfo[AVAudioSessionInterruptionOptionKey] as? UInt else { return }
        let options = AVAudioSession.InterruptionOptions(rawValue: optionsValue)
        if options.contains(.shouldResume) {
            player.play()
        }

    @unknown default:
        break
    }
}

Route Change Handling

NotificationCenter.default.addObserver(
    forName: AVAudioSession.routeChangeNotification,
    object: nil,
    queue: .main
) { notification in
    guard let userInfo = notification.userInfo,
          let reasonValue = userInfo[AVAudioSessionRouteChangeReasonKey] as? UInt,
          let reason = AVAudioSession.RouteChangeReason(rawValue: reasonValue) else {
        return
    }

    switch reason {
    case .oldDeviceUnavailable:
        // Headphones unplugged — pause playback
        player.pause()

    case .newDeviceAvailable:
        // New device connected
        break

    case .categoryChange:
        // Category changed by system or another app
        break

    default:
        break
    }
}

---

AVAudioEngine

Basic Pipeline

let engine = AVAudioEngine()

// Create nodes
let player = AVAudioPlayerNode()
let reverb = AVAudioUnitReverb()
reverb.loadFactoryPreset(.largeHall)
reverb.wetDryMix = 50

// Attach to engine
engine.attach(player)
engine.attach(reverb)

// Connect: player → reverb → mixer → output
engine.connect(player, to: reverb, format: nil)
engine.connect(reverb, to: engine.mainMixerNode, format: nil)

// Start
engine.prepare()
try engine.start()

// Play file
let url = Bundle.main.url(forResource: "audio", withExtension: "m4a")!
let file = try AVAudioFile(forReading: url)
player.scheduleFile(file, at: nil)
player.play()

Node Types

Installing Taps (Audio Analysis)

let inputNode = engine.inputNode
let format = inputNode.outputFormat(forBus: 0)

inputNode.installTap(onBus: 0, bufferSize: 1024, format: format) { buffer, time in
    // Process audio buffer
    guard let channelData = buffer.floatChannelData?[0] else { return }
    let frameLength = Int(buffer.frameLength)

    // Calculate RMS level
    var sum: Float = 0
    for i in 0..<frameLength {
        sum += channelData[i] * channelData[i]
    }
    let rms = sqrt(sum / Float(frameLength))
    let dB = 20 * log10(rms)

    DispatchQueue.main.async {
        self.levelMeter = dB
    }
}

// Don't forget to remove when done
inputNode.removeTap(onBus: 0)

Format Conversion

// AVAudioEngine mic input is always 44.1kHz/32-bit float
// Use AVAudioConverter for other formats

let inputFormat = engine.inputNode.outputFormat(forBus: 0)
let outputFormat = AVAudioFormat(
    commonFormat: .pcmFormatInt16,
    sampleRate: 48000,
    channels: 1,
    interleaved: false
)!

let converter = AVAudioConverter(from: inputFormat, to: outputFormat)!

// In tap callback:
let outputBuffer = AVAudioPCMBuffer(
    pcmFormat: outputFormat,
    frameCapacity: AVAudioFrameCount(outputFormat.sampleRate * 0.1)
)!

var error: NSError?
converter.convert(to: outputBuffer, error: &error) { inNumPackets, outStatus in
    outStatus.pointee = .haveData
    return inputBuffer
}

---

Bit-Perfect Audio / DAC Output

iOS Behavior

iOS provides bit-perfect output by default to USB DACs — no resampling occurs. The DAC receives the source sample rate directly.

// iOS automatically matches source sample rate to DAC
// No special configuration needed for bit-perfect output

let player = AVAudioPlayerNode()
// File at 96kHz → DAC receives 96kHz

Avoiding Resampling

// Check hardware sample rate
let hardwareSampleRate = AVAudioSession.sharedInstance().sampleRate

// Match your audio format to hardware when possible
let format = AVAudioFormat(
    standardFormatWithSampleRate: hardwareSampleRate,
    channels: 2
)

USB DAC Routing

// List available outputs
let currentRoute = AVAudioSession.sharedInstance().currentRoute
for output in currentRoute.outputs {
    print("Output: \(output.portName), Type: \(output.portType)")
    // USB DAC shows as .usbAudio
}

// Prefer USB output
try AVAudioSession.sharedInstance().setPreferredInput(usbPort)

Sample Rate Considerations

---

iOS 26+ Input Selection

AVInputPickerInteraction

Native input device selection with live metering:

import AVKit

class RecordingViewController: UIViewController {
    let inputPicker = AVInputPickerInteraction()

    override func viewDidLoad() {
        super.viewDidLoad()

        // Configure audio session first
        try? AVAudioSession.sharedInstance().setCategory(.playAndRecord)
        try? AVAudioSession.sharedInstance().setActive(true)

        // Setup picker
        inputPicker.delegate = self
        selectMicButton.addInteraction(inputPicker)
    }

    @IBAction func selectMicTapped(_ sender: UIButton) {
        inputPicker.present()
    }
}

extension RecordingViewController: AVInputPickerInteractionDelegate {
    // Implement delegate methods as needed
}

Features:

• Live sound level metering
• Microphone mode selection
• System remembers selection per app

---

iOS 26+ AirPods High Quality Recording

LAV-microphone equivalent quality for content creators:

// AVAudioSession approach
try AVAudioSession.sharedInstance().setCategory(
    .playAndRecord,
    options: [
        .bluetoothHighQualityRecording,  // New in iOS 26
        .allowBluetoothA2DP              // Fallback
    ]
)

// AVCaptureSession approach
let captureSession = AVCaptureSession()
captureSession.configuresApplicationAudioSessionForBluetoothHighQualityRecording = true

Notes:

• Uses dedicated Bluetooth link optimized for AirPods
• Falls back to HFP if device doesn't support HQ mode
• Supports AirPods stem controls for start/stop recording

---

Spatial Audio Capture (iOS 26+)

First Order Ambisonics (FOA)

Record 3D spatial audio using device microphone array:

// With AVCaptureMovieFileOutput (simple)
let audioInput = AVCaptureDeviceInput(device: audioDevice)
audioInput.multichannelAudioMode = .firstOrderAmbisonics

// With AVAssetWriter (full control)
// Requires two AudioDataOutputs: FOA (4ch) + Stereo (2ch)

AVAssetWriter Spatial Audio Setup

// Configure two AudioDataOutputs
let foaOutput = AVCaptureAudioDataOutput()
foaOutput.spatialAudioChannelLayoutTag = kAudioChannelLayoutTag_HOA_ACN_SN3D | 4  // 4 channels (FOA)

let stereoOutput = AVCaptureAudioDataOutput()
stereoOutput.spatialAudioChannelLayoutTag = kAudioChannelLayoutTag_Stereo    // 2 channels

// Create metadata generator
let metadataGenerator = AVCaptureSpatialAudioMetadataSampleGenerator()

// Feed FOA buffers to generator
func captureOutput(_ output: AVCaptureOutput,
                   didOutput sampleBuffer: CMSampleBuffer,
                   from connection: AVCaptureConnection) {
    metadataGenerator.append(sampleBuffer)
    // Also write to FOA AssetWriterInput
}

// When recording stops, get metadata sample
let metadataSample = metadataGenerator.createMetadataSample()
// Write to metadata track

Output File Structure

Spatial audio files contain: 1. Stereo AAC track — Compatibility fallback 2. APAC track — Spatial audio (FOA) 3. Metadata track — Audio Mix tuning parameters

File formats: .mov, .mp4, .qta (QuickTime Audio, iOS 26+)

---

ASAF / APAC (Apple Spatial Audio)

Overview

APAC Capabilities

• Bitrates: 64 kbps to 768 kbps
• Supports: Channels, Objects, Higher Order Ambisonics, Dialogue, Binaural
• Head-tracked rendering adaptive to listener position/orientation
• Required for Apple Immersive Video

Playback

// Standard AVPlayer handles APAC automatically
let player = AVPlayer(url: spatialAudioURL)
player.play()

// Head tracking enabled automatically on AirPods

Platform Support

All Apple platforms except watchOS support APAC playback.

---

Audio Mix (Cinematic Framework)

Separate and remix speech vs ambient sounds in spatial recordings:

AVPlayer Integration

import Cinematic

// Load spatial audio asset
let asset = AVURLAsset(url: spatialAudioURL)
let audioInfo = try await CNAssetSpatialAudioInfo(asset: asset)

// Configure mix parameters
let intensity: Float = 0.5  // 0.0 to 1.0
let style = CNSpatialAudioRenderingStyle.cinematic

// Create and apply audio mix
let audioMix = audioInfo.audioMix(
    effectIntensity: intensity,
    renderingStyle: style
)
playerItem.audioMix = audioMix

Rendering Styles

AUAudioMix (Direct AudioUnit)

For apps not using AVPlayer:

// Input: 4 channels FOA
// Output: Separated speech + ambient

// Get tuning metadata from file
let audioInfo = try await CNAssetSpatialAudioInfo(asset: asset)
let remixMetadata = audioInfo.spatialAudioMixMetadata as CFData

// Apply to AudioUnit via AudioUnitSetProperty

---

Common Patterns

Background Audio Playback

// 1. Set category
try AVAudioSession.sharedInstance().setCategory(.playback)

// 2. Enable background mode in Info.plist
// <key>UIBackgroundModes</key>
// <array><string>audio</string></array>

// 3. Set Now Playing info (recommended)
let nowPlayingInfo: [String: Any] = [
    MPMediaItemPropertyTitle: "Song Title",
    MPMediaItemPropertyArtist: "Artist",
    MPNowPlayingInfoPropertyElapsedPlaybackTime: player.currentTime,
    MPMediaItemPropertyPlaybackDuration: duration
]
MPNowPlayingInfoCenter.default().nowPlayingInfo = nowPlayingInfo

Ducking Other Audio

try AVAudioSession.sharedInstance().setCategory(
    .playback,
    options: .duckOthers
)

// When done, restore others
try AVAudioSession.sharedInstance().setActive(false, options: .notifyOthersOnDeactivation)

Bluetooth Device Handling

// Allow all Bluetooth
try AVAudioSession.sharedInstance().setCategory(
    .playAndRecord,
    options: [.allowBluetoothHFP, .allowBluetoothA2DP]
)

// Check current Bluetooth route
let route = AVAudioSession.sharedInstance().currentRoute
let hasBluetoothOutput = route.outputs.contains {
    $0.portType == .bluetoothA2DP || $0.portType == .bluetoothHFP
}

---

Anti-Patterns

Wrong Category

// WRONG — music player using ambient (silenced by switch)
try AVAudioSession.sharedInstance().setCategory(.ambient)

// CORRECT — music needs .playback
try AVAudioSession.sharedInstance().setCategory(.playback)

Missing Interruption Handling

// WRONG — no interruption observer
// Audio stops on phone call and never resumes

// CORRECT — always handle interruptions
NotificationCenter.default.addObserver(
    forName: AVAudioSession.interruptionNotification,
    // ... handle began/ended
)

Tap Memory Leaks

// WRONG — tap installed, never removed
engine.inputNode.installTap(onBus: 0, bufferSize: 1024, format: format) { ... }

// CORRECT — remove tap when done
deinit {
    engine.inputNode.removeTap(onBus: 0)
}

Format Mismatch Crashes

// WRONG — connecting nodes with incompatible formats
engine.connect(playerNode, to: mixerNode, format: wrongFormat)  // Crash!

// CORRECT — use nil for automatic format negotiation, or match exactly
engine.connect(playerNode, to: mixerNode, format: nil)

Forgetting to Activate Session

// WRONG — configure but don't activate
try AVAudioSession.sharedInstance().setCategory(.playback)
// Audio doesn't work!

// CORRECT — always activate
try AVAudioSession.sharedInstance().setCategory(.playback)
try AVAudioSession.sharedInstance().setActive(true)

---

Resources

WWDC: 2025-251, 2025-403, 2019-510

Docs: /avfoundation, /avkit, /cinematic

---

Targets: iOS 12+ (core), iOS 26+ (spatial features) Frameworks: AVFoundation, AVKit, Cinematic (iOS 26+) History: See git log for changes

AVFoundation Video & Media-Engine Reference

Companion to avfoundation-ref.md (which is audio-only). Covers AVFoundation's async video write / export / render engine, the genuinely new iOS 27 capabilities, and the Swift-only deprecations that retire the old callback/KVO surface. Signatures verified against the iPhoneOS 26 and 27 SDKs (Xcode 26.6 / 27.0).

Availability is per-section — read the tags. The async write/export engine (start(), inputReceiver(for:), export(to:as:), states(updateInterval:)) is an iOS 13–26 baseline, NOT new in 27. Only resumable export, the writing planner, the renderer Receiver pipeline, Apple Log 2, and audio-detach are OS27. New-in-27 types are watchOS-unavailable unless noted.

Quick Reference

The async write/export engine (iOS 26 baseline — not new in 27)

These shipped before iOS 27. They matter here because iOS 27 deprecates their predecessors in Swift (next section) — but the replacements work on your iOS 26 (or older) deployment floor, so migrating off the deprecated calls needs no `@available` gate.

// Writing — start() throws (iOS 26), receiver-based append (iOS 26):
try writer.start()                              // was: writer.startWriting() -> Bool
let receiver = writer.inputReceiver(for: input) // AVAssetWriterInput.SampleBufferReceiver
try await receiver.append(readySampleBuffer)    // CMReadySampleBuffer<…>; nonisolated(nonsending)
// receiver.appendImmediately(_:) throws -> Bool   // non-async fast path

// Exporting — export(to:as:) async (iOS 13), states stream (iOS 18):
let monitor = Task {
    for await state in session.states(updateInterval: 0.5) { _ = state }  // AVAssetExportSession.State
}
try await session.export(to: url, as: .mp4)
monitor.cancel()

startSessionAtSourceTime: / endSessionAtSourceTime: are not deprecated and still bound a writing session. inputReceiverRequestingMultiPass(for:) returns (SampleBufferReceiver, MultiPassController) for multi-pass encodes.

Swift-only deprecations → migration

Every row is a Swift-only deprecation (#if __swift__): ObjC callers are unaffected; Swift callers get a warning. The replacements all predate iOS 27, so this migration is safe on an iOS 26 floor.

NEW in iOS 27

Resumable export (OS27)

@available(iOS 27, *)
func resume(_ session: AVAssetExportSession, to url: URL) async throws {
    let state = await session.configureForResumableExport()   // async, NON-throwing
    _ = state                                                 // AVAssetExportSession.ResumptionState
    try await session.export(to: url, as: .mp4)               // resumes rather than restarting
}

configureForResumableExport() async -> AVAssetExportSession.ResumptionState lets an interrupted export pick up where it left off instead of re-encoding from zero.

Segment-based writing: AVAssetWritingPlanner (OS27)

The planner is constructed with a temp directory; the segment shape lives on AVAssetVideoTrackPlan; plan(_:segmentHandler:) drives generation and executePlan() assembles the result.

@available(iOS 27, *)
func writeSegments(_ trackID: CMPersistentTrackID,
                   _ segmentConfigs: [AVPlannedVideoSegmentConfiguration],
                   tmp: URL) async throws -> AVComposition {
    let planner = try AVAssetWritingPlanner(directoryForTemporaryFiles: tmp)
    let trackPlan = AVAssetVideoTrackPlan(
        videoCodecType: .hevc,
        encoderSpecification: nil,
        mediaType: .video,
        segmentConfigurations: segmentConfigs,
        assemblyTrackID: trackID)
    planner.plan(trackPlan) { request in        // @Sendable (AVPlannedSegmentWritingRequest) async throws -> SegmentResult
        // …write the requested segment…
        return .success
    }
    return try await planner.executePlan()      // -> AVComposition
}

AVAssetWritingPlanner.segmentBoundaryGuidelinesForVideo(codecType:encoderSpecification:) returns guidance for choosing cut points.

Concurrency-native rendering: sample-buffer Receiver (OS27)

An AVSampleBufferRenderSynchronizer vends a Receiver for a renderer; the receiver's enqueue is async and returns an EnqueueResult, so backpressure is explicit rather than a silent drop. AVSampleBufferAudioRenderer has a parallel API.

@available(iOS 27, *)
func feed(_ synchronizer: AVSampleBufferRenderSynchronizer,
          _ renderer: AVSampleBufferVideoRenderer,
          _ buffer: CMReadySampleBuffer<CMSampleBuffer.DynamicContent>) async throws {
    let receiver = synchronizer.sampleBufferReceiver(adding: renderer)   // -> Receiver (sending)
    let result = try await receiver.enqueue(buffer)                      // -> EnqueueResult; nonisolated(nonsending)
    _ = result
    for await event in receiver.renderingEventsAfterFinishedEnqueuing { _ = event }  // RenderingEvent
    _ = await synchronizer.removeReceiver(receiver: receiver, at: .zero) // async -> Bool
}

receiver.enqueueImmediately(_:) -> EnqueueResult is the synchronous variant. sampleBufferReceiver(adding:) / removeReceiver live on AVSampleBufferRenderSynchronizer; the audio overload uses the unlabeled removeReceiver(_:at:) form.

Apple Log 2 capture color (OS27)

let settings: [String: Any] = [
    AVVideoCodecKey: AVVideoCodecType.hevc,
    AVVideoLogTransferFunctionKey: AVVideoLogTransferFunction_AppleLog2,  // or _AppleLog
]

Values: AVVideoLogTransferFunction_AppleLog, AVVideoLogTransferFunction_AppleLog2.

Detach playback from system audio (OS27, not macOS)

@available(iOS 27, tvOS 27, watchOS 27, visionOS 27, *), @available(macOS, unavailable), @available(macCatalyst, unavailable) — note the inverse platform set (watchOS yes, macOS/Catalyst no):

player.setDisconnectedFromSystemAudio(true) { /* completion */ }
let detached = player.disconnectedFromSystemAudio   // Bool, nonisolated

Concurrency posture

AVFoundation's media engine is async- and Sendable-first without broad @MainActor isolation. The async append/enqueue entry points are nonisolated(nonsending) — they run on the caller's executor, integrating into actor-isolated code without forcing a hop; ownership crosses boundaries via sending parameters. In Swift 6 code prefer the async forms over the deprecated callback/KVO surface.

Resources

WWDC: 2026-256

Docs: /avfoundation/avassetexportsession, /avfoundation/avassetwriter, /avfoundation/avassetwritingplanner, /avfoundation/avsamplebuffervideorenderer

Skills: avfoundation-ref (audio), camera-capture-ref, axiom-concurrency (Swift 6 async/Sendable)

Camera Capture Diagnostics

Systematic troubleshooting for AVFoundation camera issues: frozen preview, wrong rotation, slow capture, session interruptions, permission problems, slow launch, and dropped frames during recording.

Overview

Core Principle: When camera doesn't work, the problem is usually: 1. Threading (session work on main thread) - 35% 2. Session lifecycle (not started, interrupted, not configured) - 25% 3. Rotation (deprecated APIs, missing coordinator) - 20% 4. Permissions (denied, not requested) - 15% 5. Configuration (wrong preset, missing input/output) - 5%

Always check threading and session state BEFORE debugging capture logic. (For launch speed and recording sustainability, see Patterns 16-17.)

Red Flags

Symptoms that indicate camera-specific issues:

Mandatory First Steps

Before investigating code, run these diagnostics:

Step 1: Check Session State

print("📷 Session state:")
print("  isRunning: \(session.isRunning)")
print("  inputs: \(session.inputs.count)")
print("  outputs: \(session.outputs.count)")

for input in session.inputs {
    if let deviceInput = input as? AVCaptureDeviceInput {
        print("  Input: \(deviceInput.device.localizedName)")
    }
}

for output in session.outputs {
    print("  Output: \(type(of: output))")
}

Expected output:

• ✅ isRunning: true, inputs ≥ 1, outputs ≥ 1 → Session working
• ⚠️ isRunning: false → Session not started or interrupted
• ❌ inputs: 0 → Camera not added (permission? configuration?)

Step 2: Check Threading

print("🧵 Thread check:")

// When setting up session
sessionQueue.async {
    print("  Setup thread: \(Thread.isMainThread ? "❌ MAIN" : "✅ Background")")
}

// When starting session
sessionQueue.async {
    print("  Start thread: \(Thread.isMainThread ? "❌ MAIN" : "✅ Background")")
}

Expected output:

• ✅ All background → Correct
• ❌ Any main thread → UI will freeze

Step 3: Check Permissions

let status = AVCaptureDevice.authorizationStatus(for: .video)
print("🔐 Camera permission: \(status.rawValue)")

switch status {
case .authorized: print("  ✅ Authorized")
case .notDetermined: print("  ⚠️ Not yet requested")
case .denied: print("  ❌ Denied by user")
case .restricted: print("  ❌ Restricted (parental controls?)")
@unknown default: print("  ❓ Unknown")
}

Step 4: Check for Interruptions

// Add temporary observer to see interruptions
NotificationCenter.default.addObserver(
    forName: .AVCaptureSessionWasInterrupted,
    object: session,
    queue: .main
) { notification in
    if let reason = notification.userInfo?[AVCaptureSessionInterruptionReasonKey] as? Int {
        print("🚨 Interrupted: reason \(reason)")
    }
}

Decision Tree

Camera not working as expected?
│
├─ Black/frozen preview?
│  ├─ Check Step 1 (session state)
│  │  ├─ isRunning = false → See Pattern 1 (session not started)
│  │  ├─ inputs = 0 → See Pattern 2 (no camera input)
│  │  └─ isRunning = true, inputs > 0 → See Pattern 3 (preview layer)
│
├─ UI freezes when opening camera?
│  └─ Check Step 2 (threading)
│     └─ Main thread → See Pattern 4 (move to session queue)
│
├─ Camera freezes during use?
│  ├─ After phone call → See Pattern 5 (interruption handling)
│  ├─ In Split View (iPad) → See Pattern 6 (multitasking)
│  └─ Random freezes → See Pattern 7 (thermal pressure)
│
├─ Preview/photo rotated wrong?
│  ├─ Preview rotated → See Pattern 8 (RotationCoordinator preview)
│  ├─ Captured photo rotated → See Pattern 9 (capture rotation)
│  └─ Front camera "wrong" → See Pattern 10 (mirroring expected)
│
├─ Capture too slow?
│  ├─ 2+ seconds delay → See Pattern 11 (quality prioritization)
│  └─ Slight delay → See Pattern 12 (deferred processing)
│
├─ Launch too slow (preview late)?
│  └─ See Pattern 16 (deferred start, iOS 26+)
│
├─ Recording drops frames / session unsustainable?
│  └─ See Pattern 17 (hardware cost, system pressure, Pro Video Storage)
│
├─ Permission issues?
│  ├─ Status: notDetermined → See Pattern 13 (request permission)
│  └─ Status: denied → See Pattern 14 (settings prompt)
│
└─ Crash on some devices?
   └─ See Pattern 15 (API availability)

Diagnostic Patterns

Pattern 1: Session Not Started

Symptom: Black preview, isRunning = false

Common causes: 1. startRunning() never called 2. startRunning() called but session has no inputs 3. Session stopped and never restarted

Diagnostic:

// Check if startRunning was called
print("isRunning before start: \(session.isRunning)")
session.startRunning()
print("isRunning after start: \(session.isRunning)")

Fix:

// Ensure session is started on session queue
func startSession() {
    sessionQueue.async { [self] in
        guard !session.isRunning else { return }

        // Verify we have inputs before starting
        guard !session.inputs.isEmpty else {
            print("❌ Cannot start - no inputs configured")
            return
        }

        session.startRunning()
    }
}

Time to fix: 10 min

Pattern 2: No Camera Input

Symptom: session.inputs.count = 0

Common causes: 1. Camera permission denied 2. AVCaptureDeviceInput creation failed 3. canAddInput() returned false 4. Configuration not committed

Diagnostic:

// Step through input setup
guard let camera = AVCaptureDevice.default(for: .video) else {
    print("❌ No camera device found")
    return
}
print("✅ Camera: \(camera.localizedName)")

do {
    let input = try AVCaptureDeviceInput(device: camera)
    print("✅ Input created")

    if session.canAddInput(input) {
        print("✅ Can add input")
    } else {
        print("❌ Cannot add input - check session preset compatibility")
    }
} catch {
    print("❌ Input creation failed: \(error)")
}

Fix: Ensure permission is granted BEFORE creating input, and wrap in configuration block:

session.beginConfiguration()
// Add input here
session.commitConfiguration()

Time to fix: 15 min

Pattern 3: Preview Layer Not Connected

Symptom: isRunning = true, inputs configured, but preview is black

Common causes: 1. Preview layer session not set 2. Preview layer not in view hierarchy 3. Preview layer frame is zero

Diagnostic:

print("Preview layer session: \(previewLayer.session != nil)")
print("Preview layer superlayer: \(previewLayer.superlayer != nil)")
print("Preview layer frame: \(previewLayer.frame)")
print("Preview layer connection: \(previewLayer.connection != nil)")

Fix:

// Ensure preview layer is properly configured
previewLayer.session = session
previewLayer.videoGravity = .resizeAspectFill

// Ensure frame is set (common in SwiftUI)
previewLayer.frame = view.bounds

Time to fix: 10 min

Pattern 4: Main Thread Blocking

Symptom: UI freezes for 1-3 seconds when camera opens

Root cause: startRunning() is a blocking call executed on main thread

Diagnostic:

// If this prints on main thread, that's the problem
print("startRunning on thread: \(Thread.current)")
session.startRunning()

Fix:

// Create dedicated serial queue
private let sessionQueue = DispatchQueue(label: "camera.session")

func startSession() {
    sessionQueue.async { [self] in
        session.startRunning()
    }
}

Time to fix: 15 min

Pattern 5: Phone Call Interruption

Symptom: Camera works, then freezes when phone call comes in

Root cause: Session interrupted but no handling/UI feedback

Diagnostic:

// Check if session is still running after returning from call
print("Session running: \(session.isRunning)")
// Will be false during active call, true after call ends

Fix: Add interruption observers (see camera-capture skill Pattern 5)

Key point: Session AUTOMATICALLY resumes after interruption ends. You don't need to call startRunning() again. Just update your UI.

Time to fix: 30 min

Pattern 6: Split View Camera Unavailable

Symptom: Camera stops working when iPad enters Split View

Root cause: Camera not available with multiple foreground apps

Diagnostic:

// Check interruption reason
// InterruptionReason.videoDeviceNotAvailableWithMultipleForegroundApps

Fix: Show appropriate UI message and resume when user exits Split View:

case .videoDeviceNotAvailableWithMultipleForegroundApps:
    showMessage("Camera unavailable in Split View. Use full screen.")

Time to fix: 15 min

Pattern 7: Thermal Pressure

Symptom: Camera stops randomly, especially after prolonged use

Root cause: Device getting hot, system reducing resources

Diagnostic:

// Check thermal state
print("Thermal state: \(ProcessInfo.processInfo.thermalState.rawValue)")
// 0 = nominal, 1 = fair, 2 = serious, 3 = critical

Fix: Reduce quality or show cooling message:

case .videoDeviceNotAvailableDueToSystemPressure:
    // Reduce quality
    session.sessionPreset = .medium
    showMessage("Camera quality reduced due to device temperature")

Time to fix: 20 min

Pattern 8: Preview Rotation Wrong

Symptom: Preview is rotated 90° from expected

Root cause: Not using RotationCoordinator (iOS 17+) or not observing updates

Diagnostic:

print("Preview connection rotation: \(previewLayer.connection?.videoRotationAngle ?? -1)")

Fix:

// Create and observe RotationCoordinator
let coordinator = AVCaptureDevice.RotationCoordinator(device: camera, previewLayer: previewLayer)

// Set initial rotation
previewLayer.connection?.videoRotationAngle = coordinator.videoRotationAngleForHorizonLevelPreview

// Observe changes
observation = coordinator.observe(\.videoRotationAngleForHorizonLevelPreview) { [weak previewLayer] coord, _ in
    DispatchQueue.main.async {
        previewLayer?.connection?.videoRotationAngle = coord.videoRotationAngleForHorizonLevelPreview
    }
}

Time to fix: 30 min

Pattern 9: Captured Photo Rotation Wrong

Symptom: Preview looks correct, but captured photo is rotated

Root cause: Rotation angle not applied to photo output connection

Diagnostic:

if let connection = photoOutput.connection(with: .video) {
    print("Photo connection rotation: \(connection.videoRotationAngle)")
}

Fix:

func capturePhoto() {
    // Apply current rotation to capture
    if let connection = photoOutput.connection(with: .video) {
        connection.videoRotationAngle = rotationCoordinator.videoRotationAngleForHorizonLevelCapture
    }

    photoOutput.capturePhoto(with: settings, delegate: self)
}

Time to fix: 15 min

Pattern 10: Front Camera Mirroring

Symptom: Designer says "front camera photo doesn't match preview"

Reality: This is CORRECT behavior, not a bug.

Explanation:

• Preview is mirrored (like looking in a mirror - user expectation)
• Captured photo is NOT mirrored (text reads correctly when shared)
• This matches the system Camera app behavior

If business requires mirrored photos (selfie apps):

func mirrorImage(_ image: UIImage) -> UIImage? {
    guard let cgImage = image.cgImage else { return nil }
    return UIImage(cgImage: cgImage, scale: image.scale, orientation: .upMirrored)
}

Time to fix: 5 min (explanation) or 15 min (if mirroring required)

Pattern 11: Slow Capture (Quality Priority)

Symptom: Photo capture takes 2+ seconds

Root cause: photoQualityPrioritization = .quality (default for some devices)

Diagnostic:

print("Max quality prioritization: \(photoOutput.maxPhotoQualityPrioritization.rawValue)")
// Check what you're requesting in AVCapturePhotoSettings

Fix:

var settings = AVCapturePhotoSettings()

// For fast capture (social/sharing)
settings.photoQualityPrioritization = .speed

// For balanced (general use)
settings.photoQualityPrioritization = .balanced

// Only use .quality when image quality is critical

Time to fix: 5 min

Pattern 12: Deferred Processing

Symptom: Want maximum responsiveness (zero-shutter-lag)

Solution: Enable deferred processing (iOS 17+)

photoOutput.isAutoDeferredPhotoDeliveryEnabled = true

// Then handle proxy in delegate:
// - didFinishProcessingPhoto gives proxy for immediate display
// - didFinishCapturingDeferredPhotoProxy gives final image later

Time to fix: 30 min

Pattern 13: Permission Not Requested

Symptom: authorizationStatus = .notDetermined

Fix:

// Must request before setting up session
Task {
    let granted = await AVCaptureDevice.requestAccess(for: .video)
    if granted {
        setupSession()
    }
}

Time to fix: 10 min

Pattern 14: Permission Denied

Symptom: authorizationStatus = .denied

Fix: Show settings prompt

func showSettingsPrompt() {
    let alert = UIAlertController(
        title: "Camera Access Required",
        message: "Please enable camera access in Settings to use this feature.",
        preferredStyle: .alert
    )
    alert.addAction(UIAlertAction(title: "Settings", style: .default) { _ in
        if let url = URL(string: UIApplication.openSettingsURLString) {
            UIApplication.shared.open(url)
        }
    })
    alert.addAction(UIAlertAction(title: "Cancel", style: .cancel))
    present(alert, animated: true)
}

Time to fix: 15 min

Pattern 15: API Availability Crash

Symptom: Crash on iOS 16 or earlier

Root cause: Using iOS 17+ APIs without availability check

Fix:

if #available(iOS 17.0, *) {
    // Use RotationCoordinator
    let coordinator = AVCaptureDevice.RotationCoordinator(device: camera, previewLayer: preview)
} else {
    // Fallback to deprecated videoOrientation
    if let connection = previewLayer.connection {
        connection.videoOrientation = .portrait
    }
}

Time to fix: 20 min

Pattern 16: Slow Camera Launch (Preview Appears Late)

Symptom: Noticeable blank preview after app launch; users miss the moment

Root causes (in order of impact): 1. All capture outputs initialize before the first preview frame (the most expensive launch stage) 2. Session created synchronously on the main thread during UI setup 3. Multiple commitConfiguration() calls during launch 4. Non-critical UI (mode pickers, image wells) built before preview renders

Diagnostic:

// Time the stages: app launch → session configured/started → outputs initialized → first frame
let t0 = CACurrentMediaTime()
// ...after first preview frame renders:
print("Launch to preview: \(CACurrentMediaTime() - t0)s")
// ~1s without deferred start is typical; deferred start roughly halves it (WWDC 2026-303)

Fix (iOS 26+): adopt deferred start — set isDeferredStartEnabled = true on every output not needed for preview, leave it false on the preview output, commit once, and pair with responsive capture so taps buffer while the photo output finishes initializing. Full pattern: camera-capture skill Pattern 8.

Time to fix: 1 hour

Pattern 17: Recording Drops Frames / Unsustainable Session

Symptom: High-data-rate recording (ProRes) stutters or drops frames; session stops after prolonged use

Diagnostic (check in this order):

// 1. Configuration over hardware budget? (> 1.0 won't even start — runtime error)
print("hardwareCost: \(session.hardwareCost)")          // iOS 16+
// Multi-cam: also systemPressureCost (> 1.0 = will run, but not sustainably)

// 2. System pressure rising during use?
print("pressure: \(device.systemPressureState.level), factors: \(device.systemPressureState.factors)")
// .systemStress factor (27 SDK) = ~30s from unexpected power-off — back off NOW

Fixes: 1. hardwareCost > 1.0 → lower format resolution, use binned formats, or set AVCaptureDeviceInput.videoMinFrameDurationOverride (cost assumes the format's max frame rate) 2. Pressure rising → reduce frame rate, throttle GPU/Neural Engine work, minimize UI work 3. ProRes file-write stutter on iOS 27 → adopt Pro Video Storage (pre-allocated, deterministic I/O) — camera-capture skill Pattern 9

Time to fix: 30-60 min

Quick Reference Table

Checklist

Before escalating camera issues:

Basics:

• ☑ Session has at least one input
• ☑ Session has at least one output
• ☑ Session isRunning = true
• ☑ Preview layer connected to session
• ☑ Preview layer has non-zero frame

Threading:

• ☑ All session work on sessionQueue
• ☑ startRunning() on background thread
• ☑ UI updates on main thread

Permissions:

• ☑ Authorization status checked
• ☑ Permission requested if notDetermined
• ☑ Graceful UI for denied state

Rotation:

• ☑ RotationCoordinator created with device AND previewLayer
• ☑ Observation set up for preview angle changes
• ☑ Capture angle applied when taking photos

Interruptions:

• ☑ Interruption observer registered
• ☑ UI feedback for interrupted state
• ☑ Tested with incoming phone call

Resources

WWDC: 2021-10247, 2023-10105, 2026-303

Docs: /avfoundation/avcapturesession, /avfoundation/avcapturesessionwasinterruptednotification

Skills: skills/camera-capture.md, skills/camera-capture-ref.md

Camera Capture API Reference

Quick Reference

// SESSION SETUP
import AVFoundation

let session = AVCaptureSession()
let sessionQueue = DispatchQueue(label: "camera.session")

sessionQueue.async {
    session.beginConfiguration()
    session.sessionPreset = .photo

    guard let camera = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .back),
          let input = try? AVCaptureDeviceInput(device: camera),
          session.canAddInput(input) else { return }
    session.addInput(input)

    let photoOutput = AVCapturePhotoOutput()
    if session.canAddOutput(photoOutput) {
        session.addOutput(photoOutput)
    }

    session.commitConfiguration()
    session.startRunning()
}

// CAPTURE PHOTO
var settings = AVCapturePhotoSettings()
settings.photoQualityPrioritization = .balanced
photoOutput.capturePhoto(with: settings, delegate: self)

// ROTATION (iOS 17+)
let coordinator = AVCaptureDevice.RotationCoordinator(device: camera, previewLayer: previewLayer)
previewLayer.connection?.videoRotationAngle = coordinator.videoRotationAngleForHorizonLevelPreview

---

AVCaptureSession

Central coordinator for capture data flow.

Session Presets

Session Configuration

// Batch configuration (atomic)
session.beginConfiguration()
defer { session.commitConfiguration() }

// Check preset support
if session.canSetSessionPreset(.hd4K3840x2160) {
    session.sessionPreset = .hd4K3840x2160
}

// Add input/output
if session.canAddInput(input) {
    session.addInput(input)
}

if session.canAddOutput(output) {
    session.addOutput(output)
}

Session Lifecycle

// Start (ALWAYS on background queue)
sessionQueue.async {
    session.startRunning()  // Blocking call
}

// Stop
sessionQueue.async {
    session.stopRunning()
}

// Check state
session.isRunning      // true/false
session.isInterrupted  // true during phone calls, etc.

Deferred Start (iOS 26+)

Postpones output initialization until after the first preview frame, cutting launch time roughly in half (WWDC 2026-303). Not on visionOS/watchOS.

session.beginConfiguration()
session.automaticallyRunsDeferredStart = true  // default true when linked against iOS 26 SDK+

let previewLayer = AVCaptureVideoPreviewLayer(session: session)
previewLayer.isDeferredStartEnabled = false    // preview must NOT be deferred

let photoOutput = AVCapturePhotoOutput()
session.addOutput(photoOutput)
photoOutput.isDeferredStartEnabled = true      // defer everything not needed for first frame

session.setDeferredStartDelegate(delegate, deferredStartDelegateCallbackQueue: sessionQueue)
session.commitConfiguration()
session.startRunning()

Manual-mode trigger from a CAMetalLayer — run deferred start after the first drawable is presented:

guard let drawable = layer.nextDrawable() else { return }
if !firstFramePresented {
    drawable.addPresentedHandler { _ in
        captureSession.runDeferredStartWhenNeeded()
    }
    firstFramePresented = true
}

Session Cost and System Pressure

// After commitConfiguration(), before startRunning()
session.hardwareCost          // target <= 1.0; > 1.0 = configuration can't run (iOS 16+)
// Contributors: camera count, active formats (1080p vs 4K), format max frame rate
// (cost assumes the format's max — set AVCaptureDeviceInput.videoMinFrameDurationOverride
// to the reciprocal of the frame rate you actually use), binned formats

// Multi-cam sessions also expose sustainability
multiCamSession.systemPressureCost  // > 1.0 = unsustainable (AVCaptureMultiCamSession, iOS 13+)

// Adapt at runtime: KVO the device's pressure state
let obs = device.observe(\.systemPressureState, options: [.initial, .new]) { device, _ in
    // Reduce frame rate, throttle GPU/ANE work, minimize UI work as pressure rises
}

Session Notifications

// Session started
NotificationCenter.default.addObserver(
    forName: .AVCaptureSessionDidStartRunning,
    object: session, queue: .main) { _ in }

// Session stopped
NotificationCenter.default.addObserver(
    forName: .AVCaptureSessionDidStopRunning,
    object: session, queue: .main) { _ in }

// Session interrupted (phone call, etc.)
NotificationCenter.default.addObserver(
    forName: .AVCaptureSessionWasInterrupted,
    object: session, queue: .main) { notification in
        let reason = notification.userInfo?[AVCaptureSessionInterruptionReasonKey] as? Int
    }

// Interruption ended
NotificationCenter.default.addObserver(
    forName: .AVCaptureSessionInterruptionEnded,
    object: session, queue: .main) { _ in }

// Runtime error
NotificationCenter.default.addObserver(
    forName: .AVCaptureSessionRuntimeError,
    object: session, queue: .main) { notification in
        let error = notification.userInfo?[AVCaptureSessionErrorKey] as? Error
    }

Interruption Reasons

---

AVCaptureDevice

Represents a physical capture device (camera, microphone).

Getting Devices

// Default back camera
AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .back)

// Default front camera
AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .front)

// Default microphone
AVCaptureDevice.default(for: .audio)

// Discovery session for all cameras
let discoverySession = AVCaptureDevice.DiscoverySession(
    deviceTypes: [.builtInWideAngleCamera, .builtInUltraWideCamera, .builtInTelephotoCamera],
    mediaType: .video,
    position: .unspecified
)
let cameras = discoverySession.devices

Device Types

Device Configuration

do {
    try device.lockForConfiguration()
    defer { device.unlockForConfiguration() }

    // Focus
    if device.isFocusModeSupported(.continuousAutoFocus) {
        device.focusMode = .continuousAutoFocus
    }

    // Exposure
    if device.isExposureModeSupported(.continuousAutoExposure) {
        device.exposureMode = .continuousAutoExposure
    }

    // Torch (flashlight)
    if device.hasTorch && device.isTorchModeSupported(.on) {
        device.torchMode = .on
    }

    // Zoom
    device.videoZoomFactor = 2.0  // 2x zoom

} catch {
    print("Failed to configure device: \(error)")
}

Switching Cameras

// Switch between front and back during active session
func switchCamera() {
    sessionQueue.async { [self] in
        session.beginConfiguration()
        defer { session.commitConfiguration() }

        // Remove current camera input
        if let currentInput = session.inputs.first(where: { ($0 as? AVCaptureDeviceInput)?.device.hasMediaType(.video) == true }) as? AVCaptureDeviceInput {
            session.removeInput(currentInput)

            // Get opposite camera
            let newPosition: AVCaptureDevice.Position = currentInput.device.position == .back ? .front : .back
            guard let newDevice = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: newPosition),
                  let newInput = try? AVCaptureDeviceInput(device: newDevice) else { return }

            if session.canAddInput(newInput) {
                session.addInput(newInput)
            }
        }
    }
}

Important: Always switch on the session queue, within beginConfiguration/commitConfiguration.

Authorization

// Check status
let status = AVCaptureDevice.authorizationStatus(for: .video)

switch status {
case .authorized: break
case .notDetermined:
    await AVCaptureDevice.requestAccess(for: .video)
case .denied, .restricted:
    // Show settings prompt
@unknown default: break
}

Center Stage Front Camera (iOS 26+, iPhone 17 / iPhone Air / iPhone 17 Pro)

The Center Stage front camera has a square sensor (any aspect ratio without rotating the phone) with a 95° field of view, exposed as the front `.builtInUltraWideCamera` (WWDC 2026-341). Three new iOS 26, iOS-only API families support it, alongside the existing Center Stage controls:

Dynamic Aspect Ratio

Crops your chosen aspect ratio out of the square sensor without rebuilding the session or interrupting preview.

let discovery = AVCaptureDevice.DiscoverySession(
    deviceTypes: [.builtInUltraWideCamera], mediaType: .video, position: .front)
guard let camera = discovery.devices.first else { return }

// Find a format supporting the target ratio
for format in camera.formats where format.supportedDynamicAspectRatios.contains(.ratio4x3) {
    try camera.lockForConfiguration()
    camera.activeFormat = format
    camera.unlockForConfiguration()
    break
}

try camera.lockForConfiguration()
defer { camera.unlockForConfiguration() }
let syncTime = try await camera.setDynamicAspectRatio(.ratio4x3)  // returns first-buffer timestamp

Video recording: QuickTime tracks require constant dimensions — AVCaptureMovieFileOutput recordings stop automatically when the ratio changes. With AVCaptureVideoDataOutput + AVAssetWriter, use the completion timestamp to end one recording and start the next.

Smart Framing Monitor (Auto Zoom / Auto Rotate)

Face/gaze-driven framing recommendations. Photo-capture oriented: recommendations only on the 4032 photo format.

for format in camera.formats where format.isSmartFramingSupported {
    try camera.lockForConfiguration()
    camera.activeFormat = format
    camera.unlockForConfiguration()
    break
}

guard let monitor = camera.smartFramingMonitor else { return }  // nil if unsupported
try camera.lockForConfiguration()
monitor.enabledFramings = monitor.supportedFramings  // default: empty — nothing recommended
camera.unlockForConfiguration()

observation = monitor.observe(\.recommendedFraming, options: [.new]) { monitor, _ in
    guard let framing = monitor.recommendedFraming else { return }
    Task {
        try camera.lockForConfiguration()
        defer { camera.unlockForConfiguration() }
        // Apple recommends ratio first, then zoom, for a smooth preview transition
        try await camera.setDynamicAspectRatio(framing.aspectRatio)
        camera.videoZoomFactor = CGFloat(framing.zoomFactor)
    }
}

try monitor.startMonitoring()   // before or after session.startRunning()
// later: observation?.invalidate(); monitor.stopMonitoring()

AVCaptureFraming = aspectRatio + zoomFactor (Float). Set enabledFramings before running the session; you can change it any time while monitoring.

Sensor Orientation Compensation

Historically front sensors are mounted landscape-left; the Center Stage front camera is mounted portrait. AVCapturePhotoOutput compensates by default — photos are physically rotated and EXIF-updated to landscape-left, so existing rotation code keeps working. Applies to HEIC/JPEG/uncompressed processed photos only — never Bayer RAW or ProRAW.

photoOutput.isCameraSensorOrientationCompensationSupported  // iOS 26+, iOS-only
// Disabling skips the rotation pass (best performance) — verify orientation stays correct
photoOutput.isCameraSensorOrientationCompensationEnabled = false

Center Stage Toggle and Stabilization

// Center Stage (system video effect, per process). VoIP-background-mode apps get it for
// free via Control Center; otherwise enable in-app:
for format in camera.formats where format.isCenterStageSupported {
    try camera.lockForConfiguration()
    camera.activeFormat = format
    camera.unlockForConfiguration()
    break
}
AVCaptureDevice.centerStageControlMode = .cooperative  // or .app — set BEFORE enabling
AVCaptureDevice.isCenterStageEnabled = true

// Real-time low-latency stabilization for video calls (iOS 26+, off by default)
connection.preferredVideoStabilizationMode = .lowLatency

// Recording: .cinematicExtended / .cinematicExtendedEnhanced are face-aware on this camera

Bonus (iOS 26+, iOS-only): device.nominalFocalLengthIn35mmFilm — nominal 35mm-equivalent focal length (0 for virtual/external devices).

---

AVCaptureDevice.RotationCoordinator (iOS 17+)

Automatically tracks device orientation and provides rotation angles.

Setup

// Create with device and preview layer
let coordinator = AVCaptureDevice.RotationCoordinator(
    device: captureDevice,
    previewLayer: previewLayer
)

Properties

Observation

// KVO observation for preview updates
let observation = coordinator.observe(
    \.videoRotationAngleForHorizonLevelPreview,
    options: [.new]
) { [weak previewLayer] coordinator, _ in
    DispatchQueue.main.async {
        previewLayer?.connection?.videoRotationAngle = coordinator.videoRotationAngleForHorizonLevelPreview
    }
}

// Set initial value
previewLayer.connection?.videoRotationAngle = coordinator.videoRotationAngleForHorizonLevelPreview

Applying to Capture

func capturePhoto() {
    if let connection = photoOutput.connection(with: .video) {
        connection.videoRotationAngle = coordinator.videoRotationAngleForHorizonLevelCapture
    }
    photoOutput.capturePhoto(with: settings, delegate: self)
}

---

AVCapturePhotoOutput

Output for capturing still photos.

Configuration

let photoOutput = AVCapturePhotoOutput()

// High resolution (iOS 16+) — isHighResolutionCaptureEnabled is deprecated.
// Set the output's max dimensions to one of the active format's supported values.
photoOutput.maxPhotoDimensions = camera.activeFormat.supportedMaxPhotoDimensions.last!

// Max quality prioritization
photoOutput.maxPhotoQualityPrioritization = .quality

// Deferred processing (iOS 17+)
photoOutput.isAutoDeferredPhotoDeliveryEnabled = true

// Live Photo
photoOutput.isLivePhotoCaptureEnabled = true

// Depth
photoOutput.isDepthDataDeliveryEnabled = true

// Portrait Effects Matte
photoOutput.isPortraitEffectsMatteDeliveryEnabled = true

Supported Features

// Check support before enabling
camera.activeFormat.supportedMaxPhotoDimensions  // [CMVideoDimensions] — pick one for maxPhotoDimensions
photoOutput.isLivePhotoCaptureSupported
photoOutput.isDepthDataDeliverySupported
photoOutput.isPortraitEffectsMatteDeliverySupported
photoOutput.maxPhotoQualityPrioritization  // .speed, .balanced, .quality

Responsive Capture APIs (iOS 17+)

// Zero Shutter Lag - uses ring buffer for instant capture
photoOutput.isZeroShutterLagSupported
photoOutput.isZeroShutterLagEnabled  // true by default for iOS 17+ apps

// Responsive Capture - overlapping captures
photoOutput.isResponsiveCaptureSupported
photoOutput.isResponsiveCaptureEnabled

// Fast Capture Prioritization - adapts quality for burst-like capture
photoOutput.isFastCapturePrioritizationSupported
photoOutput.isFastCapturePrioritizationEnabled

// Deferred Processing - proxy + background processing
photoOutput.isAutoDeferredPhotoDeliverySupported
photoOutput.isAutoDeferredPhotoDeliveryEnabled

On iOS 27 (iPhone 16/17), the system also routes balanced fast captures through deferred processing, further shrinking shot-to-shot delay during rapid capture (WWDC 2026-304).

High-Resolution Capture (24/48 MP)

Only the .photo session preset supports 24/48 MP. Resolution availability by quality prioritization (WWDC 2026-304):

48 MP quad-sensor: iPhone 14 Pro+. 24 MP: iPhone 15+ (Camera-app default). 24/48 MP also on the telephoto (iPhone 16 Pro) and ultra-wide (iPhone 17) cameras. Deferred processing is what makes the multi-frame 18/24 MP resolutions practical — processing happens in the background without holding capture-session memory.

// 1. Configure the output for the largest dimensions you'll request (before commit —
//    changing maxPhotoDimensions after commit triggers a lengthy reconfiguration)
let dims = camera.activeFormat.supportedMaxPhotoDimensions  // [CMVideoDimensions], iOS 16+
photoOutput.maxPhotoDimensions = dims.max { Int($0.width) * Int($0.height) < Int($1.width) * Int($1.height) }!
photoOutput.maxPhotoQualityPrioritization = .quality

// 2. Pre-allocate capture resources as soon as the user enters the mode —
//    otherwise allocation happens at capture time and slows the first shot
let prepareSettings = AVCapturePhotoSettings()
prepareSettings.maxPhotoDimensions = photoOutput.maxPhotoDimensions
prepareSettings.photoQualityPrioritization = .quality
photoOutput.setPreparedPhotoSettingsArray([prepareSettings]) { prepared, error in /* ... */ }

// 3. Capture with a NEW settings object matching the prepared configuration
//    (prepared settings objects cannot be reused for capture)
let settings = AVCapturePhotoSettings()
settings.maxPhotoDimensions = photoOutput.maxPhotoDimensions  // request, not guarantee
settings.photoQualityPrioritization = .quality
photoOutput.capturePhoto(with: settings, delegate: self)

// Actual dimensions + expected processing time arrive in the delegate's
// AVCaptureResolvedPhotoSettings (photoProcessingTimeRange)

---

AVCapturePhotoOutputReadinessCoordinator (iOS 17+)

Provides synchronous shutter button state updates.

Setup

let coordinator = AVCapturePhotoOutputReadinessCoordinator(photoOutput: photoOutput)
coordinator.delegate = self

Tracking Captures

// Call BEFORE capturePhoto()
coordinator.startTrackingCaptureRequest(using: settings)
photoOutput.capturePhoto(with: settings, delegate: self)

Delegate

func readinessCoordinator(_ coordinator: AVCapturePhotoOutputReadinessCoordinator,
                          captureReadinessDidChange captureReadiness: AVCapturePhotoOutput.CaptureReadiness) {
    switch captureReadiness {
    case .ready:                         // Can capture immediately
    case .notReadyMomentarily:           // Brief delay, prevent double-tap
    case .notReadyWaitingForCapture:     // Flash firing, sensor reading
    case .notReadyWaitingForProcessing:  // Processing previous photo
    case .sessionNotRunning:             // Session stopped
    @unknown default: break
    }
}

---

AVCapturePhotoSettings

Configuration for a single photo capture.

Basic Settings

// Standard JPEG
var settings = AVCapturePhotoSettings()

// HEIF format
settings = AVCapturePhotoSettings(format: [AVVideoCodecKey: AVVideoCodecType.hevc])

// RAW
settings = AVCapturePhotoSettings(rawPixelFormatType: kCVPixelFormatType_14Bayer_BGGR)

// RAW + JPEG
settings = AVCapturePhotoSettings(
    rawPixelFormatType: kCVPixelFormatType_14Bayer_BGGR,
    processedFormat: [AVVideoCodecKey: AVVideoCodecType.jpeg]
)

Quality Prioritization

settings.photoQualityPrioritization = .speed

Flash

settings.flashMode = .auto  // .off, .on, .auto

Apple ProRAW and HDR

// Check ProRAW support
if photoOutput.isAppleProRAWSupported {
    photoOutput.isAppleProRAWEnabled = true

    // Capture ProRAW
    let query = photoOutput.isAppleProRAWEnabled
        ? AVCapturePhotoOutput.AppleProRAWQuery(photoOutput)
        : nil
    if let rawType = query?.availableRawPixelFormatTypes.first {
        let settings = AVCapturePhotoSettings(
            rawPixelFormatType: rawType,
            processedFormat: [AVVideoCodecKey: AVVideoCodecType.hevc]
        )
    }
}

// HDR configuration
settings.photoQualityPrioritization = .quality  // Enables computational photography/HDR
// HDR is automatic with .balanced or .quality — no separate toggle needed

Note: ProRAW requires iPhone 12 Pro or later. HDR is automatic with quality prioritization — Apple's Deep Fusion and Smart HDR are controlled by the system based on the quality setting.

Resolution

// Per-shot max dimensions (iOS 16+) — isHighResolutionPhotoEnabled is deprecated.
// Must match one of camera.activeFormat.supportedMaxPhotoDimensions, and be no
// larger than photoOutput.maxPhotoDimensions.
settings.maxPhotoDimensions = CMVideoDimensions(width: 4032, height: 3024)

Note: isHighResolutionPhotoEnabled is deprecated since iOS 16 — use maxPhotoDimensions only.

Preview/Thumbnail

// Preview for immediate display
settings.previewPhotoFormat = [
    kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA
]

// Thumbnail
settings.embeddedThumbnailPhotoFormat = [
    AVVideoCodecKey: AVVideoCodecType.jpeg,
    AVVideoWidthKey: 160,
    AVVideoHeightKey: 120
]

Important Notes

// Settings cannot be reused
// Each capture needs a NEW settings instance
let settings1 = AVCapturePhotoSettings()  // Use once
let settings2 = AVCapturePhotoSettings()  // Use for second capture

// Copy settings for similar captures
let settings2 = AVCapturePhotoSettings(from: settings1)

---

AVCapturePhotoCaptureDelegate

Delegate for photo capture events.

extension CameraManager: AVCapturePhotoCaptureDelegate {

    // Photo capture will begin
    func photoOutput(_ output: AVCapturePhotoOutput,
                     willBeginCaptureFor resolvedSettings: AVCaptureResolvedPhotoSettings) {
        // Show shutter animation
    }

    // Photo capture finished
    func photoOutput(_ output: AVCapturePhotoOutput,
                     didFinishProcessingPhoto photo: AVCapturePhoto,
                     error: Error?) {
        guard error == nil else {
            print("Capture error: \(error!)")
            return
        }

        // Get JPEG data
        if let data = photo.fileDataRepresentation() {
            savePhoto(data)
        }

        // Or get raw pixel buffer
        if let pixelBuffer = photo.pixelBuffer {
            processBuffer(pixelBuffer)
        }
    }

    // Deferred processing proxy (iOS 17+)
    func photoOutput(_ output: AVCapturePhotoOutput,
                     didFinishCapturingDeferredPhotoProxy deferredPhotoProxy: AVCaptureDeferredPhotoProxy,
                     error: Error?) {
        guard error == nil, let data = deferredPhotoProxy.fileDataRepresentation() else { return }
        replaceThumbnailWithFinal(data)
    }
}

---

AVCaptureMovieFileOutput

Output for recording video to file.

Setup

let movieOutput = AVCaptureMovieFileOutput()

if session.canAddOutput(movieOutput) {
    session.addOutput(movieOutput)
}

// Add audio input
if let microphone = AVCaptureDevice.default(for: .audio),
   let audioInput = try? AVCaptureDeviceInput(device: microphone),
   session.canAddInput(audioInput) {
    session.addInput(audioInput)
}

Recording

// Start recording
let outputURL = FileManager.default.temporaryDirectory
    .appendingPathComponent(UUID().uuidString)
    .appendingPathExtension("mov")

// Apply rotation
if let connection = movieOutput.connection(with: .video) {
    connection.videoRotationAngle = rotationCoordinator.videoRotationAngleForHorizonLevelCapture
}

movieOutput.startRecording(to: outputURL, recordingDelegate: self)

// Stop recording
movieOutput.stopRecording()

// Check state
movieOutput.isRecording
movieOutput.recordedDuration
movieOutput.recordedFileSize

Delegate

extension CameraManager: AVCaptureFileOutputRecordingDelegate {

    func fileOutput(_ output: AVCaptureFileOutput,
                    didStartRecordingTo fileURL: URL,
                    from connections: [AVCaptureConnection]) {
        // Recording started
    }

    func fileOutput(_ output: AVCaptureFileOutput,
                    didFinishRecordingTo outputFileURL: URL,
                    from connections: [AVCaptureConnection],
                    error: Error?) {
        if let error = error {
            print("Recording failed: \(error)")
            return
        }

        // Video saved to outputFileURL
        saveToPhotoLibrary(outputFileURL)
    }
}

Pro Video Storage OS27

Pre-allocated, system-wide storage for high-data-rate captures (e.g. ProRes) giving deterministic file-write performance — normal file I/O is non-deterministic under load (WWDC 2026-303). User controls capacity in Camera settings. Not on visionOS/watchOS.

Guided adoption flow: camera-capture.md Pattern 9.

---

Other 27-Cycle Capture Additions OS27

---

AVCaptureVideoPreviewLayer

Layer for displaying camera preview.

Setup

let previewLayer = AVCaptureVideoPreviewLayer(session: session)
previewLayer.videoGravity = .resizeAspectFill
previewLayer.frame = view.bounds
view.layer.addSublayer(previewLayer)

Video Gravity

SwiftUI Integration

struct CameraPreview: UIViewRepresentable {
    let session: AVCaptureSession

    func makeUIView(context: Context) -> PreviewView {
        let view = PreviewView()
        view.previewLayer.session = session
        view.previewLayer.videoGravity = .resizeAspectFill
        return view
    }

    func updateUIView(_ uiView: PreviewView, context: Context) {}

    class PreviewView: UIView {
        override class var layerClass: AnyClass { AVCaptureVideoPreviewLayer.self }
        var previewLayer: AVCaptureVideoPreviewLayer { layer as! AVCaptureVideoPreviewLayer }
    }
}

---

Common Code Patterns

Complete Camera Manager

import AVFoundation

@MainActor
class CameraManager: NSObject, ObservableObject {
    let session = AVCaptureSession()
    let photoOutput = AVCapturePhotoOutput()
    private let sessionQueue = DispatchQueue(label: "camera.session")
    private var rotationCoordinator: AVCaptureDevice.RotationCoordinator?
    private var rotationObservation: NSKeyValueObservation?

    @Published var isSessionRunning = false

    func setup() async -> Bool {
        guard await AVCaptureDevice.requestAccess(for: .video) else { return false }

        return await withCheckedContinuation { continuation in
            sessionQueue.async { [self] in
                session.beginConfiguration()
                defer { session.commitConfiguration() }

                session.sessionPreset = .photo

                guard let camera = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, position: .back),
                      let input = try? AVCaptureDeviceInput(device: camera),
                      session.canAddInput(input) else {
                    continuation.resume(returning: false)
                    return
                }
                session.addInput(input)

                guard session.canAddOutput(photoOutput) else {
                    continuation.resume(returning: false)
                    return
                }
                session.addOutput(photoOutput)
                photoOutput.maxPhotoQualityPrioritization = .quality

                continuation.resume(returning: true)
            }
        }
    }

    func start() {
        sessionQueue.async { [self] in
            session.startRunning()
            DispatchQueue.main.async {
                self.isSessionRunning = self.session.isRunning
            }
        }
    }

    func stop() {
        sessionQueue.async { [self] in
            session.stopRunning()
            DispatchQueue.main.async {
                self.isSessionRunning = false
            }
        }
    }

    func capturePhoto() {
        var settings = AVCapturePhotoSettings()
        settings.photoQualityPrioritization = .balanced

        if let connection = photoOutput.connection(with: .video),
           let angle = rotationCoordinator?.videoRotationAngleForHorizonLevelCapture {
            connection.videoRotationAngle = angle
        }

        photoOutput.capturePhoto(with: settings, delegate: self)
    }
}

extension CameraManager: AVCapturePhotoCaptureDelegate {
    nonisolated func photoOutput(_ output: AVCapturePhotoOutput,
                                  didFinishProcessingPhoto photo: AVCapturePhoto,
                                  error: Error?) {
        guard let data = photo.fileDataRepresentation() else { return }
        // Handle photo data
    }
}

---

Resources

WWDC: 2023-10105, 2026-303, 2026-304, 2026-341

Docs: /avfoundation/avcapturesession, /avfoundation/avcapturedevice, /avfoundation/avcapturephotosettings, /avfoundation/avcapturedevice/rotationcoordinator, /avfoundation/avprovideostorage, /avfoundation/avcapturesmartframingmonitor

Skills: skills/camera-capture.md, skills/camera-capture-diag.md

Camera Capture with AVFoundation

Guides you through implementing camera capture: session setup, photo capture, video recording, responsive capture UX, rotation handling, and session lifecycle management.

When to Use This Skill

Use when you need to:

• ☑ Build a custom camera UI (not system picker)
• ☑ Capture photos with quality/speed tradeoffs
• ☑ Record video with audio
• ☑ Handle device rotation correctly (RotationCoordinator)
• ☑ Make capture feel responsive (zero-shutter-lag)
• ☑ Make camera launch fast — preview up in half the time (deferred start, iOS 26+)
• ☑ Handle session interruptions (phone calls, multitasking)
• ☑ Switch between front/back cameras
• ☑ Configure capture quality and resolution (incl. 24/48 MP — see camera-capture-ref)
• ☑ Support the Center Stage front camera (iPhone 17 — see camera-capture-ref)
• ☑ Record high-data-rate video (ProRes) without dropped frames (Pro Video Storage OS27)

Example Prompts

"How do I set up a camera preview in SwiftUI?" "My camera freezes when I get a phone call" "The photo preview is rotated wrong on front camera" "How do I make photo capture feel instant?" "Should I use deferred processing?" "My camera takes too long to capture" "My camera app launches slowly — the preview takes forever to appear" "How do I capture 48 megapixel photos?" "How do I support the Center Stage front camera?" "My ProRes recording drops frames" "How do I switch between front and back cameras?" "How do I record video with audio?"

Red Flags

Signs you're making this harder than it needs to be:

• ❌ Calling startRunning() on main thread (blocks UI for seconds)
• ❌ Initializing every output before the first preview frame (slow launch — defer non-preview outputs on iOS 26+)
• ❌ Creating non-critical UI (mode pickers, image wells) before preview renders
• ❌ Using deprecated videoOrientation instead of RotationCoordinator (iOS 17+)
• ❌ Not observing session interruptions (app freezes on phone call)
• ❌ Creating new AVCaptureSession for each capture (expensive)
• ❌ Using .photo preset for video (wrong format)
• ❌ Ignoring photoQualityPrioritization (slow captures)
• ❌ Not handling .notAuthorized permission state
• ❌ Modifying session without beginConfiguration()/commitConfiguration()
• ❌ Using UIImagePickerController for custom camera UI (limited control)

Mandatory First Steps

Before implementing any camera feature:

1. Choose Your Capture Mode

What do you need?

┌─ Just let user pick a photo?
│  └─ Don't use AVFoundation - use PHPicker or PhotosPicker
│     See: skills/photo-library.md
│
├─ Simple photo/video capture with system UI?
│  └─ UIImagePickerController (but limited customization)
│
├─ Custom camera UI with photo capture?
│  └─ AVCaptureSession + AVCapturePhotoOutput
│     → Continue with this skill
│
├─ Custom camera UI with video recording?
│  └─ AVCaptureSession + AVCaptureMovieFileOutput
│     → Continue with this skill
│
└─ Both photo and video in same session?
   └─ AVCaptureSession + both outputs
      → Continue with this skill

2. Request Camera Permission

import AVFoundation

func requestCameraAccess() async -> Bool {
    let status = AVCaptureDevice.authorizationStatus(for: .video)

    switch status {
    case .authorized:
        return true
    case .notDetermined:
        return await AVCaptureDevice.requestAccess(for: .video)
    case .denied, .restricted:
        // Show settings prompt
        return false
    @unknown default:
        return false
    }
}

Info.plist required:

<key>NSCameraUsageDescription</key>
<string>Take photos and videos</string>

For audio (video recording):

<key>NSMicrophoneUsageDescription</key>
<string>Record audio with video</string>

3. Understand Session Architecture

AVCaptureSession
    ├─ Inputs
    │   ├─ AVCaptureDeviceInput (camera)
    │   └─ AVCaptureDeviceInput (microphone, for video)
    │
    ├─ Outputs
    │   ├─ AVCapturePhotoOutput (photos)
    │   ├─ AVCaptureMovieFileOutput (video files)
    │   └─ AVCaptureVideoDataOutput (raw frames)
    │
    └─ Connections (automatic between compatible input/output)

Key rule: All session configuration happens on a dedicated serial queue, never main thread.

Core Patterns

Pattern 1: Basic Session Setup

Use case: Set up camera preview with photo capture capability.

import AVFoundation

class CameraManager: NSObject {
    let session = AVCaptureSession()
    let photoOutput = AVCapturePhotoOutput()

    // CRITICAL: Dedicated serial queue for session work
    private let sessionQueue = DispatchQueue(label: "camera.session")

    func setupSession() {
        sessionQueue.async { [self] in
            session.beginConfiguration()
            defer { session.commitConfiguration() }

            // 1. Set session preset
            session.sessionPreset = .photo

            // 2. Add camera input
            guard let camera = AVCaptureDevice.default(.builtInWideAngleCamera,
                                                        for: .video,
                                                        position: .back),
                  let input = try? AVCaptureDeviceInput(device: camera),
                  session.canAddInput(input) else {
                return
            }
            session.addInput(input)

            // 3. Add photo output
            guard session.canAddOutput(photoOutput) else { return }
            session.addOutput(photoOutput)

            // 4. Configure photo output
            photoOutput.maxPhotoDimensions = camera.activeFormat.supportedMaxPhotoDimensions.last!
            photoOutput.maxPhotoQualityPrioritization = .quality
        }
    }

    func startSession() {
        sessionQueue.async { [self] in
            if !session.isRunning {
                session.startRunning()  // Blocking call - never on main thread!
            }
        }
    }

    func stopSession() {
        sessionQueue.async { [self] in
            if session.isRunning {
                session.stopRunning()
            }
        }
    }
}

Cost: 30 min implementation

Pattern 2: SwiftUI Camera Preview

Use case: Display camera preview in SwiftUI view.

import SwiftUI
import AVFoundation

struct CameraPreview: UIViewRepresentable {
    let session: AVCaptureSession

    func makeUIView(context: Context) -> PreviewView {
        let view = PreviewView()
        view.previewLayer.session = session
        view.previewLayer.videoGravity = .resizeAspectFill
        return view
    }

    func updateUIView(_ uiView: PreviewView, context: Context) {}

    class PreviewView: UIView {
        override class var layerClass: AnyClass { AVCaptureVideoPreviewLayer.self }
        var previewLayer: AVCaptureVideoPreviewLayer { layer as! AVCaptureVideoPreviewLayer }
    }
}

// Usage in SwiftUI
struct CameraView: View {
    @StateObject private var camera = CameraManager()

    var body: some View {
        CameraPreview(session: camera.session)
            .ignoresSafeArea()
            .onAppear { camera.startSession() }
            .onDisappear { camera.stopSession() }
    }
}

Cost: 20 min implementation

Pattern 3: Rotation Handling with RotationCoordinator (iOS 17+)

Use case: Keep preview and captured photos correctly oriented regardless of device rotation.

Why RotationCoordinator: Deprecated videoOrientation requires manual observation of device orientation. RotationCoordinator automatically tracks gravity and provides angles.

import AVFoundation

class CameraManager {
    private var rotationCoordinator: AVCaptureDevice.RotationCoordinator?
    private var rotationObservation: NSKeyValueObservation?

    func setupRotationCoordinator(device: AVCaptureDevice, previewLayer: AVCaptureVideoPreviewLayer) {
        // Create coordinator with device and preview layer
        rotationCoordinator = AVCaptureDevice.RotationCoordinator(
            device: device,
            previewLayer: previewLayer
        )

        // Observe preview rotation changes
        rotationObservation = rotationCoordinator?.observe(
            \.videoRotationAngleForHorizonLevelPreview,
            options: [.new]
        ) { [weak previewLayer] coordinator, _ in
            // Update preview layer rotation on main thread
            DispatchQueue.main.async {
                previewLayer?.connection?.videoRotationAngle = coordinator.videoRotationAngleForHorizonLevelPreview
            }
        }

        // Set initial rotation
        previewLayer.connection?.videoRotationAngle = rotationCoordinator!.videoRotationAngleForHorizonLevelPreview
    }

    func captureRotationAngle() -> CGFloat {
        // Use this angle when capturing photos
        rotationCoordinator?.videoRotationAngleForHorizonLevelCapture ?? 0
    }
}

When capturing:

func capturePhoto() {
    let settings = AVCapturePhotoSettings()

    // Apply rotation angle from coordinator
    if let connection = photoOutput.connection(with: .video) {
        connection.videoRotationAngle = captureRotationAngle()
    }

    photoOutput.capturePhoto(with: settings, delegate: self)
}

Cost: 45 min implementation, prevents 2+ hours debugging rotation issues

Pattern 4: Responsive Capture Pipeline (iOS 17+)

Use case: Make photo capture feel instant with zero-shutter-lag, overlapping captures, and responsive button states.

iOS 17+ introduces four complementary APIs that work together for maximum responsiveness:

4a. Zero Shutter Lag

Uses a ring buffer of recent frames to "time travel" back to the exact moment you tapped the shutter. Enabled automatically for iOS 17+ apps.

// Check if supported for current format
if photoOutput.isZeroShutterLagSupported {
    // Enabled by default for apps linking iOS 17+
    // Opt out if causing issues:
    // photoOutput.isZeroShutterLagEnabled = false
}

Why it matters: Without ZSL, there's a delay between tap and frame capture. For action shots, the moment is already over.

Requirements: iPhone XS and newer. Does NOT apply to flash captures, manual exposure, bracketed captures, or constituent photo delivery.

4b. Responsive Capture (Overlapping Captures)

Allows a new capture to start while the previous one is still processing:

// Check support first
if photoOutput.isZeroShutterLagSupported {
    photoOutput.isZeroShutterLagEnabled = true  // Required for responsive capture

    if photoOutput.isResponsiveCaptureSupported {
        photoOutput.isResponsiveCaptureEnabled = true
    }
}

Tradeoff: Increases peak memory usage. If your app is memory-constrained, consider leaving disabled.

Requirements: A12 Bionic (iPhone XS) and newer.

4c. Fast Capture Prioritization

Automatically adapts quality when taking multiple photos rapidly (like burst mode):

if photoOutput.isFastCapturePrioritizationSupported {
    photoOutput.isFastCapturePrioritizationEnabled = true
    // When enabled, rapid captures use "balanced" quality instead of "quality"
    // to maintain consistent shot-to-shot time
}

When to enable: User-facing toggle ("Prioritize Faster Shooting" in Camera.app). Off by default because it reduces quality.

4d. Readiness Coordinator (Button State Management)

Critical for UX: Provides synchronous updates for shutter button state without async lag.

class CameraManager {
    private var readinessCoordinator: AVCapturePhotoOutputReadinessCoordinator!

    func setupReadinessCoordinator() {
        readinessCoordinator = AVCapturePhotoOutputReadinessCoordinator(photoOutput: photoOutput)
        readinessCoordinator.delegate = self
    }

    func capturePhoto() {
        var settings = AVCapturePhotoSettings()
        settings.photoQualityPrioritization = .balanced

        // Tell coordinator to track this capture BEFORE calling capturePhoto
        readinessCoordinator.startTrackingCaptureRequest(using: settings)

        photoOutput.capturePhoto(with: settings, delegate: self)
    }
}

extension CameraManager: AVCapturePhotoOutputReadinessCoordinatorDelegate {
    func readinessCoordinator(_ coordinator: AVCapturePhotoOutputReadinessCoordinator,
                              captureReadinessDidChange captureReadiness: AVCapturePhotoOutput.CaptureReadiness) {
        DispatchQueue.main.async {
            switch captureReadiness {
            case .ready:
                self.shutterButton.isEnabled = true
                self.shutterButton.alpha = 1.0

            case .notReadyMomentarily:
                // Brief delay - disable to prevent double-tap
                self.shutterButton.isEnabled = false

            case .notReadyWaitingForCapture:
                // Flash is firing - dim button
                self.shutterButton.alpha = 0.5

            case .notReadyWaitingForProcessing:
                // Processing previous photo - show spinner
                self.showProcessingIndicator()

            case .sessionNotRunning:
                self.shutterButton.isEnabled = false

            @unknown default:
                break
            }
        }
    }
}

Why use Readiness Coordinator: Without it, you'd need to track capture state manually and users might spam the shutter button during processing.

Quality Prioritization (Baseline)

Still useful even without the new APIs:

func capturePhoto() {
    var settings = AVCapturePhotoSettings()

    // Speed vs Quality tradeoff
    // .speed     - Fastest capture, lower quality
    // .balanced  - Good default
    // .quality   - Best quality, may have delay
    settings.photoQualityPrioritization = .speed

    // For specific use cases:
    // - Social sharing: .speed (users expect instant)
    // - Document scanning: .quality (accuracy matters)
    // - General photography: .balanced

    photoOutput.capturePhoto(with: settings, delegate: self)
}

Deferred Processing (iOS 17+):

For maximum responsiveness, capture returns immediately with proxy image, full Deep Fusion processing happens in background:

// Check support and enable deferred processing
if photoOutput.isAutoDeferredPhotoDeliverySupported {
    photoOutput.isAutoDeferredPhotoDeliveryEnabled = true
}

Delegate callbacks with deferred processing:

// Called for BOTH regular photos AND deferred proxies
func photoOutput(_ output: AVCapturePhotoOutput,
                 didFinishProcessingPhoto photo: AVCapturePhoto,
                 error: Error?) {
    guard error == nil else { return }

    // Non-deferred photo - save directly
    if !photo.isRawPhoto, let data = photo.fileDataRepresentation() {
        savePhotoToLibrary(data)
    }
}

// Called ONLY for deferred proxies - save to PhotoKit for later processing
func photoOutput(_ output: AVCapturePhotoOutput,
                 didFinishCapturingDeferredPhotoProxy deferredPhotoProxy: AVCaptureDeferredPhotoProxy,
                 error: Error?) {
    guard error == nil else { return }

    // CRITICAL: Save proxy to library ASAP before app is backgrounded
    // App may be force-quit if memory pressure is high during backgrounding
    guard let proxyData = deferredPhotoProxy.fileDataRepresentation() else { return }

    Task {
        try await PHPhotoLibrary.shared().performChanges {
            let request = PHAssetCreationRequest.forAsset()
            // Use .photoProxy resource type - triggers deferred processing in Photos
            request.addResource(with: .photoProxy, data: proxyData, options: nil)
        }
    }
}

When final processing happens:

• On-demand when image is requested from PhotoKit
• Or automatically when device is idle (plugged in, not in use)

Fetching images with deferred processing awareness:

// Request with secondary degraded image for smoother UX
let options = PHImageRequestOptions()
options.allowSecondaryDegradedImage = true  // New in iOS 17

PHImageManager.default().requestImage(
    for: asset,
    targetSize: targetSize,
    contentMode: .aspectFill,
    options: options
) { image, info in
    let isDegraded = info?[PHImageResultIsDegradedKey] as? Bool ?? false

    if isDegraded {
        // First: Low quality (immediate)
        // Second: Medium quality (new - while processing)
        // Third callback will be final quality
        self.showTemporaryImage(image)
    } else {
        // Final quality - processing complete
        self.showFinalImage(image)
    }
}

Requirements: iPhone 11 Pro and newer. Not used for flash captures or formats that don't benefit from extended processing.

Important considerations:

• Can't apply pixel buffer customizations (filters, metadata changes) to deferred photos
• Use PhotoKit adjustments after processing for edits
• Get proxy into library ASAP - limited time when backgrounded

Cost: 1 hour implementation, prevents "camera feels slow" complaints

Pattern 5: Session Interruption Handling

Use case: Handle phone calls, multitasking, system camera usage.

class CameraManager {
    private var interruptionObservers: [NSObjectProtocol] = []

    func setupInterruptionHandling() {
        // Session was interrupted
        let interruptedObserver = NotificationCenter.default.addObserver(
            forName: .AVCaptureSessionWasInterrupted,
            object: session,
            queue: .main
        ) { [weak self] notification in
            guard let reason = notification.userInfo?[AVCaptureSessionInterruptionReasonKey] as? Int,
                  let interruptionReason = AVCaptureSession.InterruptionReason(rawValue: reason) else {
                return
            }

            switch interruptionReason {
            case .videoDeviceNotAvailableInBackground:
                // App went to background - normal, will resume
                self?.showPausedOverlay()

            case .audioDeviceInUseByAnotherClient:
                // Another app using audio
                self?.showInterruptedBanner("Audio in use by another app")

            case .videoDeviceInUseByAnotherClient:
                // Another app using camera
                self?.showInterruptedBanner("Camera in use by another app")

            case .videoDeviceNotAvailableWithMultipleForegroundApps:
                // Split View/Slide Over - camera not available
                self?.showInterruptedBanner("Camera unavailable in Split View")

            case .videoDeviceNotAvailableDueToSystemPressure:
                // Thermal state - reduce quality or stop
                self?.handleThermalPressure()

            @unknown default:
                self?.showInterruptedBanner("Camera interrupted")
            }
        }
        interruptionObservers.append(interruptedObserver)

        // Session interruption ended
        let endedObserver = NotificationCenter.default.addObserver(
            forName: .AVCaptureSessionInterruptionEnded,
            object: session,
            queue: .main
        ) { [weak self] _ in
            self?.hideInterruptedBanner()
            self?.hidePausedOverlay()
            // Session automatically resumes - no need to call startRunning()
        }
        interruptionObservers.append(endedObserver)
    }

    deinit {
        interruptionObservers.forEach { NotificationCenter.default.removeObserver($0) }
    }
}

Cost: 30 min implementation, prevents "camera freezes" bug reports

Pattern 6: Camera Switching (Front/Back)

Use case: Toggle between front and back cameras.

func switchCamera() {
    sessionQueue.async { [self] in
        guard let currentInput = session.inputs.first as? AVCaptureDeviceInput else {
            return
        }

        let currentPosition = currentInput.device.position
        let newPosition: AVCaptureDevice.Position = currentPosition == .back ? .front : .back

        guard let newDevice = AVCaptureDevice.default(
            .builtInWideAngleCamera,
            for: .video,
            position: newPosition
        ) else {
            return
        }

        session.beginConfiguration()
        defer { session.commitConfiguration() }

        // Remove old input
        session.removeInput(currentInput)

        // Add new input
        do {
            let newInput = try AVCaptureDeviceInput(device: newDevice)
            if session.canAddInput(newInput) {
                session.addInput(newInput)

                // Update rotation coordinator for new device
                if let previewLayer = previewLayer {
                    setupRotationCoordinator(device: newDevice, previewLayer: previewLayer)
                }
            } else {
                // Fallback: restore old input
                session.addInput(currentInput)
            }
        } catch {
            session.addInput(currentInput)
        }
    }
}

Front camera mirroring: Front camera preview is mirrored by default (matches user expectation). Captured photos are NOT mirrored (correct for sharing). This is intentional.

Cost: 20 min implementation

Pattern 7: Video Recording

Use case: Record video with audio to file.

class CameraManager: NSObject {
    let movieOutput = AVCaptureMovieFileOutput()
    private var currentRecordingURL: URL?

    func setupVideoRecording() {
        sessionQueue.async { [self] in
            session.beginConfiguration()
            defer { session.commitConfiguration() }

            // Set video preset
            session.sessionPreset = .high  // Or .hd1920x1080, .hd4K3840x2160

            // Add microphone input
            if let microphone = AVCaptureDevice.default(for: .audio),
               let audioInput = try? AVCaptureDeviceInput(device: microphone),
               session.canAddInput(audioInput) {
                session.addInput(audioInput)
            }

            // Add movie output
            if session.canAddOutput(movieOutput) {
                session.addOutput(movieOutput)
            }
        }
    }

    func startRecording() {
        guard !movieOutput.isRecording else { return }

        let outputURL = FileManager.default.temporaryDirectory
            .appendingPathComponent(UUID().uuidString)
            .appendingPathExtension("mov")

        currentRecordingURL = outputURL

        // Apply rotation
        if let connection = movieOutput.connection(with: .video) {
            connection.videoRotationAngle = captureRotationAngle()
        }

        movieOutput.startRecording(to: outputURL, recordingDelegate: self)
    }

    func stopRecording() {
        guard movieOutput.isRecording else { return }
        movieOutput.stopRecording()
    }
}

extension CameraManager: AVCaptureFileOutputRecordingDelegate {
    func fileOutput(_ output: AVCaptureFileOutput,
                    didFinishRecordingTo outputFileURL: URL,
                    from connections: [AVCaptureConnection],
                    error: Error?) {
        if let error = error {
            print("Recording error: \(error)")
            return
        }

        // Video saved to outputFileURL
        saveVideoToPhotoLibrary(outputFileURL)
    }
}

Cost: 45 min implementation

Pattern 8: Fast Camera Launch with Deferred Start (iOS 26+)

Use case: Get the preview frame on screen as fast as possible — Apple measured launch cut roughly in half with deferred start (WWDC 2026-303). The single most important factor in a camera launch feeling fast is how quickly preview appears.

The launch sequence: app launch → session configuration/start → output initialization (the most expensive stage) → preview streaming. Deferred start postpones output initialization until after preview is up.

Split your launch into two phases: 1. Critical for preview: session creation (off the main thread, in parallel with UI setup), camera input, the preview output, shutter button 2. Everything else after preview renders: photo/movie outputs, mode pickers, image wells, preferences

// Automatic mode — system runs deferred start shortly after preview appears.
// automaticallyRunsDeferredStart is true by default for apps linked against the iOS 26 SDK+.
session.beginConfiguration()
session.automaticallyRunsDeferredStart = true

let previewLayer = AVCaptureVideoPreviewLayer(session: session)
previewLayer.isDeferredStartEnabled = false   // the preview output must NOT be deferred

let photoOutput = AVCapturePhotoOutput()
guard session.canAddOutput(photoOutput) else { return }
session.addOutput(photoOutput)
photoOutput.isDeferredStartEnabled = true     // defer everything not needed for first frame

session.setDeferredStartDelegate(deferredStartDelegate,
                                 deferredStartDelegateCallbackQueue: sessionQueue)
session.commitConfiguration()                 // ONE commit — multiple commits extend launch
session.startRunning()                        // still on the session queue, never main thread

class DeferredStartDelegate: NSObject, AVCaptureSessionDeferredStartDelegate {
    func sessionWillRunDeferredStart(_ session: AVCaptureSession) {
        // Before deferred output initialization — create background resources here
    }
    func sessionDidRunDeferredStart(_ session: AVCaptureSession) {
        // All outputs initialized and ready to capture
    }
}

Manual mode — for apps that render preview with AVCaptureVideoDataOutput (automatic deferred start doesn't apply there) or need to finish their own startup work first. Check isManualDeferredStartSupported, set automaticallyRunsDeferredStart = false, keep isDeferredStartEnabled = false on the video data output, then call runDeferredStartWhenNeeded() once your first frame is presented — e.g. from a CAMetalLayer drawable's addPresentedHandler. Trigger snippet and full API table: camera-capture-ref "Deferred Start".

The catch: deferring the photo output speeds up preview but the time to first capture stays the same — the output still has to initialize before a capture can begin. Pair deferred start with isResponsiveCaptureEnabled = true (Pattern 4b) so taps buffer while the photo output finishes initializing.

Cost: 1 hour implementation; cuts perceived launch time ~2x

Pattern 9: Deterministic High-Data-Rate Recording OS27

Use case: ProRes or other high-bandwidth recordings drop frames because file I/O is non-deterministic under load (competing writes, fragmentation, storage wear).

Pro Video Storage is a shared, system-wide pool of pre-allocated storage; the user sizes it in Camera settings. Recordings write into the pre-allocated space, then move to your destination URL when capture finishes. Works with AVCaptureMovieFileOutput and AVAssetWriter. Not on visionOS/watchOS.

guard AVProVideoStorage.isSupported, let storage = AVProVideoStorage.shared else {
    return // fall back to normal recording
}
guard storage.remainingCapacity > 0 else {   // 0 = unconfigured, -1 = read failure
    storage.openSettings()                   // let the user allocate space
    return
}

// movieOutput: the AVCaptureMovieFileOutput already added to the session (Pattern 7)
guard movieOutput.isProVideoStorageSupported else { return }  // or setting the flag raises
guard !storage.isBusy else { return }  // resizing/file ops in flight; capture would raise
movieOutput.usesProVideoStorage = true
movieOutput.startRecording(to: movieFileURL, recordingDelegate: delegate)

See camera-capture-ref "Session Cost and System Pressure" for the companion APIs (hardwareCost, systemPressureState) that keep long recordings sustainable.

Cost: 30 min implementation

Anti-Patterns

Anti-Pattern 1: Session Work on Main Thread

Wrong:

func startCamera() {
    session.startRunning()  // Blocks UI for 1-3 seconds!
}

Right:

func startCamera() {
    sessionQueue.async { [self] in
        session.startRunning()
    }
}

Why it matters: startRunning() is blocking. On main thread, UI freezes.

Anti-Pattern 2: Using Deprecated videoOrientation

Wrong (pre-iOS 17):

// Manually tracking orientation
NotificationCenter.default.addObserver(
    forName: UIDevice.orientationDidChangeNotification,
    object: nil,
    queue: .main
) { _ in
    // Manual rotation logic...
}

Right (iOS 17+):

let coordinator = AVCaptureDevice.RotationCoordinator(device: camera, previewLayer: preview)
// Automatically tracks gravity, provides angles

Why it matters: RotationCoordinator handles edge cases (face-up, face-down) that manual tracking misses.

Anti-Pattern 3: Ignoring Session Interruptions

Wrong:

// No interruption handling - camera freezes on phone call

Right:

NotificationCenter.default.addObserver(
    forName: .AVCaptureSessionWasInterrupted,
    object: session,
    queue: .main
) { notification in
    // Show UI feedback
}

Why it matters: Without handling, camera appears frozen when interrupted.

Anti-Pattern 4: Modifying Session Without Configuration Block

Wrong:

session.removeInput(oldInput)
session.addInput(newInput)  // May fail mid-stream

Right:

session.beginConfiguration()
session.removeInput(oldInput)
session.addInput(newInput)
session.commitConfiguration()  // Atomic change

Why it matters: Without configuration block, session may enter invalid state between calls.

Pressure Scenarios

Scenario 1: "Just Make the Camera Work by Friday"

Context: Product wants camera feature shipped. You're considering skipping interruption handling.

Pressure: "It works when I test it, let's ship."

Reality: First user who gets a phone call while using camera will see frozen UI. App Store review may catch this.

Correct action: 1. Implement interruption handling (30 min) 2. Test by calling your test device during camera use 3. Verify UI shows appropriate feedback

Push-back template: "Camera captures work, but the app freezes if a phone call comes in. I need 30 minutes to handle interruptions properly and avoid 1-star reviews."

Scenario 2: "The Camera is Too Slow"

Context: QA reports photo capture feels sluggish. PM wants it "instant like the system camera."

Pressure: "Just make it faster somehow."

Reality: Default settings prioritize quality over speed. System camera uses deferred processing.

Correct action: 1. Set photoQualityPrioritization = .speed for social/sharing use cases 2. Consider deferred processing for maximum responsiveness 3. Show capture animation immediately (before processing completes)

Push-back template: "We're currently optimizing for image quality. I can make capture feel instant by prioritizing speed and showing the preview immediately while processing continues in background. This is what the system Camera app does."

Scenario 3: "Why is the Front Camera Photo Mirrored?"

Context: Designer reports front camera photos look "wrong" - they're not mirrored like the preview.

Pressure: "The preview shows it one way, the photo should match."

Reality: Preview is mirrored (user expectation - like a mirror). Photo is NOT mirrored (correct for sharing - text reads correctly). This is intentional behavior matching system camera.

Correct action: 1. Explain this is Apple's standard behavior 2. If business requires mirrored photos (selfie apps), manually mirror in post-processing 3. Never mirror the preview differently than expected

Push-back template: "This is intentional Apple behavior. The preview is mirrored like a mirror so users can frame themselves, but the captured photo is unmirrored so text reads correctly when shared. We can add optional mirroring in post-processing if our use case requires it."

Checklist

Before shipping camera features:

Session Setup:

• ☑ All session work on dedicated serial queue
• ☑ startRunning() never called on main thread
• ☑ Session preset matches use case (.photo for photos, .high for video)
• ☑ Configuration changes wrapped in beginConfiguration()/commitConfiguration()

Permissions:

• ☑ Camera permission requested before session setup
• ☑ NSCameraUsageDescription in Info.plist
• ☑ NSMicrophoneUsageDescription if recording audio
• ☑ Graceful handling of denied permission

Rotation:

• ☑ RotationCoordinator used (not deprecated videoOrientation)
• ☑ Preview layer rotation updated via observation
• ☑ Capture rotation angle applied when taking photos
• ☑ Tested in all orientations (portrait, landscape, face-up)

Responsiveness:

• ☑ photoQualityPrioritization set appropriately for use case
• ☑ Capture button shows immediate feedback
• ☑ Deferred processing considered for maximum speed

Launch Performance (iOS 26+):

• ☑ Non-preview outputs have isDeferredStartEnabled = true
• ☑ Session created off the main thread, in parallel with UI setup
• ☑ Single commitConfiguration() during launch
• ☑ Non-critical UI created after preview renders
• ☑ Responsive capture enabled alongside deferred photo output (otherwise taps are rejected until the photo output finishes initializing)
• ☑ Manual deferred start used when rendering preview via AVCaptureVideoDataOutput

Interruptions:

• ☑ Session interruption observer registered
• ☑ UI feedback shown when interrupted
• ☑ Tested with incoming phone call
• ☑ Tested in Split View (iPad)

Camera Switching:

• ☑ Front/back switch updates rotation coordinator
• ☑ Switch happens on session queue
• ☑ Fallback if new camera unavailable

Video Recording (if applicable):

• ☑ Microphone input added
• ☑ Recording delegate handles completion
• ☑ File cleanup for temporary recordings

Resources

WWDC: 2021-10247, 2023-10105, 2026-303, 2026-304, 2026-341

Docs: /avfoundation/avcapturesession, /avfoundation/avcapturedevice/rotationcoordinator, /avfoundation/avcapturephotosettings, /avfoundation/avcapturephotooutputreadinesscoordinator, /avfoundation/avprovideostorage

Skills: skills/camera-capture-ref.md, skills/camera-capture-diag.md, skills/photo-library.md

Music Understanding (On-Device Musical Analysis) OS27

import MusicUnderstanding — a new framework (OS27, all platforms: iOS 27, macOS 27, watchOS 27, tvOS 27, visionOS 27) that extracts musical features from audio entirely on-device: it works offline, the audio never leaves the device, and you need no signal-processing or ML expertise. Apple's Final Cut Pro uses it for beat detection and the iPad montage feature.

It analyzes six areas: key, rhythm, structure, pace, instrument activity, and loudness.

When to Use

• Sync visuals/edits to a song's beat, sections, loudness, or pace (video editors, montage, audio-reactive animation/games)
• Organize a catalog by tempo or key (DJ / library apps)
• Pre-compute and bundle analysis data to drive playback-time effects

For identifying which song is playing, use ShazamKit instead (skills/shazamkit.md) — that is catalog matching, a different problem.

Quick Start (analyze a file)

import MusicUnderstanding
import AVFoundation

// Set PreferPreciseDurationAndTiming for the most accurate results.
let asset = AVURLAsset(
    url: songURL,
    options: [AVURLAssetPreferPreciseDurationAndTimingKey: true]
)
let session = try await MusicUnderstandingSession(asset: asset)

// Analyze ALL six areas (every SessionResult field is populated):
let result = try await session.analyze()
if let bpm = result.rhythm?.beatsPerMinute {
    print("Tempo: \(bpm) bpm")
}

MusicUnderstandingSession is an actor. It has two convenience initializers (no public designated init):

analyze() vs analyze(for:)

Both are @discardableResult ... async throws -> SessionResult.

• analyze() — analyzes all six areas; every SessionResult field is non-nil.
• analyze(for: Set<AnalysisType>) — only the requested areas; unrequested fields come back `nil`. Use this to skip unnecessary computation.

AnalysisType values: .key, .rhythm, .structure, .pace, .instrumentActivity, .loudness.

let result = try await session.analyze(for: [.key, .rhythm])
// result.key and result.rhythm are populated; result.loudness, .structure, etc. are nil.

SessionResult fields (all optional): key, rhythm, structure, pace, instrumentActivity, loudness.

Result Types

Two helpers tie data to time (both nested in MusicUnderstandingSession, both generic, both Codable):

• `TimedValue<Value>` — { time: CMTime, value: Value }
• `RangedValue<Value>` — { range: CMTimeRange, value: Value }

// KeyResult — a timeline of key signatures
if let key = result.key {
    for ranged in key.ranges {                 // [RangedValue<KeyResult.KeySignature>]
        let sig = ranged.value                 // KeySignature { tonic, mode }
        print("\(sig.tonic) \(sig.mode)")      // e.g. .dFlat .major
    }
}
// Tonic: the 17 chromatic spellings (.c, .cSharp, .dFlat, …). Mode: .major / .minor.

// RhythmResult — beat/bar grid + global tempo
if let r = result.rhythm {
    let beats: [CMTime] = r.beats
    let bars:  [CMTime] = r.bars
    let bpm:   Float?    = r.beatsPerMinute     // nil if fewer than 2 beats were found
}

// StructureResult — three nested levels, each an array of ranges
if let s = result.structure {
    let sections = s.sections   // [CMTimeRange] — chorus/verse/intro/bridge
    let segments = s.segments   // [CMTimeRange]
    let phrases  = s.phrases    // [CMTimeRange]
}

// PaceResult — how fast the music *feels* over time (energy)
if let p = result.pace {
    let energy = p.ranges       // [RangedValue<Double>] — higher = more energetic
}

// InstrumentActivityResult — per-instrument presence and intensity
if let inst = result.instrumentActivity {
    let drumWhen  = inst.ranges[.drum]          // [CMTimeRange]? — when the drum is present
    let vocalCurve = inst.activity[.vocal]      // [TimedValue<Float>]? — intensity 0…1 over time
}
// Instrument: .vocal, .drum, .bass, .other.

// LoudnessResult — LUFS perceptual loudness (peak in dB)
if let loud = result.loudness {
    let overall  = loud.integrated.value        // TimedValue<Float> — whole-song average
    let momentary = loud.momentary              // [TimedValue<Float>] — 400 ms window, every 100 ms
    let shortTerm = loud.shortTerm              // [TimedValue<Float>] — 3 s window, smoother
    let peakDB   = loud.peak.value              // absolute peak, in decibels
}

Streaming Loudness

MusicUnderstandingSession exposes loudnessResults, an AsyncSequence<LoudnessResult, any Error> that emits as each 100 ms of audio is analyzed — useful for live meters. Consume the stream and drive analysis concurrently:

let session = MusicUnderstandingSession(audioProvider: liveBuffers)

// One task consumes the stream; analysis drives it on the current task.
let meter = Task {
    for try await loud in session.loudnessResults {
        await updateMeter(loud.momentary.last?.value)
    }
}
try await session.analyze(for: [.loudness])
try await meter.value

Custom Audio Input (AudioProvider)

Instead of an asset, feed buffers from any AsyncSequence whose Element is AVReadOnlyAudioPCMBuffer and whose Failure is Never. Send a final `nil` to signal completion so analysis can finish.

let session = MusicUnderstandingSession(audioProvider: myBufferSequence)
let result = try await session.analyze()

Export

Every result type is Codable. Encode the whole SessionResult to JSON:

let data = try JSONEncoder().encode(result)

Errors & Lifecycle

Call await session.cancel() to stop an in-flight analysis.

Resources

WWDC: 2026-253

Docs: /musicunderstanding

Skills: shazamkit, avfoundation-ref