Axiom Vision

Computer Vision

You MUST use this skill for ANY computer vision work using the Vision framework.

Quick Reference

Decision Tree

digraph vision {
    start [label="Computer vision task" shape=ellipse];
    what [label="What do you need?" shape=diamond];

    start -> what;
    what -> "skills/vision-framework.md" [label="implement feature"];
    what -> "skills/vision-ref.md" [label="API reference"];
    what -> "skills/vision-ref.md" [label="Visual Intelligence"];
    what -> "skills/vision-ref.md" [label="tap-to-segment / watchOS / FM tools (27)"];
    what -> "skills/vision-diag.md" [label="something broken"];
}

1. Implementing (pose, segmentation, OCR, barcodes, documents, live scanning)? → skills/vision-framework.md 2. Visual Intelligence system integration (camera/screenshot search; iOS 26+, iPadOS27/macOS27)? → skills/vision-ref.md (Visual Intelligence section) 3. Tap-to-segment, Vision on watchOS, or Vision tools for Foundation Models (27 cycle)? → skills/vision-ref.md 4. Need API reference / code examples? → skills/vision-ref.md 5. Debugging issues (detection failures, confidence, coordinates)? → skills/vision-diag.md

Critical Patterns

Implementation (skills/vision-framework.md):

• Decision tree for choosing the right Vision API
• Subject segmentation with VisionKit
• Isolating objects while excluding hands (combining APIs)
• Hand/body pose detection (21/19 landmarks)
• Text recognition (fast vs accurate modes)
• Barcode detection with symbology selection
• Document scanning and structured extraction (iOS 26+)
• Live scanning with DataScannerViewController
• CoreImage HDR compositing

Diagnostics (skills/vision-diag.md):

• Subject detection failures (edge of frame, lighting)
• Landmark tracking issues (confidence thresholds)
• Performance optimization (frame skipping, downscaling)
• Coordinate conversion (lower-left vs top-left origin)
• Text recognition failures (language, contrast)
• Barcode detection issues (symbology, size, glare)
• DataScanner troubleshooting (availability, data types)

Anti-Rationalization

Example Invocations

User: "How do I detect hand pose in an image?" → See skills/vision-framework.md

User: "Isolate a subject but exclude the user's hands" → See skills/vision-framework.md

User: "How do I read text from an image?" → See skills/vision-framework.md

User: "Scan QR codes with the camera" → See skills/vision-framework.md

User: "Subject detection isn't working" → See skills/vision-diag.md

User: "Text recognition returns wrong characters" → See skills/vision-diag.md

User: "Show me VNDetectHumanBodyPoseRequest examples" → See skills/vision-ref.md

User: "How do I make my app work with Visual Intelligence?" → See skills/vision-ref.md

User: "Let users tap an object in a photo to cut it out" → See skills/vision-ref.md (Iterative Segmentation)

User: "Can I use Vision in my watchOS app?" → See skills/vision-ref.md (Vision on watchOS)

User: "RecognizeDocumentsRequest API reference" → See skills/vision-ref.md

Vision Framework Diagnostics

Systematic troubleshooting for Vision framework issues: subjects not detected, missing landmarks, low confidence, performance problems, coordinate mismatches, text recognition failures, barcode detection issues, and document scanning problems.

Overview

Core Principle: When Vision doesn't work, the problem is usually: 1. Environment (lighting, occlusion, edge of frame) - 30% 2. Confidence threshold (ignoring low confidence data) - 25% 3. Orientation (handler defaults to .up; rotated photos look sideways) - 15% 4. Threading (blocking main thread causes frozen UI) - 15% 5. Coordinates (mixing lower-left and top-left origins) - 10% 6. API availability (using iOS 17+ APIs on older devices) - 5%

Always check environment, confidence, AND orientation BEFORE debugging code. "Nothing detected" on a photo that obviously contains the subject is almost always orientation, not lighting.

Red Flags

Symptoms that indicate Vision-specific issues:

Mandatory First Steps

Before investigating code, run these diagnostics:

Step 0: Verify Orientation

The single most common cause of "nothing detected" on a photo that obviously contains the subject. VNImageRequestHandler and ImageRequestHandler default to .up when you omit orientation. A photo shot in portrait or landscape carries EXIF rotation, so without it Vision analyzes a sideways image — and orientation-sensitive detectors (faces, text, body pose) find nothing.

// ❌ WRONG — handler assumes .up, ignores the photo's EXIF rotation
let handler = VNImageRequestHandler(cgImage: cgImage)

// ✅ CORRECT — derive CGImagePropertyOrientation from the source, then pass it
let cgOrientation = CGImagePropertyOrientation(uiImage.imageOrientation)
let handler = VNImageRequestHandler(cgImage: cgImage, orientation: cgOrientation)

Caveat — never cast raw values. UIImage.Orientation and CGImagePropertyOrientation use different raw-value orderings, so CGImagePropertyOrientation(rawValue: uiImage.imageOrientation.rawValue) silently produces the wrong rotation. Map the cases explicitly:

extension CGImagePropertyOrientation {
    init(_ orientation: UIImage.Orientation) {
        switch orientation {
        case .up: self = .up
        case .down: self = .down
        case .left: self = .left
        case .right: self = .right
        case .upMirrored: self = .upMirrored
        case .downMirrored: self = .downMirrored
        case .leftMirrored: self = .leftMirrored
        case .rightMirrored: self = .rightMirrored
        @unknown default: self = .up
        }
    }
}

Consequence for coordinates: once you pass orientation, Vision returns normalized coordinates in the oriented space, so convert overlays against the oriented (display) dimensions — not the raw pixel buffer's width/height. See Pattern 6.

Step 1: Verify Detection with Diagnostic Code

let request = VNGenerateForegroundInstanceMaskRequest()  // Or hand/body pose
// Pass orientation — see Step 0. Omitting it defaults to .up.
let handler = VNImageRequestHandler(cgImage: testImage, orientation: cgOrientation)

do {
    try handler.perform([request])

    if let results = request.results {
        print("✅ Request succeeded")
        print("Result count: \(results.count)")

        if let observation = results.first as? VNInstanceMaskObservation {
            print("All instances: \(observation.allInstances)")
            print("Instance count: \(observation.allInstances.count)")
        }
    } else {
        print("⚠️ Request succeeded but no results")
    }
} catch {
    print("❌ Request failed: \(error)")
}

Expected output:

• ✅ Request succeeded, instance count > 0 → Detection working
• ⚠️ Request succeeded, instance count = 0 → Nothing detected (see Decision Tree)
• ❌ Request failed → API availability issue

Step 2: Check Confidence Scores

// For hand/body pose
if let observation = request.results?.first as? VNHumanHandPoseObservation {
    let allPoints = try observation.recognizedPoints(.all)

    for (key, point) in allPoints {
        print("\(key): confidence \(point.confidence)")

        if point.confidence < 0.3 {
            print("  ⚠️ LOW CONFIDENCE - unreliable")
        }
    }
}

Expected output:

• Most landmarks > 0.5 confidence → Good detection
• Many landmarks < 0.3 → Poor lighting, occlusion, or edge of frame

Step 3: Verify Threading

print("🧵 Thread: \(Thread.current)")

if Thread.isMainThread {
    print("❌ Running on MAIN THREAD - will block UI!")
} else {
    print("✅ Running on background thread")
}

Expected output:

• ✅ Background thread → Correct
• ❌ Main thread → Move to DispatchQueue.global()

Decision Tree

Vision not working as expected?
│
├─ No results returned?
│  ├─ From a portrait/landscape photo? → Check Step 0 (orientation) FIRST
│  ├─ Check Step 1 output
│  │  ├─ "Request failed" → See Pattern 1a (API availability)
│  │  ├─ "No results" → See Pattern 1b (nothing detected)
│  │  └─ Results but count = 0 → See Pattern 1c (edge of frame)
│
├─ Landmarks have nil/low confidence?
│  ├─ Hand pose → See Pattern 2 (hand detection issues)
│  ├─ Body pose → See Pattern 3 (body detection issues)
│  └─ Face detection → See Pattern 4 (face detection issues)
│
├─ UI freezing/slow?
│  ├─ Check Step 3 (threading)
│  │  ├─ Main thread → See Pattern 5a (move to background)
│  │  └─ Background thread → See Pattern 5b (performance tuning)
│
├─ Overlays in wrong position?
│  └─ See Pattern 6 (coordinate conversion)
│
├─ Person segmentation missing people?
│  └─ See Pattern 7 (crowded scenes)
│
├─ VisionKit not working?
│  └─ See Pattern 8 (VisionKit specific)
│
├─ Text recognition issues?
│  ├─ No text detected → See Pattern 9a (image quality)
│  ├─ Wrong characters → See Pattern 9b (language/correction)
│  └─ Too slow → See Pattern 9c (recognition level)
│
├─ Barcode detection issues?
│  ├─ Barcode not detected → See Pattern 10a (symbology/size)
│  └─ Wrong payload → See Pattern 10b (barcode quality)
│
├─ DataScannerViewController issues?
│  ├─ Blank screen → See Pattern 11a (availability check)
│  └─ Items not detected → See Pattern 11b (data types)
│
└─ Document scanning issues?
   ├─ Edges not detected → See Pattern 12a (contrast/shape)
   └─ Perspective wrong → See Pattern 12b (corner points)

Modern Swift Vision API (iOS 18+)

The patterns below use the legacy VN* API (VNImageRequestHandler, VNRequest, request.results) because it covers iOS 13–17. On an iOS 18+ target, prefer the modern Swift Vision API — it eliminates two whole classes of bug from this skill: the Y-flip and the off-main-thread mistake.

// Modern: typed async request, typed results, no manual Y-flip
let handler = ImageRequestHandler(cgImage, orientation: cgOrientation)
let request = DetectFaceRectanglesRequest()
let faces = try await handler.perform(request)  // [FaceObservation]

for face in faces {
    let rect = face.boundingBox.toImageCoordinates(
        imageSize, origin: .upperLeft  // handles bottom-left → top-left flip
    )
}

ImageRequestHandler still takes orientation in its initializer, so Step 0 applies to both APIs.

On watchOS (watchOS27) the modern Swift API is the only Vision API — legacy VN* request classes don't exist there, and the request set is a subset (no text recognition, no pose). See vision-ref.md (Vision on watchOS) before debugging "missing API" errors on the watch.

Diagnostic Patterns

Pattern 1a: Request Failed (API Availability)

Symptom: try handler.perform([request]) throws error

Common errors:

"VNGenerateForegroundInstanceMaskRequest is only available on iOS 17.0 or newer"
"VNDetectHumanBodyPose3DRequest is only available on iOS 17.0 or newer"

Root cause: Using iOS 17+ APIs on older deployment target

Fix:

if #available(iOS 17.0, *) {
    let request = VNGenerateForegroundInstanceMaskRequest()
    // ...
} else {
    // Fallback for iOS 14-16
    let request = VNGeneratePersonSegmentationRequest()
    // ...
}

Prevention: Check API availability in skills/vision-ref.md before implementing

Also check (`OS27`): downloadable-asset requests (GenerateIterativeSegmentationRequest) fail on first use until the model is downloaded — VNErrorResourceUnavailable / VNErrorResourceCorrupted. Check await request.assetStatus and call try await request.downloadAssets() before the first perform.

Time to fix: 10 min

Pattern 1b: No Results (Nothing Detected)

Symptom: request.results == nil or results.isEmpty

Diagnostic:

// 1. Save debug image to Photos
UIImageWriteToSavedPhotosAlbum(debugImage, nil, nil, nil)

// 2. Inspect visually
// - Is subject too small? (< 10% of image)
// - Is subject blurry?
// - Poor contrast with background?

Common causes (check in this order):

• Orientation not passed — handler defaults to .up, sees a rotated photo sideways (see Step 0). Check this FIRST: it's the cheapest fix and the most common cause for camera/library photos.
• Subject too small (resize or crop closer)
• Subject too blurry (increase lighting, stabilize camera)
• Low contrast (subject same color as background)

Fix:

// Crop image to focus on region of interest — keep passing orientation
let croppedImage = cropImage(sourceImage, to: regionOfInterest)
let handler = VNImageRequestHandler(cgImage: croppedImage, orientation: cgOrientation)

Time to fix: 30 min (2 min if it turns out to be orientation)

Pattern 1c: Edge of Frame Issues

Symptom: Subject detected intermittently as object moves across frame

Root cause: Partial occlusion when subject touches image edges

Diagnostic:

// Check if subject is near edges
if let observation = results.first as? VNInstanceMaskObservation {
    // Bounds only need the raw instance-label buffer — no handler or mask generation required
    let bounds = calculateMaskBounds(observation.instanceMask)

    if bounds.minX < 0.1 || bounds.maxX > 0.9 ||
       bounds.minY < 0.1 || bounds.maxY > 0.9 {
        print("⚠️ Subject too close to edge")
    }
}

Fix:

// Add padding to capture area
let paddedRect = captureRect.insetBy(dx: -20, dy: -20)

// OR guide user with on-screen overlay
overlayView.addSubview(guideBox)  // Visual boundary

Time to fix: 20 min

Pattern 2: Hand Pose Issues

Symptom: VNDetectHumanHandPoseRequest returns nil or low confidence landmarks

Diagnostic:

if let observation = request.results?.first as? VNHumanHandPoseObservation {
    let thumbTip = try? observation.recognizedPoint(.thumbTip)
    let wrist = try? observation.recognizedPoint(.wrist)

    print("Thumb confidence: \(thumbTip?.confidence ?? 0)")
    print("Wrist confidence: \(wrist?.confidence ?? 0)")

    // Check hand orientation
    if let thumb = thumbTip, let wristPoint = wrist {
        let angle = atan2(
            thumb.location.y - wristPoint.location.y,
            thumb.location.x - wristPoint.location.x
        )
        print("Hand angle: \(angle * 180 / .pi) degrees")

        if abs(angle) > 80 && abs(angle) < 100 {
            print("⚠️ Hand parallel to camera (hard to detect)")
        }
    }
}

Common causes:

Fix for parallel hand:

// Detect and warn user
if avgConfidence < 0.4 {
    showWarning("Rotate your hand toward the camera")
}

Time to fix: 45 min

Pattern 3: Body Pose Issues

Symptom: VNDetectHumanBodyPoseRequest skips frames or returns low confidence

Diagnostic:

if let observation = request.results?.first as? VNHumanBodyPoseObservation {
    let nose = try? observation.recognizedPoint(.nose)
    let root = try? observation.recognizedPoint(.root)

    if let nosePoint = nose, let rootPoint = root {
        let bodyAngle = atan2(
            nosePoint.location.y - rootPoint.location.y,
            nosePoint.location.x - rootPoint.location.x
        )

        let angleFromVertical = abs(bodyAngle - .pi / 2)

        if angleFromVertical > .pi / 4 {
            print("⚠️ Person bent over or upside down")
        }
    }
}

Common causes:

Time to fix: 1 hour

Pattern 4: Face Detection Issues

Symptom: VNDetectFaceRectanglesRequest misses faces or returns wrong count

Diagnostic:

if let faces = request.results as? [VNFaceObservation] {
    print("Detected \(faces.count) faces")

    for face in faces {
        print("Face bounds: \(face.boundingBox)")
        print("Confidence: \(face.confidence)")

        if face.boundingBox.width < 0.1 {
            print("⚠️ Face too small")
        }
    }
}

Common causes:

• Face < 10% of image (crop closer)
• Profile view (use face landmarks request instead)
• Poor lighting (increase exposure)

Time to fix: 30 min

Pattern 5a: UI Freezing (Main Thread)

Symptom: App freezes ~1s when performing Vision request

Diagnostic (Step 3 above confirms main thread)

Red Flag — "run it on the main thread, it's fast": Vision requests routinely take 200ms–1s+ on real images, and longer on older devices. perform is synchronous and blocks the calling thread for its full duration. A 1-second main-thread stall is a visible freeze and a documented App Review rejection reason — never run Vision on the main thread, regardless of how fast it looks on your dev device. The modern ImageRequestHandler.perform is async, which keeps this off the main actor by construction (see Modern API note).

Fix:

// BEFORE (wrong)
let request = VNGenerateForegroundInstanceMaskRequest()
try handler.perform([request])  // Blocks UI for the request's full duration

// AFTER (correct)
DispatchQueue.global(qos: .userInitiated).async {
    let request = VNGenerateForegroundInstanceMaskRequest()
    try? handler.perform([request])

    DispatchQueue.main.async {
        // Update UI
    }
}

Time to fix: 15 min

Pattern 5b: Performance Issues (Background Thread)

Symptom: Already on background thread but still slow / dropping frames

Diagnostic:

let start = CFAbsoluteTimeGetCurrent()

try handler.perform([request])

let elapsed = CFAbsoluteTimeGetCurrent() - start
print("Request took \(elapsed * 1000)ms")

if elapsed > 0.2 {  // 200ms = too slow for real-time
    print("⚠️ Request too slow for real-time processing")
}

Common causes & fixes:

Fix for real-time camera:

// Skip frames
frameCount += 1
guard frameCount % 3 == 0 else { return }

// OR downscale
let scaledImage = resizeImage(sourceImage, to: CGSize(width: 1280, height: 720))

// OR set lower hand count
request.maximumHandCount = 2  // Instead of default

Time to fix: 1 hour

Pattern 6: Coordinate Conversion

Symptom: UI overlays appear in wrong position

Diagnostic:

// Vision point (lower-left origin, normalized)
let visionPoint = recognizedPoint.location
print("Vision point: \(visionPoint)")  // e.g., (0.5, 0.8)

// Convert to UIKit
let uiX = visionPoint.x * imageWidth
let uiY = (1 - visionPoint.y) * imageHeight  // FLIP Y
print("UIKit point: (\(uiX), \(uiY))")

// Verify overlay
overlayView.center = CGPoint(x: uiX, y: uiY)

Common mistakes:

// ❌ WRONG (no Y flip)
let uiPoint = CGPoint(
    x: visionPoint.x * width,
    y: visionPoint.y * height
)

// ❌ WRONG (forgot to scale from normalized)
let uiPoint = CGPoint(
    x: visionPoint.x,
    y: 1 - visionPoint.y
)

// ✅ CORRECT
let uiPoint = CGPoint(
    x: visionPoint.x * width,
    y: (1 - visionPoint.y) * height
)

For bounding boxes, let the framework do the flip instead of hand-rolling Y math:

// Legacy: scales normalized rect to pixels AND flips to top-left origin
let rectInPixels = VNImageRectForNormalizedRect(
    observation.boundingBox, Int(width), Int(height)
)

// Modern (iOS 18+): NormalizedRect handles the bottom-left→top-left flip
let rectInPixels = observation.boundingBox.toImageCoordinates(
    CGSize(width: width, height: height), origin: .upperLeft
)

Two non-obvious traps:

• Orientation space: if you passed orientation to the handler (Step 0), Vision's normalized coordinates are in the oriented image's space. Convert against the oriented (display) width/height, not the raw pixel buffer's.
• Aspect-fit letterboxing: when the image is displayed with .scaledToFit (UIView.ContentMode.scaleAspectFit), it is letterboxed — the image rect is smaller than the view. Compute that rect with AVMakeRect(aspectRatio:insideRect:) and convert into it, not the full view bounds, or every overlay is offset.

let imageRect = AVMakeRect(aspectRatio: image.size, insideRect: imageView.bounds)

Time to fix: 20 min

Pattern 7: Crowded Scenes (>4 People)

Symptom: VNGeneratePersonInstanceMaskRequest misses people or combines them

Diagnostic:

// Count faces
let faceRequest = VNDetectFaceRectanglesRequest()
try handler.perform([faceRequest])

let faceCount = faceRequest.results?.count ?? 0
print("Detected \(faceCount) faces")

// Person instance segmentation
let personRequest = VNGeneratePersonInstanceMaskRequest()
try handler.perform([personRequest])

let personCount = (personRequest.results?.first as? VNInstanceMaskObservation)?.allInstances.count ?? 0
print("Detected \(personCount) people")

if faceCount > 4 && personCount <= 4 {
    print("⚠️ Crowded scene - some people combined or missing")
}

Fix:

if faceCount > 4 {
    // Fallback: Use single mask for all people
    let singleMaskRequest = VNGeneratePersonSegmentationRequest()
    try handler.perform([singleMaskRequest])

    // OR guide user
    showWarning("Please reduce number of people in frame (max 4)")
}

Time to fix: 30 min

Pattern 8: VisionKit Specific Issues

Symptom: ImageAnalysisInteraction not showing subject lifting UI

Diagnostic:

// 1. Check interaction types
print("Interaction types: \(interaction.preferredInteractionTypes)")

// 2. Check if analysis is set
print("Analysis: \(interaction.analysis != nil ? "set" : "nil")")

// 3. Check if view supports interaction
if let view = interaction.view {
    print("View: \(view)")
} else {
    print("❌ View not set")
}

Common causes:

Fix:

// Ensure analysis is set
let analyzer = ImageAnalyzer()
let analysis = try await analyzer.analyze(image, configuration: config)

interaction.analysis = analysis  // Required!
interaction.preferredInteractionTypes = .imageSubject

Time to fix: 20 min

Pattern 9a: Text Not Detected (Image Quality)

Symptom: VNRecognizeTextRequest returns no results or empty strings

Diagnostic:

let request = VNRecognizeTextRequest()
request.recognitionLevel = .accurate

try handler.perform([request])

if request.results?.isEmpty ?? true {
    print("❌ No text detected")

    // Check image quality
    print("Image size: \(image.size)")
    print("Minimum text height: \(request.minimumTextHeight)")
}

for obs in request.results as? [VNRecognizedTextObservation] ?? [] {
    let top = obs.topCandidates(3)
    for candidate in top {
        print("'\(candidate.string)' confidence: \(candidate.confidence)")
    }
}

Common causes:

Fix for small text:

// Lower minimum text height (default ignores very small text)
request.minimumTextHeight = 0.02  // 2% of image height

Time to fix: 30 min

Pattern 9b: Wrong Characters (Language/Correction)

Symptom: Text is detected but characters are wrong (e.g., "C001" → "COOL")

Diagnostic:

// Check all candidates, not just first
for observation in results {
    let candidates = observation.topCandidates(5)
    for (i, candidate) in candidates.enumerated() {
        print("Candidate \(i): '\(candidate.string)' (\(candidate.confidence))")
    }
}

Common causes:

Fix for codes/serial numbers:

let request = VNRecognizeTextRequest()
request.usesLanguageCorrection = false  // Don't "fix" codes

// Post-process with domain knowledge
func correctSerialNumber(_ text: String) -> String {
    text.replacingOccurrences(of: "O", with: "0")
        .replacingOccurrences(of: "l", with: "1")
        .replacingOccurrences(of: "S", with: "5")
}

Time to fix: 30 min

Pattern 9c: Text Recognition Too Slow

Symptom: Text recognition takes >500ms, real-time camera drops frames

Diagnostic:

let start = CFAbsoluteTimeGetCurrent()
try handler.perform([request])
let elapsed = CFAbsoluteTimeGetCurrent() - start

print("Recognition took \(elapsed * 1000)ms")
print("Recognition level: \(request.recognitionLevel == .fast ? "fast" : "accurate")")
print("Language correction: \(request.usesLanguageCorrection)")

Common causes & fixes:

Fix for real-time:

request.recognitionLevel = .fast
request.usesLanguageCorrection = false
request.regionOfInterest = CGRect(x: 0.1, y: 0.3, width: 0.8, height: 0.4)

// Skip frames
frameCount += 1
guard frameCount % 3 == 0 else { return }

Time to fix: 30 min

Pattern 10a: Barcode Not Detected (Symbology/Size)

Symptom: VNDetectBarcodesRequest returns no results

Diagnostic:

let request = VNDetectBarcodesRequest()
// Don't specify symbologies to detect all types
try handler.perform([request])

if let results = request.results as? [VNBarcodeObservation] {
    print("Found \(results.count) barcodes")
    for barcode in results {
        print("Type: \(barcode.symbology)")
        print("Payload: \(barcode.payloadStringValue ?? "nil")")
        print("Bounds: \(barcode.boundingBox)")
    }
} else {
    print("❌ No barcodes detected")
}

Common causes:

Fix for small barcodes:

// Crop to barcode region for better detection
let croppedHandler = VNImageRequestHandler(
    cgImage: croppedImage,
    options: [:]
)

Time to fix: 20 min

Pattern 10b: Wrong Barcode Payload

Symptom: Barcode detected but payloadStringValue is wrong or nil

Diagnostic:

if let barcode = results.first {
    print("String payload: \(barcode.payloadStringValue ?? "nil")")
    print("Raw payload: \(barcode.payloadData ?? Data())")
    print("Symbology: \(barcode.symbology)")
    print("Confidence: Implicit (always 1.0 for barcodes)")
}

Common causes:

Time to fix: 15 min

Pattern 11a: DataScanner Blank Screen

Symptom: DataScannerViewController shows black/blank when presented

Diagnostic:

// Check support first
print("isSupported: \(DataScannerViewController.isSupported)")
print("isAvailable: \(DataScannerViewController.isAvailable)")

// Check camera permission
let status = AVCaptureDevice.authorizationStatus(for: .video)
print("Camera access: \(status.rawValue)")

Common causes:

Fix:

guard DataScannerViewController.isSupported else {
    showError("Scanning not supported on this device")
    return
}

guard DataScannerViewController.isAvailable else {
    // Request camera access
    AVCaptureDevice.requestAccess(for: .video) { granted in
        // Retry after access granted
    }
    return
}

Time to fix: 15 min

Pattern 11b: DataScanner Items Not Detected

Symptom: DataScanner shows camera but doesn't recognize items

Diagnostic:

// Check recognized data types
print("Data types: \(scanner.recognizedDataTypes)")

// Add delegate to see what's happening
func dataScanner(_ scanner: DataScannerViewController,
                 didAdd items: [RecognizedItem],
                 allItems: [RecognizedItem]) {
    print("Added \(items.count) items, total: \(allItems.count)")
    for item in items {
        switch item {
        case .text(let text): print("Text: \(text.transcript)")
        case .barcode(let barcode): print("Barcode: \(barcode.payloadStringValue ?? "")")
        @unknown default: break
        }
    }
}

Common causes:

Time to fix: 20 min

Pattern 12a: Document Edges Not Detected

Symptom: VNDetectDocumentSegmentationRequest returns no results

Diagnostic:

let request = VNDetectDocumentSegmentationRequest()
try handler.perform([request])

if let observation = request.results?.first {
    print("Document found at: \(observation.boundingBox)")
    print("Corners: TL=\(observation.topLeft), TR=\(observation.topRight)")
} else {
    print("❌ No document detected")
}

Common causes:

Fix: Use VNDocumentCameraViewController for guided user experience with live feedback.

Time to fix: 15 min

Pattern 12b: Perspective Correction Wrong

Symptom: Document extracted but distorted

Diagnostic:

// Verify corner order
print("TopLeft: \(observation.topLeft)")
print("TopRight: \(observation.topRight)")
print("BottomLeft: \(observation.bottomLeft)")
print("BottomRight: \(observation.bottomRight)")

// Check if corners are in expected positions
// TopLeft should have larger Y than BottomLeft (Vision uses lower-left origin)

Common causes:

Fix:

// Scale normalized to image coordinates
func scaled(_ point: CGPoint, to size: CGSize) -> CGPoint {
    CGPoint(x: point.x * size.width, y: point.y * size.height)
}

Time to fix: 20 min

Production Crisis Scenario

Situation: App Store review rejected for "app freezes when tapping analyze button"

Triage (5 min): 1. Confirm Vision running on main thread → Pattern 5a 2. Verify on older device (iPhone 12) → Freezes 3. Check profiling: 800ms on main thread

Fix (15 min):

@IBAction func analyzeTapped(_ sender: UIButton) {
    showLoadingIndicator()

    DispatchQueue.global(qos: .userInitiated).async { [weak self] in
        let request = VNGenerateForegroundInstanceMaskRequest()
        // ... perform request

        DispatchQueue.main.async {
            self?.hideLoadingIndicator()
            self?.updateUI(with: results)
        }
    }
}

Communicate to PM: "App Store rejection due to Vision processing on main thread. Fixed by moving to background queue (industry standard). Testing on iPhone 12 confirms fix. Safe to resubmit."

Quick Reference Table

Resources

WWDC: 2019-234, 2021-10041, 2022-10024, 2022-10025, 2025-272, 2023-10176, 2020-10653, 2026-237

Docs: /vision, /vision/vnrecognizetextrequest, /visionkit

Skills: skills/vision-framework.md, skills/vision-ref.md

Vision Framework Computer Vision

Guides you through implementing computer vision: subject segmentation, hand/body pose detection, person detection, text recognition, barcode detection, document scanning, and combining Vision APIs to solve complex problems.

When to Use This Skill

Use when you need to:

• ☑ Isolate subjects from backgrounds (subject lifting)
• ☑ Detect and track hand poses for gestures
• ☑ Detect and track body poses for fitness/action classification
• ☑ Segment multiple people separately
• ☑ Exclude hands from object bounding boxes (combining APIs)
• ☑ Choose between VisionKit and Vision framework
• ☑ Combine Vision with CoreImage for compositing
• ☑ Decide which Vision API solves your problem
• ☑ Recognize text in images (OCR)
• ☑ Detect barcodes and QR codes
• ☑ Scan documents with perspective correction
• ☑ Extract structured data from documents (iOS 26+)
• ☑ Build live scanning experiences (DataScannerViewController)

Example Prompts

"How do I isolate a subject from the background?" "I need to detect hand gestures like pinch" "How can I get a bounding box around an object without including the hand holding it?" "Should I use VisionKit or Vision framework for subject lifting?" "How do I segment multiple people separately?" "I need to detect body poses for a fitness app" "How do I preserve HDR when compositing subjects on new backgrounds?" "How do I recognize text in an image?" "I need to scan QR codes from camera" "How do I extract data from a receipt?" "Should I use DataScannerViewController or Vision directly?" "How do I scan documents and correct perspective?" "I need to extract table data from a document"

Red Flags

Signs you're making this harder than it needs to be:

• ❌ Manually implementing subject segmentation with CoreML models
• ❌ Using ARKit just for body pose (Vision works offline)
• ❌ Writing gesture recognition from scratch (use hand pose + simple distance checks)
• ❌ Processing on main thread (blocks UI - Vision is resource intensive)
• ❌ Training custom models when Vision APIs already exist
• ❌ Counting reps on an instantaneous per-frame threshold (e.g. hipY < kneeY) → phantom reps when the user is still; use a joint-angle hysteresis state machine (Pattern 8b)
• ❌ Comparing raw normalized landmark coordinates for "depth" — use a joint angle (scale- and position-invariant), not pixel Y
• ❌ Not checking confidence scores (low confidence = unreliable landmarks)
• ❌ Forgetting to convert coordinates (lower-left origin vs UIKit top-left)
• ❌ Building custom text recognizer when VNRecognizeTextRequest exists
• ❌ Using AVFoundation + Vision when DataScannerViewController suffices
• ❌ Processing every camera frame for scanning (skip frames, use region of interest)
• ❌ Enabling all barcode symbologies when you only need one (performance hit)
• ❌ Ignoring RecognizeDocumentsRequest when you need table/list structure (iOS 26+)

Mandatory First Steps

Before implementing any Vision feature:

1. Choose the Right API (Decision Tree)

What do you need to do?

┌─ Isolate subject(s) from background?
│  ├─ Need system UI + out-of-process → VisionKit
│  │  └─ ImageAnalysisInteraction (iOS/iPadOS)
│  │  └─ ImageAnalysisOverlayView (macOS)
│  ├─ Need custom pipeline / HDR / large images → Vision
│  │  └─ VNGenerateForegroundInstanceMaskRequest
│  ├─ User picks the object (tap/box/scribble), OS27 → GenerateIterativeSegmentationRequest
│  │  └─ Tap-to-segment + refine; see vision-ref.md (Iterative Segmentation)
│  └─ Need to EXCLUDE hands from object → Combine APIs
│     └─ Subject mask + Hand pose + custom masking (see Pattern 1)
│
├─ Segment people?
│  ├─ All people in one mask → VNGeneratePersonSegmentationRequest
│  └─ Separate mask per person (up to 4) → VNGeneratePersonInstanceMaskRequest
│
├─ Detect hand pose/gestures?
│  ├─ Just hand location → VNDetectHumanRectanglesRequest
│  └─ 21 hand landmarks → VNDetectHumanHandPoseRequest
│     └─ Gesture recognition → Hand pose + distance checks
│
├─ Detect body pose?
│  ├─ 2D normalized landmarks → VNDetectHumanBodyPoseRequest
│  ├─ 3D real-world coordinates → VNDetectHumanBodyPose3DRequest
│  ├─ Count reps of ONE known movement → joint angle + state machine, no training (Pattern 8b)
│  └─ Classify MANY exercises / nuanced form → Body pose + CreateML model (Pattern 8)
│
├─ Face detection?
│  ├─ Just bounding boxes → VNDetectFaceRectanglesRequest
│  └─ Detailed landmarks → VNDetectFaceLandmarksRequest
│
├─ Person detection (location only)?
│  └─ VNDetectHumanRectanglesRequest
│
├─ Recognize text in images?
│  ├─ Real-time from camera + need UI → DataScannerViewController (iOS 16+)
│  ├─ Processing captured image → VNRecognizeTextRequest
│  │  ├─ Need speed (real-time camera) → recognitionLevel = .fast
│  │  └─ Need accuracy (documents) → recognitionLevel = .accurate
│  └─ Need structured documents (iOS 26+) → RecognizeDocumentsRequest
│
├─ Detect barcodes/QR codes?
│  ├─ Real-time camera + need UI → DataScannerViewController (iOS 16+)
│  └─ Processing image → VNDetectBarcodesRequest
│
└─ Scan documents?
   ├─ Need built-in UI + perspective correction → VNDocumentCameraViewController
   ├─ Need structured data (tables, lists) → RecognizeDocumentsRequest (iOS 26+)
   └─ Custom pipeline → VNDetectDocumentSegmentationRequest + perspective correction

2. Set Up Background Processing

NEVER run Vision on main thread:

let processingQueue = DispatchQueue(label: "com.yourapp.vision", qos: .userInitiated)

processingQueue.async {
    do {
        let request = VNGenerateForegroundInstanceMaskRequest()
        let handler = VNImageRequestHandler(cgImage: image)
        try handler.perform([request])

        // Process observations...

        DispatchQueue.main.async {
            // Update UI
        }
    } catch {
        // Handle error
    }
}

3. Choose the Right Request Handler

Processing video frames? Use VNSequenceRequestHandler (maintains inter-frame state for temporal smoothing). For single images, use VNImageRequestHandler. Creating a new VNImageRequestHandler per frame discards temporal context and causes jittery results. See skills/vision-ref.md for full comparison and code examples.

4. Verify Platform Availability

Common Patterns

Pattern 1: Isolate Object While Excluding Hand

User's original problem: Getting a bounding box around an object held in hand, without including the hand.

Root cause: VNGenerateForegroundInstanceMaskRequest is class-agnostic and treats hand+object as one subject.

Solution: Combine subject mask with hand pose to create exclusion mask.

// 1. Get subject instance mask (modern async Vision API)
let handler = ImageRequestHandler(sourceImage)

guard let subjectObservation = try await handler.perform(GenerateForegroundInstanceMaskRequest()) else {
    fatalError("No subject detected")
}

// 2. Get hand pose landmarks (hand pose still uses the legacy request handler)
let handRequest = VNDetectHumanHandPoseRequest()
handRequest.maximumHandCount = 2
let poseHandler = VNImageRequestHandler(cgImage: sourceImage)
try poseHandler.perform([handRequest])

guard let handObservation = handRequest.results?.first as? VNHumanHandPoseObservation else {
    // No hand detected - use full subject mask
    let mask = try subjectObservation.generateScaledMask(
        for: subjectObservation.allInstances,
        scaledToImageFrom: handler
    )
    return mask
}

// 3. Create hand exclusion region from landmarks
let handPoints = try handObservation.recognizedPoints(.all)
let handBounds = calculateConvexHull(from: handPoints)  // Your implementation

// 4. Subtract hand region from subject mask using CoreImage
let subjectMask = try subjectObservation.generateScaledMask(
    for: subjectObservation.allInstances,
    scaledToImageFrom: handler
)

let subjectCIMask = CIImage(cvPixelBuffer: subjectMask)
let handMask = createMaskFromRegion(handBounds, size: sourceImage.size)
let finalMask = subtractMasks(handMask: handMask, from: subjectCIMask)

// 5. Calculate bounding box from final mask
let objectBounds = calculateBoundingBox(from: finalMask)

Helper: Convex Hull

func calculateConvexHull(from points: [VNRecognizedPointKey: VNRecognizedPoint]) -> CGRect {
    // Get high-confidence points
    let validPoints = points.values.filter { $0.confidence > 0.5 }

    guard !validPoints.isEmpty else { return .zero }

    // Simple bounding rect (for more accuracy, use actual convex hull algorithm)
    let xs = validPoints.map { $0.location.x }
    let ys = validPoints.map { $0.location.y }

    let minX = xs.min()!
    let maxX = xs.max()!
    let minY = ys.min()!
    let maxY = ys.max()!

    return CGRect(
        x: minX,
        y: minY,
        width: maxX - minX,
        height: maxY - minY
    )
}

Cost: 2-5 hours initial implementation, 30 min ongoing maintenance

Pattern 2: VisionKit Simple Subject Lifting

Use case: Add system-like subject lifting UI with minimal code.

// iOS
let interaction = ImageAnalysisInteraction()
interaction.preferredInteractionTypes = .imageSubject
imageView.addInteraction(interaction)

// macOS
let overlayView = ImageAnalysisOverlayView()
overlayView.preferredInteractionTypes = .imageSubject
nsView.addSubview(overlayView)

When to use:

• ✓ Want system behavior (long-press to select, drag to share)
• ✓ Don't need custom processing pipeline
• ✓ Image size within VisionKit limits (out-of-process)

Cost: 15 min implementation, 5 min ongoing

Pattern 3: Programmatic Subject Access (VisionKit)

Use case: Need subject images/bounds without UI interaction.

let analyzer = ImageAnalyzer()
let configuration = ImageAnalyzer.Configuration([.text, .visualLookUp])

let analysis = try await analyzer.analyze(sourceImage, configuration: configuration)

// Subjects come from ImageAnalysisInteraction, NOT the ImageAnalyzer.analyze result.
let interaction = ImageAnalysisInteraction()
interaction.analysis = analysis

// Get all subjects (`subjects` is async; `subject.image` is async throws)
for subject in await interaction.subjects {
    let subjectImage = try await subject.image
    let subjectBounds = subject.bounds

    // Process subject...
}

// Tap-based lookup
if let subject = await interaction.subject(at: tapPoint) {
    let compositeImage = try await interaction.image(for: [subject])
}

Cost: 30 min implementation, 10 min ongoing

Pattern 4: Vision Instance Mask for Custom Pipeline

Use case: HDR preservation, large images, custom compositing.

let handler = ImageRequestHandler(sourceImage)

guard let observation = try await handler.perform(GenerateForegroundInstanceMaskRequest()) else {
    return
}

// Get soft segmentation mask
let mask = try observation.generateScaledMask(
    for: observation.allInstances,
    scaledToImageFrom: handler  // Soft mask at input resolution, for compositing
)

// Use with CoreImage for HDR preservation
let filter = CIFilter(name: "CIBlendWithMask")!
filter.setValue(CIImage(cgImage: sourceImage), forKey: kCIInputImageKey)
filter.setValue(CIImage(cvPixelBuffer: mask), forKey: kCIInputMaskImageKey)
filter.setValue(newBackground, forKey: kCIInputBackgroundImageKey)

let compositedImage = filter.outputImage

Cost: 1 hour implementation, 15 min ongoing

Pattern 5: Tap-to-Select Instance

Use case: User taps to select which subject/person to lift.

// Tap-to-select uses instanceAtPoint + generateMask, which exist ONLY on the
// modern InstanceMaskObservation (iOS 18+). Obtain it via the async handler:
let observation = try await ImageRequestHandler(sourceImage)
    .perform(GenerateForegroundInstanceMaskRequest())
guard let observation else { return }

// instanceAtPoint takes a NormalizedPoint and returns an IndexSet
let tapped = observation.instanceAtPoint(tapPoint)  // tapPoint: NormalizedPoint

let mask: CVPixelBuffer
if tapped.isEmpty {
    mask = try observation.generateMask(for: observation.allInstances)  // nothing tapped → all
} else {
    mask = try observation.generateMask(for: tapped)                    // selected instance(s)
}

Alternative: Raw label access

// allInstancesMask is a PixelBufferObservation (UInt8 label buffer).
// Read the label at the tap directly — no CVPixelBuffer locking required.
let labelMask = observation.allInstancesMask
let label = labelMask.pixel(at: tapPoint)  // Float; 0 = background

// Or scan every label byte via the unsafe pointer:
labelMask.withUnsafePointer { raw in
    let first = raw.load(fromByteOffset: 0, as: UInt8.self)
    _ = first
}

Cost: 45 min implementation, 10 min ongoing

Pattern 6: Hand Gesture Recognition (Pinch)

Use case: Detect pinch gesture for custom camera trigger or UI control.

let request = VNDetectHumanHandPoseRequest()
request.maximumHandCount = 1

try handler.perform([request])

guard let observation = request.results?.first as? VNHumanHandPoseObservation else {
    return
}

let thumbTip = try observation.recognizedPoint(.thumbTip)
let indexTip = try observation.recognizedPoint(.indexTip)

// Check confidence
guard thumbTip.confidence > 0.5, indexTip.confidence > 0.5 else {
    return
}

// Calculate distance (normalized coordinates)
let dx = thumbTip.location.x - indexTip.location.x
let dy = thumbTip.location.y - indexTip.location.y
let distance = sqrt(dx * dx + dy * dy)

let isPinching = distance < 0.05  // Adjust threshold

// State machine for evidence accumulation
if isPinching {
    pinchFrameCount += 1
    if pinchFrameCount >= 3 {
        state = .pinched
    }
} else {
    pinchFrameCount = max(0, pinchFrameCount - 1)
    if pinchFrameCount == 0 {
        state = .apart
    }
}

Cost: 2 hours implementation, 20 min ongoing

Pattern 7: Separate Multiple People

Use case: Apply different effects to each person or count people.

let handler = ImageRequestHandler(sourceImage)

guard let observation = try await handler.perform(GeneratePersonInstanceMaskRequest()) else {
    return
}

let peopleCount = observation.allInstances.count  // Up to 4

for personIndex in observation.allInstances {
    let personMask = try observation.generateScaledMask(
        for: IndexSet(integer: personIndex),
        scaledToImageFrom: handler
    )

    // Apply effect to this person only
    applyEffect(to: personMask, personIndex: personIndex)
}

Crowded scenes (>4 people):

// Count faces to detect crowding
let faceRequest = VNDetectFaceRectanglesRequest()
try handler.perform([faceRequest])

let faceCount = faceRequest.results?.count ?? 0

if faceCount > 4 {
    // Fallback: Use single mask for all people
    let singleMaskRequest = VNGeneratePersonSegmentationRequest()
    try handler.perform([singleMaskRequest])
}

Cost: 1.5 hours implementation, 15 min ongoing

Pattern 8: Body Pose for Action Classification

Use case: Fitness app that recognizes exercises (jumping jacks, squats, etc.)

// 1. Collect body pose observations
var poseObservations: [VNHumanBodyPoseObservation] = []

let request = VNDetectHumanBodyPoseRequest()
try handler.perform([request])

if let observation = request.results?.first as? VNHumanBodyPoseObservation {
    poseObservations.append(observation)
}

// 2. When you have 60 frames of poses, prepare for CreateML model
if poseObservations.count == 60 {
    var multiArray = try MLMultiArray(
        shape: [60, 19, 3],  // 60 frames, 19 joints, (x, y, confidence)
        dataType: .double
    )

    for (frameIndex, observation) in poseObservations.enumerated() {
        let allPoints = try observation.recognizedPoints(.all)

        for (jointIndex, (_, point)) in allPoints.enumerated() {
            multiArray[[frameIndex, jointIndex, 0] as [NSNumber]] = NSNumber(value: point.location.x)
            multiArray[[frameIndex, jointIndex, 1] as [NSNumber]] = NSNumber(value: point.location.y)
            multiArray[[frameIndex, jointIndex, 2] as [NSNumber]] = NSNumber(value: point.confidence)
        }
    }

    // 3. Run inference with CreateML model
    let input = YourActionClassifierInput(poses: multiArray)
    let output = try actionClassifier.prediction(input: input)

    let action = output.label  // "jumping_jacks", "squats", etc.
}

Cost: 3-4 hours implementation, 1 hour ongoing

When to use Pattern 8 vs 8b Use the CreateML classifier (Pattern 8) when you must distinguish multiple exercise types or judge nuanced form. To count reps of one known movement, you do NOT need a model — Pattern 8b is more reliable, debuggable, and ships today with zero training data.

Pattern 8b: Rep Counting Without Training (joint angle + hysteresis state machine)

Use case: Count squats/curls/pushups from the live camera. The naive approach — if hipY < kneeY { reps += 1 } — produces phantom reps the instant a noisy landmark crosses the threshold (it counts while the user stands still), and raw normalized Y is meaningless for depth (Vision's origin is bottom-left, and pixel distance changes with how far the user stands from the camera).

Two ideas fix it: measure a joint angle (scale- and position-invariant), and count only a completed transition through a hysteresis state machine — never an instantaneous predicate.

// 1. Joint angle at vertex B for A-B-C (e.g. hip-knee-ankle). Origin-agnostic.
func angle(_ a: CGPoint, _ b: CGPoint, _ c: CGPoint) -> Double {
    let v1 = CGVector(dx: a.x - b.x, dy: a.y - b.y)
    let v2 = CGVector(dx: c.x - b.x, dy: c.y - b.y)
    let mag = hypot(v1.dx, v1.dy) * hypot(v2.dx, v2.dy)
    guard mag > 0 else { return 180 }
    let cosine = max(-1, min(1, (v1.dx * v2.dx + v1.dy * v2.dy) / mag))
    return acos(cosine) * 180 / .pi
}

// 2. Pure, unit-testable counter. A rep = standing → bottom → standing.
//    Two thresholds (the hysteresis gap) reject jitter near a single edge.
struct SquatRepCounter {
    enum Phase { case standing, bottom }
    private(set) var phase: Phase = .standing
    private(set) var reps = 0
    private var framesAtBottom = 0
    let downAngle = 100.0, upAngle = 160.0, confirmFrames = 3

    mutating func update(kneeAngle: Double) -> Bool {
        switch phase {
        case .standing where kneeAngle <= downAngle:
            framesAtBottom += 1
            if framesAtBottom >= confirmFrames { phase = .bottom; framesAtBottom = 0 }
        case .bottom where kneeAngle >= upAngle:
            phase = .standing; reps += 1; return true   // the ONLY place a rep counts
        default: framesAtBottom = 0
        }
        return false
    }
}

// 3. Per frame: gate on confidence, average both legs, feed the counter.
func kneeAngle(_ obs: VNHumanBodyPoseObservation, minConfidence: VNConfidence = 0.5) -> Double? {
    func leg(_ h: VNHumanBodyPoseObservation.JointName,
             _ k: VNHumanBodyPoseObservation.JointName,
             _ a: VNHumanBodyPoseObservation.JointName) -> Double? {
        guard let h = try? obs.recognizedPoint(h), h.confidence > minConfidence,
              let k = try? obs.recognizedPoint(k), k.confidence > minConfidence,
              let a = try? obs.recognizedPoint(a), a.confidence > minConfidence else { return nil }
        return angle(h.location, k.location, a.location)
    }
    let l = leg(.leftHip, .leftKnee, .leftAnkle), r = leg(.rightHip, .rightKnee, .rightAnkle)
    switch (l, r) { case let (l?, r?): return (l + r) / 2; case let (l?, nil): return l
                    case let (nil, r?): return r; default: return nil }   // no confident pose
}

Why this is reliable

• Angle, not pixels — independent of camera distance and where the user stands in frame.
• Hysteresis + frame confirmation — standing still keeps the angle near 175°; noise can't cross both the 100° down gate and the 160° up gate, so reps fire only on a real down-and-up.
• Confidence gating — drop frames where joints are occluded instead of counting garbage.
• Pure counter — SquatRepCounter and angle have no camera dependency; unit-test them with synthetic angle streams (standing → 0 reps, ten clean reps → 10, a single spike → 0).

Production add-ons Smooth the angle (EMA or 3-5 sample median) before the counter; calibrate thresholds per user from a 1-2s "stand still" baseline; require a minimum rep duration (~600 ms) to reject bouncing. For real-time video, drive this from a single VNSequenceRequestHandler (see Mandatory First Steps #3) and run detection off the main thread.

Cost: ~1 day, no training data, no model.

Pattern 9: Text Recognition (OCR)

Use case: Extract text from images, receipts, signs, documents.

let request = VNRecognizeTextRequest()
request.recognitionLevel = .accurate  // Or .fast for real-time
request.recognitionLanguages = ["en-US"]  // Specify known languages
request.usesLanguageCorrection = true  // Helps accuracy

let handler = VNImageRequestHandler(cgImage: image)
try handler.perform([request])

guard let observations = request.results as? [VNRecognizedTextObservation] else {
    return
}

for observation in observations {
    // Get top candidate (most likely)
    guard let candidate = observation.topCandidates(1).first else { continue }

    let text = candidate.string
    let confidence = candidate.confidence

    // Get bounding box for specific substring
    if let range = text.range(of: searchTerm) {
        if let boundingBox = try? candidate.boundingBox(for: range) {
            // Use for highlighting
        }
    }
}

Fast vs Accurate:

• Fast: Real-time camera, large legible text (signs, billboards), character-by-character
• Accurate: Documents, receipts, small text, handwriting, ML-based word/line recognition

Language tips:

• Order matters: first language determines ML model for accurate path
• Use automaticallyDetectsLanguage = true only when language unknown
• Query supportedRecognitionLanguages for current revision

Cost: 30 min basic implementation, 2 hours with language handling

Pattern 10: Barcode/QR Code Detection

Use case: Scan product barcodes, QR codes, healthcare codes.

let request = VNDetectBarcodesRequest()
request.revision = VNDetectBarcodesRequestRevision3  // ML-based, iOS 16+
request.symbologies = [.qr, .ean13]  // Specify only what you need!

let handler = VNImageRequestHandler(cgImage: image)
try handler.perform([request])

guard let observations = request.results as? [VNBarcodeObservation] else {
    return
}

for barcode in observations {
    let payload = barcode.payloadStringValue  // Decoded content
    let symbology = barcode.symbology  // Type of barcode
    let bounds = barcode.boundingBox  // Location (normalized)

    print("Found \(symbology): \(payload ?? "no string")")
}

Performance tip: Specifying fewer symbologies = faster scanning

Revision differences:

• Revision 1: One code at a time, 1D codes return lines
• Revision 2: Codabar, GS1Databar, MicroPDF, MicroQR, better with ROI
• Revision 3: ML-based, multiple codes at once, better bounding boxes, fewer duplicates

Cost: 15 min implementation

Pattern 11: DataScannerViewController (Live Scanning)

Use case: Camera-based text/barcode scanning with built-in UI (iOS 16+).

import VisionKit

// Check support
guard DataScannerViewController.isSupported,
      DataScannerViewController.isAvailable else {
    // Not supported or camera access denied
    return
}

// Configure what to scan
let recognizedDataTypes: Set<DataScannerViewController.RecognizedDataType> = [
    .barcode(symbologies: [.qr]),
    .text(textContentType: .URL)  // Or nil for all text
]

// Create and present
let scanner = DataScannerViewController(
    recognizedDataTypes: recognizedDataTypes,
    qualityLevel: .balanced,  // Or .fast, .accurate
    recognizesMultipleItems: false,  // Center-most if false
    isHighFrameRateTrackingEnabled: true,  // For smooth highlights
    isPinchToZoomEnabled: true,
    isGuidanceEnabled: true,
    isHighlightingEnabled: true
)

scanner.delegate = self
present(scanner, animated: true) {
    try? scanner.startScanning()
}

Delegate methods:

func dataScanner(_ scanner: DataScannerViewController,
                 didTapOn item: RecognizedItem) {
    switch item {
    case .text(let text):
        print("Tapped text: \(text.transcript)")
    case .barcode(let barcode):
        print("Tapped barcode: \(barcode.payloadStringValue ?? "")")
    @unknown default: break
    }
}

// For custom highlights
func dataScanner(_ scanner: DataScannerViewController,
                 didAdd addedItems: [RecognizedItem],
                 allItems: [RecognizedItem]) {
    for item in addedItems {
        let highlight = createHighlight(for: item)
        scanner.overlayContainerView.addSubview(highlight)
    }
}

Async stream alternative:

for await items in scanner.recognizedItems {
    // Process current items
}

Cost: 45 min implementation with custom highlights

Pattern 12: Document Scanning with VNDocumentCameraViewController

Use case: Scan paper documents with automatic edge detection and perspective correction.

import VisionKit

let documentCamera = VNDocumentCameraViewController()
documentCamera.delegate = self
present(documentCamera, animated: true)

// In delegate
func documentCameraViewController(_ controller: VNDocumentCameraViewController,
                                   didFinishWith scan: VNDocumentCameraScan) {
    controller.dismiss(animated: true)

    // Process each page
    for pageIndex in 0..<scan.pageCount {
        let image = scan.imageOfPage(at: pageIndex)

        // Now run text recognition on the corrected image
        let handler = VNImageRequestHandler(cgImage: image.cgImage!)
        let textRequest = VNRecognizeTextRequest()
        try? handler.perform([textRequest])
    }
}

Cost: 30 min implementation

Pattern 13: Document Segmentation (Custom Pipeline)

Use case: Detect document edges programmatically for custom camera UI.

let request = VNDetectDocumentSegmentationRequest()
let handler = VNImageRequestHandler(ciImage: inputImage)
try handler.perform([request])

guard let observation = request.results?.first,
      let document = observation as? VNRectangleObservation else {
    return
}

// Get corner points (normalized coordinates)
let topLeft = document.topLeft
let topRight = document.topRight
let bottomLeft = document.bottomLeft
let bottomRight = document.bottomRight

// Apply perspective correction with CoreImage
let correctedImage = inputImage
    .cropped(to: document.boundingBox.scaled(to: imageSize))
    .applyingFilter("CIPerspectiveCorrection", parameters: [
        "inputTopLeft": CIVector(cgPoint: topLeft.scaled(to: imageSize)),
        "inputTopRight": CIVector(cgPoint: topRight.scaled(to: imageSize)),
        "inputBottomLeft": CIVector(cgPoint: bottomLeft.scaled(to: imageSize)),
        "inputBottomRight": CIVector(cgPoint: bottomRight.scaled(to: imageSize))
    ])

VNDetectDocumentSegmentationRequest vs VNDetectRectanglesRequest:

• Document: ML-based, trained on documents, handles non-rectangles, returns one document
• Rectangle: Edge-based, finds any quadrilateral, returns multiple, CPU-only

Cost: 1-2 hours implementation

Pattern 14: Structured Document Extraction (iOS 26+)

Use case: Extract tables, lists, paragraphs with semantic understanding.

// iOS 26+
let request = RecognizeDocumentsRequest()
let observations = try await request.perform(on: imageData)

guard let document = observations.first?.document else {
    return
}

// Extract tables
for table in document.tables {
    for row in table.rows {
        for cell in row {
            let text = cell.content.text.transcript
            print("Cell: \(text)")
        }
    }
}

// Get detected data (emails, phones, URLs, dates)
let allDetectedData = document.text.detectedData
for data in allDetectedData {
    switch data.match.details {
    case .emailAddress(let email):
        print("Email: \(email.emailAddress)")
    case .phoneNumber(let phone):
        print("Phone: \(phone.phoneNumber)")
    case .link(let link):
        print("URL: \(link.url)")
    default: break
    }
}

Document hierarchy:

• Document → containers (text, tables, lists, barcodes)
• Table → rows → cells → content
• Content → text (transcript, lines, paragraphs, words, detectedData)

Cost: 1 hour implementation

Pattern 15: Real-time Phone Number Scanner

Use case: Scan phone numbers from camera like barcode scanner (from WWDC 2019).

// 1. Use region of interest to guide user
let textRequest = VNRecognizeTextRequest { request, error in
    guard let observations = request.results as? [VNRecognizedTextObservation] else { return }

    for observation in observations {
        guard let candidate = observation.topCandidates(1).first else { continue }

        // Use domain knowledge to filter
        if let phoneNumber = self.extractPhoneNumber(from: candidate.string) {
            self.stringTracker.add(phoneNumber)
        }
    }

    // Build evidence over frames
    if let stableNumber = self.stringTracker.getStableString(threshold: 10) {
        self.foundPhoneNumber(stableNumber)
    }
}

textRequest.recognitionLevel = .fast  // Real-time
textRequest.usesLanguageCorrection = false  // Codes, not natural text
textRequest.regionOfInterest = guidanceBox  // Crop to user's focus area

// 2. String tracker for stability
class StringTracker {
    private var seenStrings: [String: Int] = [:]

    func add(_ string: String) {
        seenStrings[string, default: 0] += 1
    }

    func getStableString(threshold: Int) -> String? {
        seenStrings.first { $0.value >= threshold }?.key
    }
}

Key techniques from WWDC 2019:

• Use .fast recognition level for real-time
• Disable language correction for codes/numbers
• Use region of interest to improve speed and focus
• Build evidence over multiple frames (string tracker)
• Apply domain knowledge (phone number regex)

Cost: 2 hours implementation

Anti-Patterns

Anti-Pattern 1: Processing on Main Thread

Wrong:

let request = VNGenerateForegroundInstanceMaskRequest()
let handler = VNImageRequestHandler(cgImage: image)
try handler.perform([request])  // Blocks UI!

Right:

DispatchQueue.global(qos: .userInitiated).async {
    let request = VNGenerateForegroundInstanceMaskRequest()
    let handler = VNImageRequestHandler(cgImage: image)
    try handler.perform([request])

    DispatchQueue.main.async {
        // Update UI
    }
}

Why it matters: Vision is resource-intensive. Blocking main thread freezes UI.

Anti-Pattern 2: Ignoring Confidence Scores

Wrong:

let thumbTip = try observation.recognizedPoint(.thumbTip)
let location = thumbTip.location  // May be unreliable!

Right:

let thumbTip = try observation.recognizedPoint(.thumbTip)
guard thumbTip.confidence > 0.5 else {
    // Low confidence - landmark unreliable
    return
}
let location = thumbTip.location

Why it matters: Low confidence points are inaccurate (occlusion, blur, edge of frame).

Anti-Pattern 3: Forgetting Coordinate Conversion

Wrong (mixing coordinate systems):

// Vision uses lower-left origin
let visionPoint = recognizedPoint.location  // (0, 0) = bottom-left

// UIKit uses top-left origin
let uiPoint = CGPoint(x: visionPoint.x, y: visionPoint.y)  // WRONG!

Right:

let visionPoint = recognizedPoint.location

// Convert to UIKit coordinates
let uiPoint = CGPoint(
    x: visionPoint.x * imageWidth,
    y: (1 - visionPoint.y) * imageHeight  // Flip Y axis
)

Why it matters: Mismatched origins cause UI overlays to appear in wrong positions.

Anti-Pattern 4: Setting maximumHandCount Too High

Wrong:

let request = VNDetectHumanHandPoseRequest()
request.maximumHandCount = 10  // "Just in case"

Right:

let request = VNDetectHumanHandPoseRequest()
request.maximumHandCount = 2  // Only compute what you need

Why it matters: Performance scales with maximumHandCount. Pose computed for all detected hands ≤ max.

Anti-Pattern 5: Using ARKit When Vision Suffices

Wrong (if you don't need AR):

// Requires AR session just for body pose
let arSession = ARBodyTrackingConfiguration()

Right:

// Vision works offline on still images
let request = VNDetectHumanBodyPoseRequest()

Why it matters: ARKit body pose requires rear camera, AR session, supported devices. Vision works everywhere (even offline).

Pressure Scenarios

Scenario 1: "Just Ship the Feature"

Context: Product manager wants subject lifting "like in Photos app" by Friday. You're considering skipping background processing.

Pressure: "It's working on my iPhone 15 Pro, let's ship it."

Reality: Vision blocks UI on older devices. Users on iPhone 12 will experience frozen app.

Correct action: 1. Implement background queue (15 min) 2. Add loading indicator (10 min) 3. Test on iPhone 12 or earlier (5 min)

Push-back template: "Subject lifting works, but it freezes the UI on older devices. I need 30 minutes to add background processing and prevent 1-star reviews."

Scenario 2: "Training Our Own Model"

Context: Designer wants to exclude hands from subject bounding box. Engineer suggests training custom CoreML model for specific object detection.

Pressure: "We need perfect bounds, let's train a model."

Reality: Training requires labeled dataset (weeks), ongoing maintenance, and still won't generalize to new objects. Built-in Vision APIs + hand pose solve it in 2-5 hours.

Correct action: 1. Explain Pattern 1 (combine subject mask + hand pose) 2. Prototype in 1 hour to demonstrate 3. Compare against training timeline (weeks vs hours)

Push-back template: "Training a model takes weeks and only works for specific objects. I can combine Vision APIs to solve this in a few hours and it'll work for any object."

Scenario 3: "We Can't Wait for iOS 17"

Context: You need instance masks but app supports iOS 15+.

Pressure: "Just use iOS 15 person segmentation and ship it."

Reality: VNGeneratePersonSegmentationRequest (iOS 15) returns single mask for all people. Doesn't solve multi-person use case.

Correct action: 1. Raise minimum deployment target to iOS 17 (best UX) 2. OR implement fallback: use iOS 15 API but disable multi-person features 3. OR use @available to conditionally enable features

Push-back template: "Person segmentation on iOS 15 combines all people into one mask. We can either require iOS 17 for the best experience, or disable multi-person features on older OS versions. Which do you prefer?"

Checklist

Before shipping Vision features:

Performance:

• ☑ All Vision requests run on background queue
• ☑ UI shows loading indicator during processing
• ☑ Tested on iPhone 12 or earlier (not just latest devices)
• ☑ maximumHandCount set to minimum needed value

Accuracy:

• ☑ Confidence scores checked before using landmarks
• ☑ Fallback behavior for low confidence observations
• ☑ Handles case where no subjects/hands/people detected

Coordinates:

• ☑ Vision coordinates (lower-left origin) converted to UIKit (top-left)
• ☑ Normalized coordinates scaled to pixel dimensions
• ☑ UI overlays aligned correctly with image

Platform Support:

• ☑ @available checks for iOS 17+ APIs (instance masks)
• ☑ Fallback for iOS 14-16 (or raised deployment target)
• ☑ Tested on actual devices, not just simulator

Edge Cases:

• ☑ Handles images with no detectable subjects
• ☑ Handles partially occluded hands/bodies
• ☑ Handles hands/bodies near image edges
• ☑ Handles >4 people for person instance segmentation

CoreImage Integration (if applicable):

• ☑ HDR preservation verified with high dynamic range images
• ☑ Mask resolution matches source image
• ☑ Use generateScaledMask(for:scaledToImageFrom:) for compositing; reserve generateMaskedImage(for:imageFrom:croppedToInstancesExtent:) (with croppedToInstancesExtent: true) for a tight-cropped masked image

Text/Barcode Recognition (if applicable):

• ☑ Recognition level matches use case (fast for real-time, accurate for documents)
• ☑ Language correction disabled for codes/serial numbers
• ☑ Barcode symbologies limited to actual needs (performance)
• ☑ Region of interest used to focus scanning area
• ☑ Multiple candidates checked (not just top candidate)
• ☑ Evidence accumulated over frames for real-time (string tracker)
• ☑ DataScannerViewController availability checked before presenting

Resources

WWDC: 2019-234, 2021-10041, 2022-10024, 2022-10025, 2025-272, 2023-10176, 2023-111241, 2020-10653, 2026-237

Docs: /vision, /visionkit, /vision/vnrecognizetextrequest, /vision/vndetectbarcodesrequest, /vision/generateiterativesegmentationrequest

Skills: skills/vision-ref.md, skills/vision-diag.md

Vision Framework API Reference

Comprehensive reference for Vision framework computer vision: subject segmentation, hand/body pose detection, person detection, face analysis, text recognition (OCR), barcode detection, document scanning, Visual Intelligence integration, and the 27-cycle additions (tap-to-segment, Vision on watchOS, Foundation Models tools).

When to Use This Reference

• Implementing subject lifting using VisionKit or Vision
• Detecting hand/body poses for gesture recognition or fitness apps
• Segmenting people from backgrounds or separating multiple individuals
• Face detection and landmarks for AR effects or authentication
• Combining Vision APIs to solve complex computer vision problems
• Looking up specific API signatures and parameter meanings
• Recognizing text in images (OCR) with VNRecognizeTextRequest
• Detecting barcodes and QR codes with VNDetectBarcodesRequest
• Building live scanners with DataScannerViewController
• Scanning documents with VNDocumentCameraViewController
• Extracting structured document data with RecognizeDocumentsRequest (iOS 26+)
• Integrating with Visual Intelligence — camera/screenshot search surfacing your app's content (iOS 26+, iPadOS27/macOS27)
• Tap-to-segment any object with GenerateIterativeSegmentationRequest OS27
• Using Vision on watchOS watchOS27
• Giving Foundation Models vision tools (BarcodeReaderTool, OCRTool) OS27

Related skills: See skills/vision-framework.md for decision trees and patterns, skills/vision-diag.md for troubleshooting

Vision Framework Overview

Vision provides computer vision algorithms for still images and video:

Core workflow: 1. Create request (e.g., VNDetectHumanHandPoseRequest()) 2. Create handler with image (VNImageRequestHandler(cgImage: image)) 3. Perform request (try handler.perform([request])) 4. Access observations from request.results

Coordinate system: Lower-left origin, normalized (0.0-1.0) coordinates

Performance: Run on background queue - resource intensive, blocks UI if on main thread

Request Handlers

Vision provides two request handlers for different scenarios.

VNImageRequestHandler

Analyzes a single image. Initialize with the image, perform requests against it, discard.

let handler = VNImageRequestHandler(cgImage: image)
try handler.perform([request1, request2])  // Multiple requests, one image

Initialize with: CGImage, CIImage, CVPixelBuffer, Data, or URL

Rule: One handler per image. Reusing a handler with a different image is unsupported.

VNSequenceRequestHandler

Analyzes a sequence of frames (video, camera feed). Initialize empty, pass each frame to perform(). Maintains inter-frame state for temporal smoothing.

let sequenceHandler = VNSequenceRequestHandler()

// In your camera/video frame callback:
func processFrame(_ pixelBuffer: CVPixelBuffer) throws {
    try sequenceHandler.perform([request], on: pixelBuffer)
}

Rule: Create once, reuse across frames. The handler tracks state between calls.

When to Use Which

Requests That Benefit from Sequence Handling

These requests use inter-frame state when run through VNSequenceRequestHandler:

• VNDetectHumanBodyPoseRequest — Smoother joint tracking
• VNDetectHumanHandPoseRequest — Smoother landmark tracking
• VNGeneratePersonSegmentationRequest — Temporally consistent masks
• VNGeneratePersonInstanceMaskRequest — Stable person identity across frames
• VNDetectDocumentSegmentationRequest — Stable document edges
• Any VNStatefulRequest subclass — Designed for sequences

Common Mistake

Creating a new VNImageRequestHandler per video frame discards temporal context. Pose landmarks jitter, segmentation masks flicker, and you lose the smoothing that sequence handling provides.

// Wrong — loses temporal context every frame
func processFrame(_ buffer: CVPixelBuffer) throws {
    let handler = VNImageRequestHandler(cvPixelBuffer: buffer)
    try handler.perform([poseRequest])
}

// Right — maintains inter-frame state
let sequenceHandler = VNSequenceRequestHandler()
func processFrame(_ buffer: CVPixelBuffer) throws {
    try sequenceHandler.perform([poseRequest], on: buffer)
}

Subject Segmentation APIs

VNGenerateForegroundInstanceMaskRequest

Availability: iOS 17+, macOS 14+, tvOS 17+, visionOS 1+

Generates class-agnostic instance mask of foreground objects (people, pets, buildings, food, shoes, etc.)

Basic Usage

let handler = ImageRequestHandler(image)

guard let observation = try await handler.perform(GenerateForegroundInstanceMaskRequest()) else {
    return
}

InstanceMaskObservation

allInstances: IndexSet containing all foreground instance indices (excludes background 0)

allInstancesMask: PixelBufferObservation holding the UInt8 label buffer (0 = background, 1+ = instance indices). Read a single label with pixel(at:) (takes a NormalizedPoint, returns Float) or scan the whole buffer — pre-27 via withUnsafePointer(_:); on 27 via the new pixelBuffer property (CVReadOnlyPixelBuffer, OS27) and pixelBuffer.withUnsafeBuffer, which deprecates withUnsafePointer. There is no mutable CVPixelBuffer accessor.

instanceAtPoint(_:): Takes a NormalizedPoint and returns the IndexSet of instances at that point. iOS 18+ modern InstanceMaskObservation only. (For raw label lookup you can instead use allInstancesMask.pixel(at:) — see below.)

// Center of image; returns an IndexSet (empty = background)
let instances = observation.instanceAtPoint(NormalizedPoint(x: 0.5, y: 0.5))

if instances.isEmpty {
    print("Background tapped")
} else {
    print("Instances \(instances) tapped")
}

Generating Masks

Three methods generate output from the selected instances. All are throwing and require the request handler that performed the request (VNImageRequestHandler/ImageRequestHandler).

generateScaledMask(for:scaledToImageFrom:) → soft segmentation mask

Parameters:

• for: IndexSet of instances to include
• scaledToImageFrom: the request handler that performed the request

Returns: single-channel floating-point CVPixelBuffer (soft mask) at the input image's resolution. No crop option — use it for compositing.

generateMaskedImage(for:imageFrom:croppedToInstancesExtent:) → masked image

Parameters:

• for: IndexSet of instances to include
• imageFrom: the request handler that performed the request
• croppedToInstancesExtent: false (default) = full image; true = tight crop around the selected instances

Returns: the masked image as a CVPixelBuffer, optionally cropped.

generateMask(for:) → handler-free soft mask (modern InstanceMaskObservation only)

Returns a soft mask without needing a handler. For input-resolution scaling use generateScaledMask(for:scaledToImageFrom:) instead.

// All instances, soft mask at input resolution (handler in scope)
let mask = try observation.generateScaledMask(
    for: observation.allInstances,
    scaledToImageFrom: handler
)

// Single instance, masked image cropped to its extent
let instances = IndexSet(integer: 1)
let croppedImage = try observation.generateMaskedImage(
    for: instances,
    imageFrom: handler,
    croppedToInstancesExtent: true
)

Instance Mask Hit Testing

The simplest path is instanceAtPoint(_:), which maps a NormalizedPoint straight to an IndexSet. To read the raw label yourself, the allInstancesMask (PixelBufferObservation) gives you pixel(at:) for a single point; for whole-buffer scans use withUnsafePointer(_:) pre-27, or pixelBuffer.withUnsafeBuffer on 27 (pixelBuffer is the OS27 read-only buffer accessor; withUnsafePointer is deprecated in 27 and renamed to it).

let labelMask = observation.allInstancesMask  // PixelBufferObservation

// Single point: read the label directly (returns Float; 0 = background)
let label = labelMask.pixel(at: NormalizedPoint(x: normalizedX, y: normalizedY))

let instances = label == 0
    ? observation.allInstances
    : IndexSet(integer: Int(label))

// Whole buffer: scan all labels via the unsafe pointer (pre-27;
// on 27 use labelMask.pixelBuffer.withUnsafeBuffer instead)
labelMask.withUnsafePointer { raw in
    let first = raw.load(fromByteOffset: 0, as: UInt8.self)
    // ... iterate label bytes as needed
    _ = first
}

VisionKit Subject Lifting

ImageAnalysisInteraction (iOS)

Availability: iOS 16+, iPadOS 16+

Adds system-like subject lifting UI to views:

let interaction = ImageAnalysisInteraction()
interaction.preferredInteractionTypes = .imageSubject  // Or .automatic
imageView.addInteraction(interaction)

Interaction types:

• .automatic: Subject lifting + Live Text + data detectors
• .imageSubject: Subject lifting only (no interactive text)

ImageAnalysisOverlayView (macOS)

Availability: macOS 13+

let overlayView = ImageAnalysisOverlayView()
overlayView.preferredInteractionTypes = .imageSubject
nsView.addSubview(overlayView)

Programmatic Access

ImageAnalyzer

let analyzer = ImageAnalyzer()
let configuration = ImageAnalyzer.Configuration([.text, .visualLookUp])

let analysis = try await analyzer.analyze(image, configuration: configuration)

ImageAnalysisInteraction (subjects)

Subject APIs live on ImageAnalysisInteraction (a @MainActor UIInteraction), NOT on the ImageAnalysis returned by analyzer.analyze(...). The ImageAnalysis result only exposes transcript and hasResults(for:).

subjects: Set<ImageAnalysisInteraction.Subject> - All subjects in image (async — read with await)

highlightedSubjects: Set<ImageAnalysisInteraction.Subject> - Currently highlighted (user long-pressed)

subject(at:): Async lookup of subject at a point (returns nil if none)

image(for:): Async composite of the given subjects

// Get all subjects (Set — cannot subscript by Int). `subjects` is async.
let subjects = await interaction.subjects

// Look up subject at tap
if let subject = await interaction.subject(at: tapPoint) {
    // Process subject
}

// Change highlight state (take first two safely)
interaction.highlightedSubjects = Set(subjects.prefix(2))

Subject Struct

The type is ImageAnalysisInteraction.Subject (nested on the interaction).

image: UIImage - Extracted subject with transparency. The accessor is async throws — read it with try await.

bounds: CGRect - Subject boundaries in image coordinates (@MainActor-isolated)

// Single subject image (`image` is async throws)
let subjectImage = try await subject.image

// Composite multiple subjects
let compositeImage = try await interaction.image(for: [subject1, subject2])

Out-of-process: VisionKit analysis happens out-of-process (performance benefit, image size limited)

Person Segmentation APIs

VNGeneratePersonSegmentationRequest

Availability: iOS 15+, macOS 12+

Returns single mask containing all people in image:

let request = VNGeneratePersonSegmentationRequest()
// Configure quality level if needed
try handler.perform([request])

guard let observation = request.results?.first as? VNPixelBufferObservation else {
    return
}

let personMask = observation.pixelBuffer  // CVPixelBuffer

VNGeneratePersonInstanceMaskRequest

Availability: iOS 17+, macOS 14+

Returns separate masks for up to 4 people:

let handler = ImageRequestHandler(image)

guard let observation = try await handler.perform(GeneratePersonInstanceMaskRequest()) else {
    return
}

// Same InstanceMaskObservation API as foreground instance masks
let allPeople = observation.allInstances  // Up to 4 people (1-4)

// Get mask for person 1
let person1Mask = try observation.generateScaledMask(
    for: IndexSet(integer: 1),
    scaledToImageFrom: handler
)

Limitations:

• Segments up to 4 people
• With >4 people: may miss people or combine them (typically background people)
• Use VNDetectFaceRectanglesRequest to count faces if you need to handle crowded scenes

Iterative Segmentation (Tap-to-Segment) OS27

GenerateIterativeSegmentationRequest segments any object the user selects — by tap, bounding box, or scribble/lasso — and refines the mask interactively. Modern Swift API; not available on watchOS. Apple ships a tap-to-segment sample app (WWDC 2026-237).

Seeding and Refining

let handler = ImageRequestHandler(image)
let request = GenerateIterativeSegmentationRequest(seedPoint: point)  // NormalizedPoint
let observation = try await handler.perform(request)  // PixelBufferObservation?
let mask = observation?.pixelBuffer  // CVReadOnlyPixelBuffer (or try .cgImage)

// Refine: include the plate, exclude the cup — then perform again
try request.addIncludedPoint(platePoint)
try request.addExcludedPoint(cupPoint)
let refined = try await handler.perform(request)

Seed initializers (each takes an optional trailing Revision argument):

• qualityLevel — .fast / .balanced / .accurate
• Result is PixelBufferObservation? — pixels in the mask belong to the selected object
• Coordinates are normalized (0–1) with lower-left origin — the same conversion gotchas as every Vision request
• Scribble/lasso strokes must be at least 1% of the image width wide; thinner strokes degrade results

Model Download (DownloadableAssetsRequest)

The segmentation model is not on-device by default — first use requires a download. GenerateIterativeSegmentationRequest is the first request conforming to the new DownloadableAssetsRequest protocol (OS27):

switch await request.assetStatus {  // DownloadableAssetsRequestStatus
case .notReady:
    try await request.downloadAssets()  // or downloadAssets(progress: consuming Subprogress)
case .ready:
    break
case .error(let error):
    throw error  // surface or retry the download
@unknown default:
    break
}

Performing without the model downloaded fails; the legacy VNError domain adds VNErrorResourceUnavailable and VNErrorResourceCorrupted codes in 27 for asset failures. Check assetStatus before the first perform.

Hand Pose Detection

VNDetectHumanHandPoseRequest

Availability: iOS 14+, macOS 11+

Detects 21 hand landmarks per hand:

let request = VNDetectHumanHandPoseRequest()
request.maximumHandCount = 2  // Default: 2, increase if needed

let handler = VNImageRequestHandler(cgImage: image)
try handler.perform([request])

for observation in request.results as? [VNHumanHandPoseObservation] ?? [] {
    // Process each hand
}

Performance note: maximumHandCount affects latency. Pose computed only for hands ≤ maximum. Set to lowest acceptable value.

Hand Landmarks (21 points)

Wrist: 1 landmark

Thumb (4 landmarks):

• .thumbTip
• .thumbIP (interphalangeal joint)
• .thumbMP (metacarpophalangeal joint)
• .thumbCMC (carpometacarpal joint)

Fingers (4 landmarks each):

• Tip (.indexTip, .middleTip, .ringTip, .littleTip)
• DIP (distal interphalangeal joint)
• PIP (proximal interphalangeal joint)
• MCP (metacarpophalangeal joint)

Group Keys

Access landmark groups:

// Get all points
let allPoints = try observation.recognizedPoints(.all)

// Get index finger points only
let indexPoints = try observation.recognizedPoints(.indexFinger)

// Get specific point
let thumbTip = try observation.recognizedPoint(.thumbTip)
let indexTip = try observation.recognizedPoint(.indexTip)

// Check confidence
guard thumbTip.confidence > 0.5 else { return }

// Access location (normalized coordinates, lower-left origin)
let location = thumbTip.location  // CGPoint

Gesture Recognition Example (Pinch)

let thumbTip = try observation.recognizedPoint(.thumbTip)
let indexTip = try observation.recognizedPoint(.indexTip)

guard thumbTip.confidence > 0.5, indexTip.confidence > 0.5 else {
    return
}

let distance = hypot(
    thumbTip.location.x - indexTip.location.x,
    thumbTip.location.y - indexTip.location.y
)

let isPinching = distance < 0.05  // Normalized threshold

Chirality (Handedness)

let chirality = observation.chirality  // .left or .right or .unknown

Body Pose Detection

VNDetectHumanBodyPoseRequest (2D)

Availability: iOS 14+, macOS 11+

Detects 19 body landmarks (2D normalized coordinates):

let request = VNDetectHumanBodyPoseRequest()
try handler.perform([request])

for observation in request.results as? [VNHumanBodyPoseObservation] ?? [] {
    // Process each person
}

Body Landmarks (19 points)

Face (5 landmarks):

• .nose, .leftEye, .rightEye, .leftEar, .rightEar

Arms (6 landmarks):

• Left: .leftShoulder, .leftElbow, .leftWrist
• Right: .rightShoulder, .rightElbow, .rightWrist

Torso (7 landmarks):

• .neck (between shoulders)
• .leftShoulder, .rightShoulder (also in arm groups)
• .leftHip, .rightHip
• .root (between hips)

Legs (6 landmarks):

• Left: .leftHip, .leftKnee, .leftAnkle
• Right: .rightHip, .rightKnee, .rightAnkle

Note: Shoulders and hips appear in multiple groups

Group Keys (Body)

// Get all body points
let allPoints = try observation.recognizedPoints(.all)

// Get left arm only
let leftArmPoints = try observation.recognizedPoints(.leftArm)

// Get specific joint
let leftWrist = try observation.recognizedPoint(.leftWrist)

VNDetectHumanBodyPose3DRequest (3D)

Availability: iOS 17+, macOS 14+

Returns 3D skeleton with 17 joints in meters (real-world coordinates):

let request = VNDetectHumanBodyPose3DRequest()
try handler.perform([request])

guard let observation = request.results?.first as? VNHumanBodyPose3DObservation else {
    return
}

// Get 3D joint position
let leftWrist = try observation.recognizedPoint(.leftWrist)
let position = leftWrist.position  // simd_float4x4 matrix
let localPosition = leftWrist.localPosition  // Relative to parent joint

3D Body Landmarks (17 joints, VNHumanBodyPose3DObservation.JointName): root, spine, centerShoulder, centerHead, topHead, plus left/right shoulder, elbow, wrist, hip, knee, ankle. The 3D skeleton is NOT the 2D set minus ears — it omits the 2D face/head joints (nose, eyes, ears, neck) and adds spine, centerShoulder, centerHead, topHead. (2D set = 19 joints via VNHumanBodyPoseObservation.JointName.)

3D Observation Properties

bodyHeight: Estimated height in meters

• With depth data: Measured height
• Without depth data: Reference height (1.8m)

heightEstimation: .measured or .reference

cameraOriginMatrix: simd_float4x4 camera position/orientation relative to subject

pointInImage(\_:): Project 3D joint back to 2D image coordinates

let wrist2D = try observation.pointInImage(leftWrist)

3D Point Classes

VNPoint3D: Base class with simd_float4x4 position matrix

VNRecognizedPoint3D: Adds identifier (joint name)

VNHumanBodyRecognizedPoint3D: Adds localPosition and parentJoint

// Position relative to skeleton root (center of hip)
let modelPosition = leftWrist.position

// Position relative to parent joint (left elbow)
let relativePosition = leftWrist.localPosition

Depth Input

Vision accepts depth data alongside images:

// From AVDepthData
let handler = VNImageRequestHandler(
    cvPixelBuffer: imageBuffer,
    depthData: depthData,
    orientation: orientation
)

// From file (automatic depth extraction)
let handler = VNImageRequestHandler(url: imageURL)  // Depth auto-fetched

Depth formats: Disparity or Depth (interchangeable via AVFoundation)

LiDAR: Use in live capture sessions for accurate scale/measurement

Face Detection & Landmarks

VNDetectFaceRectanglesRequest

Availability: iOS 11+

Detects face bounding boxes:

let request = VNDetectFaceRectanglesRequest()
try handler.perform([request])

for observation in request.results as? [VNFaceObservation] ?? [] {
    let faceBounds = observation.boundingBox  // Normalized rect
}

VNDetectFaceLandmarksRequest

Availability: iOS 11+

Detects face with detailed landmarks:

let request = VNDetectFaceLandmarksRequest()
try handler.perform([request])

for observation in request.results as? [VNFaceObservation] ?? [] {
    if let landmarks = observation.landmarks {
        let leftEye = landmarks.leftEye
        let nose = landmarks.nose
        let leftPupil = landmarks.leftPupil  // Revision 2+
    }
}

Revisions:

• Revision 1: Basic landmarks
• Revision 2: Detects upside-down faces
• Revision 3+: Pupil locations

Person Detection

VNDetectHumanRectanglesRequest

Availability: iOS 13+

Detects human bounding boxes (torso detection):

let request = VNDetectHumanRectanglesRequest()
try handler.perform([request])

for observation in request.results as? [VNHumanObservation] ?? [] {
    let humanBounds = observation.boundingBox  // Normalized rect
}

Use case: Faster than pose detection when you only need location

CoreImage Integration

CIBlendWithMask Filter

Composite subject on new background using Vision mask:

// 1. Get mask from Vision (`handler` is the ImageRequestHandler that performed the request)
let handler = ImageRequestHandler(sourceImage)
guard let observation = try await handler.perform(GenerateForegroundInstanceMaskRequest()) else { return }
let visionMask = try observation.generateScaledMask(
    for: observation.allInstances,
    scaledToImageFrom: handler
)

// 2. Convert to CIImage
let maskImage = CIImage(cvPixelBuffer: visionMask)

// 3. Apply filter
let filter = CIFilter(name: "CIBlendWithMask")!
filter.setValue(sourceImage, forKey: kCIInputImageKey)
filter.setValue(maskImage, forKey: kCIInputMaskImageKey)
filter.setValue(newBackground, forKey: kCIInputBackgroundImageKey)

let output = filter.outputImage  // Composited result

Parameters:

• Input image: Original image to mask
• Mask image: Vision's soft segmentation mask
• Background image: New background (or empty image for transparency)

HDR preservation: CoreImage preserves high dynamic range from input (Vision/VisionKit output is SDR)

Text Recognition APIs

VNRecognizeTextRequest

Availability: iOS 13+, macOS 10.15+

Recognizes text in images with configurable accuracy/speed trade-off.

Basic Usage

let request = VNRecognizeTextRequest()
request.recognitionLevel = .accurate  // Or .fast
request.recognitionLanguages = ["en-US", "de-DE"]  // Order matters
request.usesLanguageCorrection = true

let handler = VNImageRequestHandler(cgImage: image)
try handler.perform([request])

for observation in request.results as? [VNRecognizedTextObservation] ?? [] {
    // Get top candidates
    let candidates = observation.topCandidates(3)
    let bestText = candidates.first?.string ?? ""
}

Recognition Levels

Fast path: Character-by-character recognition (Neural Network → Character Detection)

Accurate path: Full-line ML recognition (Neural Network → Line/Word Recognition)

Properties

Language Support

// Check supported languages for current settings
let languages = try VNRecognizeTextRequest.supportedRecognitionLanguages(
    for: .accurate,
    revision: VNRecognizeTextRequestRevision3
)

Language correction: Improves accuracy but takes processing time. Disable for codes/serial numbers.

Custom words: Add domain-specific vocabulary for better recognition (medical terms, product codes).

VNRecognizedTextObservation

boundingBox: Normalized rect containing recognized text

topCandidates(_:): Returns [VNRecognizedText] ordered by confidence

VNRecognizedText

// Get bounding box for substring
let text = candidate.string
if let range = text.range(of: "invoice") {
    let box = try candidate.boundingBox(for: range)
}

Barcode Detection APIs

VNDetectBarcodesRequest

Availability: iOS 11+, macOS 10.13+

Detects and decodes barcodes and QR codes.

Basic Usage

let request = VNDetectBarcodesRequest()
request.symbologies = [.qr, .ean13, .code128]  // Specific codes

let handler = VNImageRequestHandler(cgImage: image)
try handler.perform([request])

for barcode in request.results as? [VNBarcodeObservation] ?? [] {
    let payload = barcode.payloadStringValue
    let type = barcode.symbology
    let bounds = barcode.boundingBox
}

Symbologies

1D Barcodes:

• .codabar (iOS 15+)
• .code39, .code39Checksum, .code39FullASCII, .code39FullASCIIChecksum
• .code93, .code93i
• .code128
• .ean8, .ean13
• .gs1DataBar, .gs1DataBarExpanded, .gs1DataBarLimited (iOS 15+)
• .i2of5, .i2of5Checksum
• .itf14
• .upce

2D Codes:

• .aztec
• .dataMatrix
• .microPDF417 (iOS 15+)
• .microQR (iOS 15+)
• .pdf417
• .qr

Performance: Specifying fewer symbologies = faster detection

Revisions

VNBarcodeObservation

VisionKit Scanner APIs

DataScannerViewController

Availability: iOS 16+

Camera-based live scanner with built-in UI for text and barcodes.

Check Availability

// Hardware support
DataScannerViewController.isSupported

// Runtime availability (camera access, parental controls)
DataScannerViewController.isAvailable

Configuration

import VisionKit

let dataTypes: Set<DataScannerViewController.RecognizedDataType> = [
    .barcode(symbologies: [.qr, .ean13]),
    .text(textContentType: .URL),  // Or nil for all text
    // .text(languages: ["ja"])  // Filter by language
]

let scanner = DataScannerViewController(
    recognizedDataTypes: dataTypes,
    qualityLevel: .balanced,  // .fast, .balanced, .accurate
    recognizesMultipleItems: true,
    isHighFrameRateTrackingEnabled: true,
    isPinchToZoomEnabled: true,
    isGuidanceEnabled: true,
    isHighlightingEnabled: true
)

scanner.delegate = self
present(scanner, animated: true) {
    try? scanner.startScanning()
}

RecognizedDataType

Delegate Protocol

protocol DataScannerViewControllerDelegate {
    func dataScanner(_ dataScanner: DataScannerViewController,
                     didTapOn item: RecognizedItem)

    func dataScanner(_ dataScanner: DataScannerViewController,
                     didAdd addedItems: [RecognizedItem],
                     allItems: [RecognizedItem])

    func dataScanner(_ dataScanner: DataScannerViewController,
                     didUpdate updatedItems: [RecognizedItem],
                     allItems: [RecognizedItem])

    func dataScanner(_ dataScanner: DataScannerViewController,
                     didRemove removedItems: [RecognizedItem],
                     allItems: [RecognizedItem])

    func dataScanner(_ dataScanner: DataScannerViewController,
                     becameUnavailableWithError error: DataScannerViewController.ScanningUnavailable)
}

RecognizedItem

enum RecognizedItem {
    case text(RecognizedItem.Text)
    case barcode(RecognizedItem.Barcode)

    var id: UUID { get }
    var bounds: RecognizedItem.Bounds { get }
}

// Text item
struct Text {
    let transcript: String
}

// Barcode item
struct Barcode {
    let payloadStringValue: String?
    let observation: VNBarcodeObservation
}

Async Stream

// Alternative to delegate
for await items in scanner.recognizedItems {
    // Current recognized items
}

Custom Highlights

// Add custom views over recognized items
scanner.overlayContainerView.addSubview(customHighlight)

// Capture still photo
let photo = try await scanner.capturePhoto()

VNDocumentCameraViewController

Availability: iOS 13+

Document scanning with automatic edge detection, perspective correction, and lighting adjustment.

Basic Usage

import VisionKit

let camera = VNDocumentCameraViewController()
camera.delegate = self
present(camera, animated: true)

Delegate Protocol

protocol VNDocumentCameraViewControllerDelegate {
    func documentCameraViewController(_ controller: VNDocumentCameraViewController,
                                       didFinishWith scan: VNDocumentCameraScan)

    func documentCameraViewControllerDidCancel(_ controller: VNDocumentCameraViewController)

    func documentCameraViewController(_ controller: VNDocumentCameraViewController,
                                       didFailWithError error: Error)
}

VNDocumentCameraScan

func documentCameraViewController(_ controller: VNDocumentCameraViewController,
                                   didFinishWith scan: VNDocumentCameraScan) {
    controller.dismiss(animated: true)

    for i in 0..<scan.pageCount {
        let pageImage = scan.imageOfPage(at: i)
        // Process with VNRecognizeTextRequest
    }
}

Document Analysis APIs

VNDetectDocumentSegmentationRequest

Availability: iOS 15+, macOS 12+

Detects document boundaries for custom camera UIs or post-processing.

let request = VNDetectDocumentSegmentationRequest()
let handler = VNImageRequestHandler(ciImage: image)
try handler.perform([request])

guard let observation = request.results?.first as? VNRectangleObservation else {
    return  // No document found
}

// Get corner points (normalized)
let corners = [
    observation.topLeft,
    observation.topRight,
    observation.bottomLeft,
    observation.bottomRight
]

vs VNDetectRectanglesRequest:

• Document: ML-based, trained specifically on documents
• Rectangle: Edge-based, finds any quadrilateral

RecognizeDocumentsRequest (iOS 26+)

Availability: iOS 26+, macOS 26+

Structured document understanding with semantic parsing.

Basic Usage

let request = RecognizeDocumentsRequest()
let observations = try await request.perform(on: imageData)

guard let document = observations.first?.document else {
    return
}

DocumentObservation Hierarchy

DocumentObservation
└── document: DocumentObservation.Container
    ├── text: Container.Text
    ├── paragraphs: [Container.Text]
    ├── tables: [Container.Table]
    └── lists: [Container.List]

Table Extraction

for table in document.tables {
    for row in table.rows {
        for cell in row {
            let text = cell.content.text.transcript
            let detectedData = cell.content.text.detectedData
        }
    }
}

Detected Data Types

for data in document.text.detectedData {
    switch data.match.details {
    case .emailAddress(let email):
        let address = email.emailAddress
    case .phoneNumber(let phone):
        let number = phone.phoneNumber
    case .link(let link):
        let url = link.url
    case .postalAddress(let address):
        let components = address
    case .calendarEvent(let event):
        let dates = (event.startDate, event.endDate)
    default:
        break
    }
}

Container.Text Hierarchy

Container.Text
├── transcript: String
├── lines: [RecognizedTextObservation]
├── words: [RecognizedTextObservation]?
└── detectedData: [DataDetectorMatch]

Visual Intelligence Integration

Visual Intelligence is a system-level feature (iOS 26+; expands to iPadOS27/macOS27) that lets users point their camera at real-world objects — or highlight a screenshot — and find matching content across apps. This is distinct from the Vision framework (VNRequest-based image analysis) covered above. Vision analyzes images within your app; Visual Intelligence lets the system invoke your app when users search with the camera or screenshots.

The same IntentValueQuery, entities, and OpenIntent code works unchanged across iOS, iPadOS, and macOS. Platform differences worth handling:

• iOS — primary entry point is the camera (physical objects: posters, products, artwork)
• iPad/Mac — primary entry point is screenshots (digital media); make sure your search handles both kinds of content
• Mac — the input pixel buffer can be much larger than on iPhone; consider resizing before analysis

How It Works

1. User activates Visual Intelligence camera or takes a screenshot 2. System analyzes what the user is looking at 3. System queries participating apps via IntentValueQuery 4. Your app receives a SemanticContentDescriptor with labels and/or pixel data 5. Your app searches its content and returns matching AppEntity results 6. Results appear in the Visual Intelligence UI with your app's branding

Required Frameworks

import VisualIntelligence
import AppIntents

SemanticContentDescriptor

The core object the system provides to describe what the user is looking at.

Use labels for fast keyword matching against your content catalog. Use the pixel buffer for image-similarity search when labels are insufficient.

Matching by Image Similarity (Feature Prints)

For visual matching against your own catalog, compare Vision feature prints. Pre-compute prints for catalog items (never at query time); at query time, convert the descriptor's pixel buffer to a CGImage and rank by distance:

import Vision
import VideoToolbox

var cgImage: CGImage?
_ = pixelBuffer.withUnsafeBuffer {
    VTCreateCGImageFromCVPixelBuffer($0, options: nil, imageOut: &cgImage)
}
guard let cgImage else { return [] }

let queryPrint = try await GenerateImageFeaturePrintRequest().perform(on: cgImage)
let distance = try queryPrint.distance(to: entry.featurePrint)  // pre-computed FeaturePrintObservation; smaller = more similar

Filter by a maximum distance, sort ascending, and cap the result count — return results fast and ranked. Returning an empty array is fine; the system handles the empty state.

IntentValueQuery

The entry point for Visual Intelligence to communicate with your app. Implement values(for:) to receive search requests and return matching entities.

struct LandmarkIntentValueQuery: IntentValueQuery {
    @Dependency var modelData: ModelData

    func values(for input: SemanticContentDescriptor) async throws -> [LandmarkEntity] {
        if !input.labels.isEmpty {
            return try await modelData.search(matching: input.labels)
        }
        guard let pixelBuffer = input.pixelBuffer else { return [] }
        return try await modelData.search(matching: pixelBuffer)
    }
}

Returning Multiple Result Types

Use @UnionValue when your app can return different entity types from a single search.

@UnionValue
enum VisualSearchResult {
    case landmark(LandmarkEntity)
    case collection(CollectionEntity)
}

Display Representation

Visual Intelligence uses your entity's DisplayRepresentation to show results. Provide a title, subtitle, and image for each result.

struct LandmarkEntity: AppEntity {
    var id: String
    var name: String
    var location: String

    static var typeDisplayRepresentation: TypeDisplayRepresentation {
        TypeDisplayRepresentation(
            name: LocalizedStringResource("Landmark", table: "AppIntents"),
            numericFormat: "\(placeholder: .int) landmarks"
        )
    }

    var displayRepresentation: DisplayRepresentation {
        DisplayRepresentation(
            title: "\(name)",
            subtitle: "\(location)",
            image: .init(named: thumbnailImageName)
        )
    }
}

Deep Linking from Results

When a user taps a result, your app should open to the relevant content. Provide an appLinkURL on your entity.

var appLinkURL: URL? {
    URL(string: "yourapp://landmark/\(id)")
}

"More Results" — Continue Search in Your App

Adopt the semanticContentSearch schema so the system's More results button lands users in your full in-app search, pre-populated from the captured context. The system provides the semanticContent property automatically — no @Parameter needed:

@AppIntent(schema: .visualIntelligence.semanticContentSearch)
struct SemanticContentSearchIntent: AppIntent {
    static let title: LocalizedStringResource = "Search in app"
    static let openAppWhenRun: Bool = true

    var semanticContent: SemanticContentDescriptor

    func perform() async throws -> some IntentResult {
        guard let pixelBuffer = semanticContent.pixelBuffer else { return .result() }
        // Search, then navigate to your full search UI pre-populated with results
        return .result()
    }
}

Your in-app search can show far more than the Visual Intelligence results sheet — filters, categories, the full depth of your content.

System Store Integrations

Image Search is your app providing results to Visual Intelligence. The reverse also works: Visual Intelligence actions write data to system stores your app may already read — making Visual Intelligence a new input source with zero Visual Intelligence code. New in the 27 cycle: adding to contacts, saving multiple calendar events, and medical-device logging (WWDC 2026-297).

Observe store-change notifications (e.g. .EKEventStoreChanged) so entries created by Visual Intelligence appear in your app without a manual refresh.

Best Practices

• Return results quickly — Visual Intelligence expects low-latency responses. Limit to 10-20 most relevant results; returning an empty array is fine
• Prefer labels first — Label matching is faster than pixel buffer analysis. Fall back to pixel buffer when labels are empty or insufficient
• Serve thumbnail-sized images — the results sheet shows ~3 lines of text + a thumbnail in a two-column layout; a single result's image spans the full sheet width
• Reuse your existing `OpenIntent` — if you already have one for the entity, Visual Intelligence uses it; don't write a separate one
• Keep `perform()` lightweight — it runs as your app comes to the foreground; navigate first, defer heavy loading until the view appears
• Localize everything — Display representations appear in the system UI. Use LocalizedStringResource for all user-facing text

Testing

1. Build and run on a physical device (iPhone, iPad, or Mac) 2. Activate Visual Intelligence camera or take a screenshot of relevant content 3. Perform a visual search and verify your app's results appear (ordering among providers is decided by the system) 4. Tap results to verify deep linking opens the correct content 5. On Mac, test with large screenshots — input pixel buffers are bigger than iPhone's

Vision on watchOS watchOS27

Vision arrives on watchOS in the 27 cycle — the framework does not exist in earlier watchOS SDKs. The watch gets the modern Swift API only (no legacy VN* request classes) and a subset of requests:

Canonical watch use case from WWDC 2026-237 — saliency-based cropping so small screens feature the subject prominently:

// Crop to the most prominent subject for a small watch screen
let request = GenerateObjectnessBasedSaliencyImageRequest()
let observation = try await request.perform(on: image)  // SaliencyImageObservation
let crop = observation.salientObjects.first?.boundingBox  // NormalizedRect

salientObjects is [RectangleObservation] — take .boundingBox (or the quadrilateral corners) for the crop rect.

Vision Tools for Foundation Models OS27

Vision ships two ready-made FoundationModels.Tool implementations via a cross-import overlay (import Vision + import FoundationModels) so the on-device LLM can call computer vision on attached images:

import FoundationModels
import Vision

let session = LanguageModelSession(tools: [BarcodeReaderTool()])
let response = try await session.respond(generating: EventInfo.self) {  // EventInfo: your @Generable struct
    "Get the date, location, and website from this flyer"
    Attachment(image)
        .label("flyer")  // labels are how the model picks which image to pass to a tool
}

Both tools accept optional init(name:description:) overrides. For image inputs, ImageReference tool arguments, and the rest of the Foundation Models surface, see axiom-ai foundation-models-ref.md (Built-in System Tools).

API Quick Reference

Subject Segmentation

Pose Detection

Face & Person Detection

Text & Barcode

Visual Intelligence

Observation Types

Sensitive Content Analysis (SensitiveContentAnalysis framework)

A separate framework from Vision (and VisionKit), but the same job family — SensitiveContentAnalysis (iOS 17+, macOS 14+, visionOS 2+; not watchOS/tvOS) flags nudity, gore, and violence in images and video. It runs only when the user has the system Sensitive Content Warning / Communication Safety setting on: check analysisPolicy first, and treat .disabled as "feature off," not an error.

import SensitiveContentAnalysis

let analyzer = SCSensitivityAnalyzer()
guard analyzer.analysisPolicy != .disabled else { return }   // user setting gates analysis

let result = try await analyzer.analyzeImage(cgImage)         // -> SCSensitivityAnalysis
if result.isSensitive {
    if #available(iOS 27, macOS 27, visionOS 27, *) {
        let kinds = result.detectedTypes                     // Set<SCSensitivityAnalysis.ContentType>
        if kinds.contains(.goreOrViolence) { /* gore-specific UX */ }
        if kinds.contains(.sexuallyExplicit) { /* explicit-specific UX */ }
    }
}

`OS27` upgrade — categorized results. Before 27 you got only the boolean isSensitive. At 27 (iOS27/macOS27/visionOS27, not watchOS/tvOS) detectedTypes reports which kind of sensitive content was found, so you can branch handling (different messaging for gore vs. explicit). Guard it with #available and fall back to the boolean on earlier targets.

Resources

WWDC: 2019-234, 2021-10041, 2022-10024, 2022-10025, 2025-272, 2023-10176, 2023-111241, 2023-10048, 2020-10653, 2020-10043, 2020-10099, 2026-237, 2026-297

Docs: /vision, /visionkit, /visualintelligence, /visualintelligence/semanticcontentdescriptor, /visualintelligence/integrating-your-app-with-visual-intelligence, /vision/generateiterativesegmentationrequest, /vision/vnrecognizetextrequest, /vision/vndetectbarcodesrequest, /sensitivecontentanalysis

Skills: skills/vision-framework.md, skills/vision-diag.md, axiom-ai (skills/foundation-models-ref.md)