Serve Sim

serve-sim

Drive an Apple Simulator (iOS, iPad, Apple Watch) from an agent using the serve-sim CLI. serve-sim spawns a Swift helper that captures the simulator framebuffer via simctl io, exposes it as an MJPEG stream plus a binary WebSocket input channel, and serves a React preview UI on top. This skill teaches an agent the exact CLI surface, the gesture JSON shape, the gotchas, and the recommended workflows.

When to use

• The user wants an agent to tap, swipe, drag, pinch, or send hardware buttons to a running Apple Simulator.
• The user wants to stream a simulator to a browser (local, LAN, or tunneled) for review or remote control.
• The user wants to inject a synthetic camera feed (file, webcam, or animated placeholder) into a specific app on the simulator.
• The user wants to toggle CoreAnimation debug overlays (off-screen rendering, blended layers, slow animations) for performance work.
• The user wants to simulate a memory warning or rotate the device programmatically.
• The user wants to read the simulator's accessibility tree to find UI elements without pixel hunting.
• The user wants to grant, revoke, or reset an app's privacy permissions — camera, photos, location, contacts, or push notifications.

When NOT to use

• Android emulators → use adb shell tooling.
• Building or installing an iOS app → use xcodebuild or xcrun simctl install.
• React Native in-app runtime debugging (Redux state, network inspection, component tree) → use rn-debugger tooling.
• Real iOS hardware devices → use xcrun devicectl or Xcode.

Prerequisites

Before any other action, verify the host satisfies these. If something is missing, tell the user exactly what to install — do not proceed.

A bundled helper script is available: scripts/check-prereqs.sh. Run it; if it exits non-zero, surface the message to the user.

A booted simulator is required for most subcommands. Check with xcrun simctl list devices booted. If none are booted, tell the user to open Xcode → Simulator or to run xcrun simctl boot <UDID>.

Mental model

┌──────────────┐  simctl io  ┌─────────────────┐  MJPEG / WS  ┌─────────┐
│ iOS Simulator│ ──────────► │ serve-sim-bin   │ ───────────► │ Browser │
└──────────────┘   (Swift)   │ (per-device)    │              └─────────┘
                             └─────────────────┘
                                     ▲
                              state file in
                            $TMPDIR/serve-sim/
                                     ▲
                            ┌──────────────────┐
                            │ serve-sim CLI    │
                            └──────────────────┘

Key invariants the agent must respect:

• All coordinates are normalized 0..1, with (0, 0) at top-left and (1, 1) at bottom-right of the display. Never pass pixel coordinates.
• One helper per device. Multiple booted simulators are supported by passing several device names or by attaching to all.
• State lives in `$TMPDIR/serve-sim/server-{udid}.json`. Use serve-sim --list to query it; do not read the JSON directly unless you know what you are doing.
• The orientation set via `rotate` is remembered by the helper, and subsequent gestures are rotated client-side. An agent that sends raw coords after a rotation does not need to compensate manually.

Common operations

Most subcommands accept -d <udid|name> to target a specific device when several are booted.

Critical gotcha: prefer tap over gesture for taps

Each serve-sim gesture call opens its own WebSocket. If you issue two back-to-back gesture calls — one with {"type":"begin",...} and one with {"type":"end",...} — the simulator receives them with enough latency between them that the touch is interpreted as a long-press, not a tap. This is a deliberate constraint of the protocol, not a bug to work around.

Rule: for any single-shot tap, use serve-sim tap <x> <y>. Only use gesture for drags, swipes, or multi-step interactions where you must thread the same socket across begin → move × N → end.

Targeting a specific device

When multiple simulators are booted, every subcommand accepts -d <udid|name>. The name match is case-insensitive against the device name returned by xcrun simctl list devices booted. Examples:

npx serve-sim tap 0.5 0.5 -d "iPhone 16 Pro"
npx serve-sim button home -d ABC12345-...
npx serve-sim --list                                # show all running streams

If the user has only one booted simulator, omit -d entirely. The skill should prefer auto-detection over hard-coding device names.

Output modes

By default, serve-sim prints human-readable status to stdout. For agent loops, prefer JSON output:

npx serve-sim --list -q          # JSON array of running streams
npx serve-sim --detach -q        # JSON with pid/port/url after spawn
npx serve-sim camera status -q   # JSON with {alive, source, mirror, ...}

Parse -q output programmatically. Never parse the non--q human output — it can change between versions.

Showing the stream in your agent's preview

When the user asks to "see the simulator here", "view it in preview", "open it in this tool", or similar, the goal is to stream the simulator into the same surface the user is chatting with. serve-sim returns a regular HTTP URL — the agent's job is to surface that URL and, if the host exposes a preview tool, hand it off.

Steps:

1. Start serve-sim and capture the URL:

   npx serve-sim --detach -q

This returns JSON like {"pid":..., "port":3200, "url":"http://localhost:3200", "streamUrl":"http://localhost:3100", ...}. The url field is the human-facing preview UI; streamUrl is the raw MJPEG endpoint.

2. Always surface the URL plainly in your response so the user can fallback to opening it manually in any browser.

3. Probe your host's preview tool and hand off the URL if one exists. Examples of tool names you may see in your toolset:

• preview_start (Claude Code) — call it with { url: "<url>" }.
• mcp__Claude_Preview__preview_start (some MCP setups).
• A browser_open, open_url, or similar URL-opening tool — pass the URL.
• Cursor / Codex CLI / others may not expose a preview tool to the agent. In that case, just print the URL and tell the user how to open it (their browser, their IDE's built-in browser pane, etc.).

4. Do not assume any specific preview tool exists. Inspect the tools available to you in the current session. If one matches the description above, use it. If none does, fall back to step 2 (print the URL prominently).

The stream stays alive until npx serve-sim --kill. Multiple clients (the host's preview + the user's browser + a tunnel) can read the same URL simultaneously.

See references/workflows.md workflow "Show the simulator stream in the host's preview" for the full recipe.

Workflows

For complete end-to-end recipes (UI automation, camera testing, accessibility-driven taps, deep-link flows, preview handoff), see references/workflows.md. The reference covers the patterns documented in serve-sim's own AGENTS.md.

Cleanup

Always stop helpers when finished, unless the user explicitly wants them to keep running:

npx serve-sim --kill            # stop all
npx serve-sim --kill "iPhone 16 Pro"  # stop one

Orphan helpers occupy ports 3200/3100 and prevent fresh starts.

Anti-patterns

• Do not pass pixel coordinates. All coords are normalized 0..1. If the user gives pixel values, divide by the screen dimensions reported by GET /config.
• Do not use `gesture` for plain taps. Use tap. See "Critical gotcha" above.
• Do not assume `npx serve-sim` is already running. Verify with --list or by checking $TMPDIR/serve-sim/server-{udid}.json. If absent, start it explicitly.
• Do not skip the prerequisites check on the first invocation in a session. Wrong macOS version, missing Xcode CLI tools, or Node <18 produce confusing errors downstream.
• Do not invent button names. Only these six are valid: home, swipe_home, app_switcher, lock, siri, side_button. See references/buttons-rotation.md for the source-of-truth list.
• Do not parse the non-quiet human output. Use -q for JSON.
• Do not leave camera helpers running across unrelated tasks. Stop them with npx serve-sim camera --stop-webcam when done.
• Do not guess coordinates when an accessibility lookup returns no match. If you fetched the AX tree (e.g. GET /ax) to find a target element and the query returned no result, fail loudly — tapping a guessed spot is almost always worse than reporting "target not found" back to the user. See references/workflows.md workflow 1 for the guard pattern.

Reference index

• references/gestures.md — exact gesture JSON shapes, edge values, multi-touch, drag/swipe recipes.
• references/buttons-rotation.md — the six valid buttons and the four orientations, with behavioral notes.
• references/camera.md — synthetic camera injection: placeholder, file, webcam, mirror modes, hot-swap.
• references/permissions.md — granting/revoking app privacy permissions, including push notifications.
• references/ca-debug.md — the five CoreAnimation debug flags and when each one helps.
• references/endpoints.md — HTTP and WebSocket endpoints for agents that bypass the CLI.
• references/workflows.md — end-to-end recipes for UI automation, camera testing, deep-link flows.

serve-sim agent skill

A portable Agent Skill that teaches AI coding agents to drive an Apple Simulator via the serve-sim CLI.

Works in Claude Code, Cursor, Codex CLI, Gemini CLI, GitHub Copilot, and any other tool that implements the open Agent Skills standard. The same SKILL.md works across all of them without modification.

What it does

Once installed, your agent knows how to:

• Tap at normalized coordinates (serve-sim tap).
• Send multi-touch / drag / swipe gestures with the correct JSON shape and edge flags.
• Press the six valid hardware buttons (home, swipe_home, app_switcher, lock, siri, side_button).
• Rotate the simulator (portrait, portrait_upside_down, landscape_left, landscape_right).
• Inject a synthetic camera feed (placeholder, image, video, or live webcam) with mirror-mode control.
• Toggle CoreAnimation debug overlays (blended layers, off-screen rendering, slow animations, …).
• Simulate a memory warning.
• Discover the running stream's URL and read the simulator's accessibility tree to find UI elements.
• Hand the stream URL off to the host agent's preview pane (preview_start in Claude Code, equivalents elsewhere) so the user sees the simulator inline.

It also teaches the agent the gotchas (use tap, not gesture, for plain taps), the prerequisites (macOS, Xcode CLI tools, Node 18+, macOS 14+ for camera), and anti-patterns to avoid.

Install

The skill lives in this repo under skills/serve-sim/, so it is discoverable by the Agent Skills tooling directly from the serve-sim repository.

Claude Code

/plugin marketplace add EvanBacon/serve-sim
/plugin install serve-sim

Any agent that supports the Agent Skills standard (Cursor, Codex CLI, Gemini CLI, …)

bunx add-skill EvanBacon/serve-sim
# or
npx skills add EvanBacon/serve-sim

Manual install

Copy this folder into your agent's skills directory:

# from a clone of this repo
cp -r skills/serve-sim ~/.claude/skills/serve-sim
# or for other agents: ~/.agents/skills/serve-sim, ~/.cursor/skills/serve-sim, etc.

The skill is a folder with a SKILL.md file plus reference documents. No build step.

Prerequisites on the user's machine

The agent checks these for you, but for reference:

• macOS host (any recent version).
• Xcode command line tools (xcode-select --install).
• Node.js 18+.
• macOS 14+ if you want camera injection.
• At least one booted iOS, iPad, or Apple Watch simulator.

serve-sim itself is invoked via npx serve-sim — no global install required.

How it's structured

serve-sim/
├── SKILL.md                          (loaded when the skill triggers)
├── references/
│   ├── gestures.md                   (gesture JSON, edges, multi-touch, recipes)
│   ├── buttons-rotation.md           (the six buttons, the four orientations)
│   ├── camera.md                     (camera injection: sources, mirroring, hot-swap)
│   ├── ca-debug.md                   (CoreAnimation debug flags)
│   ├── endpoints.md                  (HTTP + WebSocket surface)
│   └── workflows.md                  (end-to-end recipes incl. preview handoff)
├── scripts/
│   ├── check-prereqs.sh              (verify host satisfies requirements)
│   └── ensure-running.sh             (idempotent start of the helper)
└── evals/
    └── evals.json                    (6 test prompts for agent quality)

Following Anthropic's recommended structure: short SKILL.md, references one level deep, executable scripts that the agent can run without loading their source into context.

Designed around progressive disclosure

• Discovery: only the name and description from the frontmatter cost tokens at startup.
• Activation: when the agent decides the task matches, it reads SKILL.md.
• Execution: it reads only the reference files relevant to the current task.

This keeps context usage low across hundreds of installed skills.

Source of truth

Every claim in this skill — the six button names, the four orientations, the gesture JSON shape, the edge values, the HTTP endpoints — was verified against the serve-sim source at the time of authoring. The skill does not invent behavior the CLI does not expose. When the CLI changes, update the skill and the evals/ alongside it.

Evals

evals/evals.json contains six representative prompts with expected behaviors, suitable for running through Anthropic's skill-creator eval framework. When changing the skill, re-run the evals to catch regressions.

Contributing

Found a divergence between this skill and serve-sim's actual behavior? Open an issue or PR on this repo.

Want to add a workflow recipe? Add it to references/workflows.md with an explanation of when an agent would use it, and add a matching eval to evals/evals.json.

License

Apache-2.0, same as the rest of the serve-sim repository.

Buttons and rotation reference

Contents

• Valid button names
• Button behavior details
• Valid orientations
• Rotation gotchas

Valid button names

These are the only values accepted by npx serve-sim button <name>. Anything else prints [hid] Unknown button: <name> and does nothing.

Button behavior details

home vs swipe_home

• Devices with a physical Home button (iPhone SE, classic iPad layouts): use home.
• Face ID devices (iPhone X and later, modern iPads): both work, but swipe_home matches the gesture a real user performs and exercises iOS's edge-gesture recognizer. Use swipe_home for fidelity with real-user behavior.

siri requires the hold

Tapping the side button briefly does not trigger Siri. The CLI command holds for 300ms automatically — you do not need to script it.

app_switcher availability

app_switcher depends on the IndigoHIDMessageForButton symbol being resolvable in the simulator's private framework. If the simulator was launched from an older Xcode, the command may log App switcher not available and do nothing. Fall back to a gesture that swipes up to the middle of the screen and pauses, then continues — but this is fragile across iOS versions.

Lock screen capture

A common test pattern: lock → wait → lock again wakes the device into the lock screen, which is useful for testing widget and lock-screen UI.

Valid orientations

These are the only values accepted by npx serve-sim rotate <orientation>:

These correspond to UIDeviceOrientation values 1, 2, 4, and 3 respectively in iOS's internal enum.

Rotation gotchas

• `portrait_upside_down` is not honored by many apps. Apple's HIG discourages it, and most apps lock orientation in their Info.plist. The simulator dispatches the event regardless, but the app may not visibly rotate.
• *`landscape_ requires app support.** If the target app declares UISupportedInterfaceOrientations = ["UIInterfaceOrientationPortrait"]` only, the device rotates internally but the app keeps its portrait layout.
• Apple Watch does not rotate. Sending rotate to a watchOS simulator has no effect.
• Touch coordinates auto-compensate after rotation. The serve-sim client tracks the last-set orientation and rotates incoming x/y from "logical display" to "raw HID" coordinates. You can continue to think in (0,0) = top-left of the screen as the user sees it, regardless of orientation.
• Simulator.app must be running for rotate to work. The orientation event is delivered via Mach IPC through PurpleWorkspacePort, which only exists when Simulator.app is attached to the device. A pure-headless xcrun simctl boot without Simulator.app cannot receive the rotation event — the CLI will print a warning and return non-zero.

CoreAnimation debug flags reference

The npx serve-sim ca-debug <option> <on|off> command toggles CoreAnimation render-debug overlays on the running simulator. These are the same flags Simulator.app exposes under Debug → Color Blended Layers, etc. Each maps to a private -[SimDevice setCADebugOption:enabled:] call.

Valid options and aliases

On / off values

The CLI accepts any of these for the second argument, case-insensitive: on, off, 1, 0, true, false.

Typical use

# Find blended layers in your app
npx serve-sim ca-debug blended on
# ... interact with the app, screenshot the preview ...
npx serve-sim ca-debug blended off

# Slow animations 10x to inspect a transition
npx serve-sim ca-debug slow-animations on
# ... drive the app, record video of the preview ...
npx serve-sim ca-debug slow-animations off

Toggle individually — combining offscreen and blended makes the screen unreadable.

Notes

• Settings persist until the next simulator reboot or until you turn them off explicitly.
• The flags affect the simulator's rendering pipeline directly; performance numbers in Instruments will differ from a real device. Use these for qualitative debugging (find the problem), not quantitative measurement.
• These options correspond to the same flags real-device Xcode uses, so insights transfer.

Camera injection reference

Contents

• What camera injection does
• Requirements
• Sources: placeholder, file, webcam
• Mirror modes
• Hot-swapping a running source
• Status and listing
• Typical end-to-end recipe

What camera injection does

npx serve-sim camera <bundle-id> replaces the simulator's camera feed for a single app. The serve-sim helper writes BGRA frames into a POSIX shared-memory region on the host. An injected dylib (DYLD_INSERT_LIBRARIES) inside the simulator process swizzles AVFoundation so the app reads from that region instead of the simulator's default stub camera.

The helper is one-per-device and outlives any single app launch. Multiple apps on the same simulator can share the feed — run npx serve-sim camera <other-bundle-id> again to (re)launch the next app with the dylib attached.

Requirements

• macOS 14+ (Sonoma or later) on the host.
• An iOS Simulator running iOS 17+ recommended.
• The target app must use standard AVFoundation capture APIs.

If macOS is older than 14, the camera subcommand exits with an error.

Sources

npx serve-sim camera <bundle-id> [source-flag] picks one source. Without a flag, the source defaults to placeholder.

placeholder (default)

npx serve-sim camera com.acme.MyApp

Animated programmatic frames generated by the helper. Useful as a "camera is alive" sanity check.

--file <path> — image or video

npx serve-sim camera com.acme.MyApp --file ~/Pictures/face.png
npx serve-sim camera com.acme.MyApp --file ~/Movies/loop.mp4

The CLI sniffs the file kind from the extension first, then falls back to magic bytes:

• Image: PNG, JPEG, HEIC, GIF, BMP, TIFF, WebP
• Video: MP4, MOV, M4V, WebM, AVI

Videos loop at their native frame rate.

--webcam [name] — live host camera

npx serve-sim camera com.acme.MyApp --webcam
npx serve-sim camera com.acme.MyApp --webcam "MacBook Pro Camera"

Without [name], uses the host's built-in front camera. With [name], the helper performs a substring match (case-insensitive) against connected AVCaptureDevice names — built-in, Continuity Camera, USB webcams, etc.

List host cameras first if you do not know the exact name:

npx serve-sim camera --list-webcams

Mirror modes

npx serve-sim camera com.acme.MyApp --mirror auto    # default
npx serve-sim camera com.acme.MyApp --mirror on
npx serve-sim camera com.acme.MyApp --mirror off
npx serve-sim camera com.acme.MyApp --no-mirror      # shortcut for --mirror off

Mirroring applies to the preview layer only. Data-output buffers handed to the app are never auto-mirrored — apps that process frames see them in their natural orientation.

Hot-swapping a running source

Once a helper is running, you can change source or mirror without relaunching the app:

npx serve-sim camera switch placeholder
npx serve-sim camera switch webcam "MacBook Pro Camera"
npx serve-sim camera switch ~/Movies/loop.mp4

npx serve-sim camera mirror on
npx serve-sim camera mirror auto

The helper has a Unix-domain control socket; these commands send a single message and exit. The app keeps running and sees the new source on the next frame.

Status and listing

npx serve-sim camera status         # human-readable
npx serve-sim camera status -q      # JSON: {alive, source, mirror, ...}
npx serve-sim camera --list-webcams # host cameras

status reports whether the helper is alive, the current source kind and argument, and the active mirror mode. An agent should call this before issuing a switch to confirm a helper exists; otherwise spawn one via camera <bundle-id>.

Stopping the helper

npx serve-sim camera --stop-webcam              # stop for current device
npx serve-sim camera --stop-webcam -d "iPhone 16 Pro"

The injected dylib persists in the next app launch unless you also restart the simulator process. To force a clean state, terminate the target app first:

xcrun simctl terminate booted com.acme.MyApp
npx serve-sim camera --stop-webcam

Typical end-to-end recipe

Test that the target app captures a static reference image and saves it correctly:

# 1. Clean slate
npx serve-sim camera --stop-webcam
xcrun simctl terminate booted com.acme.MyApp

# 2. Inject the test image, force no-mirror so any embedded text reads right
npx serve-sim camera com.acme.MyApp --file ~/test-assets/reference.png --mirror off

# 3. Open the capture screen via deep link
xcrun simctl openurl booted "myapp://camera/capture"

# 4. Tap the shutter (assume it's at 50% horizontal, 90% vertical)
npx serve-sim tap 0.5 0.9

# 5. Confirm the helper still owns the feed
npx serve-sim camera status -q

# 6. Inspect the saved image on disk in the app's sandbox
APP_DATA=$(xcrun simctl get_app_container booted com.acme.MyApp data)
ls -la "$APP_DATA/Documents/captures/"

# 7. Tear down
npx serve-sim camera --stop-webcam

HTTP and WebSocket endpoints reference

For agents that want to bypass the CLI — for example to drive gestures from a long-running process without forking npx per call — serve-sim exposes two surfaces over HTTP.

Contents

• Stream server (Swift helper, port 3100)
• Preview middleware (port 3200)
• Authentication
• Discovering the live URLs

Stream server (Swift helper, default port 3100)

This is the per-device binary started for each booted simulator. It serves the video stream and accepts input over a binary WebSocket.

CORS is wide-open (Access-Control-Allow-Origin: *) on this server.

WebSocket message types

The /ws endpoint accepts binary frames. Two formats are in use today:

Touch (prefix `0x10`) — 12 or 13 bytes:

[0x10] [subtype:u8] [x:f32] [y:f32] [seq:u16] [edge:u8?]

• subtype: 0 = begin, 1 = move, 2 = end
• x, y: normalized floats in [0, 1]
• seq: monotonic 16-bit counter the server uses to coalesce frames
• edge: optional, 0–4 as in gestures.md

Multi-touch (prefix `0x11`) — 20 bytes:

[0x11] [subtype:u8] [x1:f32] [y1:f32] [x2:f32] [y2:f32] [seq:u16]

A JSON channel also exists with these message-type bytes; refer to the helper source if you need them:

• 0x03 — JSON touch event (legacy)
• 0x04 — JSON button event
• 0x05 — JSON multi-touch event (legacy)
• 0x06 — JSON keyboard event ({type: down|up, usage: u32}, USB HID Usage Page 0x07)
• 0x07 — JSON orientation event
• 0x08 — JSON CoreAnimation debug toggle
• 0x09 — empty body, triggers memory warning

For most agents, the CLI is the right entry point. Use the WebSocket directly only when you need sub-CLI-latency input streams (drag animations, multi-finger gestures).

Preview middleware (default port 3200)

This is a Node middleware that serves the preview UI and proxies state. It can be run standalone (npx serve-sim) or embedded in another dev server (serve-sim/middleware).

When embedding the middleware in another dev server (Metro, Vite, Express), the basePath is configurable:

import { simMiddleware } from "serve-sim/middleware";
app.use(simMiddleware({ basePath: "/.sim" }));

Authentication

• The Swift stream server has no authentication. It listens on 0.0.0.0 by default — be careful when tunneling.
• The preview middleware's /.sim/exec endpoint requires a bearer token. The token is printed when serve-sim starts and is stored in the per-device state file under $TMPDIR/serve-sim/server-{udid}.json. Use Authorization: Bearer <token>.

Discovering the live URLs

The simplest path is the CLI:

npx serve-sim --list -q

Returns a JSON array of running streams, each with url, streamUrl, wsUrl, device, pid, port. An agent should call this once on entry to discover ports — they are not guaranteed to be 3200/3100 if other processes occupy those defaults.

Alternatively, read state files directly:

ls $TMPDIR/serve-sim/server-*.json
cat $TMPDIR/serve-sim/server-<udid>.json

This is faster than spawning npx but couples you to the file format. Prefer --list -q for portability.

Gestures reference

Contents

• Coordinate system
• Single-touch JSON shape
• Multi-touch JSON shape
• Edge values (system edge gestures)
• Common recipes (swipe, drag, pinch)
• Why tap is preferred over gesture for taps

Coordinate system

All coordinates are normalized floats in [0, 1]:

• (0.0, 0.0) = top-left of the display
• (1.0, 1.0) = bottom-right of the display
• (0.5, 0.5) = center

The serve-sim client compensates for device orientation automatically. After a rotate landscape_left, coords still describe the logical display from the user's perspective — you do not rotate them manually.

To convert pixel coords to normalized, divide by the display dimensions reported by GET http://localhost:3100/config ({width, height, orientation}).

Single-touch JSON shape

{
  "type": "begin",
  "x": 0.5,
  "y": 0.5,
  "edge": 0
}

A complete touch is a sequence of begin → 0 or more move → end on the same WebSocket. The gesture CLI opens one socket per invocation, so a complete touch must fit inside a single CLI call's payload, or be issued via the WebSocket directly.

Multi-touch JSON shape

{
  "type": "begin",
  "x1": 0.4,
  "y1": 0.5,
  "x2": 0.6,
  "y2": 0.5
}

Multi-touch does not support edge. Use it for pinch, zoom, and two-finger rotation gestures.

Edge values

The edge field flags a touch as a system edge gesture. iOS interprets edge touches specially (e.g., bottom edge swipe = swipe-to-home on Face ID devices). Values:

For a swipe-to-home on a Face ID device, you can use npx serve-sim button swipe_home — it issues the correct edge-3 touch sequence for you.

Common recipes

Tap (avoid gesture — use tap)

npx serve-sim tap 0.5 0.5

Two back-to-back gesture calls for begin and end will be interpreted as a long-press because each call opens a fresh WebSocket. Always use tap.

Drag (vertical scroll down)

A drag is a sequence on a single WebSocket. With the CLI, the simplest correct approach is the streaming server's WebSocket. From an agent that only has the CLI, prefer breaking the drag into a single gesture call that contains the full sequence — or use button swipe_home for the specific case of swipe-to-home.

For ad-hoc drags via the CLI, this minimal sequence works because it issues one socket call per phase but the simulator coalesces them when issued rapidly:

npx serve-sim gesture '{"type":"begin","x":0.5,"y":0.2}'
npx serve-sim gesture '{"type":"move","x":0.5,"y":0.5}'
npx serve-sim gesture '{"type":"move","x":0.5,"y":0.8}'
npx serve-sim gesture '{"type":"end","x":0.5,"y":0.8}'

The agent should accept that this may produce a long-press start on some hosts. For reliable drags, drive the WebSocket directly (see references/endpoints.md).

Pinch to zoom in

npx serve-sim gesture '{"type":"begin","x1":0.4,"y1":0.5,"x2":0.6,"y2":0.5}'
npx serve-sim gesture '{"type":"move","x1":0.25,"y1":0.5,"x2":0.75,"y2":0.5}'
npx serve-sim gesture '{"type":"end","x1":0.25,"y1":0.5,"x2":0.75,"y2":0.5}'

Pull down Notification Center (top edge)

npx serve-sim gesture '{"type":"begin","x":0.5,"y":0.02,"edge":2}'
npx serve-sim gesture '{"type":"move","x":0.5,"y":0.4,"edge":2}'
npx serve-sim gesture '{"type":"end","x":0.5,"y":0.4,"edge":2}'

Swipe back from left edge

npx serve-sim gesture '{"type":"begin","x":0.01,"y":0.5,"edge":1}'
npx serve-sim gesture '{"type":"move","x":0.5,"y":0.5,"edge":1}'
npx serve-sim gesture '{"type":"end","x":0.5,"y":0.5,"edge":1}'

Why tap is preferred

The CLI's gesture subcommand opens a fresh WebSocket on each invocation. When you issue begin and end as two separate calls, the round-trip plus socket setup introduces a delay of tens of milliseconds. The simulator's HID layer measures touch duration; anything beyond ~100ms reads as a long-press, not a tap.

The tap subcommand exists specifically to bypass this: it issues begin and end over a single connection with negligible latency, producing a true tap every time. There is no good reason to emulate a tap with gesture.

App permissions reference

npx serve-sim permissions manages an installed app's privacy permissions on the booted simulator. It is modelled on AppleSimulatorUtils but writes the underlying state stores directly, because xcrun simctl privacy is timing- fragile and cannot touch push notifications at all.

CLI surface

npx serve-sim permissions grant  <permission> <bundle-id> [--value <v>] [-d <udid|name>]
npx serve-sim permissions revoke <permission> <bundle-id> [-d <udid|name>]
npx serve-sim permissions reset  <permission|all> <bundle-id> [-d <udid|name>]
npx serve-sim permissions list   [bundle-id] [-d <udid|name>]

• grant — allow the permission.
• revoke (alias: deny) — deny the permission.
• reset — clear it so the app is prompted again next time. reset all clears

every permission for the bundle id.

• list — print current state as JSON: a tcc map (keyed by the same

permission names grant/revoke/reset accept), plus location and notifications. Pass -q for single-line JSON.

<bundle-id> is required for every verb except list. The app must be installed; location in particular silently no-ops on an uninstalled bundle.

Supported permissions

Aliases accepted for parity with AppleSimulatorUtils / agent-device: push → notifications, photo-library/photo → photos, mic → microphone, location-always → location --value always, location-inuse → location --value inuse.

A trailing positional is also accepted as the value, e.g. permissions grant photos com.foo.bar limited.

How it works (and why)

• TCC permissions (camera, photos, contacts, …) are written straight into

the simulator's TCC.db with sqlite3. simctl privacy works for these too but races simulator boot; the direct write is deterministic.

• Notifications are written into the BulletinBoard VersionedSectionInfo.plist

(the same mechanism Settings → Notifications reads). simctl privacy has no notifications service at all.

• Location is delegated to xcrun simctl privacy, because iOS keys

locationd's clients.plist entries as i<bundleId>: — a format that plutil and PlistBuddy cannot address, but simctl privacy handles correctly.

This command manages the permission only — it does not deliver a notification. To push a test notification, use xcrun simctl push.

Examples

# Grant push notifications (including critical alerts)
npx serve-sim permissions grant notifications com.example.app
npx serve-sim permissions grant notifications com.example.app --value critical

# Location, always-on
npx serve-sim permissions grant location com.example.app --value always

# Limited photo library
npx serve-sim permissions grant photos com.example.app --value limited

# Deny the camera, then inspect everything
npx serve-sim permissions revoke camera com.example.app
npx serve-sim permissions list com.example.app -q

# Wipe every permission so the next launch re-prompts
npx serve-sim permissions reset all com.example.app

Notes

• Changes are written to disk immediately. An app that's already running may

need to be relaunched to observe the new state.

• Use a real installed bundle id. Find one with

xcrun simctl listapps booted.

Workflows reference

Recipes for common end-to-end agent tasks. These compose the primitives documented in the other reference files.

Contents

• Workflow 1: Tap on a UI element by accessibility label
• Workflow 2: Deep-link into an app and verify the screen
• Workflow 3: Test a camera capture flow
• Workflow 4: Find blended layers in a screen
• Workflow 5: Reset and re-test (clean slate)
• Workflow 6: Drive a multi-step gesture reliably
• Workflow 7: Show the simulator stream in the host's preview

Workflow 1: Tap on a UI element by accessibility label

Pixel-hunting is fragile across device sizes. Prefer driving taps from the accessibility tree.

# 1. Ensure serve-sim is running and discover the stream port
URL=$(npx serve-sim --list -q | jq -r '.[0].streamUrl')
PORT=$(echo "$URL" | sed -E 's|.*://[^:]+:([0-9]+).*|\1|')

# 2. Fetch the accessibility tree
AX=$(curl -s "http://localhost:${PORT}/ax")

# 3. Find the element by label (use jq with a query that matches your tree shape)
TARGET=$(echo "$AX" | jq '.[] | select(.label == "Submit")')

# Guard: if no element matched, fail loudly. Never guess coordinates when the
# AX tree didn't contain the target — tapping a random spot is almost always
# worse than reporting the failure.
if [ -z "$TARGET" ] || [ "$TARGET" = "null" ]; then
  echo "ERROR: no element with label 'Submit' found in accessibility tree" >&2
  exit 2
fi

CX=$(echo "$TARGET" | jq '.frame.x + .frame.width / 2')
CY=$(echo "$TARGET" | jq '.frame.y + .frame.height / 2')

# 4. Normalize coordinates against the display
CONFIG=$(curl -s "http://localhost:${PORT}/config")
W=$(echo "$CONFIG" | jq '.width')
H=$(echo "$CONFIG" | jq '.height')
NX=$(echo "scale=4; $CX / $W" | bc)
NY=$(echo "scale=4; $CY / $H" | bc)

# 5. Tap
npx serve-sim tap "$NX" "$NY"

The exact jq query depends on your tree shape — inspect with curl /ax | jq and adapt.

Workflow 2: Deep-link into an app and verify the screen

# 1. Open the URL — uses Apple's simctl, not serve-sim
xcrun simctl openurl booted "myapp://products/42"

# 2. Wait briefly for the app to handle the deep link
sleep 1

# 3. Resolve the stream endpoint — do not hardcode :3100, it may differ
PORT=$(npx serve-sim --list -q | jq -r '.[0].streamUrl' | sed -E 's|.*://[^:]+:([0-9]+).*|\1|')

# 4. Verify the frontmost app is yours
curl -s "http://localhost:${PORT}/foreground" | jq
# Expect: {"bundleId":"com.acme.myapp","pid":12345}

# 5. Verify the screen by reading the accessibility tree
curl -s "http://localhost:${PORT}/ax" | jq '.[] | select(.label | test("Product #42"))'

Workflow 3: Test a camera capture flow

# 1. Clean state
npx serve-sim camera --stop-webcam
xcrun simctl terminate booted com.acme.MyApp

# 2. Inject a known test image, force no-mirror so any text reads right
npx serve-sim camera com.acme.MyApp --file ~/test-assets/reference.png --mirror off

# 3. Navigate to the capture screen
xcrun simctl openurl booted "myapp://camera/capture"
sleep 1

# 4. Tap the shutter (50% horizontal, 90% vertical)
npx serve-sim tap 0.5 0.9
sleep 1

# 5. Inspect the saved capture in the app's sandbox
APP_DATA=$(xcrun simctl get_app_container booted com.acme.MyApp data)
ls -la "$APP_DATA/Documents/captures/"

# 6. Tear down
npx serve-sim camera --stop-webcam

Workflow 4: Find blended layers in a screen

# 1. Turn on blended-layers overlay
npx serve-sim ca-debug blended on

# 2. Navigate to the screen of interest (via openurl, taps, etc.)
xcrun simctl openurl booted "myapp://settings"

# 3. Resolve the stream endpoint — do not hardcode :3100, it may differ
PORT=$(npx serve-sim --list -q | jq -r '.[0].streamUrl' | sed -E 's|.*://[^:]+:([0-9]+).*|\1|')

# 4. Grab a screenshot from the MJPEG stream
curl -s "http://localhost:${PORT}/stream.mjpeg?raw=1" \
  --max-time 1 -o /tmp/blended.jpg

# 5. Inspect — red regions are blended (expensive); green is opaque (good)
open /tmp/blended.jpg

# 6. Turn off
npx serve-sim ca-debug blended off

The single-shot curl --max-time 1 grabs the first JPEG frame from the MJPEG stream. For a clean frame grab, prefer the preview UI's screenshot button or read multiple frames and pick a stable one.

Workflow 5: Reset and re-test (clean slate)

When tests pollute the simulator state, fastest reset:

# Kill all serve-sim helpers (frees ports, drops streams)
npx serve-sim --kill

# Terminate your app
xcrun simctl terminate booted com.acme.MyApp

# (Optional) erase the simulator entirely — destroys app data
# xcrun simctl shutdown booted
# xcrun simctl erase booted
# xcrun simctl boot <UDID>

# Restart serve-sim
npx serve-sim --detach -q

The full erase is destructive — only do it when the user explicitly asks.

Workflow 6: Drive a multi-step gesture reliably

For a complex gesture (long drag, multi-finger choreography), the CLI's gesture subcommand is unreliable because each call opens a fresh WebSocket. The reliable path is one persistent WebSocket connection.

In a Node agent:

import WebSocket from "ws";
import { encodeSingleTouch, encodeMultiTouch } from "serve-sim-client/touch-codec";

// 3100 is the default stream port — discover the real one with
// `serve-sim --list -q` (.streamUrl) when it may differ.
const ws = new WebSocket("ws://localhost:3100/ws");
await new Promise((r) => ws.once("open", r));

let seq = 0;
function send(data) {
  const buf = data.x1 !== undefined
    ? encodeMultiTouch(data, seq++)
    : encodeSingleTouch(data, seq++);
  ws.send(buf);
}

// Long vertical drag
send({ type: "begin", x: 0.5, y: 0.2 });
for (let i = 1; i <= 30; i++) {
  send({ type: "move", x: 0.5, y: 0.2 + (0.6 * i) / 30 });
  await new Promise((r) => setTimeout(r, 16));
}
send({ type: "end", x: 0.5, y: 0.8 });

ws.close();

If you do not want a Node dependency, you can build the same frames in any language — the binary format is documented in endpoints.md.

Workflow 7: Show the simulator stream in the host's preview

The user has a simulator running with their app already in the foreground. They want to see it streamed into the same tool they're chatting with the agent in (Claude Code, Cursor, Codex CLI, or any other host) so they don't have to leave the conversation to open a browser.

The trick is that the skill cannot invoke the host's preview tool directly — each host (Claude Code, Cursor, Codex, …) has its own URL-opening mechanism with a different tool name and signature. The agent must inspect its own toolset and route accordingly.

# 1. Start serve-sim in detached mode and capture the JSON
STREAM_JSON=$(npx serve-sim --detach -q)
URL=$(echo "$STREAM_JSON" | jq -r '.url')
STREAM_URL=$(echo "$STREAM_JSON" | jq -r '.streamUrl')

# 2. Surface the URL plainly to the user so they always have a manual fallback
echo "Simulator stream URL: $URL"
echo "Raw MJPEG endpoint:   $STREAM_URL"

3. Hand off to the host's preview tool, only if one is available in the current toolset:

• Claude Code: if preview_start (or mcp__Claude_Preview__preview_start) appears in the toolset, call it with { url: "$URL" }. The integrated preview pane will render the simulator stream live.
• Cursor: there is no agent-callable preview tool in Cursor today. Print the URL prominently and tell the user "open this URL in Cursor's built-in browser pane (Cmd+P → 'Open URL')".
• Codex CLI: print the URL and tell the user to open it in their browser. Codex does not expose a preview tool to the agent.
• Other hosts / MCP setups: scan available tools for names containing preview, browser, open_url, open, etc. If you find a match whose signature accepts a URL, try it; if it fails, fall back to printing.

4. Never invent a tool name. If your toolset does not include a URL-opening tool, do not pretend to call one. Tell the user explicitly: "I can't open URLs in your host's preview directly. Open $URL in your browser to see the stream."

5. The stream persists until npx serve-sim --kill. Multiple clients (the host preview + the user's browser + a tunnel) can connect to the same URL at once.

Why this is split between skill and host

The skill knows how to make a URL exist (serve-sim --detach). The host knows how to render a URL in its preview. The agent is the glue — it should detect which side of that glue is available in the current session and act accordingly. There is no single "open in preview" command because each host has reinvented it.

Quick decision tree

• Did --detach -q return a valid url? If not, stop and report the failure.
• Is there a preview_start-like tool in your current toolset? Call it.
• No such tool? Print the URL plainly and instruct the user how to open it.
• Did the preview tool reject the URL or fail? Fall back to printing.