Agent Device

agent-device

Use this skill as a router with mandatory defaults. Read this file first. For normal device tasks, always load references/bootstrap-install.md and references/exploration.md before acting. Use bootstrap to confirm or establish deterministic setup. Use exploration for UI inspection, interaction, and verification once the app session is open.

Default operating rules

• Start conservative. Prefer read-only inspection before mutating the UI.
• Use plain snapshot when the task is to verify what text or structure is currently visible on screen.
• Use snapshot -i only when you need interactive refs such as @e3 for a requested action or targeted query.
• Avoid speculative mutations. You may take the smallest reversible UI action needed to unblock inspection or complete the requested task, such as dismissing a popup, closing an alert, or clearing an unintended surface.
• Do not browse the web or use external sources unless the user explicitly asks.
• Re-snapshot after meaningful UI changes instead of reusing stale refs.
• Prefer @ref or selector targeting over raw coordinates.
• Ensure the correct target is pinned and an app session is open before interacting.
• Keep the loop short: open -> inspect/act -> verify if needed -> close.

Default flow

1. Load references/bootstrap-install.md and references/exploration.md before acting on a normal device task. 2. Use bootstrap first to confirm or establish the correct target, app install, and open app session. 3. Once the app session is open and stable, use exploration for inspection, interaction, and verification. 4. Start with plain snapshot if the goal is to read or verify what is visible. 5. Escalate to snapshot -i only if you need refs for interactive exploration or a requested action. 6. Use get, is, or find before mutating the UI when a read-only command can answer the question. 7. End by capturing proof if needed, then close.

QA modes

• Open-ended bug hunt with reporting: use ../dogfood/SKILL.md.
• Pass/fail QA from acceptance criteria: stay in this skill, start with references/bootstrap-install.md, then use the QA loop in references/exploration.md.

Required references

• For every normal device task, after reading this file, load references/bootstrap-install.md first, then references/exploration.md, before acting.
• Use bootstrap to confirm or establish deterministic setup, especially in sandbox or cloud environments.
• Use exploration once the app session is open and stable.
• Load additional references only when their scope is needed.

Decision rules

• Use plain snapshot when you need to verify whether text is visible.
• Use snapshot -i mainly for interactive exploration and choosing refs.
• Use get, is, or find when they can answer the question without changing UI state.
• Use fill to replace text.
• Use type to append text.
• When a task asks to "go back", use plain back for predictable app-owned navigation and reserve back --system for platform back gestures or button semantics.
• Use type --delay-ms or fill --delay-ms for debounced search fields that drop characters when typed too quickly.
• If there is no simulator, no app install, or no open app session yet, switch to bootstrap-install.md instead of improvising setup steps.
• Use the smallest unblock action first when transient UI blocks inspection, but do not navigate, search, or enter new text just to make the UI reveal data unless the user asked for that interaction.
• Do not use external lookups to compensate for missing on-screen data unless the user asked for them.
• If the needed information is not exposed on screen, say that plainly instead of compensating with extra navigation, text entry, or web search.
• Prefer @ref or selector targeting over raw coordinates.

Additional references

• Need logs, network, alerts, permissions, or failure triage: references/debugging.md
• Need screenshots, diff, recording, replay maintenance, or perf data: references/verification.md
• Need desktop surfaces, menu bar behavior, or macOS-specific interaction rules: references/macos-desktop.md
• Need remote HTTP transport, --remote-config launches, or tenant leases on a remote macOS host: references/remote-tenancy.md

Bootstrap and Install

When to open this file

Open this file when you still need to choose the right target, start the right session, install or relaunch the app, or pin automation to one device before interacting. This is the deterministic setup layer for sandbox, cloud, or other environments where install paths, device state, or app readiness may be uncertain.

Open-first path

• devices
• apps
• ensure-simulator
• open
• session list

Install path

• install or reinstall

Most common mistake to avoid

Do not start acting before you have pinned the correct target and opened an app session. In mixed-device environments, always pass --device, --udid, or --serial.

Deterministic setup rule

If there is no simulator, no app install, no open app session, or any uncertainty about where the app should come from, stay in this file and use deterministic setup commands or bootstrap scripts first. Do not improvise install paths or app-launch flows while exploring.

After setup is confirmed or completed, move to exploration.md before doing UI inspection or interaction.

Open-first rule

• If the user asks to test an app and does not provide an install artifact or explicit install instruction, try open <app> first.
• If open <app> fails, run agent-device apps and retry with a discovered app name before considering install steps.
• Do not install or reinstall on the first attempt unless the user explicitly asks for installation or provides a concrete artifact path or URL.
• When installation is required from a known location, prefer a checked-in shell script or other deterministic bootstrap command over ad hoc path guessing.

• If open <app> fails, or you are not sure which app name is available on the target, run agent-device apps first and choose from the discovered app list instead of guessing.
• Use apps --platform <platform> together with --device, --udid, or --serial when target selection matters.
• Once you have the correct app name, retry open with that exact discovered value.

Common starting points

These are examples, not required exact sequences. Use the smallest setup flow that matches the task.

Boot a simulator and open an app

agent-device ensure-simulator --platform ios --device "iPhone 17 Pro" --boot
agent-device open MyApp --platform ios --device "iPhone 17 Pro" --relaunch

Install an app artifact

agent-device install com.example.app ./build/app.apk --platform android --serial emulator-5554

agent-device install com.example.app ./build/MyApp.app --platform ios --device "iPhone 17 Pro"

Install guidance

• Use install <app> <path> when the app may already be installed and you do not need a fresh-state reset.
• Use reinstall <app> <path> when you explicitly need uninstall plus install as one deterministic step.
• Keep install and open as separate phases. Do not turn them into one default command flow.
• Supported binary formats:
• Android: .apk and .aab
• iOS: .app and .ipa
• For iOS .ipa files, <app> is used as the bundle id or bundle name hint when the archive contains multiple app bundles.
• After install or reinstall, later use open <app> with the exact discovered or known package/bundle identifier, not the artifact path.

Choose the right starting point

• iOS local QA: prefer simulators unless the task explicitly requires physical hardware.
• iOS in mixed simulator and device environments: run ensure-simulator first, then keep using --device or --udid.
• TV targets: use --target tv together with --platform when the task is for tvOS or Android TV rather than phone or tablet surfaces.
• Android binary flow: use install or reinstall for .apk or .aab, then open by installed package name.
• macOS desktop app flow: use open <app> --platform macos. Only load macos-desktop.md if a desktop surface or macOS-specific behavior matters.

TV example:

agent-device open MyTvApp --platform ios --target tv
agent-device open com.example.androidtv --platform android --target tv

Session rules

• Use --session <name> when you need a named session:

agent-device --session auth open Settings --platform ios
agent-device --session auth snapshot -i

• Use open <app> before interactions.
• Use close when done. Add --shutdown when you want simulators or emulators torn down with the session.
• Use semantic session names when you need multiple concurrent runs.
• Use --save-script=<path> on close when you want to keep a replay script.
• For dev loops where state can linger, prefer open <app> --relaunch.
• In iOS sessions, use open <app> for the app itself. Use open <url> for deep links, and open <app> <url> when you need to launch the app and deep link in one step.
• On iOS, appstate is session-scoped and requires the matching active session on the target device.

After a session is established

Once you have opened the correct session on the correct target, default to the conservative rule: keep the session binding on follow-up commands, and stop repeating device-routing flags unless you are intentionally retargeting.

• Prefer --session <name> on follow-up commands, or use sandboxed AGENT_DEVICE_SESSION.
• Do not keep repeating --platform, --target, --device, --udid, --serial, or similar target-selection flags on normal follow-up commands.
• Only omit follow-up session flags when the environment explicitly guarantees isolation.

Good shared-host pattern:

agent-device --session auth open Settings --platform ios --device "iPhone 17 Pro"
agent-device --session auth snapshot -i
agent-device --session auth press @e3
agent-device --session auth close

Bad shared-host pattern:

agent-device --session auth open Settings --platform ios --device "iPhone 17 Pro"
agent-device --session auth snapshot -i --platform ios --device "iPhone 17 Pro"

Use target-selection flags again only when you are choosing the target before opening a session, or when you explicitly mean to retarget.

Session-bound automation

Use this when an orchestrator must keep plain CLI calls on one session and device.

export AGENT_DEVICE_SESSION=qa-ios
export AGENT_DEVICE_PLATFORM=ios
export AGENT_DEVICE_SESSION_LOCK=strip

agent-device open MyApp --relaunch

• AGENT_DEVICE_SESSION plus AGENT_DEVICE_PLATFORM provides the default binding.
• --session-lock reject|strip controls whether conflicting per-call routing flags fail or are ignored.
• Conflicts include explicit retargeting flags such as --platform, --target, --device, --udid, --serial, --ios-simulator-device-set, and --android-device-allowlist.
• Lock policy applies to nested batch steps too.
• Compatibility aliases remain supported: --session-locked, --session-lock-conflicts, AGENT_DEVICE_SESSION_LOCKED, and AGENT_DEVICE_SESSION_LOCK_CONFLICTS.

Android emulator variant:

export AGENT_DEVICE_SESSION=qa-android
export AGENT_DEVICE_PLATFORM=android

agent-device --session-lock reject open com.example.myapp --relaunch

Scoped discovery

Use scoped discovery when one run must not see host-global device lists.

agent-device devices --platform ios --ios-simulator-device-set /tmp/tenant-a/simulators
agent-device devices --platform android --android-device-allowlist emulator-5554,device-1234

• Scope is applied before --device, --udid, and --serial.
• Out-of-scope selectors fail with DEVICE_NOT_FOUND.
• With iOS simulator-set scope enabled, iOS physical devices are not enumerated.
• If the scoped iOS simulator set is empty, the error should point at the set path and suggest creating a simulator in that set.
• Environment equivalents:
• AGENT_DEVICE_IOS_SIMULATOR_DEVICE_SET
• AGENT_DEVICE_ANDROID_DEVICE_ALLOWLIST

Session inspection and replay

agent-device session list
agent-device replay ./session.ad --session auth
agent-device replay -u ./session.ad --session auth

• iOS session entries include device_udid and ios_simulator_device_set. Use them to confirm routing in concurrent runs.
• Prefer selector-based actions and assertions in saved replay scripts.
• Tenant isolation namespaces sessions as <tenant>:<session> during tenant-scoped runs.

When to leave this file

• Once the correct target and session are pinned, move to exploration.md.
• If opening, startup, permissions, or logs become the blocker, switch to debugging.md.

Install examples

agent-device reinstall MyApp /path/to/app-debug.apk --platform android --serial emulator-5554

agent-device install com.example.app ./build/MyApp.ipa --platform ios --device "iPhone 17 Pro"

Do not use open <apk|aab> --relaunch on Android.

Security and trust notes

• Treat signing, provisioning, and daemon auth values as host secrets. Do not paste them into shared logs or commit them to source control.
• Prefer Xcode Automatic Signing over manual overrides when a physical iOS device is involved.
• Keep persistent host-specific defaults in environment variables rather than checked-in project config.

Coordinate System

When to open this file

Open this file only when you must use raw coordinates instead of selectors or @ref targeting.

Main commands to reach for first

• screenshot
• coordinate-based click or swipe

Most common mistake to avoid

Do not assume coordinates mean the same thing across platforms or runs. Prefer selectors and refs first.

Canonical loop

agent-device screenshot /tmp/current-screen.png
agent-device click 120 240

Rules

• Origin is the top-left of the device screen.
• iOS uses device points.
• Android uses pixels.
• Use screenshots to reason about coordinates before acting.

Debugging

When to open this file

Open this file when the task turns into failure triage, logs, network inspection, permission prompts, setup trouble, or unstable session behavior.

Main commands to reach for first

• logs clear --restart
• network dump
• logs path
• logs doctor
• alert wait
• alert accept or alert dismiss

Most common mistake to avoid

Do not leave logging on for normal flows or dump full log files into context. Keep debug windows short and inspect logs with grep or tail.

Canonical loop

agent-device open MyApp --platform ios
agent-device logs clear --restart
agent-device network dump 25
agent-device logs path
agent-device close

Log and network flow

Logging is off by default. Enable it only when you need a debugging window.

• Default app logs live under ~/.agent-device/sessions/<session>/app.log.
• logs clear --restart is the fastest clean repro loop.
• network dump [limit] [summary|headers|body|all] parses recent HTTP(s) entries from the same session app log.
• logs doctor checks backend and runtime readiness for the current session and device.
• logs mark "before tap" inserts a timestamped marker into the app log.
• Session app logs can contain runtime data, headers, or payload fragments. Review them before sharing.
• logs start requires an active app session and appends to app.log.
• logs stop stops streaming. close also stops logging.
• logs clear truncates app.log and removes rotated app.log.N files, and requires logging to be stopped first.
• logs path returns the log path plus metadata about the active backend and file state.
• network log is an alias for network dump.

Operational limits:

• app.log rotates to app.log.1 after 5 MB by default.
• network dump scans the last 4000 app-log lines, returns up to 200 entries, and truncates header or payload fields at 2048 characters.
• Retention knobs:
• AGENT_DEVICE_APP_LOG_MAX_BYTES
• AGENT_DEVICE_APP_LOG_MAX_FILES
• Redaction hook:
• AGENT_DEVICE_APP_LOG_REDACT_PATTERNS

Useful shell follow-up after logs path:

grep -n -E "Error|Exception|Fatal|crash" <path>
tail -50 <path>

Alerts and permissions

Use alert for iOS simulator permission dialogs instead of tapping coordinates.

agent-device alert wait 5000
agent-device alert accept

• alert is only supported on iOS simulators.
• alert accept and alert dismiss retry internally for a short window, so you usually do not need manual sleeps.
• iOS 16+ "Allow Paste" prompts are suppressed under XCUITest. Use xcrun simctl pbcopy booted when you need to seed simulator clipboard content directly.

Setup problems worth recognizing early

• iOS snapshots do not require macOS Accessibility permissions.
• iOS physical-device XCTest setup does require valid signing and provisioning.
• If physical-device runner setup fails, prefer Xcode Automatic Signing first.
• Optional overrides are:
• AGENT_DEVICE_IOS_TEAM_ID
• AGENT_DEVICE_IOS_SIGNING_IDENTITY
• AGENT_DEVICE_IOS_PROVISIONING_PROFILE
• AGENT_DEVICE_IOS_BUNDLE_ID
• If daemon startup is timing out during setup, increase AGENT_DEVICE_DAEMON_TIMEOUT_MS.
• If daemon startup fails with stale metadata hints, clean ~/.agent-device/daemon.json and ~/.agent-device/daemon.lock, then retry.
• Free Apple Developer personal-team accounts may reject generic bundle IDs. Use a unique reverse-DNS value for AGENT_DEVICE_IOS_BUNDLE_ID when that happens.

Common failure patterns

• snapshot returns 0 nodes: the app may no longer be foregrounded or the UI is not stable yet. Re-open the app or retry when state settles.
• Logs are empty: confirm you opened an app session before logs clear --restart.
• Android logs look stale after relaunch: retry the repro window after the process rebinds.
• Permission prompts block the flow: wait for the alert and handle it explicitly.
• If snapshots keep returning 0 nodes on an iOS simulator, restart Simulator and re-open the app.
• If a macOS snapshot looks incomplete, compare with snapshot --raw --platform macos to separate collector filtering from missing AX content.

Crash triage fast path

Always start from the session app log, then branch by platform.

agent-device logs path
grep -n -E "SIGABRT|SIGSEGV|EXC_|fatal|exception|terminated|killed|jetsam|memorystatus|FATAL EXCEPTION|Abort message" <path>

• iOS: if the log suggests ReportCrash, SIGABRT, or EXC_*, inspect ~/Library/Logs/DiagnosticReports.
• Android: if the app log is not enough, use adb logcat for FATAL EXCEPTION, Abort message, or signal lines around process death.
• If no crash signature appears in app logs, stop collecting broad logs and switch to the platform-native crash source.

When to leave this file

• Return to exploration.md once the app is stable again.
• Load verification.md if you need evidence artifacts after reproducing the issue.

Exploration

When to open this file

Open this file when the app or screen is already running and you need to discover the UI, choose targets, read state, wait for conditions, or perform normal interactions.

Read-only first

• If the question is what text, labels, or structure is visible on screen, start with plain snapshot.
• Escalate to snapshot -i only when you need refs such as @e3 for interactive exploration or a requested action.
• If you intend to press, fill, or otherwise interact, start with snapshot -i and fall back to plain snapshot only if interactive refs are unavailable.
• Prefer get, is, or find before mutating the UI when a read-only command can answer the question.
• You may take the smallest reversible UI action needed to unblock inspection, such as dismissing a popup, closing an alert, or backing out of an unintended surface.
• Do not type or fill text just to make hidden information easier to access unless the user asked for that interaction.
• Do not use external sources to infer missing UI state unless the user explicitly asked.
• If the answer is not visible or exposed in the UI, report that gap instead of compensating with search, navigation, or text entry.

Decision shortcut

• User asks what is visible on screen: snapshot
• User asks for exact text from a known target: get text
• User asks you to tap, type, or choose an element: snapshot -i, then act
• UI does not expose the answer: say so plainly; do not browse or force the app into a new state unless asked

Read-only commands

• snapshot
• get
• is
• find

Interaction commands

• snapshot -i
• press
• fill
• type
• wait

Most common mistake to avoid

Do not treat @ref values as durable after navigation or dynamic updates. Re-snapshot after the UI changes, and switch to selectors when the flow must stay stable.

Common example loops

These are examples, not required exact sequences. Adapt them to the app, state, and task at hand.

Interactive exploration loop

agent-device open Settings --platform ios
agent-device snapshot -i
agent-device press @e3
agent-device wait visible 'label="Privacy & Security"' 3000
agent-device get text 'label="Privacy & Security"'
agent-device close

Screen verification loop

agent-device open MyApp --platform ios
# perform the necessary actions to reach the state you need to verify
agent-device snapshot
# verify whether the expected element or text is present
agent-device close

Snapshot choices

• Use plain snapshot when you only need to verify whether visible text or structure is on screen.
• Use snapshot -i when you need refs such as @e3 for interactive exploration or for an intended interaction.
• Treat large text-surface lines in snapshot -i as discovery output. If a node shows preview or truncation metadata, use get text @ref only after you have already decided that snapshot -i is needed for that surface.
• Use snapshot -i -s "Camera" or snapshot -i -s @e3 when you want a smaller, scoped result.

Example:

agent-device snapshot -i

Sample output:

Page: com.apple.Preferences
App: com.apple.Preferences

@e1 [ioscontentgroup]
  @e2 [button] "Camera"
  @e3 [button] "Privacy & Security"

Refs vs selectors

• Use refs for discovery, debugging, and short local loops.
• Use selectors for deterministic scripts, assertions, and replay-friendly actions.
• Prefer selector or @ref targeting over raw coordinates.
• For tap interactions, press is canonical and click is an equivalent alias.

Examples:

agent-device press @e2
agent-device fill @e5 "test"
agent-device press 'id="camera_row" || label="Camera" role=button'
agent-device is visible 'id="camera_settings_anchor"'

Text entry rules

• Use fill to replace text in an editable field.
• Use type to append text to the current insertion point.
• Do not use fill or type just to make the app reveal information that is not currently visible unless the user asked for that interaction.

Query and sync rules

• Use get to read text, attrs, or state from a known target.
• Use is for assertions.
• Use wait when the UI needs time to settle after a mutation.
• Use find "<query>" click --json when you need search-driven targeting plus matched-target metadata.
• If you are forced onto raw coordinates, open coordinate-system.md first.

Example:

agent-device find "Increment" click --json

Returned metadata comes from the matched snapshot node and can be used for observability or replay maintenance.

QA from acceptance criteria

Use this loop when the task starts from acceptance criteria and you need to turn them into concrete checks.

Preferred mapping:

• visibility claim for what is on-screen now: is visible or plain snapshot
• presence claim regardless of viewport visibility: is exists
• exact text, label, or value claim: get text
• post-action state change: act, then wait, then is or get
• nearby structural UI change: diff snapshot
• proof artifact for the final result: screenshot or record

Notes:

• wait text is useful for synchronizing on text presence, but it is not the same as is visible.

Anti-hallucination rules:

• Do not invent app names, device ids, session names, refs, selectors, or package names.
• Discover them first with devices, open, snapshot -i, find, or session list.
• If refs drift after navigation, re-snapshot or switch to selectors instead of guessing.

Avoid this escalation path for visible-text questions:

• Do not jump from snapshot -i to get text @ref, then to web search, then to typing into a search box just to force the app to reveal the answer.
• Start with snapshot. If the text is not visible or exposed, report that directly.

Canonical QA loop:

agent-device open MyApp --platform ios
agent-device snapshot -i
agent-device press @e3
agent-device wait visible 'label="Success"' 3000
agent-device is visible 'label="Success"'
agent-device screenshot /tmp/qa-proof.png
agent-device close

Accessibility audit

Use this pattern when you need to find UI that is visible to a user but missing from the accessibility tree.

Audit loop:

1. Capture a screenshot to see what is visually rendered. 2. Capture a snapshot or snapshot -i to see what the accessibility tree exposes. 3. Compare the two:

• visible in screenshot and present in snapshot: exposed to accessibility
• visible in screenshot and missing from snapshot: likely accessibility gap

4. If you suspect the node exists in AX but is filtered from interactive output, retry with snapshot --raw.

Example:

agent-device screenshot /tmp/accessibility-screen.png
agent-device snapshot -i

Use screenshot as the visual source of truth and snapshot as the accessibility source of truth for this audit.

Batch only when the sequence is already known

Use batch when a short command sequence is already planned and belongs to one logical screen flow.

agent-device batch --session sim --platform ios --steps-file /tmp/batch-steps.json --json

• Keep batch size moderate, roughly 5 to 20 steps.
• Add wait or is exists guards after mutating steps.
• Do not use batch for highly dynamic flows that need replanning after each step.

Step payload contract:

[
  { "command": "open", "positionals": ["Settings"], "flags": { "platform": "ios" } },
  { "command": "wait", "positionals": ["label=\"Privacy & Security\"", "3000"], "flags": {} },
  { "command": "click", "positionals": ["label=\"Privacy & Security\""], "flags": {} },
  { "command": "get", "positionals": ["text", "label=\"Tracking\""], "flags": {} }
]

• positionals is optional and defaults to [].
• flags is optional and defaults to {}.
• Only command, positionals, flags, and runtime are accepted as top-level step keys.
• Nested batch and replay are rejected.
• Supported error mode is stop-on-first-error.

Response handling:

• Success returns fields such as total, executed, totalDurationMs, and results[].
• Human-mode batch runs also print a short per-step success summary.
• Failed runs include details.step, details.command, details.executed, and details.partialResults.
• Replan from the first failing step instead of rerunning the whole flow blindly.

Common batch error categories:

• INVALID_ARGS: fix the payload shape and retry.
• SESSION_NOT_FOUND: open or select the correct session, then retry.
• UNSUPPORTED_OPERATION: switch to a supported command or surface.
• AMBIGUOUS_MATCH: refine the selector or locator, then retry the failed step.
• COMMAND_FAILED: add sync guards and retry from the failing step.

Stop conditions

• If refs drift after transitions, switch to selectors.
• If a desktop surface or context menu is involved on macOS, load macos-desktop.md.
• If logs, network, alerts, or setup failures become the blocker, switch to debugging.md.
• If the flow is stable and you need proof or replay maintenance, switch to verification.md.

macOS Desktop

When to open this file

Open this file only when --platform macos is involved or the task needs frontmost-app, desktop, or menubar surfaces.

Main commands to reach for first

• open <app> --platform macos
• open --platform macos --surface frontmost-app|desktop|menubar
• snapshot -i
• get
• is
• click --button secondary

Most common mistake to avoid

Do not treat every macOS surface the same. Use the normal app surface when you want to act inside one app. Use frontmost-app, desktop, or menubar mainly to inspect what is visible before switching back to app for most interactions.

Canonical loop

agent-device open TextEdit --platform macos
agent-device snapshot
agent-device close

Surface rules

• app: default surface and the normal choice for click, fill, press, scroll, screenshot, and record.
• frontmost-app: inspect the currently focused app without naming it first.
• desktop: inspect visible desktop windows across apps.
• menubar: inspect the active app menu bar and system menu extras.

Use inspect-first surfaces to understand desktop-global UI, then switch back to app when you need to act in one app.

Snapshot expectations

• snapshot -i should describe UI visible to a human.
• desktop snapshots can include multiple windows from multiple apps.
• menubar snapshots can include both app-menu items and system menu extras.
• Finder-style rows, sidebar items, toolbar controls, search fields, and opened context menus should appear when visible.
• Finder and other native apps may expose duplicate-looking row, cell, and child text nodes. Treat them as distinct AX nodes unless you have a stronger selector anchor.

Context menus

Context menus are not ambient UI. Open them explicitly, then re-snapshot.

agent-device click @e66 --button secondary --platform macos
agent-device snapshot -i

Expected loop:

1. Snapshot visible content. 2. Secondary-click the target item. 3. Snapshot again. 4. Interact with the new menu-item nodes.

Targeting rules

• Prefer selectors or @ref values over raw coordinates.
• On macOS, window position can vary across runs, so coordinate-only flows are fragile.
• If the task only needs shared exploration rules, return to exploration.md.

Selector guidance:

• Good selectors usually anchor on stable labels or app-owned identifiers such as label="Downloads" or role=menu-item label="Rename".
• Avoid relying on framework-generated _NS:* identifiers as stable selectors.

Use snapshot --raw --platform macos only when debugging AX structure or collector filtering. Do not make raw snapshots the default agent loop.

Things not to rely on:

• Mobile-only helpers such as install, reinstall, or push.
• Desktop-global click or fill parity from desktop or menubar sessions.
• Raw coordinate assumptions across runs.

Troubleshooting:

• If visible content is missing from snapshot -i, re-snapshot after the UI settles.
• If desktop is too broad, retry with frontmost-app.
• If menubar is missing the expected menu, make the app frontmost first and retry.
• If the wrong menu opened, retry secondary-clicking the row or cell wrapper rather than the nested text node.
• If the app has multiple windows, make the correct window frontmost before relying on refs.

Remote Tenancy

When to open this file

Open this file for remote daemon HTTP flows, including --remote-config launches, that let an agent running in a Linux sandbox talk to another agent-device instance on a remote macOS host in order to control devices that are not available locally. This file covers daemon URL setup, authentication, lease allocation, and tenant-scoped command admission.

Main commands to reach for first

• agent-device open <app> --remote-config <path> --relaunch
• AGENT_DEVICE_DAEMON_BASE_URL=...
• AGENT_DEVICE_DAEMON_AUTH_TOKEN=...
• curl ... agent_device.lease.allocate
• curl ... agent_device.lease.heartbeat
• curl ... agent_device.lease.release
• agent-device --tenant ... --session-isolation tenant --run-id ... --lease-id ...

Most common mistake to avoid

Do not run a tenant-isolated command without matching tenant, run, and lease scope. Admission checks require all three to line up.

Preferred remote launch path

Use this when the agent needs the simplest remote control flow: a Linux sandbox agent talks over HTTP to agent-device on a remote macOS host and launches the target app through a checked-in --remote-config profile.

agent-device open com.example.myapp --remote-config ./agent-device.remote.json --relaunch

• This is the preferred remote launch path for sandbox or cloud agents.
• For Android React Native relaunch flows, install or reinstall the APK first, then relaunch by installed package name.
• Do not use open <apk|aab> --relaunch; remote runtime hints are applied through the installed app sandbox.

Lease flow example

export AGENT_DEVICE_DAEMON_BASE_URL=http://mac-host.example:4310
export AGENT_DEVICE_DAEMON_AUTH_TOKEN=<token>

curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \
  -H "content-type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{"jsonrpc":"2.0","id":"alloc-1","method":"agent_device.lease.allocate","params":{"tenantId":"acme","runId":"run-123","ttlMs":60000}}'

agent-device \
  --tenant acme \
  --session-isolation tenant \
  --run-id run-123 \
  --lease-id <lease-id> \
  session list --json

Heartbeat and release example:

curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \
  -H "content-type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{"jsonrpc":"2.0","id":"hb-1","method":"agent_device.lease.heartbeat","params":{"leaseId":"<lease-id>","ttlMs":60000}}'

curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \
  -H "content-type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{"jsonrpc":"2.0","id":"rel-1","method":"agent_device.lease.release","params":{"leaseId":"<lease-id>"}}'

Session-locked RPC command example:

curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \
  -H "content-type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{"jsonrpc":"2.0","id":"cmd-1","method":"agent_device.command","params":{"session":"qa-ios","command":"snapshot","positionals":[],"meta":{"lockPolicy":"reject","lockPlatform":"ios","tenantId":"acme","runId":"run-123","leaseId":"<lease-id>"}}}'

Transport prerequisites

• Start the daemon in HTTP mode with AGENT_DEVICE_DAEMON_SERVER_MODE=http|dual.
• Point the client at the remote host with AGENT_DEVICE_DAEMON_BASE_URL=http(s)://host:port[/base-path].
• Use AGENT_DEVICE_DAEMON_AUTH_TOKEN or --daemon-auth-token when the client should send the shared daemon token automatically.
• Direct JSON-RPC callers can authenticate with request params, Authorization: Bearer <token>, or x-agent-device-token.
• Prefer an auth hook such as AGENT_DEVICE_HTTP_AUTH_HOOK when the host needs caller validation or tenant injection.

Lease lifecycle

Use JSON-RPC methods on POST /rpc:

• agent_device.lease.allocate
• agent_device.lease.heartbeat
• agent_device.lease.release

Keep the lease alive for the duration of the run and release it when the tenant-scoped work is complete.

Host-level lease knobs:

• AGENT_DEVICE_MAX_SIMULATOR_LEASES
• AGENT_DEVICE_LEASE_TTL_MS
• AGENT_DEVICE_LEASE_MIN_TTL_MS
• AGENT_DEVICE_LEASE_MAX_TTL_MS

Command admission contract

For tenant-isolated command execution, pass all four CLI flags together:

agent-device \
  --tenant acme \
  --session-isolation tenant \
  --run-id run-123 \
  --lease-id <lease-id> \
  session list --json

The CLI sends AGENT_DEVICE_DAEMON_AUTH_TOKEN in both the JSON-RPC request token field and HTTP auth headers so existing daemon auth paths continue to work.

Failure semantics and trust notes

• Missing tenant, run, or lease fields in tenant-isolation mode should fail as INVALID_ARGS.
• Inactive or scope-mismatched leases should fail as UNAUTHORIZED.
• Inspect logs on the remote host during remote debugging. Client-side --debug does not tail a local daemon log once AGENT_DEVICE_DAEMON_BASE_URL is set.
• Treat daemon auth tokens and lease identifiers as sensitive operational data.

Verification

When to open this file

Open this file when the task needs evidence, regression checks, replay maintenance, or startup performance measurements after the main interaction flow is already working.

Main commands to reach for first

• screenshot
• diff snapshot
• record
• replay -u
• perf

Most common mistake to avoid

Do not use verification tools as the first exploration step. First get the app into the correct state with the normal interaction flow, then capture proof or maintain replay assets.

Canonical loop

agent-device open Settings --platform ios
# after using exploration to reach the state you want to verify
agent-device snapshot
agent-device screenshot /tmp/settings-proof.png
agent-device close

Structural verification with diff snapshot

Use diff snapshot when you need a compact view of how the UI changed between nearby states.

agent-device snapshot -i
agent-device press @e5
agent-device diff snapshot -i

• Initialize the baseline at a stable point.
• Perform the mutation.
• Run diff snapshot to confirm the expected structural change.
• Re-run full snapshot only when you need fresh refs.

Visual artifacts

Use screenshot when the proof needs a rendered image instead of a structural tree.

Session recording

Use record for debugging, documentation, or shareable verification artifacts.

agent-device record start ./recordings/ios.mov
agent-device open App
agent-device snapshot -i
agent-device press @e3
agent-device close
agent-device record stop

• record supports iOS simulators, iOS devices, and Android.
• On iOS, recording is a wrapper around simctl for simulators and the corresponding device capture path for physical devices.
• On Android, recording is a wrapper around adb.
• Recording writes a video artifact and a gesture-telemetry sidecar JSON.
• On macOS hosts, touch overlay burn-in is available for supported recordings.
• On non-macOS hosts, recording still succeeds but the video stays raw and record stop can return an overlayWarning.
• If the agent already knows the interaction sequence and wants a more lifelike, uninterrupted recording, drive the flow with batch while recording instead of replanning between each step.

Example:

agent-device record start ./recordings/smoke.mov
agent-device batch --session sim --platform ios --steps-file /tmp/smoke-steps.json --json
agent-device record stop

• Use this only after exploration has stabilized the flow.
• Keep the batch short and add wait or is exists guards after mutating steps so the recorded flow still tracks realistic UI timing.

Replay maintenance

Use replay updates when selectors drift but the recorded scenario is still correct.

agent-device replay -u ./session.ad
agent-device test ./smoke --platform android

• Prefer selector-based actions in recorded .ad replays.
• Use test when you already have multiple .ad flows and need a quick regression pass after updating or recording them.
• Keep the skill-level rule simple: use replay -u to maintain one script, use test to verify a folder or matcher of scripts.
• Treat test as a human and CI-facing suite runner that an agent can invoke for verification, not as the main source of product documentation.
• Failed runs keep suite artifacts under .agent-device/test-artifacts by default, which is usually enough for debugging without extra agent-side processing.
• Use update mode for maintenance, not as a substitute for fixing a broken interaction strategy.

Performance checks

Use perf --json or metrics --json when you need startup timing for the active session.

agent-device open Settings --platform ios
agent-device perf --json

• Current startup data is command round-trip timing around open.
• It is not true first-frame or first-interactive telemetry.
• fps, memory, and cpu are currently placeholders.