Desktop Control

Desktop-Control Driver

The orchestrator routed you here. Use these mechanics to execute your plan.

Drive native desktop GUI apps through upstream trycua/cua cua-driver: enumerate apps and windows, snapshot accessibility trees, click/type/scroll by element_index or pixel coordinates, and verify by re-snapshot -- all without bringing the target to the foreground.

When to use

• Automating a native desktop app (Finder, Notepad, System Settings, native editors)
• Driving native dialogs and security/permission sheets that no DOM or PTY can reach
• Visual QA of native UI: per-window screenshots, accessibility-tree assertions

If the target is a terminal TUI, use tuistory or true-input. If it is a web page or an Electron app, use agent-browser -- CDP beats accessibility trees for anything Chromium-based.

Platform support

Read the platform file for your target OS. Each contains permissions, daemon launch, and platform-specific patterns and failure modes.

Prerequisites

# one-time install: per-user, no sudo/admin
curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh | bash
# Windows (PowerShell):
#   irm https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.ps1 | iex

cua-driver doctor           # platform probes: permissions, daemon, accessibility plumbing
cua-driver skills install   # fetch the upstream skill pack to ~/.cua-driver/skills/cua-driver

The upstream pack (~/.cua-driver/skills/cua-driver/SKILL.md + your platform's doc) is the deep reference -- full tool surface, window-state behavior matrix, forbidden-command lists -- and it updates with the binary. Read it before any nontrivial workflow. This atom owns the droid-control integration: routing, run isolation, delegation, evidence handoff.

Daemon lifecycle

element_index workflows require the daemon. Without it each CLI invocation is a fresh process and the per-(pid, window_id) element cache dies between calls.

cua-driver serve            # start the daemon (macOS needs the LaunchServices form -- see platforms/macos.md)
cua-driver status           # daemon + socket health
cua-driver stop

Permissions are checked and granted through the driver, not by hand-editing system settings (macOS-only gate; a no-op surface on Windows/Linux):

cua-driver permissions status   # read-only; answers via the running daemon
cua-driver permissions grant    # attributed prompt flow -- the correct way to grant

Core loop

Tool names are snake_case and invoked directly: cua-driver <tool> '<json>'. (cua-driver call <tool> is legacy; do not use it.) cua-driver list-tools for the inventory, cua-driver describe <tool> for any schema.

Every workflow is Discover -> Observe -> Act -> Verify against an explicit (pid, window_id):

cua-driver launch_app '{"name":"TextEdit"}'
#  -> {pid: 844, windows: [{window_id: 10725, ...}]}   # list_windows only needed for long-lived pids
cua-driver get_window_state '{"pid":844,"window_id":10725}' --screenshot-out-file "${RUN_DIR}/before.png"
cua-driver click '{"pid":844,"window_id":10725,"element_index":14,"session":"'"${RUN_ID}"'-desktop"}'
cua-driver get_window_state '{"pid":844,"window_id":10725}' --screenshot-out-file "${RUN_DIR}/after.png"

Snapshot before AND after every action. The pre-action get_window_state resolves the element_index you are about to use -- indices are per-snapshot, per (pid, window_id), and stale ones fail with No cached AX state. The post-action snapshot is the evidence the action landed; without it a silent no-op looks like success.

Addressing-mode preference:

1. `element_index` (default) -- semantic, works on hidden and backgrounded windows, no foreground change. 2. Pixel click '{"pid":N,"window_id":W,"x":X,"y":Y}' -- for surfaces the tree does not reach (canvases, custom-drawn controls). Coordinates are window-local screenshot pixels, top-left origin. 3. Keyboard (press_key, hotkey) and platform fallbacks -- last resort; see the platform files.

Run isolation (ground rule 5 -> cua sessions)

cua sessions are the desktop equivalent of tctl session prefixes: a session owns its agent cursor, config overrides, and recording scope. Declare one per run, derived from the workflow's RUN_ID, and pass it on every action:

cua-driver start_session '{"session":"'"${RUN_ID}"'-desktop"}'
# ... every action carries "session":"${RUN_ID}-desktop" ...
cua-driver end_session '{"session":"'"${RUN_ID}"'-desktop"}'

Parallel workers each declare their own session and pass creates_new_application_instance: true to launch_app so each gets its own window. The element cache is keyed on (pid, window_id) and the cursor on session, so isolated workers cannot collide.

Delegation

cua-driver is on PATH -- workers need no ${DROID_PLUGIN_ROOT} resolution. As with the other drivers, give capture workers exact commands with the parent's run scope baked in:

Task prompt for a desktop capture worker:
  "Run these commands in order. Report screenshot paths and any errors.
   1. cua-driver start_session '{"session":"1712345678-42-notepad"}'
   2. cua-driver launch_app '{"name":"Notepad","creates_new_application_instance":true}'
      -> note the returned pid and window_id
   3. cua-driver get_window_state '{"pid":<pid>,"window_id":<wid>}' --screenshot-out-file /tmp/droid-run-1712345678-42-xxxx/before.png
   4. cua-driver type_text '{"pid":<pid>,"window_id":<wid>,"element_index":<text-area>,"text":"hello","session":"1712345678-42-notepad"}'
   5. cua-driver get_window_state '{"pid":<pid>,"window_id":<wid>}' --screenshot-out-file /tmp/droid-run-1712345678-42-xxxx/after.png
   6. cua-driver end_session '{"session":"1712345678-42-notepad"}'"

Evidence handoff

Hand PNG/mp4 paths to compose / verify like any other driver output. Keep raw tool output alongside screenshots whenever GUI behavior is the thing under test.

Critical rules

1. Never change the user's frontmost app. If a command says activate, foreground, raise, or make key -- stop; the per-pid event paths exist precisely so you do not need it. Platform forbidden-lists live in the upstream pack. 2. Re-snapshot after every action and report what you observed, not what you intended. An unchanged tree after an action is a finding, not a formality. 3. Destructive actions need explicit user intent. Do not delete files, send messages, or submit forms unless the workflow asked for exactly that.

Desktop-Control: Linux

cua-driver on Linux enumerates windows via X11, walks semantic trees via AT-SPI, and injects input via XSendEvent (synthetic events targeted at a window XID -- no focus change, nothing leaks to the user's focused app). Upstream calls this tier pre-release, and it shows: the lifecycle (install, daemon, doctor, sessions, one-shot CLI), window discovery, and per-window screenshots are solid; Wayland-native enumeration, AT-SPI tree quality, and input delivery are not. Plan workflows around the reliable half.

Install and daemon

Same installer and lifecycle as everywhere else (no sudo, ~/.cua-driver):

cua-driver doctor    # trustworthy probes: catches missing DISPLAY, verifies X11 + AT-SPI before you waste a run
cua-driver serve     # required for element_index workflows
cua-driver status

cua-driver permissions is a no-op surface on Linux.

The Wayland boundary

Window enumeration is X11-only. On a modern Plasma/GNOME Wayland desktop, native-Wayland windows are invisible to list_windows -- which is most windows.

• Targets running under Xwayland (or a plain X11 session) enumerate and screenshot fine.
• To drive an app that defaults to native Wayland, force its X11 backend at launch where the toolkit allows it: QT_QPA_PLATFORM=xcb (Qt), GDK_BACKEND=x11 (GTK), --ozone-platform=x11 (Chromium/Electron).
• If the target cannot be put on X11, desktop-control cannot see it -- fall back to agent-browser (web/Electron) or true-input (terminal emulators).

Semantic layer (AT-SPI) reliability

AT-SPI trees can collapse: the registry's GetChildren may time out, and Qt apps can render as a single root node even with QT_LINUX_ACCESSIBILITY_ALWAYS_ON=1. When get_window_state returns a near-empty tree:

cua-driver config set capture_mode vision   # screenshot-only snapshots

and work the pixel path (click '{"pid":N,"window_id":W,"x":X,"y":Y}') against the returned PNG. Don't burn turns re-snapshotting hoping the tree fills in -- on this tier, pixel-first is a legitimate default.

The toolkit boundary: synthetic input is silently dropped by Qt and GTK4

XSendEvent marks events with the send_event flag, and major toolkits ignore flagged input entirely. Verified on v0.5.1: Qt apps (kcalc) and GTK4 apps (zenity) no-op on every action -- pixel clicks, press_key, type_text -- while the driver reports success. There is no error to catch; only the post-action snapshot reveals it.

Practical consequence: the Act stage only works against apps that honor synthetic events (verified: winit-based apps like alacritty; generally simpler/older X11 toolkits). Probe before committing to a workflow: send one cheap keystroke, re-snapshot, and check it rendered. If the target ignores synthetic input, desktop-control cannot act on it on this tier -- Observe (screenshots, window enumeration) still works, but route the interaction through agent-browser (web/Electron) or true-input (terminal) instead.

Text input is lossy even where it lands

In apps that do accept synthetic input, typing drops and mangles characters: shifted symbols can inject as their unshifted key (* arriving as 8), trailing characters get dropped (verified: type_text "echo ok42" rendered echo ok4), and type_text_chars with generous per-char delays still loses keystrokes. hotkey chords (including paste shortcuts) and middle-click paste do not land reliably, so the clipboard is not a workaround here.

What works: short bursts plus verification. After every type_text, re-snapshot, compare the rendered text against what you sent, and repair the diff (press_key backspace, retype the missing tail). On Linux the post-action screenshot is not a formality -- it is the only way to know what actually arrived.

Failure modes

Deep mechanics live in the upstream pack: ~/.cua-driver/skills/cua-driver/LINUX.md.

Desktop-Control: macOS

cua-driver on macOS posts events per-pid through Accessibility (AX) and captures via ScreenCaptureKit. Both are gated by TCC, and TCC attributes grants to the app bundle that asks -- which is why every flow below routes through CuaDriver.app instead of your terminal.

Permissions (TCC)

cua-driver permissions grant    # LaunchServices-routed: the Accessibility + Screen Recording dialogs
                                # attribute to com.trycua.driver, then it confirms the driver's own status
cua-driver permissions status   # read-only via the daemon; reports `unknown` when no daemon is up

Do not grant by clicking through System Settings for your terminal app -- the daemon runs under the bundle identity, and terminal-attributed grants do nothing for it. The first real screen capture may trigger one extra consent sheet; accept it.

Daemon launch

Launch from the logged-in GUI session so the daemon attaches to it with the bundle's TCC identity:

open -n -g -a CuaDriver --args serve
cua-driver status
cua-driver stop

SSH-launched bare binaries often miss the GUI session and their AX/capture probes hang. (cua-driver mcp and CLI tool calls auto-proxy to a properly attributed daemon when one is reachable.)

Patterns

Reliable terminal command entry -- when type_text or raw key posting drops characters in Terminal-class apps, route through the pasteboard:

printf '%s' 'your command' | pbcopy
cua-driver hotkey    '{"pid":<term-pid>,"window_id":<wid>,"keys":["cmd","v"]}'
cua-driver press_key '{"pid":<term-pid>,"window_id":<wid>,"key":"return"}'

Native security / modal sheets (SecurityAgent, Keychain prompts, auth dialogs) -- these often report is_on_screen: false even while visible. Locate by process, then enumerate everything:

pgrep -fl SecurityAgent
cua-driver list_windows '{"pid":<sa-pid>,"on_screen_only":false}'
cua-driver get_window_state '{"pid":<sa-pid>,"window_id":<wid>}'

Only enter credentials in environments you own and were explicitly authorized to drive.

Menu commands / app shortcuts -- pass window_id so AppKit routes the key equivalent to the target app instead of the frontmost one:

cua-driver hotkey '{"pid":835,"window_id":79,"keys":["cmd","q"]}'

Backgrounded / off-space windows -- the driver acts on (pid, window_id) without raising. Enumerate with on_screen_only: false and target directly.

Failure modes

Deep mechanics (no-foreground forbidden-list, AXMenuBar navigation, SkyLight click dispatch, Apple-Events browser bridge) live in the upstream pack: ~/.cua-driver/skills/cua-driver/MACOS.md.

Desktop-Control: Windows

cua-driver on Windows walks UI Automation (UIA) trees and dispatches actions through a layered UIA + PostMessage chain -- per-window message posting, not HID synthesis, so the user's foreground app is untouched.

Install and daemon

The upstream installer is per-user (no admin elevation): binary under %LOCALAPPDATA%\Programs\Cua\cua-driver\bin, data and skill pack under %USERPROFILE%\.cua-driver, and an autostart task (cua-driver autostart status|kick|disable) registered for the daemon.

cua-driver doctor
cua-driver serve     # required for element_index workflows
cua-driver status
cua-driver stop

cua-driver permissions is a no-op surface on Windows (TCC is a macOS concept) -- there is no grant dance. The real constraint is Session 0 isolation: anything launched by a service (including some SSH daemons) lives in a session with no interactive desktop, where window enumeration returns nothing. Tool calls auto-proxy to an interactive-session daemon when one is reachable; if results come back empty, confirm the daemon was started from the logged-in interactive session, not a service context.

JSON quoting (the PowerShell 5.1 footgun)

Windows PowerShell 5.1 strips quotes around JSON field names in multi-field arguments, so positional JSON fails to parse. Pipe via stdin, or use PowerShell 7+ (pwsh):

'{"pid":1234,"window_id":5678}' | cua-driver get_window_state

From cmd.exe, escape inner quotes instead: cua-driver get_window_state "{\"pid\":1234,\"window_id\":5678}".

Patterns

UWP / packaged apps -- Store apps (Calculator, Settings) are hosted by ApplicationFrameHost.exe, so the visible window's pid is the host's, not the app process's. If list_windows against the app's own pid comes up empty, enumerate ApplicationFrameHost.exe's windows and match by title. Classic Win32 apps (Notepad, Explorer) own their windows directly.

Minimized windows -- get_window_state and element-index actions work in place, but press_key commits silently no-op (no message pump focus). Use set_value or element-index-click the commit-equivalent button instead.

Browsers / Electron -- prefer agent-browser. If you must stay in desktop-control, launch the browser with --remote-debugging-port=<port> and export CUA_DRIVER_CDP_PORT=<port> so execute_javascript / query_dom can attach; UIA covers get_text either way.

Failure modes

Deep mechanics (UIA tree semantics, click-dispatch layering, focus-steal vectors, UAC boundaries) live in the upstream pack: ~/.cua-driver/skills/cua-driver/WINDOWS.md.