Desktop Recorder

Desktop Recorder

Built on deskagent (ScreenCaptureKit + AXPress + Vision OCR). The agent never improvises during the final take - exploration is unconstrained, the recording is a deterministic JSON-script replay.

Pre-recording checklist - ASK, don't pick silently

Use AskUserQuestion (or plain text) before recording. State defaults explicitly so the user can shrug and accept.

The 6 rules

1. Explore first

Tools allowed during exploration only:

• deskagent list --json → window picking
• deskagent inspect --window $ID → element coords (AX + OCR)
• deskagent assert --window $ID --label X → cheap yes/no probe
• deskagent screenshot --window $ID → visual reference; on macOS 26/Tahoe

hangs or capture failures, use node scripts/record-screenshot.js --window $ID --out /tmp/shot.png

• Trial deskagent control runs against the live UI

Collect: click coords (window-relative), per-step waits, demo data, popups to normalize, the recording's window size (pin BEFORE inspecting), start/end states, captions.

2. The screenplay - single source of truth

One screenplay.json describes the demo end-to-end: scenes of actions to execute, plus top-level composition / zoom / speed / captions / trim directives. deskagent control reads only scenes[].actions[]; editor scripts read the top-level directives.

{
  "schema_version": 2,
  "name": "demo1",
  "coordinate_space": "window",
  "scenes": [
    { "id": "open_settings", "windowId": 245663,
      "actions": [ { "action": "click", "x": 244.5, "y": 54.5 } ] },
    { "id": "wait_load",     "windowId": 245663,
      "actions": [ { "action": "wait", "ms": 4000 } ] }
  ],
  "composition": {
    "canvas":     [2560, 1600],
    "background": "color:0a0a10",
    "layout":     "side-by-side",          // optional; auto-computes slot rects
    "padding":    60,                      // optional gap (used by layout)
    "elements": [
      { "windowId": 245663 }               // auto rect from layout
    ]
  },
  "zoom": [
    { "scale": 2.0, "follow_cursor": true,
      "fromAction": "open_settings/0", "toAction": "wait_load/0" }
  ],
  "speed": [
    { "factor": 2.5,
      "fromAction": "wait_load/0", "toAction": "wait_load/0", "endDelayMs": 4000 }
  ],
  "captions": [
    { "text": "Open Settings",       "fromAction": "open_settings/0", "durationMs": 1500 },
    { "text": "Loading the panel…",  "fromAction": "wait_load/0",     "toAction": "wait_load/0", "endDelayMs": 3000 }
  ],
  "trim": { "beforeScene": "open_settings", "afterScene": "wait_load" }
}

composition drives scripts/stages/compose.js - required for multi-clip recordings, optional when there's a single clip. zoom, speed, and captions are top-level arrays with global fromAction / toAction refs

• a single entry can span any range of actions across any scenes, with

timing offsets via startDelayMs/endDelayMs. Zoom camera modes: follow_cursor: true tracks the synthetic cursor across click-driven ranges; pan: [...] does cinematic sweeps over no-click ranges (the cursor is hidden during pans). Don't overlap a pan with clicks - use follow_cursor there. See `references/desktop.md` for the full shape.

coordinate_space: "window" keeps screenplays portable across window drags. Per-scene windowId routes that scene's action coords to the correct window in multi-window compositions. No conditionals, no wait_for, no observation loops - every action is deterministic. Recording start/stop is outside the screenplay; the agent owns the lifecycle.

3. Dry-run + state fingerprint

Run the full script with --background against the live UI (no recorder active). Iterate until clean.

Before the final take, prove state matches what the script expects:

1. deskagent screenshot --window $ID → visual record. If screenshot hangs or fails on macOS 26/Tahoe, use the streaming fallback: node scripts/record-screenshot.js --window $ID --out /tmp/shot.png. 2. At least two deskagent assert calls on labels that exist ONLY in the expected sub-state. (Section names usually persist across sub-states - they're too coarse.) 3. Refuse to record if any assert fails. Re-normalize in setup.

Per-step assertions inside recording are NOT supported - the recording is meant to be a deterministic replay, not a probe loop.

4. Pin window info BEFORE recording

Capture id, pid, and x,y for every window during exploration, then drive with --target-pid + --window-frame + --background during the take. No WindowServer lookups during the run, no chance of the wrong window being picked if focus shifted between dry-run and record. Multi-window: collect pid+origin per window in one pre-record list call; route each control invocation via its own flags.

Targeting flag rules are documented in references/deskagent.md. The short form: HID mode needs --target-window (activates the app); background mode needs --target-pid + --window-frame (delivers per-pid, no activation).

5. No live recovery - re-record

Final take failed? Stop, discard, fix the script, re-record. Don't patch live.

6. Don't resize source windows after exploration

Any osascript … set size of window belongs in setup BEFORE exploration, confirmed with the user. Never inside recording. Composition is done in the editor's compose stage; per-clip placement + canvas + background live in the screenplay's composition block, so a mid-take window resize would still mismatch the composition rects.

End-to-end pipeline

1. Explore  -> pin id/pid/origin per window, gather click coords.
2. Author   -> screenplay.json (scenes + actions, top-level composition/zoom/speed/captions/trim).
3. Normalize state (close tabs, log in, hide chrome, theme).
4. Dry-run  -> deskagent control screenplay.json --target-window $ID --timeline /tmp/dry.json
5. Fingerprint state with deskagent assert.
6. Record   -> deskagent record ./demo/raw --window $ID [--window $ID2 ...] --no-cursor --pid-file /tmp/rec.pid [--supersample 2] &
                  (writes per-source ProRes 4444 .mov clips + recording.manifest.json into ./demo/raw/)
7. Drive    -> deskagent control screenplay.json --target-pid $PID --window-frame "$ORIGIN" \
                  --background --no-activate --timeline timeline.json
                  (run in parallel per target window for multi-window demos)
8. Stop     -> kill -INT $(cat /tmp/rec.pid); wait
9. Export   -> node scripts/export.js ./demo/raw screenplay.json timeline.json demo.final.mp4 [format] [--quality high]
                  (one ffmpeg pass: compose -> highlights -> zoom -> captions -> speedups -> encode)
10. Copy    -> node scripts/generate_copy.js timeline.json prompt.txt copy.md

Every editing operation runs in a single ffmpeg invocation off the per-source clips - no intermediate mp4s in the hot path, no generation-loss from per-stage re-encodes. The final encoder choice (HEVC / H.264 / ProRes 422) is the only re-encode in the entire pipeline.

Outputs (one demo folder, default ./demo-out/<name>/)

screenplay.json        single source of truth (scenes + composition + directives)
timeline.json          execution evidence from `deskagent control`
raw/                   directory written by `deskagent record`:
  recording.manifest.json   per-clip paths + host-time alignment anchors
  window-<id>.mov           one ProRes 4444 .mov per source (alpha-preserving)
  display-<id>.mov
demo.final.mp4         the deliverable (single ffmpeg pass via scripts/export.js)
copy.md                upload copy

For per-stage debugging, each editing stage has a CLI that can either emit its filter fragment as JSON (used by the orchestrator) or render it standalone to a ProRes 4444 .mov to inspect just that stage's effect:

node scripts/stages/zoom.js generate <recDir> <screenplay> <timeline>
node scripts/stages/zoom.js generate <recDir> <screenplay> <timeline> --apply input.mov output.mov

References (load on demand)

• references/deskagent.md - CLI surface, permissions, recording.manifest.json schema
• references/desktop.md - screenplay schema (scenes + composition + captions + zoom + speed + trim), setup normalization, state-fingerprint recipe
• references/timeline.md - timeline event schema (scene_start / action / scene_end)
• references/editing.md - export orchestrator + per-stage modules (compose / highlights / zoom / captions / speedups); cursor track + cursor.hide/show
• references/web-driver.md - CDP web driver (drive-web.js): drive browser page content focus-free; trajectory/pointer actions; draw on web canvases
• remotion-template/ - optional Remotion bridge: turn a recording (manifest+timeline+screenplay) into React motion graphics. The bridge (loadRecording + <RecordingCard>/<Cursor>/<ClickRipple>/<Caption>) does the integration; you only write the creative composition. See its README.md.

Authorization rule

deskagent control mutates the user's app state (focus, tabs, open docs). --background minimizes visible impact but still mutates. Confirm before touching any app the user didn't name. Don't silently swap targets or open new tabs / windows / documents.

Quick failure map

Chromium-based browsers (Chrome, Edge, Brave, Arc)

For page content, prefer the CDP web driver (scripts/drive-web.js) over deskagent control - it injects input at the renderer, so it needs no focus and no foreground (drive an unfocused window while recording), targets the DOM by selector (no pixel-guessing / mis-clicks), and emits the same timeline.json so record + export are unchanged. It replaces control for browser scenes; full setup (launch flags, actions, coordinates) in `references/web-driver.md`. Use deskagent control only for the browser's own UI (omnibox, tabs, extensions, menus) - CDP can't reach those.

Fallback (no debug Chrome): deskagent control HID mode reaches only the browser chrome via --background; for page content record with --no-cursor and drive with --target-window (HID, cursor moves during the take). OCR beats AX for Chrome element discovery; URL-bar nav (cmd+l → type → return) beats chasing SPA links.

Keep the browser window at least partially visible during the take. When fully occluded, the renderer throttles timers / rAF / repaints (Page Visibility), so the clip captures stalled animations or stale content - even though SCK is still pulling frames. (Launch Chrome with the --disable-*backgrounding* / throttling flags - see web-driver.md - to keep an unfocused window compositing.)

Out of scope

Mobile (use mobile-recorder-skill); Linux/Windows; direct upload; AI voiceover / music; GUI video editor.

Title

{{title}}

Short post

{{short_post}}

YouTube Shorts title

{{shorts_title}}

Thumbnail text

{{thumbnail_text}}

Hashtags (optional)

{{hashtags}}

Notes

• Replace any {{...}} placeholder before posting.
• Keep title under 60 characters where possible.
• Shorts title under 40 characters.
• Thumbnail text 3–5 words, large-text safe.
• Short post should describe the flow shown in the video, not the product overall.

deskagent - CLI surface

Native macOS recorder + deterministic input replayer + AX/OCR inspector. ScreenCaptureKit captures occluded/minimized windows. macOS only.

WebView/browser caveat: capture works while occluded, but apps with web content (Chrome, Safari, Electron, Wails) throttle timers, rAF, and repaints when the window is fully occluded (Page Visibility / occlusion throttling). The clip then shows stalled animations or stale content. Keep such windows at least partially visible during the take. Native AppKit windows are unaffected.

All commands accept --json. Exit codes: 0 ok, 1 assertion-failed (assert only), 2 runtime/permission, 64 usage.

Permissions

deskagent doctor reports both. --request-accessibility surfaces the prompt without making a control call.

Discovery

deskagent list

deskagent list --json

Emits { displays: [...], windows: [{id, pid, app, bundleID, title, x, y, width, height, onScreen}] }. Window IDs are per-launch - re-list before each record. Add --all to surface occluded / minimized windows (onScreen: false).

Pick one:

INFO=$(deskagent list --all --json | jq -c '[.windows[] | select(.app=="Safari")] | first')
ID=$(echo "$INFO" | jq -r '.id')
PID=$(echo "$INFO" | jq -r '.pid')
ORIGIN=$(echo "$INFO" | jq -r '"\(.x),\(.y)"')

deskagent inspect

deskagent inspect --window <id> \
    [--ax|--no-ax] [--ocr|--no-ocr] \
    [--label "Submit" ...] [--role AXButton ...] \
    [--json]

Two complementary sources, both on by default:

• --ax - native AppKit; one entry per AX element (role, label, bbox).
• --ocr - Vision text recognition; works on any pixels (Wails/Electron/Canvas).

Filters (repeatable): --label (case-insensitive substring), --role (exact AX role). When --label matches AX, the OCR pass is skipped.

Single inspect is the most expensive op in this CLI (AX walk + Vision OCR). Cache the result and jq it locally:

SNAPSHOT=$(deskagent inspect --window $ID --json)
echo "$SNAPSHOT" | jq -r '.ax[] | select(.label=="Submit") | .center | @sh'

For Wails/Electron, --ax usually returns only the window chrome. Use --ocr for those, or rely on --background's AXPress hit-test which walks the WebKit-bridged AX tree at click time.

deskagent assert

deskagent assert --window <id> \
    [--label "X" ...] [--label-any "A,B"] [--role AXButton ...] \
    [--absent] [--no-ocr] [--json]

Exit codes: 0 found · 1 absent · 2 error. --absent inverts.

--label is AND across flags; --label-any is OR across CSV entries. JSON returns { found, source, label, role, center, bbox } for the first match - pipe straight into a click.

deskagent screenshot

deskagent screenshot --window <id> \
    [--region x,y,w,h] [--annotate-bboxes] \
    [--quality 1-100] [--max-dim N] \
    [--out path.jpg] [--json]

JPEG by default - PNG via --out *.png. Defaults: quality=85, max-dim=1568 (Claude's resize threshold), output $TMPDIR/deskagent/<window-id>-<ms>.jpg. JSON emits {path, pixelSize, windowFrameCG, backingScale, format}.

--annotate-bboxes overlays AX (cyan) + OCR (yellow) rectangles - useful for visual verification of which element a label resolves to.

macOS 26/Tahoe screenshot fallback

On macOS 26/Tahoe, deskagent screenshot can hang or fail even when doctor reports Screen Recording as granted and deskagent record succeeds. When that happens, take the visual reference through the streaming path:

node scripts/record-screenshot.js --window <id> --out /tmp/shot.png

The helper runs a short deskagent record, stops it with SIGINT, and extracts one frame with ffmpeg. Use it for fingerprints and visual checks; keep final takes on the normal deskagent record pipeline.

Recording: deskagent record

record writes one ProRes 4444 .mov per source into an output directory, plus recording.manifest.json. Per-pixel alpha is preserved on every clip, so the editor can composite windows onto any background without the OS's rounded corners or shadows showing through as black.

deskagent record /tmp/demo --window "$ID" \
    --fps 60 \
    --pid-file /tmp/rec.pid --quiet --json > /tmp/rec.json &
# … drive …
kill -INT "$(cat /tmp/rec.pid)"; wait
# /tmp/demo/ now contains window-<ID>.mov + recording.manifest.json

Never `kill -9` - the .mov's moov atom won't flush.

Composition / quality / final-encode are an editor concern; see editing.md. record always captures BGRA → ProRes 4444 .mov. The editor's export.js picks the final container, codec, and bitrate.

Source flags

Behavior flags

Output

Each source writes to <out-dir>/<kind>-<id>.mov (e.g. window-12345.mov, display-1.mov). <out-dir>/recording.manifest.json is written after all clips finalize.

JSON stdout (--json):

{
  "status": "ok",
  "directory": "/tmp/demo",
  "manifest": "/tmp/demo/recording.manifest.json",
  "durationSeconds": 12.3,
  "fps": 60,
  "clips": [
    { "path": "/tmp/demo/window-12345.mov",
      "source": "window:12345",
      "frames": 740, "dropped": 0,
      "startWallclockMs": 1716480000050 }
  ]
}

Manifest sync anchors

Every clip in recording.manifest.json carries startHostNs and endHostNs. The editor uses them to compute a shared time window:

• t0 = max(clip.startHostNs) - latest first-frame across clips.
• tEnd = min(clip.endHostNs) - earliest last-frame.
• Per clip head-trim: (t0 - clip.startHostNs) / 1e9 seconds.
• Composited duration: (tEnd - t0) / 1e9 seconds.

Alpha & color

Capture is BGRA via SCStream → ProRes 4444 (yuva444p12le) via AVAssetWriter. Each clip's alpha channel reflects the window's real shape - areas outside the window's content have alpha=0. Compositing in the editor (compose stage) uses ffmpeg's overlay, which respects the source alpha natively.

Desktop control: deskagent control

A deterministic replayer. No screen observation, no retry. Author a screenplay, dry-run, replay.

Screenplay schema

Full reference: `desktop.md`. Minimal shape:

{
  "schema_version": 2,
  "coordinate_space": "window",
  "scenes": [
    {
      "id": "open_settings",
      "caption": "Open Settings",
      "actions": [
        { "action": "click", "x": 320, "y": 180 },
        { "action": "wait",  "ms": 600 }
      ]
    }
  ]
}

Action kinds:

pointer_down → pointer_move(path) → pointer_up is one continuous stroke - draw lines/circles/bezier by sampling the curve into path points. The same actions run in the web driver (CDP) - see `web-driver.md`.

coordinate_space: "window" is the agent default - pair with deskagent inspect's window-relative coords for portability.

deskagent control ignores screenplay's editing-only fields (zoom, speed, trim, setup, preflight, ...). Editing scripts read them directly.

Invocation

deskagent control screenplay.json \
    [--target-window ID | --target-pid PID --window-frame "x,y"] \
    [--background] [--no-activate] \
    [--timeout-ms N] [--prompt-permission] \
    [--timeline /tmp/timeline.json] [--mouse-path /tmp/mp.json] [--json]

HID vs --background

HID is the safe default. --background is for "user keeps working in another app" demos.

--background caveats:

• Modifier shortcuts can drop - briefly activate the target for

essential ones, or use a menu/osascript equivalent.

• Apps with no AX exposure won't accept --background clicks. Validate

with deskagent inspect --window <id> --ax.

• Inter-character pacing: per-PID type adds 8 ms per char (target

drains slower than the global tap). HID has zero delay.

• Wails-occluded repaint lag: when the captured window is fully

occluded, the backing layer can lag the actual UI state by several frames. Clicks still register in app state, but the captured pixels may show stale content. Bring the window forward if pixel-accuracy matters.

Timeline output

Full schema: `timeline.md`. Each scene produces scene_start / action (one per executed action) / scene_end events, each tagged with scene_id / scene_index, and (for actions) canonical action_id = "<scene_id>/<action_index>". Editing scripts join screenplay directives against these IDs.

For overlay alignment use the *WallclockMs fields together with the recording's firstFrameWallclockMs. The startedAtMs field is process-local and not anchored to the video.

type MousePathSample = { tMs: number, x: number, y: number }

Recording manifest

deskagent record <out-dir> writes <out-dir>/recording.manifest.json after every clip finalizes (current schema version: 1):

{
  "version": 1,
  "createdAtWallclockMs": 1778883093800,
  "fps": 60,
  "anchorHostNs": 123456789012345,
  "durationSeconds": 18.4,
  "clips": [
    {
      "path": "window-245663.mov",
      "kind": "window",
      "id": 245663, "pid": 40115,
      "app": "MobAI", "bundleID": "run.mobai.app",
      "title": "MobAI - Untitled",
      "frameCG": [538, 90, 1190, 831],
      "pixelSize": [2380, 1662],
      "backingScale": 2.0,
      "firstFramePtsNs": 0,
      "lastFramePtsNs": 18400000000,
      "frameCount": 1104,
      "droppedFrames": 0,
      "startHostNs": 123456789012345,
      "endHostNs":   123475189012345,
      "startWallclockMs": 1778883093850,
      "endWallclockMs":   1778883112250
    }
  ]
}

scripts/export.js reads the manifest, applies the screenplay's composition block via the compose stage, then chains the other editing stages in one ffmpeg pass.

Failure modes

Screenplay format

The screenplay is the single source of truth for a demo: scenes of actions to execute, plus top-level editing directives (zoom, speed, trim) and per-scene caption. deskagent control reads only scenes[].actions[]; the editing scripts read everything.

See `deskagent.md` for the CLI, `timeline.md` for the execution-event schema, `editing.md` for the editing pipeline.

Top-level shape

{
  "schema_version": 2,
  "name": "demo1",
  "coordinate_space": "window",
  "timeout_ms": 30000,
  "sample_mouse_ms": 16,

  "scenes": [ /* ... */ ],

  "composition": { /* canvas + per-clip placement; see below */ },

  "zoom":       [ /* directive entries; see below */ ],
  "speed":      [ /* directive entries; see below */ ],
  "captions":   [ /* directive entries; see below */ ],
  "highlights": { /* cursor + ripple overrides; see below */ },

  "trim": { "beforeScene": "<sceneId>", "afterScene": "<sceneId>" }
}

Action IDs are global

Every action gets a global ID <sceneId>/<actionIndex> in the timeline. The directives reference these IDs directly - they're not scoped to any scene. "open_settings/0" works just as well from a zoom entry as from a speed entry, and a single entry can span from one scene's action to a later scene's action.

Scenes

{
  "id": "open_settings",            // unique within the screenplay
  "caption": "Open Settings",       // viewer-facing; spans the whole scene
  "note":    "verify panel state",  // author/debug only; never rendered

  "actions": [
    { "action": "click", "x": 244.5, "y": 54.5 },
    { "action": "wait",  "ms": 600 }
  ]
}

(caption on scenes is no longer consumed - captions are a top-level directive array, see below.)

Scenes no longer carry zoom or speed fields. Camera and tempo are timeline-wide concerns and live at the top level.

Actions (execution-only)

{ "action": "wait",         "ms": 500 }
{ "action": "move",         "x": 1, "y": 2, "duration_ms": 250 }
{ "action": "click",        "x": 10, "y": 20, "button": "left" }   // button: left | right | middle
{ "action": "double_click", "x": 30, "y": 40 }
{ "action": "drag",         "x": 0, "y": 0, "to_x": 100, "to_y": 200, "button": "left" }  // button optional, default left
{ "action": "type",         "text": "hello", "cpm": 300 }
{ "action": "key",          "combo": "cmd+s" }                      // keys incl. f1-f12, home, end, pageup, pagedown, arrows
{ "action": "scroll",       "dx": 0, "dy": -3 }

// Trajectory move + pointer primitives (draw shapes / compose gestures):
{ "action": "move", "path": [ {"x":110,"y":130}, {"x":400,"y":130} ], "duration_ms": 600 }   // glide along a polyline
{ "action": "pointer_down", "x": 110, "y": 130 }                    // press + hold (button optional, default left)
{ "action": "pointer_move", "path": [ ... ], "duration_ms": 800 }   // move while held = a drawn stroke
{ "action": "pointer_up" }                                          // release (defaults to current position)

move glides over duration_ms; with a path it traces that polyline (constant speed). pointer_down → pointer_move(path) → pointer_up is one continuous stroke - use it to draw lines/circles/bezier (sample the curve into a path). drag is the straight-line shorthand. click/double_click/drag take button: left/right/middle.

These run identically in native deskagent control (CGEvent) and the web driver (scripts/drive-web.js, CDP). The web driver adds page-only actions (navigate, wait_for, scroll_to, scroll_page, selector/text targets, shape sugar) - see `web-driver.md`.

No intent / caption / zoom fields on actions - those live higher up. Action records are pure execution.

coordinate_space is screenplay-wide ("window" or "screen"); the executor adds the resolved window origin at runtime for "window".

Composition

composition drives scripts/stages/compose.js - the first editing stage. It maps each clip in recording.manifest.json to a rect on a shared canvas. Required when the recording has 2+ clips; optional for single-clip recordings (the clip then fills its native pixel size).

"composition": {
  "canvas":     [1920, 1080],          // [W, H] in pixels; required for 2+ clips
  "background": "color:1a1a2e",        // optional; "none" | "dark" | "light" | "color:RRGGBB" | "image:/path/to/bg.png"
  "layout":     "side-by-side",        // optional; auto-computes slot rects (see below)
  "padding":    60,                    // optional; canvas-pixel gap when layout is set
  "elements": [
    { "windowId":  12345 },                              // auto rect from layout
    { "windowId":  67890, "weight": 2 },                 // wider slot in side-by-side / taller in stack
    { "displayId": 1,     "rect": [0, 0, 1920, 1080] }   // explicit rect overrides layout
  ]
}

Inside whichever slot it lands in, the clip is aspect-fitted (letter-/pillarboxed, centered) - never stretched.

Layout modes

Mixed mode is fine: set layout: "side-by-side", give one element an explicit rect, and the rest fill the auto-computed slots in element order.

Editing directives

Half-open ranges

Every directive entry takes a fromAction / toAction pair. The range is half-open: it starts at fromAction.tStart and ends at toAction.tStart (toAction is excluded). To extend a range "through" some action, point toAction at the next action after it.

zoom

"zoom": [
  {
    "scale":         2.0,                  // > 1; required
    "follow_cursor": false,                // optional; default false
    "x": 244.5, "y": 54.5,                 // optional center; also the implicit "afterMs=0" waypoint for pan
    "coordinate_space": "window",          // optional; defaults to top-level
    "windowId":     12345,                 // required for window-space centers in multi-window comps

    "fromAction":   "open_settings/0",     // required, global ref
    "toAction":     "save/0",              // required, half-open
    "startDelayMs": 0,                     // optional; offsets fromAction time
    "endDelayMs":   0,                     // optional; offsets toAction time

    "pan": [                               // optional; mutually exclusive with follow_cursor
      { "afterMs": 1200, "x": 800, "y": 400, "ease": "in_out" },
      { "afterMs": 4000, "x": 200, "y": 150 }
    ]
  }
]

Implementation: per-frame scale=...:eval=frame then bounded crop on the canvas. Linear ramp-in / ramp-out at segment edges (RAMP = 0.2 s, auto-clamped to half the segment length so short segments still get both ramps).

pan waypoints

pan lets the camera move within a single zoom segment. Each waypoint moves the camera to a new center, easing from the previous position.

`pan` vs `follow_cursor` - use the right one:

• follow_cursor is for ranges where clicks happen: the camera tracks

the synthetic cursor as it moves between click targets.

• pan is for ranges where no clicks happen: a cinematic "look here,

then there" sweep over static UI. The cursor sprite is hidden inside pan ranges (a parked cursor would otherwise sit off the panned view and distract).

Don't author a pan over a range that contains click actions - use follow_cursor for those. The two are mutually exclusive within one entry, and overlapping a pan range with clicks just hides the cursor for those clicks.

Semantics: from t=segment_start until pan[0].afterMs, the camera holds at the entry's x/y (the implicit "start waypoint"). Between waypoints, the camera eases from the previous position to the current one using the destination's ease. After the last waypoint, the camera holds at that position until the zoom segment ends.

Validation: every afterMs must be >= 0 and strictly less than the range's duration (after offsets). Waypoints must be in strictly increasing afterMs order. pan + follow_cursor: true is a hard error.

Continuity across scenes

Two zoom entries that touch in time (one's toAction is the next one's fromAction) both ramp at the join - A ramps out, B ramps in. For ~2 × RAMP (about 400 ms) the combined zoom amount drops, so the camera visibly unzooms and re-zooms.

To keep one continuous camera across multiple scenes, write one zoom entry whose fromAction and toAction span all of them. Use pan waypoints inside that single entry to change focus, or follow_cursor: true to glide between clicks.

speed

"speed": [
  {
    "factor":       2.5,                  // > 0 and != 1; required
    "fromAction":   "fill_list/0",        // required, global ref
    "toAction":     "fill_list/2",        // required, half-open
    "startDelayMs": 0,                    // optional
    "endDelayMs":   0                     // optional
  }
]

Factor > 1 plays faster, < 1 slower, 1 is rejected. Entries beyond the speed range play at factor 1. scripts/stages/speedups.js builds a piecewise warp; the resulting timewarp.json is consumed by scripts/export.js for trim math.

Validation: speed segments may not overlap (evaluated on the post-offset envelope). Overlapping entries are a hard error.

captions

"captions": [
  { "text":         "Press 7",
    "fromAction":   "press7/0",
    "startDelayMs": 0,                // optional, signed
    "toAction":     "plus/0",         // EITHER toAction (+ optional endDelayMs)
    "endDelayMs":   -100 },
  { "text":         "Then plus",
    "fromAction":   "plus/0",
    "durationMs":   800,              // OR durationMs (mutually exclusive with toAction)
    "y":            0.10,             // optional position (see below)
    "align":        "center" }
]

Captions render via deskagent text-png (CoreText, sidesteps the missing drawtext in some ffmpeg builds). Default position is a centered bottom strip; y/align/x move individual entries.

Validation: two captions may not overlap in time at the same position (they'd stack illegibly). Different positions can coexist (e.g. a top label over a bottom subtitle). For a same-position clash, shorten the first (endDelayMs/durationMs), push the second (startDelayMs), or move one (y/align/x).

highlights

Optional override block for the editor's cursor sprite and click-ripple. Not used by deskagent control - purely a render-time concern. Full field reference: `editing.md`.

"highlights": {
  "ripple": { /* see editing.md#highlights */ },
  "cursor": { /* see editing.md#highlights */ }
}

cursor (visibility)

Controls when the synthetic cursor is drawn. By default the cursor follows the pointer track (clicks + moves + pointer events) across the whole video. Use hide/show (action ranges, same startDelayMs/endDelayMs offsets as other directives) to gate it - e.g. hide it during a scroll, then let it reappear before the next click.

"cursor": {
  "hide": [ { "fromAction": "tour/0", "toAction": "outro/0", "endDelayMs": 1400 } ],
  "show": [ /* whitelist: if present, cursor is visible ONLY in these ranges */ ]
}

End a hide range a beat before the next click and the cursor reappears already gliding toward it (the path is continuous; hide only gates visibility). Composes with the automatic pan-range hiding. Render-time only - not read by deskagent control. See `editing.md`.

trim (top-level)

"trim": { "beforeScene": "intro", "afterScene": "outro" }

Head trim = beforeScene's tStart. Tail trim = afterScene's tEnd + 600 ms. Both fields default to first/last scene.

Validation (Swift, on load)

• schema_version == 2. v1 is rejected with a migration hint.
• Scene IDs unique.
• All actions have a valid action kind (the eight above).

Editing scripts additionally validate:

• Every zoom/speed/captions entry has fromAction AND (for zoom/speed) toAction.
• Every referenced action ID resolves.
• Every referenced scene id resolves.
• zoom: follow_cursor: true requires at least one click event in the range.
• zoom: pan: [...] waypoints must be in strictly increasing afterMs order and within the range.
• zoom: pan and follow_cursor: true are mutually exclusive.
• zoom: a window-space center (static or pan) in a multi-window comp must set the entry's windowId (scenes' windowId is not consulted for directives).
• speed: factor > 0 and != 1.
• speed: no overlapping post-offset envelopes.
• captions: entries don't overlap in time (single shared bottom strip).
• captions: each entry has either toAction (+ optional endDelayMs) OR durationMs.
• Multi-window composition needs per-scene windowId so click coords resolve to the correct window slot.

setup / preflight / validate (informational)

Authors may add free-form setup, preflight, validate arrays at the top level for their own bookkeeping (open-app shell commands, deskagent assert probes, post-demo verifications). Neither deskagent control nor the editing scripts read them - they're notes the agent re-executes manually outside the screenplay.

"setup": [
  { "action": "shell", "cmd": "open -a 'Notes'" },
  { "action": "wait",  "ms": 1500 }
],
"preflight": [
  { "assert": "label", "value": "New Note" }
]

Don't put setup actions inside a scene unless they're meant on camera.

Window size vs. format

Pick the window size BEFORE exploration so click coords stay valid.

Worked examples:

• `assets/examples/notes-demo.json` - basic scene-bound zoom and speed.
• `assets/examples/continuous-zoom-demo.json` - cross-scene zoom range with pan waypoints.

Editing & export

Inputs: the recording directory (containing recording.manifest.json and per-source ProRes 4444 .mov clips) + screenplay.json + timeline.json.

Everything runs in one ffmpeg invocation. scripts/export.js collects a filter fragment from each stage, assembles them into one filter_complex, and runs ffmpeg once. The per-source clips are decoded once, the final mp4 is encoded once - no intermediate files in the hot path, no generation loss from per-stage re-encodes.

Stage order

[compose] → [highlights] → [zoom] → [captions] → [speedups] → [final scale/pad] → encode

Each stage is a module in scripts/stages/ exporting two functions:

generate(ctx, { inputLabel }) -> {
  filters:     ["[in]...[out]", ...],       // ffmpeg filter strings, joined with ';'
  inputs:      ["[in]"],                    // upstream labels consumed
  outputs:     "[afterX]",                  // single label produced
  extraInputs: [{ argv: ["-i", "..."] }],   // additional ffmpeg `-i` blocks
  sidecars:    { captions?, timewarp? },    // out-of-band data
}

apply(ctx, inputMov, outputMov)             // debug runner; renders only this stage to a ProRes 4444 .mov

Stages reference their own extra inputs by ${capInput<N>} placeholders; the orchestrator substitutes absolute ffmpeg input indices after counting prior stages' extra inputs.

Orchestrator CLI

node scripts/export.js <recordingDir> <screenplay.json> <timeline.json> <out.mp4> [format] \
    [--quality standard|high|h264|pro]   default: high (HEVC ~200 Mbps via VideoToolbox)
    [--width N --height N]               explicit output dims (override format)
    [--skip <stage>]                     (repeatable; e.g. --skip captions)
    [--dry-run]                          print ffmpeg cmd without running
    [--debug]                            print the assembled filtergraph

format is optional. When omitted, the export sizes to the user's main display's native pixel resolution (so QuickTime plays the result 1:1 on this machine).

Named formats:

--width N --height N overrides any format choice with exact pixel dims.

Quality presets:

Composition canvas (from screenplay.composition.canvas) is letterboxed into the requested output size with scale=W:H:force_original_aspect_ratio=decrease,pad=W:H:(ow-iw)/2:(oh-ih)/2:color=black.

Per-stage debug CLI

Every stage exposes the same shape:

# Print the stage's filter fragment as JSON (no ffmpeg run).
node scripts/stages/<stage>.js generate <recDir> <screenplay> <timeline>

# Render only this stage's effect to a ProRes 4444 .mov, taking <in.mov> as
# its upstream input. Useful to verify what just one stage does.
node scripts/stages/<stage>.js generate <recDir> <screenplay> <timeline> \
    --apply <in.mov> <out.mov>

For compose, --apply takes only <out.mov> - the inputs are the per-source clips in the recording dir, not a single video.

Stage descriptions

compose

Reads composition from the screenplay, opens each per-source clip as a separate ffmpeg input, trims the head of each by shared.headTrimsByPath (so every clip's t=0 maps to the shared timeline start), aspect-fits inside its placement rect, overlays them onto the canvas background with alpha preserved.

Never upscales by default. When a clip's source pixels fit inside its slot in both axes, it sits at native pixel size centered in the slot (no scaling, sharp). Only scales down when source > slot. Opt-in via composition.upscale: true (or per-element upscale: true) to force fit-to-slot.

Layout helpers (composition.layout):

Output label: [afterCompose] (carries alpha).

highlights

Two synthetic overlays on top of the composed canvas:

1. Cursor sprite. An arrow (via deskagent cursor-png --type arrow) drawn along the pointer track - the unified timeline of every positional action (click, move, drag, pointer_*), via cursorWaypointsInCanvasSeconds. Segment easing depends on the waypoint:

• click → auto pre-arrival glide (cubic ease, ≤0.55 s at ~1400 px/s),

arriving exactly at the click so the ripple lands on a still cursor;

• move → glides over the action's own duration_ms (author-controlled

speed) - so move is the "point the viewer's eye at X" beat;

• `move`/`pointer_move` with a `path` → the polyline is spread across

the duration and interpolated linearly (constant speed), so shapes/ trajectories trace smoothly. The sprite trace is downsampled to ≤24 points per path (the cursor overlay's x/y is one ffmpeg expression term per waypoint, which has a practical size ceiling); the underlying draw/driver still used the full-resolution path. The cursor path is a flat-sum expression (one ramped term per segment), not nested - so density doesn't blow the parser. A pointing-hand sprite (--type pointing) replaces the arrow for ~220 ms around each click (moves/pointer events don't swap or ripple). 2. Click ripple. A procedural soft expanding-ring sprite (alpha .mov generated once via ffmpeg ... geq) overlay'd at each click position with -itsoffset <click.t> so each click plays its own copy. Clicks only.

User overrides on screenplay.highlights:

"highlights": {
  "ripple": {
    "sprite":     "/path/to/anim.mov",   // optional override; alpha .mov / APNG / transparent webm
    "color":      "FFFFFF",              // procedural ring color RRGGBB; default white
    "size":       160,                   // procedural sprite longest edge in canvas px; default 160
    "durationMs": 520                    // procedural sprite duration; default 520
  },
  "cursor": {
    "arrow":    "/path/to/arrow.png",    // optional; default = deskagent cursor-png --type arrow
    "pointing": "/path/to/pointing.png", // optional; default = --type pointing
    "size":     64                       // longest edge in canvas px; default 64
  }
}

Cursor visibility - screenplay.cursor gates the sprite:

• cursor.hide: [ {fromAction,toAction,startDelayMs?,endDelayMs?}, … ] - cursor

invisible inside these ranges (e.g. hide it during a scroll).

• cursor.show: [ … ] - whitelist; if present the cursor is visible ONLY in

these ranges.

Both compose with the automatic pan-range hiding. End a hide range a beat before the next click and the cursor reappears mid-glide toward it.

Implementation notes:

• Arrow's enable= is `not(click windows) not(pan ranges) visibility

gates; pointing's is (click windows) * gates`. Same path expression for both, so the swap is seamless.

• The pointer track is shared with zoom's follow_cursor: true via

lib/cursor-path.js - the camera centers on the sprite's actual position (clicks and moves), no desync during glides.

• Procedural ripple sprite is cached to

~/.cache/deskagent-skill/ripple-{size}-{durSec}-{color}.mov; only the first export of a given configuration pays the geq render cost.

• Pre-record: pass --no-cursor to deskagent record so the real OS

cursor doesn't fight the synthetic one.

Output label: [afterHighlights].

zoom

Reads screenplay.zoom[]. Each entry creates one segment whose center comes from one of three sources (mutually exclusive):

• Static center - x/y on the entry, or the first action with

coords in range.

• Follow cursor - follow_cursor: true. Camera centers on the

synthetic-cursor pointer track (clicks and moves) in range - shared with the highlights stage, so camera and sprite never desync. Needs ≥1 click or move in the range.

• Pan waypoints - pan: [...]. Explicit list of waypoints

{ afterMs, x, y, ease? }. Camera holds at the segment's start center (the entry's x/y or first action with coords in range), then eases through each waypoint, then holds at the last waypoint until the segment ends.

Use follow_cursor for click-driven ranges and pan for no-click cinematic ranges - don't mix. The highlights stage hides the cursor sprite inside any pan range (a no-click section has no cursor to show; a parked sprite off the panned view just distracts).

Window for window-space centers. When a zoom/pan center is in coordinate_space: "window" (entry-level coordinate_space, or the screenplay default) and the composition has more than one window, the entry must set windowId to name the target window. The whole entry

• the static/start center and every pan waypoint - resolves in that

one window's coordinate space; to travel a pan across windows, use coordinate_space: "screen" instead. The directive's windowId is the only lever here: unlike action coordinates, a directive is not bound to a scene, so the scene's windowId is not consulted. Omitting it in a multi-window comp is a hard error naming the entry. (Single-window comps resolve automatically.) follow_cursor and the first-action-in-range fallback are exempt - they resolve real recorded actions, which already carry their own window.

`afterMs` is video time, relative to the segment start. A waypoint's afterMs is added to the segment's tStart, where tStart is the anchor action's start as it appears in `timeline.json` (canvas/video seconds). It is not relative to deskagent control's own per-event ms, which start at 0 inside the script and are offset from video time by the recorder + control startup lead-in (~1.4 s). Compute afterMs from timeline.json event times (or as a plain delta after the anchor action begins), never from the control script's internal clock.

The per-frame scale filter (eval=frame) scales the whole canvas by the piecewise zoom factor, then bounded crop re-centers back to canvas dims. Linear ease at each segment's edges (RAMP_SEC = 0.2, clamped to half the segment length so short zooms still get both ramps).

Pan easing modes (per waypoint, default in_out):

Output label: [afterZoom].

captions

Reads screenplay.captions[] (top-level directive array - see `desktop.md`). Each entry's text is rendered to a transparent PNG via deskagent text-png and overlay'd with enable=between(t, ...) at its position: y (canvas-height fraction, default 0.88 = bottom), align (center/left/right) or an explicit x center fraction.

Two captions may overlap in time only if they sit at different positions (e.g. a top label over a bottom subtitle); a same-position time-overlap is a hard error.

Output label: [afterCaptions].

speedups

Reads screenplay.speed[]. Builds a piecewise setpts expression that compresses or expands each range by factor. Emits a sidecars.timewarp that export.js uses to size the final output -t correctly (it caps at last.dstEnd + (sourceDuration - last.srcEnd) so any post-warp tail plays at 1× and contributes to duration).

Output label: [afterSpeedups].

Skipping stages

--skip compose is rejected - compose is the source of inputs. Any other stage can be skipped; its output label short-circuits to the previous stage's output.

Sidecars

The orchestrator passes ctx (loaded once via lib/screenplay.js) and collects each stage's sidecars for cross-stage data (e.g., speedups' timewarp consumed by export's duration math). No on-disk intermediate sidecars in the hot path.

What changed vs. the older five-script pipeline

• Five add_*.js CLIs + export_video.js collapsed into one export.js

+ five stages/*.js library modules.

• No intermediate mp4s between stages; one decode → one filter graph →

one encode.

• Per-stage --apply provides the same "look at one stage's output"

affordance the old chain gave for free.

• Quality / final container is a flag on export.js rather than the

fixed libx264 crf=18 the old pipeline re-encoded between every stage.

• Default output size auto-detects the user's display so QuickTime plays

the result 1:1 on this machine.

• Captions moved from per-scene caption to a top-level

screenplay.captions[] directive array, matching zoom/speed's shape.

Timeline event schema

deskagent control screenplay.json --timeline timeline.json emits a flat JSON array of three event types, in execution order:

type TimelineEvent =
  | { type: "scene_start" } & SceneBoundary
  | { type: "scene_end"   } & SceneBoundary
  | { type: "action"      } & ActionEvent

type SceneBoundary = {
  scene_id:     string
  scene_index:  number
  startedAtMs:  number     // ms since control run start
  endedAtMs:    number     // === startedAtMs for boundary events
  startedAtWallclockMs: number    // Unix-epoch ms - anchor for video time
  endedAtWallclockMs:   number
  coordinate_space: "window" | "screen"
}

type ActionEvent = SceneBoundary & {
  action_id:    string     // canonical: "${scene_id}/${action_index}"
  action_index: number
  action:       "click" | "double_click" | "drag"
              | "type"  | "key"          | "scroll"
              | "wait"  | "move"
              | "pointer_down" | "pointer_move" | "pointer_up"
  x?: number               // CG points in coordinate_space (window/screen)
  y?: number
  path?: { x: number, y: number }[]   // trajectory polyline (move/pointer_move);
                                       // the cursor track follows it (linear)
}

Example

[
  { "type": "scene_start", "scene_id": "open_settings", "scene_index": 0,
    "startedAtMs": 500, "endedAtMs": 500,
    "startedAtWallclockMs": 1778883094300, "endedAtWallclockMs": 1778883094300,
    "coordinate_space": "window" },

  { "type": "action", "scene_id": "open_settings", "scene_index": 0,
    "action_id": "open_settings/0", "action_index": 0, "action": "click",
    "startedAtMs": 752, "endedAtMs": 800,
    "startedAtWallclockMs": 1778883094552, "endedAtWallclockMs": 1778883094600,
    "x": 244.5, "y": 54.5, "coordinate_space": "window" },

  { "type": "scene_end", "scene_id": "open_settings", "scene_index": 0,
    "startedAtMs": 850, "endedAtMs": 850,
    "startedAtWallclockMs": 1778883094650, "endedAtWallclockMs": 1778883094650,
    "coordinate_space": "window" }
]

Mapping to video time

Every editing stage anchors via the recording's recording.manifest.json. The editor picks a canvas t=0 from the shared time window across clips (t0 = max(clip.startHostNs) in host time, with the wallclock-equivalent used to convert timeline events):

t0WallMs = wallclock of the clip whose startHostNs == max(startHostNs)
videoSec = (event.startedAtWallclockMs - t0WallMs) / 1000

lib/screenplay.js's loadContext does this once; stages call ctx.actionEvents.get(actionId).tStart to read canvas-second timings. The agent never computes this by hand.

Joining timeline to screenplay

Both files agree on scene_id. Action references in screenplay directives (fromAction, toAction) use the canonical action_id = "${scene_id}/${action_index}". Resolution is hard-failure on the editing side - missing IDs print the available IDs and exit non-zero.

Coordinate space

x / y are CG screen points (not pixels) in the declared coordinate_space. Editing stages map to canvas pixels via ctx.pointToCanvasPixel(e) which finds the action's window placement (via the scene's windowId) and applies the placement's fitted rect:

windowSpace:  pixel = fit.ox + point * fit.fitW / frameCG.w
screenSpace:  pixel = fit.ox + (point - frameCG.x) * fit.fitW / frameCG.w

For multi-window compositions, scenes must carry windowId so the scene's actions are mapped to the correct window's slot on the canvas.

Invariants

1. Events arrive in execution order; *WallclockMs is monotone non-decreasing. 2. Every scene_start is followed by a matching scene_end with the same scene_id / scene_index. Actions in between carry that scene_id. 3. action_id is unique across the timeline (since scene IDs are unique and action indices are local). 4. record_start / record_stop markers are NOT in the timeline - the screenplay doesn't describe them. The recording's recording.manifest.json carries per-clip host-time + wallclock anchors that the editor uses to compute the shared canvas timeline.

Web driver - scripts/drive-web.js

A CDP-based driver for web page content. It replaces deskagent control for browser scenes: it talks to Chrome over the DevTools Protocol (zero deps, Node's built-in WebSocket) and emits the same `timeline.json` contract as deskagent control, so deskagent record and the whole export.js pipeline are unchanged.

Why it exists

deskagent control posts OS-level events (CGEvent / AX). For a browser that means:

• it can drive the browser chrome (tabs, omnibox, buttons) but **not page

content** reliably - the DOM lives in an out-of-process renderer;

• HID mode requires the window focused/frontmost and targets raw pixels.

CDP injects input at the renderer: no focus, no foreground, the real cursor never moves, and you target the DOM by selector. So Chrome can sit unfocused (even behind other windows) and still be driven while SCK records its window.

Division of labor: use the web driver for page content (scroll, click links, fill forms, draw on a web canvas); use deskagent control for the browser's own UI (omnibox, tabs, extensions, menus) - CDP can't touch those. A mixed demo can do both against the same Chrome (native posts to its pid; CDP attaches to its page target).

Launching the debug Chrome

CDP can't attach to an already-running vanilla Chrome (no debugging port). Launch a dedicated instance:

"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
  --remote-debugging-port=9222 \
  --remote-allow-origins='*' \              # REQUIRED on Chrome 111+ or the CDP WebSocket handshake is rejected (403)
  --user-data-dir=/tmp/demo/chrome-profile \ # isolated profile (clean, no extensions)
  --disable-background-timer-throttling \    # keep an UNFOCUSED window compositing
  --disable-renderer-backgrounding \         #   at full rate so scroll/animation
  --disable-backgrounding-occluded-windows \ #   stay smooth while recording
  --window-size=1280,840 --window-position=120,90 \
  --no-first-run --no-default-browser-check \
  --new-window "https://example.com"

• The isolated profile is not logged in - private/authed pages 404. For

authed content, point --user-data-dir at a persistent profile you logged into once.

• Even with the anti-throttle flags an unfocused window caps ~50-55 fps during

scroll (vs a locked 60 if focused). Hands-off background recording's tradeoff.

Workflow

Same as the native pipeline, with drive-web.js in place of control:

# 1. pre-load so the recording's first frames aren't stale (optional but clean)
node scripts/drive-web.js prenav.json --cdp-port 9222 --window-frame "120,90,1280,840" --timeline /dev/null

# 2. record the (unfocused) Chrome window
deskagent record ./demo/raw --window <ID> --no-cursor --pid-file /tmp/rec.pid &

# 3. drive page content over CDP -> timeline.json
node scripts/drive-web.js screenplay.json \
    --cdp-port 9222 --window-frame "x,y,w,h" \
    --timeline ./demo/raw/timeline.json

# 4. stop + export (unchanged)
kill -INT $(cat /tmp/rec.pid); wait
node scripts/export.js ./demo/raw screenplay.json ./demo/raw/timeline.json demo.mp4 --width W --height H

Get <ID> and the --window-frame "x,y,w,h" (CG points) from deskagent list.

CLI

node scripts/drive-web.js <screenplay.json>
    --cdp-port 9222                 # debug port (default 9222)
    --window-frame "x,y,w,h"        # REQUIRED; CG points from `deskagent list`
    --timeline <out.json>           # REQUIRED; the timeline.json to emit
    [--url-match <substring>]       # pick the page target by URL (default: first page)

Coordinates

The driver writes window-space CG points to the timeline (what pointToCanvasPixel expects). It derives the browser chrome height as chromeH = window.h - window.innerHeight and maps viewport CSS px → window points as y_window = y_css + chromeH (x is flush-left, 1:1 at 100% page zoom).

So in a screenplay, raw `x`/`y` and all `path`/`shape` points are window CG points; selector/text targets are resolved to their element centre and mapped for you.

Actions

Targets resolve by selector (CSS) or text (+ optional tag, default h1..h6; matches visible elements only, so GitHub's duplicate mobile/desktop rows don't trip it) or raw window x/y.

Trajectories (path / shape)

move and pointer_move accept a trajectory instead of a single point:

{ "action": "pointer_move", "path": [ {"x":110,"y":130}, {"x":400,"y":130} ], "duration_ms": 600 }   // polyline (window pts)
{ "action": "move", "shape": "circle", "cx": 640, "cy": 470, "r": 180, "points": 48, "duration_ms": 2200 }
{ "action": "move", "shape": "line",   "x1": 100, "y1": 100, "x2": 400, "y2": 100 }

shape is authoring sugar (circle params: cx,cy,r,points?,turns?, startDeg?,ccw?; line: x1,y1,x2,y2) - it compiles to a polyline; the timeline only ever carries points. The cursor traces the polyline at constant speed over duration_ms.

Drawing on a web canvas (e.g. jsPaint, Excalidraw): select the tool, then pointer_down → pointer_move(path/shape) → pointer_up is one continuous stroke. Same vocabulary as native control, so screenplays port across native and web.

Remotion bridge template

Turn a deskagent recording into a Remotion motion-graphics video. The bridge (src/bridge/) does the integration; you only edit the creative composition (src/Demo.tsx).

This is an optional path, separate from the lean ffmpeg export.js pipeline. It needs Node + Remotion installed (heavier deps, headless Chrome). Use it when you want React-grade motion graphics around the footage.

Use

1. Copy this folder into a working dir and npm install. 2. Feed the recording into public/ (the contract from deskagent record + the web/native driver):

• public/rec.mp4 - the clip, transcoded to a browser codec:

ffmpeg -i raw/window-<id>.mov -vf scale=<w>:<h> -c:v libx264 -pix_fmt yuv420p -an public/rec.mp4

• public/recording.manifest.json - from the recording dir
• public/timeline.json - from the driver (deskagent control or drive-web.js)
• public/screenplay.json - optional, for captions

3. Render: npx remotion render Demo out/video.mp4 (first run: npx remotion browser ensure to fetch the headless shell). 4. Iterate in npx remotion studio.

The bridge (src/bridge/)

• loadRecording(fps, {speed?, videoFile?}) - reads the contract (via

fetch(staticFile(...)), so it runs in calculateMetadata) and returns frame-indexed props: fps, durationInFrames, speed, stageWidth/Height (recording window in CG points), videoSrc, events (clicks/moves as {frame,x,y}), scenes, captions. speed > 1 plays the footage faster - duration and event frames are scaled down to match; pass rec.speed to <RecordingCard playbackRate> so the cursor stays synced with the video.

• <RecordingStage width height> - the recording's coordinate space. Put the

card + cursor + ripples inside; the creative comp scales/positions the stage as one unit and everything stays aligned with the footage.

• <RecordingCard src width height playbackRate?> - the <OffthreadVideo>, framed.
• <Cursor events> / <ClickRipple events> - timeline-driven, in stage coords.
• <Caption captions> - output-canvas captions (y/align honored).

Root.tsx wires loadRecording into calculateMetadata, so the composition duration comes from the recording and Demo receives the parsed rec prop. Coordinates: timeline x/y are recording-window CG points = the video's pixel space, so cursor/ripples line up without extra mapping.