Chrome Relay

Chrome Relay

Drives the user's real Chrome through a Chrome extension + local native host. Prefer it when logged-in browser state (auth cookies, sessions, installed extensions) matters.

Setup

1. Chrome extension 2. CLI:

   pnpm add -g chrome-relay
   chrome-relay install
   chrome-relay doctor

Verify CLI ≥ 0.7.0 — wait/get/batch/snapshot --diff landed there (0.6.0 brought the snapshot/@ref loop; ≥ 0.5.20 fixed a silent click bug on Radix/React-Aria UIs):

chrome-relay --version

The core loop

chrome-relay tabs                             # find or create a tab
chrome-relay navigate "https://kushalsm.com" --new   # background tab by default
chrome-relay snapshot --tab 1234 -i           # see the page: actionable elements get @refs
chrome-relay click @e12                       # act on refs — no --tab, no selector
chrome-relay fill @e14 "hello"
chrome-relay wait --text "Saved" --tab 1234   # block until the page reacts
chrome-relay snapshot --tab 1234 --diff       # print only what changed (~100 tokens)

Snapshot output is compact indented text (~1–15 KB for most pages) — read it directly, no jq needed:

- link "Hacker News" [ref=e4]
- textbox "Search" [ref=e41]: current value
- checkbox "Remember me" [checked, ref=e42]
- clickable "Open card" [ref=e88]        ← cursor-pointer div the AX tree missed

Refs carry their own tab. click @e12 acts on the tab that produced e12, never the active tab — safe while the user keeps browsing. A contradicting --tab errors with target_conflict.

Ref lifetime. Refs survive same-page DOM churn (cached backendNodeId, healed by role+name re-find when nodes are replaced) but die on real navigation. A dead ref returns error.code = stale_ref → re-run snapshot.

Interception. Ref clicks hit-test the point first: if an overlay / sticky header / modal owns it, you get error.code = click_intercepted naming the interceptor — dismiss it or scroll, then retry. The click was NOT delivered. fill/type skip this check (covered inputs are still writable).

Tool surface

Picking the right text tool

Element addressing — the fallback ladder

1. `@ref` from `snapshot -i` — default. Covers buttons/links/inputs, named content, cursor-pointer div-soup (the sweep), and shadow DOM. 2. CSS selector — when you know the selector statically and don't need a snapshot. 3. `js` probe → coordinate click — canvas internals and SVG chart segments (anonymous <path> elements have no DOM handle anywhere):

   chrome-relay js --tab 1234 "const r = document.querySelector('svg path').getBoundingClientRect(); return {x: r.x + r.width/2, y: r.y + r.height/2}"
   chrome-relay click --tab 1234 --x 312 --y 218

Don't poll — wait

A snapshot after every action wastes turns. The cheap loop on a changing page:

chrome-relay click @e12
chrome-relay wait --text "Saved" --tab 1234     # or wait <selector> / --url / --load
chrome-relay snapshot --tab 1234 --diff         # only the changes, refs included

Top gotchas

0. `snapshot -i` is for ACTING, not fact extraction. It prints ref-bearing elements only — non-interactive values (dashboard metrics, paragraph text, chart labels) drop out. Measured live: a Cloudflare Pages metrics page lost all its numbers under -i. To READ facts, use full snapshot, get text <target>, or a js projection. 1. `type` appends — it inserts at the caret. If the input had a value (autosaved draft, default text), clear it first via js or keys (Cmd+A then Backspace). 2. Refs die on navigation — stale_ref means the page changed under you; re-snapshot. Don't retry the same ref. 3. Coords go stale fast — read getBoundingClientRect, scroll/reflow, then click → you hit the wrong element. For autocomplete popups especially, use keyboard nav, not coord clicks. 4. Click "succeeded" but nothing happened — first diagnostic: document.elementFromPoint(x, y). If it returns a wrapper or form background, your coords are wrong. If it returns the right element but state didn't change, you're likely on chrome-relay <0.5.20 — upgrade.

More recipes: references/patterns.md Failure modes: references/troubleshooting.md

Operational guidance

• Don't give up early. A failing click is information, not a stop signal. Attach a document-level listener with capture:true and watch what fires:

  chrome-relay js --tab 1234 "
    ['pointerdown','mousedown','click'].forEach(t =>
      document.addEventListener(t, e => console.log(t, e.target.tagName, e.target.className), {capture:true})
    );
    return 'listening'
  "
  # do the action, then:
  chrome-relay console --tab 1234

• Don't echo secrets. When extracting tokens / API keys via js, write the result directly to a file. Never echo $TOKEN or interpolate into shell strings — it ends up in scrollback, logs, and tool transcripts.
• Redact `network` output. Request/response headers carry cookies, auth/CSRF tokens, account and project IDs. Never paste raw chrome-relay network output into chat, docs, issues, or commits — filter to the fields you need (url, status, timings) or redact headers first.
• Capture before irreversible actions (form submit, send message, account change). Save the screenshot path.

Guardrails

• Errors are structured: branch on relayError.code (stale_ref, click_intercepted, element_not_found, target_conflict, timeout), not on message text.
• If a flag is unclear, chrome-relay <command> --help is authoritative — these docs lag.

chrome-relay skill

Agent skill for Chrome Relay — drives the user's real Chrome session through CDP (extension + native host) so agents can read pages, click, type, fill forms, press keys, and run JS without stealing focus.

Install

npx skills add kiluazen/kstack@chrome-relay

The skill teaches agents how to use the chrome-relay CLI. Install the CLI separately:

pnpm add -g chrome-relay
chrome-relay install
chrome-relay doctor

Patterns

Recipes that took an hour to discover, written down so they don't have to be re-discovered.

Combobox / autocomplete option selection

Filter inputs with a dropdown of matching options (npm package picker, Linear assignee picker, GitHub repo search). The popup re-renders fast — coord-clicking a specific option lands on stale coordinates and hits the form background.

# 1. Click the filter input
chrome-relay click "<filter-input-selector>" --tab $TAB

# 2. Type the search query
chrome-relay type "chrome-relay" --tab $TAB

# 3. Keyboard-navigate. ArrowDown highlights the first match; Enter commits it.
chrome-relay keys "ArrowDown" --tab $TAB
chrome-relay keys "Enter" --tab $TAB

Why: comboboxes are built around keyboard nav (it's their accessibility contract). The first option auto-highlights or one ArrowDown highlights it; Enter is the canonical "select" action. Coord-clicking fights the popup's lifecycle.

Click by visible text

There's no click-text verb on purpose — js + click --x --y composes the same thing more explicitly.

COORDS=$(chrome-relay call chrome_evaluate '{"code":"
  const walker = document.createTreeWalker(document.body, NodeFilter.SHOW_ELEMENT);
  let node;
  while (node = walker.nextNode()) {
    if (node.offsetParent && (node.textContent||\"\").trim() === \"Generate token\") {
      node.scrollIntoView({block: \"center\"});
      const r = node.getBoundingClientRect();
      return {x: Math.round(r.left + r.width/2), y: Math.round(r.top + r.height/2)};
    }
  }
  return null;
"}' --tab $TAB | jq '.result')

X=$(echo "$COORDS" | jq -r '.x')
Y=$(echo "$COORDS" | jq -r '.y')
chrome-relay click --x "$X" --y "$Y" --tab $TAB

Variants: match by .includes(text) for partial, by .matches(selector) for tag constraint, by aria-label for icon buttons.

Clear, then type (overwrite a pre-filled input)

chrome-relay type inserts at the caret. If the input already has a value (autosaved draft, today's date, "Untitled"), the new text appends.

Option A — clear via JS (most reliable for React-controlled inputs):

chrome-relay js --tab $TAB "
  const el = document.getElementById('create-gat_tokenName');
  const setter = Object.getOwnPropertyDescriptor(window.HTMLInputElement.prototype, 'value').set;
  setter.call(el, '');
  el.dispatchEvent(new Event('input', {bubbles: true}));
  return 'cleared';
"
chrome-relay click "<input-selector>" --tab $TAB    # refocus
chrome-relay type "the new value" --tab $TAB

Option B — select-all + delete (works for plain inputs, may not commit cleanly in controlled components):

chrome-relay click "<input-selector>" --tab $TAB
chrome-relay keys "Cmd+a" --tab $TAB
chrome-relay keys "Backspace" --tab $TAB
chrome-relay type "the new value" --tab $TAB

Extract a value to a file without echoing it

When pulling secrets (API keys, one-time tokens) out of a page, never let them flow through shell $(...) and then echo. Route the body of the JS expression directly to a file:

chrome-relay call chrome_evaluate '{"code":"return (document.body.innerText.match(/npm_[A-Za-z0-9]{30,}/) || [\"\"])[0]"}' --tab $TAB \
  | jq -r '.result' \
  > ~/.npm-token-tmp

# Use the file, never print it:
sed -i.bak "s|//registry.npmjs.org/:_authToken=.*|//registry.npmjs.org/:_authToken=$(cat ~/.npm-token-tmp)|" ~/.npmrc
rm ~/.npm-token-tmp ~/.npmrc.bak

echo "$TOKEN", echo "captured length: ${#TOKEN}", even printf with the var — all of these get the secret into terminal scrollback, agent transcripts, and possibly logs. The jq -r '.result' > file pattern is the only path that doesn't.

Trace which events actually fire

When a click "succeeds" but the page doesn't react, the only useful question is: which events did the page actually see?

chrome-relay js --tab $TAB "
  ['pointerdown','pointerup','mousedown','mouseup','click'].forEach(t =>
    document.addEventListener(t, e => console.log('[evt]', t, e.target.tagName, e.target.className?.toString?.()?.slice(0,40)), {capture: true})
  );
  return 'listening';
"

# Now do the click that's failing:
chrome-relay click --x 506 --y 723 --tab $TAB

# Read what fired:
chrome-relay console --tab $TAB | grep '\[evt\]'

If you see mousedown but no pointerdown → you're on chrome-relay <0.5.20 (missing pointerType: "mouse" in CDP dispatch). Upgrade.

If you see no events at all → coord is wrong; check document.elementFromPoint(x, y).

If you see events on a wrapper instead of your target → coord is stale (the page reflowed between read and click). Re-read coords immediately before the click.

Use workspaces when running multiple agents

If multiple agents are driving the same Chrome, they'll fight over --tab IDs. Pin each agent to a named window:

chrome-relay workspace create research
chrome-relay --workspace research navigate "https://google.com/search?q=..." --new
chrome-relay --workspace research tabs    # only sees its own window

The --workspace flag at the top level scopes every subsequent command.

Troubleshooting

Failure modes and how to recognize them.

"Click registered, but nothing happened"

Symptom: chrome-relay click returns {clicked: true}, no error. Dropdown / menu / popover does not open. Page state unchanged.

Most common cause: chrome-relay <0.5.20. Pre-0.5.20 versions of CDP Input.dispatchMouseEvent fire only mouse events, not pointer events. Modern UI libs (Radix, React-Aria, Headless UI) listen on pointerdown and silently ignore mouse-only clicks.

Diagnostic — attach listeners and look for pointerdown (see patterns.md "Trace which events actually fire"). If you see mousedown but no pointerdown:

chrome-relay --version
pnpm add -g chrome-relay@latest
chrome-relay self-reload     # reload the extension service worker too

Other causes:

• Coords landed on a wrapper element (see "Click landed on wrong element" below)
• Element exists in DOM but is occluded by something with higher z-index — document.elementFromPoint(x, y) will tell you

"Click landed on the wrong element" (stale coords)

Symptom: you read getBoundingClientRect, click, and elementFromPoint of those coords is now a parent FORM, DIV wrapper, or even the page background.

Cause: the page reflowed between read and click. Common in:

• Autocomplete dropdowns (popup re-renders as you type)
• Pages with progressive image loading (layout shift)
• Anything that animates in/out

Fix:

• For combobox flows, use keyboard nav instead (see patterns.md "Combobox / autocomplete")
• For everything else, read coords as late as possible:

  el.scrollIntoView({block: 'center'});
  await new Promise(r => setTimeout(r, 150));  // let layout settle
  const r = el.getBoundingClientRect();        // read
  // immediately click — no awaits between

• Use chrome-relay click <selector> (selector mode) when you have one — it re-resolves coords in-page right before dispatching.

"type added on top of existing value"

Symptom: input had "Untitled", you typed "My Doc", value is now "UntitledMy Doc".

Cause: chrome-relay type uses CDP Input.insertText which inserts at the caret. It does not replace the value.

Fix: clear first. See patterns.md "Clear, then type". For most React-controlled inputs, the JS value setter + input event is the only thing that works.

"Page won't render in background"

Symptom: chrome-relay navigate --new opens a tab in the background; screenshot is blank or stuck on a loading skeleton even minutes later.

Cause until 0.5.18: chrome-relay didn't override document.visibilityState. Many SPAs (Cloudflare dashboard, Linear, Notion) gate their bootstrap JS on document.visibilityState === 'visible' and stall on backgrounded tabs.

Fix: upgrade to 0.5.18+. The fix is in the attach flow; you don't have to do anything else. If a specific page still won't render, check:

chrome-relay js --tab $TAB "return {
  visState: document.visibilityState,
  hidden: document.hidden,
  hasFocus: document.hasFocus()
}"

All three should reflect "visible" / "focused" — if not, the shim didn't apply (file a bug).

"Version mismatch" warning

Stdout prefix:

[chrome-relay] cli-outdated: 0.5.16 < extension 0.5.20; run `chrome-relay update`

The extension and CLI are versioned independently. The extension lives in Chrome; the CLI is chrome-relay on $PATH. Either may be older. Run chrome-relay update (updates the CLI) and reload the extension from chrome://extensions if needed.

"Native host not found"

chrome-relay doctor fails with native messaging host not registered or similar.

Fix:

chrome-relay install      # re-registers the native messaging manifest
chrome-relay doctor

If still failing, the extension and CLI may be talking to different host names. The install command writes ~/Library/Application Support/Google/Chrome/NativeMessagingHosts/com.chrome_relay.host.json on macOS — check it exists and points to a real binary.

"Tab ID not found"

You stored a tabId and it's now invalid (user closed the tab, restarted Chrome, etc.). Always re-list tabs at the start of a session:

chrome-relay tabs > /tmp/tabs.json
jq '.windows[].tabs[] | select(.url | test("npmjs.com"))' /tmp/tabs.json

"Click works in tests but not in the real Chrome session"

Tests run with pointerType: "mouse" (post-0.5.20). The agent's local CLI may still be older. Always check:

chrome-relay --version

Tests in apps/extension/test/ exercise the handler code directly via vitest; they don't exercise the CLI version on the agent's machine. A passing test does not mean the agent's local CLI is current.