Open Computer Use

Open Computer Use

Overview

Open Computer Use exposes Computer Use as a local CLI and stdio MCP server. It is not Codex.app-specific; adapt the commands and MCP config to the agent runtime you are operating in.

It supports the same core tool surface across macOS, Linux, and Windows: list_apps, get_app_state, click, perform_secondary_action, scroll, drag, type_text, press_key, and set_value.

Core Workflow

1. Check the CLI is installed with open-computer-use -h. If installation or setup is missing, read references/installation.md. 2. On macOS, run open-computer-use doctor before the first real GUI task. If permissions are missing, ask the user to approve Accessibility and Screen Recording in the onboarding UI. 3. Inspect available apps before acting: open-computer-use call list_apps. 4. Capture current UI state with open-computer-use call get_app_state --args '{"app":"TextEdit"}'. The default state is usually enough for UI operation. 5. When the task needs complete long text, such as chat history, email bodies, document text, or long form content, call get_app_state with show_full_text: true. 6. Prefer element-targeted actions using element_index from the latest get_app_state result. 7. For multi-step CLI work, use open-computer-use call --calls '<json-array>' so one process can reuse the latest element index mapping. 8. For agent runtimes that support local MCP servers, configure open-computer-use mcp and call the exposed Computer Use tools directly. Read references/usage.md. 9. If communication, permission, or desktop-session access fails, read references/troubleshooting.md.

Operating Rules

• Treat the target desktop as the user's real session. Do not inspect password managers, unrelated private content, or sensitive apps unless the user explicitly asked for that task.
• Ask before sending, deleting, purchasing, approving, uploading, or making other externally visible changes.
• Do not assume Codex.app plugin helpers are available. Use the installed open-computer-use CLI or an explicit MCP config.
• Always run get_app_state before using element_index; do not guess indexes across sessions or after large UI changes.
• Prefer semantic actions and set_value for editable controls. Use coordinate click, scroll, and drag only when the element tree does not expose a safer target.
• On macOS, do not enable OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1 unless the user explicitly wants diagnostic behavior that may move the real pointer.
• On Windows and Linux, confirm the command is running inside the logged-in desktop session before assuming GUI automation is available.

Common CLI Actions

open-computer-use -h
open-computer-use doctor
open-computer-use call list_apps
open-computer-use call get_app_state --args '{"app":"TextEdit"}'
open-computer-use call get_app_state --args '{"app":"TextEdit","show_full_text":true}'
open-computer-use call click --args '{"app":"TextEdit","element_index":"0"}'
open-computer-use call type_text --args '{"app":"TextEdit","text":"Hello from Open Computer Use"}'

For a short sequence that reuses state in one process:

open-computer-use call --calls '[
  {"tool":"get_app_state","args":{"app":"TextEdit"}},
  {"tool":"press_key","args":{"app":"TextEdit","key":"Return"}}
]'

MCP Usage

For runtimes that can launch local MCP servers over stdio, use:

[mcp_servers.open_computer_use]
command = "open-computer-use"
args = ["mcp"]

Read references/usage.md for JSON config examples, direct tool-call patterns, and platform notes.

References

• references/installation.md: one-time CLI install, agent MCP install commands, and macOS permissions.
• references/usage.md: MCP config, direct CLI calls, sequencing, and platform behavior.
• references/troubleshooting.md: permission, desktop-session, app discovery, and action failures.

Open Computer Use Installation

Read this reference when the user asks to install, verify, repair, or explain Open Computer Use setup.

Install The CLI

Use npm:

npm install -g open-computer-use

Verify:

open-computer-use -h
open-computer-use call list_apps

If the package is already installed and the user asks to update it:

npm update -g open-computer-use

macOS Permissions

macOS needs Accessibility and Screen Recording permissions before real app state and actions can work.

Run:

open-computer-use doctor

If permissions are missing, the onboarding UI opens. Ask the user to grant the requested permissions in System Settings. Do not try to bypass TCC prompts or silently manipulate protected settings.

Windows and Linux do not use this macOS onboarding step, but they still need a logged-in desktop session.

Install Into Agent MCP Configs

Use the built-in installers when they match the user's agent:

open-computer-use install-codex-mcp
open-computer-use install-claude-mcp
open-computer-use install-gemini-mcp
open-computer-use install-gemini-mcp --scope user
open-computer-use install-opencode-mcp

Codex App can also use the plugin installer:

open-computer-use install-codex-plugin

For any other MCP client, add a stdio server manually:

{
  "mcpServers": {
    "open-computer-use": {
      "command": "open-computer-use",
      "args": ["mcp"]
    }
  }
}

Install This Skill

Install the skill for Codex:

npx skills add iFurySt/open-codex-computer-use -g -a codex --skill open-computer-use -y
npx skills ls -g -a codex | rg 'open-computer-use'

Install the skill for Claude Code:

npx skills add iFurySt/open-codex-computer-use -g -a claude-code --skill open-computer-use -y

Update an existing global skill install:

npx skills update open-computer-use -g -y
npx skills upgrade open-computer-use -g -y

Verification

After CLI and MCP setup:

open-computer-use call list_apps
open-computer-use call get_app_state --args '{"app":"TextEdit"}'

If this fails, read troubleshooting.md.

Open Computer Use Troubleshooting

Read this reference when setup, permission checks, app discovery, snapshots, or actions fail.

First Checks

Start with:

open-computer-use -h
open-computer-use doctor
open-computer-use call list_apps

On macOS, doctor reports Accessibility and Screen Recording status. If either is missing, ask the user to approve the onboarding UI.

App Not Found

If get_app_state cannot find an app:

1. Run open-computer-use call list_apps. 2. Use the app name or bundle identifier from that result. 3. Confirm the app is running and has a visible, non-minimized window. 4. On macOS, rerun open-computer-use doctor.

Do not silently switch to a different app when the requested target is not available.

Empty Or Missing Snapshot

Common causes:

• The app has no visible window.
• The window is minimized, hidden, or on an unavailable desktop.
• macOS Screen Recording permission is missing.
• Windows or Linux commands are running outside the logged-in desktop session.
• Linux screenshot support is blocked by the compositor or desktop portal state.

Ask the user to bring the target app/window into a visible state when automation cannot do so safely.

Truncated Text

Snapshot text is limited to 500 characters by default. If a visible chat message, email body, document paragraph, or form value ends with ..., do not assume the page itself is missing content.

Request full accessibility text explicitly:

open-computer-use call get_app_state --args '{"app":"TextEdit","show_full_text":true}'
open-computer-use snapshot --show-full-text TextEdit

show_full_text only disables the text character limit. It does not remove node count, tree depth, screenshot size, permission, or desktop-session protections.

Element Action Fails

If an element-targeted action fails:

1. Re-run get_app_state. 2. Confirm the element_index still exists and refers to the intended UI element. 3. Prefer set_value for settable text/value controls. 4. Prefer perform_secondary_action only for actions exposed in the state result. 5. Use coordinate click, scroll, or drag only after the semantic route is unavailable.

Desktop Session Issues

Windows UI Automation and Linux AT-SPI require a live user desktop. SSH sessions, CI jobs, launch daemons, or services often do not have access to the GUI session even when the CLI binary starts successfully.

If desktop access is missing, ask the user to run the command from the logged-in desktop session or start the target app visibly in that session.

Permission And Safety Issues

• Do not bypass macOS TCC prompts.
• Do not enable global pointer fallbacks unless the user asks for low-level diagnostic behavior.
• Do not interact with password managers or sensitive apps unless the user explicitly requests it.
• Pause before submitting, sending, deleting, purchasing, or approving anything externally visible.

Open Computer Use Usage

Read this reference when the task requires direct Computer Use tool calls, MCP configuration, or platform-specific behavior.

MCP Server

For MCP clients that support stdio servers:

[mcp_servers.open_computer_use]
command = "open-computer-use"
args = ["mcp"]

Equivalent JSON shape:

{
  "mcpServers": {
    "open-computer-use": {
      "command": "open-computer-use",
      "args": ["mcp"]
    }
  }
}

The MCP server exposes:

list_apps
get_app_state
click
perform_secondary_action
scroll
drag
type_text
press_key
set_value

Direct CLI Tool Calls

Use call for one-off checks:

open-computer-use call list_apps
open-computer-use call get_app_state --args '{"app":"TextEdit"}'
open-computer-use call set_value --args '{"app":"TextEdit","element_index":"1","value":"Draft"}'

Use --calls for short action sequences that need to reuse the same process state:

open-computer-use call --calls '[
  {"tool":"get_app_state","args":{"app":"TextEdit"}},
  {"tool":"click","args":{"app":"TextEdit","element_index":"1"}},
  {"tool":"type_text","args":{"app":"TextEdit","text":"Hello"}}
]'

Use --calls-file when the sequence is too large for a readable shell command:

open-computer-use call --calls-file examples/textedit-overlay-seq.json --sleep 0.5

Full Text Snapshots

Snapshot text is truncated to 500 characters by default and ends with ... when truncation happens. This keeps normal UI state compact for agent planning and element-targeted actions.

Use full-text mode when the task depends on complete semantic text, such as chat histories, email bodies, document text, or long form content:

open-computer-use call get_app_state --args '{"app":"TextEdit","show_full_text":true}'
open-computer-use snapshot --show-full-text TextEdit

The same show_full_text tool argument and --show-full-text snapshot flag apply on macOS, Linux, and Windows.

Action tools return refreshed app state with the default 500 character text limit. If full text is still needed after an action, run get_app_state again with show_full_text: true.

Choosing Targets

• Prefer app names or bundle identifiers returned by list_apps.
• Run get_app_state immediately before element-targeted actions.
• Re-run get_app_state after navigation, modal changes, page reloads, or failed actions.
• Use coordinate actions only when the rendered tree does not expose the target as an element.

Platform Notes

macOS

The macOS runtime uses Accessibility, ScreenCaptureKit, and targeted input events. It normally avoids moving the user's real pointer. The visual cursor overlay is part of the Open Computer Use experience and can be disabled by the surrounding runtime only when needed.

Windows

The Windows runtime uses UI Automation and Win32 message fallbacks. It must run in a logged-in desktop session. A detached SSH or service context may start the CLI but fail to see top-level windows.

Linux

The Linux runtime uses AT-SPI2 through the desktop session bus. It must run in a logged-in graphical session with usable accessibility services. Wayland screenshot and coordinate input support is compositor-dependent and best-effort.

Safety

Pause and ask the user before actions that affect external systems or sensitive local state, including sending messages, submitting forms, deleting files, approving prompts, uploading files, or interacting with password managers.