Now liveThe Skillselion MCP - thousands of ranked skills, loaded into your agent mid-task. No install.Get it →
aefmind avatar

Windows Desktop Automation

  • 1 installs
  • Updated January 14, 2026
  • aefmind/windows-automation-skill

windows-desktop-automation is a Claude Code skill that automates Windows GUI apps through a multi-engine orchestrator with mouse, keyboard, OCR, and vision commands.

About

windows-desktop-automation is a Claude Code skill that drives Windows desktop GUI automation through a multi-engine orchestrator. A developer uses it to explore windows, focus a target, find elements, and click, type, or run hotkeys, with a vision and OCR layer for apps like Flutter, Electron, and Canvas that do not expose standard UI trees. It sends commands via node scripts and verifies results with screenshots.

  • Controls Windows GUI apps: explore, focus, click, type, and manage windows
  • Exposes 98 JSON commands plus 18 HTTP endpoints across mouse, keyboard, OCR, and vision
  • Vision layer plus OCR for Flutter, Electron, and Canvas apps that lack accessibility trees

Windows Desktop Automation by the numbers

  • 1 all-time installs (skills.sh)
  • Ranked #1,983 of 2,715 Automation & Workflows skills by installs in the Skillselion catalog
  • Data as of Jul 7, 2026 (Skillselion catalog sync)
At a glance

windows-desktop-automation capabilities & compatibility

Capabilities
gui automation · window management · ocr region · screenshot verify
Use cases
testing · web scraping
Platforms
Windows
Pricing
Free
From the docs

What windows-desktop-automation says it does

AI agent capability for Windows desktop GUI automation via multi-engine orchestrator.
SKILL.md
Control Windows GUI: explore, click, type, manage windows.
SKILL.md
Vision layer for Flutter/Electron/Canvas apps.
SKILL.md
npx skills add https://github.com/aefmind/windows-automation-skill --skill windows-desktop-automation

Add your badge

Show developers this skill is listed on Skillselion. Paste this into your README.

Listed on Skillselion
Installs1
Last updatedJanuary 14, 2026
Repositoryaefmind/windows-automation-skill

What it does

Automate and test Windows desktop applications by exploring, clicking, typing, and reading the screen.

Who is it for?

Automating or QA-testing Windows desktop applications, including Flutter, Electron, and Canvas UIs.

Skip if: macOS or Linux desktop automation, or web-only automation.

When should I use this skill?

The user needs to explore, click, type into, or read a Windows desktop application.

What you get

A Windows app is explored, interacted with, and verified by screenshot from agent commands.

  • Automated Windows GUI interactions
  • Screenshots and OCR text for verification

By the numbers

  • 98 JSON commands plus 18 HTTP endpoints
  • OCR region reads in ~60ms via WinRT native

Files

SKILL.mdMarkdownGitHub ↗

Windows Desktop Automation

Control Windows GUI: explore, click, type, manage windows. Vision layer for Flutter/Electron/Canvas apps. Agent vision mode for AI-driven screen analysis.

Quick Start

node scripts/send-command.cjs '{"action": "explore"}'
node scripts/get-command-help.cjs <command>          # Full command details
node scripts/get-command-help.cjs --list             # All commands
node scripts/get-command-help.cjs --category mouse   # By category

Essential Workflow

1. explore              -> List windows
2. focus_window         -> Activate target
3. explore_window       -> Find elements (or get_window_summary for ~300 tokens)
4. click/type/hotkey    -> Interact
5. screenshot           -> Verify

---

Command Reference (98 JSON + 18 HTTP endpoints)

Discovery (9)

CommandDescription
exploreSTART HERE. List all visible windows
explore_windowExplore elements inside a window (max_depth param)
find_elementFind element with detailed info + clickable coordinates
get_window_infoGet window details (title, bounds, state)
get_element_boundsGet element bounding box and center point
wait_for_elementWait for element to appear (timeout param)
wait_for_stateWait for state: enabled/disabled/visible/hidden/exists
wait_for_colorWait for pixel color at coordinates
wait_for_idleWait for UI to become responsive

Window Management (9)

CommandDescription
focus_windowREQUIRED before interaction. Bring to foreground
close_windowClose window gracefully
wait_for_windowWait for window to appear after launch
move_windowMove window to x,y coordinates
resize_windowResize window to width,height
minimize_windowMinimize window
maximize_windowMaximize window
restore_windowRestore from minimized/maximized
move_to_monitorMove to specific monitor (multi-display)

Application Control (3)

CommandDescription
launch_appStart app by path, optional wait_for_window
list_processesList running processes, filter by name
kill_processKill by PID or name, force option

Mouse Actions (19)

CommandDescription
clickClick element by selector
double_clickDouble-click element
right_clickRight-click for context menu
long_pressPress and hold (duration param)
click_atClick at absolute x,y coordinates
click_relativeClick at offset from element anchor
mouse_moveMove cursor to x,y or selector
mouse_move_easedMove with easing: linear, ease-in-out, bounce, elastic
hoverHover over element for duration
drag_and_dropDrag between selectors or coordinates
swipeTouch-style swipe gesture
scrollScroll wheel up/down by amount
mouse_pathMove through array of waypoints
mouse_bezierMove along cubic bezier curve
drawHold button, move path, release (for drawing apps)
draw_bezierHold button, bezier curve path, release
mouse_downPress and hold button
mouse_upRelease held button
get_cursor_positionGet current cursor x,y

Keyboard Actions (6)

CommandDescription
typeType text into element by selector
type_hereType at current cursor position
hotkeySend combo: ctrl+c, alt+f4, ctrl+shift+s
key_pressSingle key: enter, tab, escape, f1-f12, arrows
key_downPress and hold key
key_upRelease held key

Clipboard (3)

CommandDescription
get_clipboardRead clipboard text
set_clipboardWrite text to clipboard
clipboard_imageCopy image to/from clipboard (set/get operation)

Text & OCR (3)

CommandDescription
read_textRead text from element or region
ocr_regionOCR screen region (WinRT native, ~60ms)
wait_for_textWait for OCR text to appear

Screenshots (4)

CommandDescription
screenshotCapture full screen
element_screenshotCapture specific element only
list_monitorsList connected displays
screenshot_monitorCapture specific monitor by index

File Dialogs (1)

CommandDescription
file_dialogAutomate Open/Save: detect, set_path, set_filename, confirm, cancel

Events (3)

CommandDescription
subscribeSubscribe to element events (focus, invoke, property_change)
unsubscribeUnsubscribe by subscription_id
get_subscriptionsList active subscriptions

Diagnostics (5)

CommandDescription
get_metricsGet command performance stats
clear_metricsReset performance metrics
set_debug_modeEnable/disable verbose logging
get_cache_statsGet element cache stats
clear_cacheClear element cache

Human-like Behavior (5)

CommandDescription
set_human_modeConfigure: jitter, typing_error_rate, fatigue, thinking_delay
get_human_modeGet current human mode settings
human_clickClick with bezier mouse path + random pauses
human_typeType with variable speed, typos, auto-corrections
human_moveMove mouse with curves and micro-jitter

Token-Efficient Commands (6)

CommandTokensDescription
element_exists~30Boolean check only
get_element_brief~100Just name, type, coords
get_window_summary~300Overview vs ~2000 for explore_window
get_interactive_elements~500Only clickable/typeable items
describe_element~100Vision-based description at x,y
fuzzy_find_element~200Levenshtein fuzzy matching

Smart Fallback (3)

CommandDescription
smart_clickFlaUI first -> Vision fallback. Returns method_used. Supports agent_fallback param
smart_typeFlaUI first -> Vision fallback. Supports clear and agent_fallback params
vision_clickClick by OCR text (Flutter/Electron/Canvas)

Agent Vision Commands (7)

CommandDescription
vision_screenshotGet optimized base64 JPEG screenshot for AI vision
vision_screenshot_regionGet cropped region screenshot (token-efficient)
vision_configGet/set vision mode (local/agent/auto)
vision_analyze_smartSmart analysis respecting current vision mode
vision_screenshot_cache_statsGet screenshot cache statistics
vision_screenshot_cache_clearClear screenshot cache
vision_streamGet WebSocket URL for real-time screenshot streaming

Win32 Low-Level Commands (7)

CommandDescription
scroll_sendinputLow-level scroll using Win32 SendInput (bypasses UI Automation)
type_sendinputType Unicode text via Win32 SendInput (direct keyboard events)
set_always_on_topPin/unpin window always on top using SetWindowPos
flash_windowFlash window taskbar/caption for user attention
set_window_opacitySet window transparency (0-255 alpha)
fast_screenshotFast GDI BitBlt screenshot (faster than WinForms)
clipboard_imageCopy image to/from clipboard (set/get operations)

Batch & System (2)

CommandDescription
batchExecute command array, stop_on_error option
healthSystem health check, bridge status

---

Vision/Context HTTP Endpoints (Python Bridge)

The Python bridge runs on http://127.0.0.1:5001 and provides vision, OCR, and context management capabilities.

Vision - Local Processing

POST /vision/detect      # OmniParser UI detection
POST /vision/ocr         # WinRT OCR (166ms full, 60ms region)
POST /vision/find_text   # Locate text on screen
POST /vision/click_text  # Find and click text
POST /vision/analyze     # Combined detection + OCR

Vision - Agent Mode

Returns screenshots for AI vision analysis instead of local processing.

GET  /vision/config                  # Get current mode/settings
POST /vision/config                  # Set mode: local|agent|auto
POST /vision/screenshot              # Optimized base64 JPEG for AI
POST /vision/screenshot_region       # Cropped region (token-efficient)
POST /vision/analyze_or_screenshot   # Smart: local first, screenshot fallback

Vision Modes:

ModeBehavior
localFull OmniParser + OCR processing (default)
agentSkip local processing, return optimized screenshots
autoTry local first; include screenshot if results are limited

Screenshot Caching

Reduces redundant screen captures for rapid successive operations.

GET  /vision/screenshot_cache/stats  # Cache statistics
POST /vision/screenshot_cache/clear  # Clear cached screenshots
POST /vision/screenshot_cache/config # Update TTL/max_entries settings
POST /vision/screenshot_cached       # Screenshot with cache support
Environment VariableDefaultDescription
SCREENSHOT_CACHE_TTL2.0Seconds before cached screenshots expire
SCREENSHOT_CACHE_MAX_ENTRIES5Maximum cached screenshots (LRU eviction)

WebSocket Streaming

WS /vision/stream?fps=5&quality=70   # Real-time JPEG stream

Context/Cache Management

GET  /context/stats      # Cache hit rate
POST /context/clear      # Clear all caches
POST /context/enable     # Enable caching
POST /context/disable    # Disable caching
POST /context/invalidate # Invalidate specific window

---

Usage Examples

Agent Vision Mode

# Switch to agent mode
curl -X POST http://127.0.0.1:5001/vision/config \
  -H "Content-Type: application/json" \
  -d '{"mode": "agent", "jpeg_quality": 75}'

# Get optimized screenshot for AI vision
curl -X POST http://127.0.0.1:5001/vision/screenshot \
  -H "Content-Type: application/json" \
  -d '{"max_width": 1920, "jpeg_quality": 70}'

Screenshot Response:

{
  "screenshot": {
    "data": "base64...",
    "width": 1920,
    "height": 1080,
    "size_bytes": 185000,
    "compression_ratio": 35.2,
    "format": "jpeg"
  }
}

Smart Fallback with Agent Vision

When smart_click or smart_type fails to find an element, enable agent_fallback to get a screenshot for AI analysis.

node scripts/send-command.cjs '{
  "action": "smart_click",
  "search_text": "Submit Button",
  "window": "MyApp",
  "agent_fallback": true,
  "jpeg_quality": 75
}'

Success Response:

{
  "status": "success",
  "action": "smart_click",
  "method_used": "flaui",
  "x": 450,
  "y": 320
}

Agent Fallback Response (element not found):

{
  "status": "agent_fallback",
  "code": "ELEMENT_NOT_FOUND",
  "message": "Element 'Submit Button' not found with any method",
  "suggestion": "Analyze the screenshot and provide click coordinates using click_at command",
  "screenshot": {
    "data": "base64...",
    "width": 1920,
    "height": 1080,
    "format": "jpeg"
  }
}

Win32 Low-Level Commands

# Low-level scroll (bypasses UI Automation)
node scripts/send-command.cjs '{
  "action": "scroll_sendinput",
  "delta": -120,
  "x": 500,
  "y": 400
}'

# Unicode typing including emoji
node scripts/send-command.cjs '{
  "action": "type_sendinput",
  "text": "Hello World! 🎉"
}'

# Pin window on top
node scripts/send-command.cjs '{
  "action": "set_always_on_top",
  "selector": "Notepad",
  "enable": true
}'

# Set window transparency
node scripts/send-command.cjs '{
  "action": "set_window_opacity",
  "selector": "Paint",
  "alpha": 180
}'

# Fast GDI screenshot
node scripts/send-command.cjs '{
  "action": "fast_screenshot",
  "region": [0, 0, 800, 600],
  "format": "png"
}'

---

Error Recovery

CodeRecovery Action
WINDOW_NOT_FOUNDRun explore to list available windows
ELEMENT_NOT_FOUNDTry explore_window, vision_click, or click_at
TIMEOUTIncrease timeout parameter
VISION_NOT_AVAILABLEStart Python bridge with scripts/start-all.ps1
ALL_METHODS_FAILEDVerify element exists, try screenshot + click_at

Agent Fallback: When using smart_click/smart_type with agent_fallback: true, a status: "agent_fallback" response means the element wasn't found but a screenshot is included. Analyze it and call click_at with coordinates.

Full error reference: node scripts/get-command-help.cjs --errors

---

Architecture

AI Agent --> send-command.cjs --> MainAgentService (C#/.NET 9)
                                        |
                                 +------+------+
                                 |             |
                              FlaUI        Python Bridge
                              (UIA3)       (localhost:5001)
                                 |             |
                           Windows UI    +-----+-----+
                           Automation    |     |     |
                                     pywinauto Vision Context
                                              |      Manager
                                         +----+----+
                                         |         |
                                      WinRT    OmniParser
                                       OCR       ONNX
LayerTechnologyPurpose
PrimaryFlaUI (UIA3)Windows UI Automation - native, fast
SecondaryPython Bridgepywinauto + Vision + WinRT OCR + Context
VisionOmniParser (ONNX)UI element detection for non-accessible apps
OCRWinRT OCRFast native text recognition (~60ms)

---

Tips

1. Always `explore` first - Never assume window state 2. `focus_window` before interact - Required for clicks/typing 3. *Use `wait_for_ commands** - Apps need time to respond 4. **Token-efficient first** - Prefer get_window_summary over explore_window 5. **Use smart_click/smart_type** - Auto FlaUI -> Vision fallback 6. **Check health** - Diagnose bridge issues 7. **Use agent_fallback: true`** - Get screenshots when elements aren't found

---

Full command schema and examples: `node scripts/get-command-help.cjs <command>`

Related skills

FAQ

What apps can it automate?

Windows GUI apps, including Flutter, Electron, and Canvas apps handled by its vision layer.

How many commands does it expose?

98 JSON commands plus 18 HTTP endpoints across discovery, window, mouse, keyboard, OCR, and vision categories.

This week in AI coding

Five minutes, every Monday - the tools, releases and tactics for developers.

unsubscribe anytime.