
Windows Desktop Automation
- 1 installs
- Updated January 14, 2026
- aefmind/windows-automation-skill
windows-desktop-automation is a Claude Code skill that automates Windows GUI apps through a multi-engine orchestrator with mouse, keyboard, OCR, and vision commands.
About
windows-desktop-automation is a Claude Code skill that drives Windows desktop GUI automation through a multi-engine orchestrator. A developer uses it to explore windows, focus a target, find elements, and click, type, or run hotkeys, with a vision and OCR layer for apps like Flutter, Electron, and Canvas that do not expose standard UI trees. It sends commands via node scripts and verifies results with screenshots.
- Controls Windows GUI apps: explore, focus, click, type, and manage windows
- Exposes 98 JSON commands plus 18 HTTP endpoints across mouse, keyboard, OCR, and vision
- Vision layer plus OCR for Flutter, Electron, and Canvas apps that lack accessibility trees
Windows Desktop Automation by the numbers
- 1 all-time installs (skills.sh)
- Ranked #1,983 of 2,715 Automation & Workflows skills by installs in the Skillselion catalog
- Data as of Jul 7, 2026 (Skillselion catalog sync)
windows-desktop-automation capabilities & compatibility
- Capabilities
- gui automation · window management · ocr region · screenshot verify
- Use cases
- testing · web scraping
- Platforms
- Windows
- Pricing
- Free
What windows-desktop-automation says it does
AI agent capability for Windows desktop GUI automation via multi-engine orchestrator.
Control Windows GUI: explore, click, type, manage windows.
Vision layer for Flutter/Electron/Canvas apps.
npx skills add https://github.com/aefmind/windows-automation-skill --skill windows-desktop-automationAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 1 |
|---|---|
| Last updated | January 14, 2026 |
| Repository | aefmind/windows-automation-skill ↗ |
What it does
Automate and test Windows desktop applications by exploring, clicking, typing, and reading the screen.
Who is it for?
Automating or QA-testing Windows desktop applications, including Flutter, Electron, and Canvas UIs.
Skip if: macOS or Linux desktop automation, or web-only automation.
When should I use this skill?
The user needs to explore, click, type into, or read a Windows desktop application.
What you get
A Windows app is explored, interacted with, and verified by screenshot from agent commands.
- Automated Windows GUI interactions
- Screenshots and OCR text for verification
By the numbers
- 98 JSON commands plus 18 HTTP endpoints
- OCR region reads in ~60ms via WinRT native
Files
Windows Desktop Automation
Control Windows GUI: explore, click, type, manage windows. Vision layer for Flutter/Electron/Canvas apps. Agent vision mode for AI-driven screen analysis.
Quick Start
node scripts/send-command.cjs '{"action": "explore"}'
node scripts/get-command-help.cjs <command> # Full command details
node scripts/get-command-help.cjs --list # All commands
node scripts/get-command-help.cjs --category mouse # By categoryEssential Workflow
1. explore -> List windows
2. focus_window -> Activate target
3. explore_window -> Find elements (or get_window_summary for ~300 tokens)
4. click/type/hotkey -> Interact
5. screenshot -> Verify---
Command Reference (98 JSON + 18 HTTP endpoints)
Discovery (9)
| Command | Description |
|---|---|
explore | START HERE. List all visible windows |
explore_window | Explore elements inside a window (max_depth param) |
find_element | Find element with detailed info + clickable coordinates |
get_window_info | Get window details (title, bounds, state) |
get_element_bounds | Get element bounding box and center point |
wait_for_element | Wait for element to appear (timeout param) |
wait_for_state | Wait for state: enabled/disabled/visible/hidden/exists |
wait_for_color | Wait for pixel color at coordinates |
wait_for_idle | Wait for UI to become responsive |
Window Management (9)
| Command | Description |
|---|---|
focus_window | REQUIRED before interaction. Bring to foreground |
close_window | Close window gracefully |
wait_for_window | Wait for window to appear after launch |
move_window | Move window to x,y coordinates |
resize_window | Resize window to width,height |
minimize_window | Minimize window |
maximize_window | Maximize window |
restore_window | Restore from minimized/maximized |
move_to_monitor | Move to specific monitor (multi-display) |
Application Control (3)
| Command | Description |
|---|---|
launch_app | Start app by path, optional wait_for_window |
list_processes | List running processes, filter by name |
kill_process | Kill by PID or name, force option |
Mouse Actions (19)
| Command | Description |
|---|---|
click | Click element by selector |
double_click | Double-click element |
right_click | Right-click for context menu |
long_press | Press and hold (duration param) |
click_at | Click at absolute x,y coordinates |
click_relative | Click at offset from element anchor |
mouse_move | Move cursor to x,y or selector |
mouse_move_eased | Move with easing: linear, ease-in-out, bounce, elastic |
hover | Hover over element for duration |
drag_and_drop | Drag between selectors or coordinates |
swipe | Touch-style swipe gesture |
scroll | Scroll wheel up/down by amount |
mouse_path | Move through array of waypoints |
mouse_bezier | Move along cubic bezier curve |
draw | Hold button, move path, release (for drawing apps) |
draw_bezier | Hold button, bezier curve path, release |
mouse_down | Press and hold button |
mouse_up | Release held button |
get_cursor_position | Get current cursor x,y |
Keyboard Actions (6)
| Command | Description |
|---|---|
type | Type text into element by selector |
type_here | Type at current cursor position |
hotkey | Send combo: ctrl+c, alt+f4, ctrl+shift+s |
key_press | Single key: enter, tab, escape, f1-f12, arrows |
key_down | Press and hold key |
key_up | Release held key |
Clipboard (3)
| Command | Description |
|---|---|
get_clipboard | Read clipboard text |
set_clipboard | Write text to clipboard |
clipboard_image | Copy image to/from clipboard (set/get operation) |
Text & OCR (3)
| Command | Description |
|---|---|
read_text | Read text from element or region |
ocr_region | OCR screen region (WinRT native, ~60ms) |
wait_for_text | Wait for OCR text to appear |
Screenshots (4)
| Command | Description |
|---|---|
screenshot | Capture full screen |
element_screenshot | Capture specific element only |
list_monitors | List connected displays |
screenshot_monitor | Capture specific monitor by index |
File Dialogs (1)
| Command | Description |
|---|---|
file_dialog | Automate Open/Save: detect, set_path, set_filename, confirm, cancel |
Events (3)
| Command | Description |
|---|---|
subscribe | Subscribe to element events (focus, invoke, property_change) |
unsubscribe | Unsubscribe by subscription_id |
get_subscriptions | List active subscriptions |
Diagnostics (5)
| Command | Description |
|---|---|
get_metrics | Get command performance stats |
clear_metrics | Reset performance metrics |
set_debug_mode | Enable/disable verbose logging |
get_cache_stats | Get element cache stats |
clear_cache | Clear element cache |
Human-like Behavior (5)
| Command | Description |
|---|---|
set_human_mode | Configure: jitter, typing_error_rate, fatigue, thinking_delay |
get_human_mode | Get current human mode settings |
human_click | Click with bezier mouse path + random pauses |
human_type | Type with variable speed, typos, auto-corrections |
human_move | Move mouse with curves and micro-jitter |
Token-Efficient Commands (6)
| Command | Tokens | Description |
|---|---|---|
element_exists | ~30 | Boolean check only |
get_element_brief | ~100 | Just name, type, coords |
get_window_summary | ~300 | Overview vs ~2000 for explore_window |
get_interactive_elements | ~500 | Only clickable/typeable items |
describe_element | ~100 | Vision-based description at x,y |
fuzzy_find_element | ~200 | Levenshtein fuzzy matching |
Smart Fallback (3)
| Command | Description |
|---|---|
smart_click | FlaUI first -> Vision fallback. Returns method_used. Supports agent_fallback param |
smart_type | FlaUI first -> Vision fallback. Supports clear and agent_fallback params |
vision_click | Click by OCR text (Flutter/Electron/Canvas) |
Agent Vision Commands (7)
| Command | Description |
|---|---|
vision_screenshot | Get optimized base64 JPEG screenshot for AI vision |
vision_screenshot_region | Get cropped region screenshot (token-efficient) |
vision_config | Get/set vision mode (local/agent/auto) |
vision_analyze_smart | Smart analysis respecting current vision mode |
vision_screenshot_cache_stats | Get screenshot cache statistics |
vision_screenshot_cache_clear | Clear screenshot cache |
vision_stream | Get WebSocket URL for real-time screenshot streaming |
Win32 Low-Level Commands (7)
| Command | Description |
|---|---|
scroll_sendinput | Low-level scroll using Win32 SendInput (bypasses UI Automation) |
type_sendinput | Type Unicode text via Win32 SendInput (direct keyboard events) |
set_always_on_top | Pin/unpin window always on top using SetWindowPos |
flash_window | Flash window taskbar/caption for user attention |
set_window_opacity | Set window transparency (0-255 alpha) |
fast_screenshot | Fast GDI BitBlt screenshot (faster than WinForms) |
clipboard_image | Copy image to/from clipboard (set/get operations) |
Batch & System (2)
| Command | Description |
|---|---|
batch | Execute command array, stop_on_error option |
health | System health check, bridge status |
---
Vision/Context HTTP Endpoints (Python Bridge)
The Python bridge runs on http://127.0.0.1:5001 and provides vision, OCR, and context management capabilities.
Vision - Local Processing
POST /vision/detect # OmniParser UI detection
POST /vision/ocr # WinRT OCR (166ms full, 60ms region)
POST /vision/find_text # Locate text on screen
POST /vision/click_text # Find and click text
POST /vision/analyze # Combined detection + OCRVision - Agent Mode
Returns screenshots for AI vision analysis instead of local processing.
GET /vision/config # Get current mode/settings
POST /vision/config # Set mode: local|agent|auto
POST /vision/screenshot # Optimized base64 JPEG for AI
POST /vision/screenshot_region # Cropped region (token-efficient)
POST /vision/analyze_or_screenshot # Smart: local first, screenshot fallbackVision Modes:
| Mode | Behavior |
|---|---|
local | Full OmniParser + OCR processing (default) |
agent | Skip local processing, return optimized screenshots |
auto | Try local first; include screenshot if results are limited |
Screenshot Caching
Reduces redundant screen captures for rapid successive operations.
GET /vision/screenshot_cache/stats # Cache statistics
POST /vision/screenshot_cache/clear # Clear cached screenshots
POST /vision/screenshot_cache/config # Update TTL/max_entries settings
POST /vision/screenshot_cached # Screenshot with cache support| Environment Variable | Default | Description |
|---|---|---|
SCREENSHOT_CACHE_TTL | 2.0 | Seconds before cached screenshots expire |
SCREENSHOT_CACHE_MAX_ENTRIES | 5 | Maximum cached screenshots (LRU eviction) |
WebSocket Streaming
WS /vision/stream?fps=5&quality=70 # Real-time JPEG streamContext/Cache Management
GET /context/stats # Cache hit rate
POST /context/clear # Clear all caches
POST /context/enable # Enable caching
POST /context/disable # Disable caching
POST /context/invalidate # Invalidate specific window---
Usage Examples
Agent Vision Mode
# Switch to agent mode
curl -X POST http://127.0.0.1:5001/vision/config \
-H "Content-Type: application/json" \
-d '{"mode": "agent", "jpeg_quality": 75}'
# Get optimized screenshot for AI vision
curl -X POST http://127.0.0.1:5001/vision/screenshot \
-H "Content-Type: application/json" \
-d '{"max_width": 1920, "jpeg_quality": 70}'Screenshot Response:
{
"screenshot": {
"data": "base64...",
"width": 1920,
"height": 1080,
"size_bytes": 185000,
"compression_ratio": 35.2,
"format": "jpeg"
}
}Smart Fallback with Agent Vision
When smart_click or smart_type fails to find an element, enable agent_fallback to get a screenshot for AI analysis.
node scripts/send-command.cjs '{
"action": "smart_click",
"search_text": "Submit Button",
"window": "MyApp",
"agent_fallback": true,
"jpeg_quality": 75
}'Success Response:
{
"status": "success",
"action": "smart_click",
"method_used": "flaui",
"x": 450,
"y": 320
}Agent Fallback Response (element not found):
{
"status": "agent_fallback",
"code": "ELEMENT_NOT_FOUND",
"message": "Element 'Submit Button' not found with any method",
"suggestion": "Analyze the screenshot and provide click coordinates using click_at command",
"screenshot": {
"data": "base64...",
"width": 1920,
"height": 1080,
"format": "jpeg"
}
}Win32 Low-Level Commands
# Low-level scroll (bypasses UI Automation)
node scripts/send-command.cjs '{
"action": "scroll_sendinput",
"delta": -120,
"x": 500,
"y": 400
}'
# Unicode typing including emoji
node scripts/send-command.cjs '{
"action": "type_sendinput",
"text": "Hello World! 🎉"
}'
# Pin window on top
node scripts/send-command.cjs '{
"action": "set_always_on_top",
"selector": "Notepad",
"enable": true
}'
# Set window transparency
node scripts/send-command.cjs '{
"action": "set_window_opacity",
"selector": "Paint",
"alpha": 180
}'
# Fast GDI screenshot
node scripts/send-command.cjs '{
"action": "fast_screenshot",
"region": [0, 0, 800, 600],
"format": "png"
}'---
Error Recovery
| Code | Recovery Action |
|---|---|
WINDOW_NOT_FOUND | Run explore to list available windows |
ELEMENT_NOT_FOUND | Try explore_window, vision_click, or click_at |
TIMEOUT | Increase timeout parameter |
VISION_NOT_AVAILABLE | Start Python bridge with scripts/start-all.ps1 |
ALL_METHODS_FAILED | Verify element exists, try screenshot + click_at |
Agent Fallback: When using smart_click/smart_type with agent_fallback: true, a status: "agent_fallback" response means the element wasn't found but a screenshot is included. Analyze it and call click_at with coordinates.
Full error reference: node scripts/get-command-help.cjs --errors
---
Architecture
AI Agent --> send-command.cjs --> MainAgentService (C#/.NET 9)
|
+------+------+
| |
FlaUI Python Bridge
(UIA3) (localhost:5001)
| |
Windows UI +-----+-----+
Automation | | |
pywinauto Vision Context
| Manager
+----+----+
| |
WinRT OmniParser
OCR ONNX| Layer | Technology | Purpose |
|---|---|---|
| Primary | FlaUI (UIA3) | Windows UI Automation - native, fast |
| Secondary | Python Bridge | pywinauto + Vision + WinRT OCR + Context |
| Vision | OmniParser (ONNX) | UI element detection for non-accessible apps |
| OCR | WinRT OCR | Fast native text recognition (~60ms) |
---
Tips
1. Always `explore` first - Never assume window state 2. `focus_window` before interact - Required for clicks/typing 3. *Use `wait_for_ commands** - Apps need time to respond 4. **Token-efficient first** - Prefer get_window_summary over explore_window 5. **Use smart_click/smart_type** - Auto FlaUI -> Vision fallback 6. **Check health** - Diagnose bridge issues 7. **Use agent_fallback: true`** - Get screenshots when elements aren't found
---
Full command schema and examples: `node scripts/get-command-help.cjs <command>`
# Python
.venv/
__pycache__/
*.pyc
*.pyo
*.pyd
# Node
node_modules/
# .NET
bin/
obj/
# OS / tooling
.DS_Store
Thumbs.db
# Logs
*.log
# Embedded Python (should not be committed)
src/python-embedded/
# Misc
.vscode/
# Weird Windows reserved name that shouldn't exist as a file
nul
AGENTS.md
This repository is designed for collaborative development by humans and AI coding agents.
Project Overview
Windows Desktop Automation Skill - A .NET 9 Windows application that provides GUI automation capabilities via FlaUI and Win32 APIs, with a Python bridge for vision/OCR functionality.
Repository Structure
windows-desktop-automation/
├── src/
│ ├── src/MainAgentService/ # .NET 9 orchestrator (FlaUI + Win32)
│ └── bridge_python/ # Python Flask service (vision + OCR)
├── scripts/
│ ├── send-command.cjs # Primary command interface
│ ├── get-command-help.cjs # Command documentation helper
│ ├── test-commands.cjs # Test runner
│ ├── start-all.ps1 # Start all services
│ ├── stop-all.ps1 # Stop all services
│ └── auto-setup.ps1 # One-time environment setup
├── docs/ # Additional documentation
├── SKILL.md # Command reference (for AI agents)
├── README.md # User-facing documentation
├── DEVELOPMENT.md # Developer setup guide
└── AGENTS.md # This fileKey Files
| File | Lines | Purpose |
|---|---|---|
src/src/MainAgentService/Program.cs | ~6,400 | Main orchestrator (all command handlers) |
src/bridge_python/server.py | ~800 | Python Flask server (vision/OCR) |
SKILL.md | ~350 | Command reference for AI consumption |
Scope & Boundaries
In Scope
- Orchestrator:
src/src/MainAgentService/(.NET + FlaUI + Win32) - Python Bridge:
src/bridge_python/(Flask + pywinauto + WinRT OCR) - Scripts:
scripts/*.ps1andscripts/*.cjs - Documentation:
SKILL.md,README.md,DEVELOPMENT.md
Out of Scope / Safety Boundaries
- Do NOT automate destructive OS actions without explicit user intent
- Do NOT run UI automation against sensitive apps (password managers, banking) by default
- Prefer idempotent scripts (safe to re-run)
- Avoid modifying system settings or registry
Development Guidelines
Prerequisites
- Windows 10/11
- .NET SDK 9+
- Python 3.11+ (available as
pyorpython) - Node.js (for scripts)
Setup
# One-time setup
powershell -File scripts/auto-setup.ps1
# Start services
powershell -File scripts/start-all.ps1
# Stop services
powershell -File scripts/stop-all.ps1Build & Test
# Build .NET orchestrator
dotnet build src/src/MainAgentService/MainAgentService.csproj
# Run tests
node scripts/test-commands.cjs --list # List all tests
node scripts/test-commands.cjs # Run all tests
node scripts/test-commands.cjs --category mouse # Run categoryCode Style
- .NET: Follow standard C# conventions
- Python: Use
rufffor formatting/linting - Scripts: Node.js CommonJS (
.cjsextension)
Definition of Done
A task is complete when:
1. scripts/start-all.ps1 works with system Python (py or python) 2. No embedded Python runtime is required or referenced 3. Documentation is updated if behavior changes 4. git status shows only intentional changes 5. Build passes: dotnet build with 0 errors, 0 warnings
Git Hygiene
.gitignore Coverage
The following are excluded from version control:
.venv/ # Python virtual environment
__pycache__/ # Python bytecode
*.pyc, *.pyo # Python compiled files
node_modules/ # Node dependencies
bin/, obj/ # .NET build output
*.log # Log files
src/python-embedded/ # Embedded Python (not used)Commit Guidelines
- Write clear commit messages describing "why" not "what"
- Keep commits atomic (one logical change per commit)
- Verify clean build before committing
Agent-Specific Instructions
When Exploring the Codebase
1. Start with SKILL.md for command reference 2. Use Program.cs for implementation details (~6,400 lines, well-structured) 3. Check scripts/send-command.cjs for the command interface
When Making Changes
1. Build after every change: dotnet build src/src/MainAgentService/ 2. Make small, surgical edits - avoid large refactors 3. Test changes: Use scripts/send-command.cjs or test-commands.cjs 4. Update docs if adding/modifying commands
When Adding Commands
1. Add handler method in Program.cs 2. Add case in the main switch statement 3. Update SKILL.md with command documentation 4. Update docs/command-schemas.json with the new command schema 5. Add test case in scripts/test-commands.cjs (optional)
Keeping Documentation in Sync
When modifying commands, ensure these files stay synchronized:
| File | What to Update |
|---|---|
SKILL.md | Command reference tables, version number |
docs/command-schemas.json | Full JSON schema, version number, categories |
README.md | User-facing docs if behavior changes |
Version Numbers: When adding new commands or significant features: 1. Update version in SKILL.md front matter 2. Update version in docs/command-schemas.json 3. Update description in docs/command-schemas.json to mention new features
Common Patterns in Program.cs
// Handler method pattern
private void HandleMyCommand(JsonElement root)
{
// 1. Parse required parameters
if (!root.TryGetProperty("param", out var paramEl))
{
WriteMissingParam("param");
return;
}
// 2. Execute action
// ...
// 3. Write response
WriteSuccess(new { result = "value" });
}Review Checklist
Before submitting changes, verify:
- [ ]
dotnet buildpasses with 0 errors, 0 warnings - [ ]
git statusshows only intentional files - [ ] Entry points work:
scripts/start-all.ps1,scripts/auto-setup.ps1 - [ ] Documentation updated if needed (
SKILL.md,README.md) - [ ] No secrets or credentials committed
Troubleshooting
Python not found
# Verify Python is installed
py -3 --version
# or
python --versionPort 5001 already in use
scripts/stop-all.ps1
# Then retry
scripts/start-all.ps1Build errors
# Clean and rebuild
dotnet clean src/src/MainAgentService/
dotnet build src/src/MainAgentService/Development
Prerequisites
- Windows 10/11
- .NET SDK 9+
- Python 3.11+ (recommended) available as
pyorpython - Node.js (for
scripts/*.cjshelpers/tests)
Setup (recommended)
# from repo root
powershell -File scripts/auto-setup.ps1This will:
- create
.venv/ - install Python dependencies into
.venv/ - (optionally) download OmniParser model via
scripts/download-omniparser.ps1 - build the .NET orchestrator
Manual Python setup
py -3 -m venv .venv
.\.venv\Scripts\python.exe -m pip install --upgrade pip
.\.venv\Scripts\python.exe -m pip install flask pywinauto comtypes onnxruntime numpy PillowRun
powershell -File scripts/start-all.ps1Stop
powershell -File scripts/stop-all.ps1Build (.NET)
dotnet build src\src\MainAgentService\MainAgentService.csprojTests (Node)
node scripts\test-commands.cjs --list
node scripts\test-commands.cjs --category errorHandling
node scripts\test-commands.cjsTroubleshooting
Python not found
- Install Python 3.11+ from https://www.python.org/downloads/
- On Windows, ensure the Python Launcher is installed so
py -3works.
Port 5001 already in use
Run scripts/stop-all.ps1 and retry.
Vision Layer Performance Analysis
Executive Summary
The Vision Layer (OmniParser + ONNX + WinRT OCR) delivers sub-300ms full analysis with a combined pipeline taking ~250ms. After WinRT native OCR optimization, we achieved 3.2x faster OCR by bypassing PowerShell subprocess overhead.
Key Achievement: Full screen OCR in 166ms (was 530ms), region OCR in 60ms (was 120ms).
---
Performance Benchmarks (v4.2 - WinRT Native)
Timing Breakdown
| Component | Time (ms) | % of Total |
|---|---|---|
| Screenshot Capture | 49 | 20% |
| Image Preprocessing | 17 | 7% |
| ONNX Inference | 66 | 26% |
| Postprocessing (NMS) | 12 | 5% |
| OmniParser Total | ~76 | 30% |
| WinRT OCR | ~166 | 66% |
| Full Analyze | ~250 | 100% |
Key Metrics
Screen Resolution: 1920x1080
Model Input Size: 640x640 (letterboxed)
Scale Factor: 3.0x
Padding: (0, 140) - top/bottom for 16:9 aspect
Confidence Threshold: 0.10 (optimized from 0.30)---
OmniParser ONNX Analysis
Model Specifications
- Model:
omniparser-icon_detect_fp16.onnx(6MB, FP16) - Architecture: YOLO-style detector
- Output Shape:
[1, 5, 8400](8400 candidate boxes) - Format:
[x_center, y_center, width, height, confidence]
Raw Output Statistics
Total Predictions: 8400
Confidence Distribution:
Min: 0.0000
Max: 0.4087
Mean: 0.0044
Std: 0.0203
Detections by Threshold:
>= 0.01: 675
>= 0.05: 182
>= 0.10: 90 ← Current threshold
>= 0.20: 21
>= 0.30: 3
>= 0.50: 0
>= 0.70: 0After NMS (Non-Maximum Suppression)
| Threshold | Detections | Processing Time | Recommendation |
|---|---|---|---|
| 0.01 | 174 | 85ms | Too many false positives |
| 0.10 | 37 | 12ms | Optimal (current) |
| 0.30 | 3 | 7ms | Too aggressive |
Detected Elements (Sample @ 0.10 threshold)
[0] pos=(1228, 1058) size=33x44 conf=0.409 # Taskbar icon
[1] pos=(1272, 1057) size=34x47 conf=0.354 # Taskbar icon
[2] pos=(1418, 380) size=462x23 conf=0.319 # UI element
[3] pos=(1153, 307) size=48x25 conf=0.286 # Button
[4] pos=(1185, 1057) size=29x37 conf=0.268 # Taskbar icon
[5] pos=(958, 977) size=38x42 conf=0.250 # Icon
[6] pos=(1878, 1054) size=56x32 conf=0.250 # System tray
[7] pos=(1275, 52) size=373x24 conf=0.245 # Title bar
[8] pos=(867, 1058) size=45x39 conf=0.229 # Taskbar icon
[9] pos=(38, 210) size=40x48 conf=0.224 # Desktop icon---
Windows OCR Analysis
Performance (v4.2 - WinRT Native)
- Time: 166ms for full 1920x1080 screen (was 530-570ms with PowerShell)
- Region OCR: 60ms for small areas (taskbar, dialogs)
- Cached: <1ms for repeated queries
- Text Regions Detected: 340-370 (depends on screen content)
- Confidence: 1.0 for all detected text (Windows OCR is binary)
OCR Optimization (v4.2 - WinRT Native)
After extensive benchmarking, we implemented several optimizations including WinRT native OCR which bypasses PowerShell subprocess overhead:
| Optimization | Time | Improvement |
|---|---|---|
| WinRT Native OCR | 166ms avg | 3.2x faster than PowerShell |
| Caching | <1ms | 3400x faster for cache hits |
| Region-based OCR | 60ms | 8.8x faster for small areas |
Benchmark Results (v4.2)
WinRT Full Screen OCR: 166ms (avg), 156-200ms range
PowerShell OCR: 535ms (baseline - no longer used by default)
Cached OCR: 0.16ms (instant for repeated queries)
Region OCR (taskbar): 60ms (was 120ms with PowerShell)WinRT Native OCR Implementation
We replaced PowerShell subprocess calls with direct Windows Runtime API access via winrt-runtime:
# Old approach (PowerShell subprocess) - 535ms
subprocess.run(["powershell", "-File", "ocr_script.ps1", image_path])
# New approach (WinRT native) - 166ms
from winrt.windows.media.ocr import OcrEngine
from winrt.windows.graphics.imaging import BitmapDecoder
engine = OcrEngine.try_create_from_language(Language("en-US"))
result = await engine.recognize_async(bitmap)Packages installed:
winrt-runtimewinrt-Windows.Media.Ocrwinrt-Windows.Graphics.Imagingwinrt-Windows.Storagewinrt-Windows.Globalizationwinrt-Windows.Foundation
PowerShell Bottleneck (Solved!)
The primary bottleneck was PowerShell subprocess overhead:
- PowerShell startup: ~200-250ms (now bypassed)
- Windows.Media.Ocr processing: ~166ms (direct access)
- Total savings: ~370ms per OCR call
Alternative OCR Engines Tested
| Engine | Time | Regions | Result |
|---|---|---|---|
| WinRT Native | 166ms | 344 | ✅ Best - 3.2x faster |
| Windows OCR (PowerShell) | 530ms | 354 | ✅ Fallback option |
| RapidOCR (ONNX) | 7,400ms | 135 | ❌ Too slow on CPU |
| Image scaling (50%) | 445ms | 19 | ❌ Loses 95% of text |
| pythonnet (WinRT via CLR) | 1,600ms | 0 | ❌ Doesn't work |
Coverage Quality
The OCR detected all visible text including:
- Application titles ("Calculator", "OpenCode")
- Button labels ("CE", numbers 0-9)
- Menu items and status bar text
- System tray information
- Even small text in complex UIs
---
Comparison: OmniParser vs OCR
| Aspect | OmniParser (th=0.10) | Windows OCR |
|---|---|---|
| Speed | ~76ms | ~650ms |
| Elements Found | 37 | 185-207 |
| Best For | Icons, buttons, UI regions | Text labels, menus |
| Confidence Range | 0.10-0.41 | 1.0 (binary) |
| Training Domain | Web/mobile UI icons | Any text |
---
Threshold Optimization Results
Before Optimization (v4.0 initial)
Threshold: 0.30
Detections: 2-5
Status: Missed most UI elementsAfter Optimization (v4.0 updated)
Threshold: 0.10
Detections: 37
Status: Good coverage of icons and UI regions
Improvement: 7-18x more detectionsFiles Modified
src/bridge_python/bridge.py:confidence_threshold=0.10src/bridge_python/vision/detector.py: default0.15src/bridge_python/vision/vision_service.py: default0.15
---
What Gets Detected
Well Detected (confidence > 0.20)
- Taskbar icons
- Desktop shortcuts
- Application icons in title bars
- Buttons with distinct graphical elements
- Large UI regions/panels
Partially Detected (confidence 0.10-0.20)
- Text fields with icons
- Status bar icons
- Tool buttons
- Navigation elements
Not Detected (use OCR)
- Text-only buttons
- Windows native controls (checkboxes, sliders)
- Flat/minimal UI elements without icons
---
Recommendations
1. Current Configuration (Optimal)
# Production settings
confidence_threshold = 0.10 # 37 detections, good balance2. Hybrid Strategy (Implemented)
The current implementation correctly uses hybrid detection:
1. OmniParser for icon/button detection (37 elements) 2. Windows OCR for text-based elements (200+ regions) 3. Smart fallback when FlaUI fails
3. Use Case Guide
| Task | Best Method | Expected Performance |
|---|---|---|
| Click button by text | WinRT OCR + find_text | 166ms |
| Click icon | OmniParser + vision_click | 76ms |
| Read screen content | OCR (cached) | <1ms cached |
| Search in taskbar | Region-based OCR | 60ms |
| Automated UI testing | FlaUI (first) → Vision (fallback) | <50ms / 250ms |
4. OCR Optimization Strategy
Choose the right OCR method based on your use case:
┌─────────────────────────────────────────────────────────────────┐
│ OCR DECISION TREE (v4.2) │
├─────────────────────────────────────────────────────────────────┤
│ Q: Is this a repeated query (same screen)? │
│ YES → Use /vision/ocr_cached (<1ms) │
│ NO ↓ │
│ │
│ Q: Do you know where the text should be? │
│ YES → Use /vision/ocr_region with coordinates (60ms) │
│ NO ↓ │
│ │
│ Q: Do you have location hints? │
│ YES → Use /vision/find_text_fast with hint_regions (60ms) │
│ NO → Use /vision/ocr (166ms full scan - WinRT) │
└─────────────────────────────────────────────────────────────────┘5. Future Improvements
1. ~~pythonnet direct Windows API: Bypass PowerShell overhead~~ ✅ Done - WinRT native 2. Adaptive thresholding: Automatically lower threshold if < 10 detections 3. Alternative models: Florence-2, Grounding DINO for better Windows UI detection 4. Custom fine-tuning: Train on Windows 11 UI specifically
---
Conclusion
The Vision Layer performs excellently after WinRT native OCR optimization:
| Metric | v4.0 | v4.1 | v4.2 WinRT | Assessment |
|---|---|---|---|---|
| Full analysis time | ~720ms | ~600ms | ~250ms | 3x faster |
| OmniParser inference | ~90ms | ~76ms | ~76ms | Very fast |
| Icon detection | 2-5 | 37 | 37 | 7x improvement |
| Text detection | 185+ | 354+ | 344 | Comprehensive |
| OCR (full screen) | 530ms | 530ms | 166ms | 3.2x faster |
| OCR (cached) | N/A | <1ms | <1ms | Instant |
| OCR (region) | N/A | 120ms | 60ms | 8.8x faster |
Key Achievements in v4.2:
- ✅ WinRT native OCR bypasses PowerShell - 3.2x faster (166ms vs 535ms)
- ✅ OCR caching provides instant results for repeated queries (<1ms)
- ✅ Region-based OCR is 8.8x faster for targeted searches (60ms)
- ✅ New optimized API endpoints for fine-grained control
- ✅ PowerShell bottleneck SOLVED via winrt-runtime package
The hybrid approach (FlaUI → Vision fallback) with optimized OmniParser and Windows OCR provides robust automation capabilities for Windows desktop applications.
---
Test Commands
# Run full diagnostic (use your venv)
& ".\\.venv\\Scripts\\python.exe" `
".\\src\\bridge_python\\vision\\diagnostic.py"
# Test vision endpoints
curl -X POST http://127.0.0.1:5001/vision/detect -H "Content-Type: application/json" -d "{}"
curl -X POST http://127.0.0.1:5001/vision/analyze -H "Content-Type: application/json" -d "{}"
# Check health
curl http://127.0.0.1:5001/health
curl http://127.0.0.1:5001/context/stats---
OCR API Reference (v4.1)
Standard Endpoints
| Endpoint | Method | Description | Performance |
|---|---|---|---|
/vision/ocr | POST | Full screen OCR | ~530ms |
/vision/find_text | POST | Find element by text | ~650ms |
/vision/click_text | POST | Click on text | ~700ms |
/vision/analyze | POST | Full screen analysis | ~720ms |
Optimized Endpoints (New in v4.1)
| Endpoint | Method | Description | Performance |
|---|---|---|---|
/vision/ocr_cached | POST | OCR with caching | <1ms cached |
/vision/ocr_region | POST | Region-based OCR | ~120ms |
/vision/find_text_fast | POST | Fast text search with hints | ~120-380ms |
/vision/cache_stats | GET | Cache statistics | <1ms |
/vision/cache_clear | POST | Clear OCR cache | <1ms |
/vision/ocr_cached
OCR with explicit cache control.
Request:
{
"use_cache": true,
"clear_cache": false,
"full_text": false
}Response:
{
"status": "success",
"elapsed_ms": 0,
"region_count": 354,
"cache_stats": {"entries": 1, "ttl_seconds": 2.0}
}/vision/ocr_region
OCR only a specific screen region (4x faster).
Request:
{
"x": 0,
"y": 1040,
"width": 1920,
"height": 40
}Response:
{
"status": "success",
"elapsed_ms": 122,
"region_count": 7,
"search_region": {"x": 0, "y": 1040, "width": 1920, "height": 40}
}/vision/find_text_fast
Fast text search with optional region hints.
Request:
{
"text": "Search",
"hint_regions": [
{"x": 0, "y": 1040, "width": 1920, "height": 40}
]
}Common hint regions:
- Taskbar:
{"x": 0, "y": 1040, "width": 1920, "height": 40} - Title bar:
{"x": 0, "y": 0, "width": 1920, "height": 50} - Center dialog:
{"x": 660, "y": 340, "width": 600, "height": 400}
---
_Analysis performed: January 11, 2026_ _Vision Layer v4.1 (OCR optimized)_
Windows Desktop Automation Skill
Hybrid automation engine for Windows using .NET 9 + FlaUI with Python vision fallback. Supports modern (UWP/WPF), legacy (Win32), and visual-only apps (Flutter, Electron).
Quick Start
# One-time setup
powershell -File scripts/auto-setup.ps1
# Start services
powershell -File scripts/start-all.ps1
# Send commands
node scripts/send-command.cjs '{"action":"explore"}'
# Stop services
powershell -File scripts/stop-all.ps1Architecture
AI Agent → JSON Commands → MainAgentService (.NET 9 + FlaUI)
├── Layer 1: FlaUI (UIA3)
└── Layer 2: Python Bridge (:5001)
├── pywinauto (Win32)
├── OmniParser (Vision)
└── WinRT OCRFallback Chain: FlaUI → pywinauto → Vision (OmniParser + OCR)
Commands Reference
Discovery
| Command | Description |
|---|---|
explore | List visible windows |
explore_window | List UI elements in window |
find_element | Find element with coordinates |
get_window_info | Window details |
wait_for_element | Wait for element (with timeout) |
element_exists | Boolean check (~30 tokens) |
get_interactive_elements | Clickable/typeable only |
get_window_summary | Structured overview |
fuzzy_find_element | Fuzzy text matching |
Smart Commands (Auto-Fallback)
| Command | Description |
|---|---|
smart_click | Click with FlaUI→Vision fallback |
smart_type | Type with FlaUI→Vision fallback |
vision_click | Vision-only click by OCR text |
Agent Vision (v4.3+)
| Command | Description |
|---|---|
vision_screenshot | Base64 JPEG for AI vision |
vision_screenshot_region | Cropped region screenshot |
vision_config | Get/set vision mode |
vision_analyze_smart | Smart analysis with mode |
vision_screenshot_cache_stats | Cache statistics |
vision_screenshot_cache_clear | Clear cache |
vision_stream | WebSocket URL for streaming |
Win32 Low-Level (v5.0)
| Command | Description |
|---|---|
scroll_sendinput | Low-level scroll (bypasses UIA) |
type_sendinput | Unicode + emoji via SendInput |
set_always_on_top | Pin/unpin window on top |
flash_window | Flash taskbar/caption |
set_window_opacity | Window transparency (0-255) |
fast_screenshot | Fast GDI BitBlt capture |
clipboard_image | Image to/from clipboard |
Window Management
| Command | Description |
|---|---|
focus_window | Bring to foreground |
close_window | Close gracefully |
move_window | Move to X,Y |
resize_window | Resize to W×H |
minimize_window / maximize_window / restore_window | State control |
move_to_monitor | Move to specific display |
wait_for_window | Wait for window to appear |
Mouse
| Command | Description |
|---|---|
click / double_click / right_click | Element clicks |
click_at | Click at X,Y coordinates |
mouse_move | Move cursor |
drag_and_drop | Drag between positions |
scroll | Mouse wheel |
mouse_path | Move along waypoints |
mouse_bezier / draw_bezier | Bezier curve movement/drawing |
draw | Hold + move + release |
mouse_down / mouse_up | Button hold/release |
click_relative | Click relative to anchor |
Keyboard
| Command | Description |
|---|---|
type | Type into focused element |
type_here | Type at cursor position |
hotkey | Key combinations (Ctrl+C, etc.) |
key_press | Single key |
key_down / key_up | Key hold/release |
Clipboard & OCR
| Command | Description |
|---|---|
get_clipboard / set_clipboard | Text clipboard |
clipboard_image | Image clipboard (v5.0) |
read_text | Read from element or OCR |
ocr_region | OCR on screen region |
Other
| Command | Description |
|---|---|
screenshot / screenshot_monitor | Screen capture |
list_monitors | Enumerate displays |
launch_app / list_processes / kill_process | Process control |
file_dialog | Automate Open/Save dialogs |
batch | Execute multiple commands |
health | System diagnostics |
Example Commands
# Window discovery and focus
node scripts/send-command.cjs '{"action":"explore"}'
node scripts/send-command.cjs '{"action":"focus_window", "selector":"Notepad"}'
node scripts/send-command.cjs '{"action":"explore_window", "selector":"Notepad", "max_depth":3}'
# Launch and interact
node scripts/send-command.cjs '{"action":"launch_app", "path":"notepad.exe", "wait_for_window":"Notepad"}'
node scripts/send-command.cjs '{"action":"type", "selector":"Notepad", "text":"Hello World"}'
node scripts/send-command.cjs '{"action":"hotkey", "keys":"ctrl+s"}'
# Smart commands with fallback
node scripts/send-command.cjs '{"action":"smart_click", "selector":"Submit", "window":"MyApp"}'
node scripts/send-command.cjs '{"action":"smart_click", "search_text":"Submit", "agent_fallback":true}'
# Win32 low-level (v5.0)
node scripts/send-command.cjs '{"action":"type_sendinput", "text":"Hello 🎉"}'
node scripts/send-command.cjs '{"action":"set_always_on_top", "selector":"Notepad", "enable":true}'
node scripts/send-command.cjs '{"action":"set_window_opacity", "selector":"Paint", "alpha":180}'
# Vision screenshots for AI
node scripts/send-command.cjs '{"action":"vision_screenshot"}'
node scripts/send-command.cjs '{"action":"vision_screenshot_region", "x":100, "y":100, "width":600, "height":400}'
# Batch execution
node scripts/send-command.cjs '{"action":"batch", "commands":[{"action":"launch_app","path":"notepad.exe"},{"action":"wait_for_window","selector":"Notepad"},{"action":"type","selector":"Notepad","text":"Hello!"}]}'Testing
node scripts/test-commands.cjs # Run all tests
node scripts/test-commands.cjs --list # List tests
node scripts/test-commands.cjs --category mouse # Run categoryPython Bridge API (Port 5001)
| Category | Endpoints |
|---|---|
| Core | /health, /explore, /click, /type, /hotkey, /focus_window |
| Vision | /vision/detect, /vision/ocr, /vision/find_text, /vision/screenshot |
| Context | /context/stats, /context/clear, /context/invalidate |
| Cache | /vision/screenshot_cache/stats, /vision/screenshot_cache/clear |
Troubleshooting
| Problem | Solution |
|---|---|
| Build fails (locked) | scripts/stop-all.ps1 |
| Python bridge offline | Check port 5001, run start-all.ps1 |
| Window not found | Use partial match: "Note" → "Notepad - Untitled" |
| Element click fails | focus_window first, then explore_window |
Version History
| Version | Highlights |
|---|---|
| 5.0.0 | Win32 Low-Level Commands (7 new: scroll_sendinput, type_sendinput, set_always_on_top, flash_window, set_window_opacity, fast_screenshot, clipboard_image) |
| 4.3 | Agent Vision Commands, Screenshot Caching, Removed Java/SikuliX |
| 4.0 | Vision Layer (OmniParser + WinRT OCR), Token-efficient commands, Smart fallback |
| 3.3 | Advanced mouse (bezier, paths), Key hold/release, OCR region |
| 3.2 | Multi-monitor, File dialog automation, Batch commands |
---
Full API details: See SKILL.md for complete command schemas and parameters.
# Auto-Setup Script for Windows Automation Skill
# Downloads and configures EVERYTHING automatically.
$ErrorActionPreference = "Stop"
$ProgressPreference = "SilentlyContinue"
Write-Host ">>> Iniciando Auto-Setup Windows Automation Skill (system Python) <<<" -ForegroundColor Cyan
# 0. Definir rutas
$scriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
$skillRoot = Resolve-Path (Join-Path $scriptDir "..") # skill root
$javaLibDir = Join-Path $skillRoot "src\bridge_java\lib"
$modelsDir = Join-Path $skillRoot "src\bridge_python\models"
$sikuliUrl = "https://launchpad.net/sikuli/sikulix/2.0.5/+download/sikulixapi-2.0.5.jar"
$omniparserUrl = "https://huggingface.co/microsoft/OmniParser/resolve/main/icon_detect/model.onnx"
# 1. Verificar Python (system installed)
Write-Host "[1/4] Verificando Python instalado en el sistema..." -ForegroundColor Yellow
$PythonExe = $null
if (Get-Command py -ErrorAction SilentlyContinue) {
$PythonExe = "py"
} elseif (Get-Command python -ErrorAction SilentlyContinue) {
$PythonExe = "python"
}
if (-not $PythonExe) {
Write-Host "[ERROR] Python no encontrado. Instala Python 3.11+ y asegfarate de tener 'py' o 'python' en PATH." -ForegroundColor Red
throw "Python missing"
}
Write-Host "[OK] Python detectado: $PythonExe" -ForegroundColor Green
# 2. Crear venv local (recomendado)
Write-Host "[2/4] Creando entorno virtual (.venv)..." -ForegroundColor Yellow
$venvDir = Join-Path $skillRoot ".venv"
if (-not (Test-Path $venvDir)) {
if ($PythonExe -eq "py") {
& py -3 -m venv $venvDir
} else {
& python -m venv $venvDir
}
} else {
Write-Host "[SKIP] .venv ya existe." -ForegroundColor Gray
}
$VenvPython = Join-Path $venvDir "Scripts\python.exe"
if (-not (Test-Path $VenvPython)) {
throw "No se pudo crear .venv (faltfa $VenvPython)"
}
# 3. Instalar dependencias Python
Write-Host "[3/4] Instalando dependencias Python (Flask, pywinauto, etc.)..." -ForegroundColor Yellow
try {
& $VenvPython -m pip install --upgrade pip --quiet
& $VenvPython -m pip install flask pywinauto comtypes onnxruntime numpy Pillow --quiet
Write-Host "[OK] Dependencias Python instaladas." -ForegroundColor Green
} catch {
Write-Host "[WARN] No se pudieron instalar dependencias Python: $_" -ForegroundColor Yellow
}
# 4. Instalar Vision Layer Dependencies (OmniParser is optional)
Write-Host "[4/4] Vision Layer: dependencias listas (modelo OmniParser opcional)." -ForegroundColor Yellow
# 3. Descargar OmniParser Model
if (-not (Test-Path $modelsDir)) {
New-Item -ItemType Directory -Force -Path $modelsDir | Out-Null
}
$omniparserPath = Join-Path $modelsDir "omniparser-icon_detect.onnx"
if (-not (Test-Path $omniparserPath)) {
Write-Host "[3/5] Descargando OmniParser model (~6MB)..." -ForegroundColor Yellow
try {
$webClient = New-Object System.Net.WebClient
$webClient.DownloadFile($omniparserUrl, $omniparserPath)
$fileSize = (Get-Item $omniparserPath).Length / 1MB
Write-Host "[OK] OmniParser descargado ($([math]::Round($fileSize, 2)) MB)." -ForegroundColor Green
} catch {
Write-Host "[WARN] Falló descarga de OmniParser. La Vision Layer estará deshabilitada." -ForegroundColor Yellow
Write-Host " Descarga manual: $omniparserUrl" -ForegroundColor Gray
}
} else {
Write-Host "[SKIP] OmniParser ya existe." -ForegroundColor Gray
}
# 4. Configurar SikuliX
if (-not (Test-Path $javaLibDir)) {
New-Item -ItemType Directory -Force -Path $javaLibDir | Out-Null
}
$sikuliJar = Join-Path $javaLibDir "sikulixapi.jar"
if (-not (Test-Path $sikuliJar)) {
Write-Host "[4/5] Descargando SikuliX API JAR..." -ForegroundColor Yellow
# SikuliX launchpad redirects can be tricky, using direct link logic or fallback
# Using a known reliable direct link structure or user prompt if fails.
# Trying direct download:
try {
Invoke-WebRequest -Uri $sikuliUrl -OutFile $sikuliJar
Write-Host "[OK] SikuliX descargado." -ForegroundColor Green
} catch {
Write-Host "[ERROR] Falló descarga automática de SikuliX. Por favor descarga $sikuliUrl manualmente a $javaLibDir" -ForegroundColor Red
}
} else {
Write-Host "[SKIP] SikuliX ya existe." -ForegroundColor Gray
}
# 5. Compilar .NET Project
Write-Host "[5/5] Compilando MainAgentService (.NET)..." -ForegroundColor Yellow
$projPath = Join-Path $skillRoot "src\src\MainAgentService\MainAgentService.csproj"
dotnet build $projPath
if ($LASTEXITCODE -eq 0) {
Write-Host "[OK] Compilación exitosa." -ForegroundColor Green
} else {
Write-Host "[ERROR] Falló la compilación." -ForegroundColor Red
}
Write-Host "`n>>> Instalación v4.0 COMPLETADA <<<" -ForegroundColor Cyan
Write-Host "Componentes instalados/configurados:" -ForegroundColor White
Write-Host " - Python del sistema + .venv" -ForegroundColor Gray
Write-Host " - Dependencias Python (Flask, pywinauto, ONNX Runtime, etc.)" -ForegroundColor Gray
Write-Host " - OmniParser ONNX model (~6MB, opcional)" -ForegroundColor Gray
Write-Host " - SikuliX API JAR (si aplica)" -ForegroundColor Gray
Write-Host " - MainAgentService (.NET 9)" -ForegroundColor Gray
Write-Host "`nAhora ejecuta: .\scripts\start-all.ps1" -ForegroundColor White
# Download OmniParser Model Script
# Downloads the OmniParser icon detection ONNX model for the Vision Layer
$ErrorActionPreference = "Stop"
$ProgressPreference = "SilentlyContinue"
Write-Host ">>> OmniParser Model Download <<<" -ForegroundColor Cyan
# Define paths
$scriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
$skillRoot = Resolve-Path (Join-Path $scriptDir "..")
$modelsDir = Join-Path $skillRoot "src\bridge_python\models"
$modelPath = Join-Path $modelsDir "omniparser-icon_detect.onnx"
# Model URL (Microsoft OmniParser from HuggingFace)
$modelUrl = "https://huggingface.co/microsoft/OmniParser/resolve/main/icon_detect/model.onnx"
# Create models directory if needed
if (-not (Test-Path $modelsDir)) {
Write-Host "[INFO] Creating models directory..." -ForegroundColor Yellow
New-Item -ItemType Directory -Force -Path $modelsDir | Out-Null
}
# Download model
if (-not (Test-Path $modelPath)) {
Write-Host "[1/1] Downloading OmniParser model (~6MB)..." -ForegroundColor Yellow
Write-Host " URL: $modelUrl" -ForegroundColor Gray
Write-Host " Destination: $modelPath" -ForegroundColor Gray
try {
# Use Invoke-WebRequest with longer timeout for HuggingFace
$webClient = New-Object System.Net.WebClient
$webClient.DownloadFile($modelUrl, $modelPath)
# Verify file size (should be ~6MB)
$fileSize = (Get-Item $modelPath).Length / 1MB
if ($fileSize -lt 1) {
throw "Downloaded file is too small ($fileSize MB). Expected ~6MB."
}
Write-Host "[OK] OmniParser model downloaded successfully ($([math]::Round($fileSize, 2)) MB)" -ForegroundColor Green
} catch {
Write-Host "[ERROR] Failed to download OmniParser model: $_" -ForegroundColor Red
Write-Host "[INFO] Please download manually from:" -ForegroundColor Yellow
Write-Host " $modelUrl" -ForegroundColor White
Write-Host " Save to: $modelPath" -ForegroundColor White
exit 1
}
} else {
$fileSize = (Get-Item $modelPath).Length / 1MB
Write-Host "[SKIP] OmniParser model already exists ($([math]::Round($fileSize, 2)) MB)" -ForegroundColor Gray
}
Write-Host "`n>>> OmniParser Download Complete <<<" -ForegroundColor Cyan
#!/usr/bin/env node
/**
* get-command-help.cjs - Lazy-load command documentation
*
* Query command schemas, examples, and tips from command-schemas.json
* without loading the entire documentation into context.
*
* Usage:
* node scripts/get-command-help.cjs <command> # Full details for a command
* node scripts/get-command-help.cjs --list # List all commands
* node scripts/get-command-help.cjs --category <name> # Commands in a category
* node scripts/get-command-help.cjs --schema <command> # Just the JSON schema
* node scripts/get-command-help.cjs --example <command> # Just the example
* node scripts/get-command-help.cjs --errors # List all error codes
* node scripts/get-command-help.cjs --categories # List all categories
* node scripts/get-command-help.cjs --search <term> # Search commands by keyword
*/
const fs = require('fs');
const path = require('path');
// Load schemas
const schemasPath = path.join(__dirname, '..', 'docs', 'command-schemas.json');
let schemas;
try {
schemas = JSON.parse(fs.readFileSync(schemasPath, 'utf8'));
} catch (e) {
console.error(`Error loading schemas: ${e.message}`);
console.error(`Expected file at: ${schemasPath}`);
process.exit(1);
}
// Parse arguments
const args = process.argv.slice(2);
if (args.length === 0) {
printUsage();
process.exit(0);
}
// Handle flags
const flag = args[0];
switch (flag) {
case '--help':
case '-h':
printUsage();
break;
case '--list':
case '-l':
listAllCommands();
break;
case '--categories':
case '-c':
listCategories();
break;
case '--category':
if (!args[1]) {
console.error('Error: --category requires a category name');
console.error('Use --categories to see available categories');
process.exit(1);
}
showCategory(args[1]);
break;
case '--schema':
case '-s':
if (!args[1]) {
console.error('Error: --schema requires a command name');
process.exit(1);
}
showSchema(args[1]);
break;
case '--example':
case '-e':
if (!args[1]) {
console.error('Error: --example requires a command name');
process.exit(1);
}
showExample(args[1]);
break;
case '--errors':
listErrors();
break;
case '--search':
if (!args[1]) {
console.error('Error: --search requires a search term');
process.exit(1);
}
searchCommands(args[1]);
break;
default:
// Assume it's a command name
if (flag.startsWith('-')) {
console.error(`Unknown flag: ${flag}`);
printUsage();
process.exit(1);
}
showCommandDetails(flag);
}
// === Functions ===
function printUsage() {
console.log(`
Windows Desktop Automation - Command Help (v${schemas.version})
${'='.repeat(50)}
Usage:
node get-command-help.cjs <command> Full details for a command
node get-command-help.cjs --list List all commands
node get-command-help.cjs --categories List all categories
node get-command-help.cjs --category <name> Commands in a category
node get-command-help.cjs --schema <command> Just the JSON schema
node get-command-help.cjs --example <command> Just the example
node get-command-help.cjs --errors List all error codes
node get-command-help.cjs --search <term> Search commands by keyword
Examples:
node get-command-help.cjs click
node get-command-help.cjs --category mouse
node get-command-help.cjs --schema explore_window
node get-command-help.cjs --search window
`);
}
function listAllCommands() {
console.log(`\nAll Commands (${Object.keys(schemas.commands).length} total):`);
console.log('='.repeat(60));
// Group by category
for (const [catId, cat] of Object.entries(schemas.categories)) {
console.log(`\n${cat.name}:`);
for (const cmd of cat.commands) {
const cmdData = schemas.commands[cmd];
const desc = cmdData ? cmdData.description.substring(0, 50) : 'N/A';
console.log(` ${cmd.padEnd(20)} ${desc}...`);
}
}
}
function listCategories() {
console.log('\nAvailable Categories:');
console.log('='.repeat(40));
for (const [catId, cat] of Object.entries(schemas.categories)) {
console.log(`\n${catId.padEnd(18)} ${cat.name}`);
console.log(`${''.padEnd(18)} ${cat.description}`);
console.log(`${''.padEnd(18)} Commands: ${cat.commands.length}`);
}
}
function showCategory(categoryName) {
const catId = categoryName.toLowerCase().replace(/\s+/g, '_');
const cat = schemas.categories[catId];
if (!cat) {
console.error(`Category not found: ${categoryName}`);
console.error('\nAvailable categories:');
for (const id of Object.keys(schemas.categories)) {
console.error(` - ${id}`);
}
process.exit(1);
}
console.log(`\n${cat.name} Commands`);
console.log('='.repeat(50));
console.log(`${cat.description}\n`);
for (const cmdName of cat.commands) {
const cmd = schemas.commands[cmdName];
if (cmd) {
console.log(`${cmdName}`);
console.log(` ${cmd.description}`);
console.log(` Required: ${cmd.required.join(', ') || 'none'}`);
console.log(` Optional: ${cmd.optional.join(', ') || 'none'}`);
console.log();
}
}
}
function showSchema(commandName) {
const cmd = schemas.commands[commandName];
if (!cmd) {
console.error(`Command not found: ${commandName}`);
suggestSimilar(commandName);
process.exit(1);
}
console.log(`\nSchema for: ${commandName}`);
console.log('='.repeat(40));
console.log(JSON.stringify(cmd.schema, null, 2));
console.log(`\nRequired: ${cmd.required.join(', ')}`);
console.log(`Optional: ${cmd.optional.join(', ') || 'none'}`);
}
function showExample(commandName) {
const cmd = schemas.commands[commandName];
if (!cmd) {
console.error(`Command not found: ${commandName}`);
suggestSimilar(commandName);
process.exit(1);
}
console.log(`\nExample for: ${commandName}`);
console.log('='.repeat(40));
console.log(JSON.stringify(cmd.example, null, 2));
}
function showCommandDetails(commandName) {
const cmd = schemas.commands[commandName];
if (!cmd) {
console.error(`Command not found: ${commandName}`);
suggestSimilar(commandName);
process.exit(1);
}
console.log(`
${'='.repeat(60)}
Command: ${commandName}
Category: ${cmd.category}
${'='.repeat(60)}
${cmd.description}
REQUIRED PARAMETERS:
${formatParameters(cmd.schema, cmd.required)}
OPTIONAL PARAMETERS:
${cmd.optional.length > 0 ? formatParameters(cmd.schema, cmd.optional) : ' (none)'}
EXAMPLE:
${JSON.stringify(cmd.example, null, 2)}
RETURNS:
${JSON.stringify(cmd.returns, null, 2)}
TIPS:
${cmd.tips ? cmd.tips.map(t => ` - ${t}`).join('\n') : ' (none)'}
`);
}
function formatParameters(schema, params) {
if (!params || params.length === 0) return ' (none)';
return params.map(p => {
const paramSchema = schema[p];
if (!paramSchema) return ` - ${p}`;
const type = paramSchema.type || 'any';
const desc = paramSchema.description || '';
const def = paramSchema.default !== undefined ? ` (default: ${paramSchema.default})` : '';
const constVal = paramSchema.const ? ` = "${paramSchema.const}"` : '';
return ` - ${p} (${type})${constVal}: ${desc}${def}`;
}).join('\n');
}
function listErrors() {
console.log('\nError Codes:');
console.log('='.repeat(60));
for (const [code, info] of Object.entries(schemas.error_codes)) {
console.log(`\n${code}`);
console.log(` Meaning: ${info.meaning}`);
console.log(` Recovery: ${info.recovery}`);
}
}
function searchCommands(term) {
const searchTerm = term.toLowerCase();
const matches = [];
for (const [name, cmd] of Object.entries(schemas.commands)) {
const searchableText = `${name} ${cmd.description} ${cmd.category}`.toLowerCase();
if (searchableText.includes(searchTerm)) {
matches.push({ name, cmd });
}
}
if (matches.length === 0) {
console.log(`\nNo commands found matching: "${term}"`);
return;
}
console.log(`\nCommands matching "${term}" (${matches.length} found):`);
console.log('='.repeat(50));
for (const { name, cmd } of matches) {
console.log(`\n${name} [${cmd.category}]`);
console.log(` ${cmd.description}`);
}
}
function suggestSimilar(commandName) {
const allCommands = Object.keys(schemas.commands);
const similar = allCommands.filter(cmd =>
cmd.includes(commandName) ||
commandName.includes(cmd) ||
levenshteinDistance(cmd, commandName) <= 3
);
if (similar.length > 0) {
console.error('\nDid you mean:');
for (const s of similar.slice(0, 5)) {
console.error(` - ${s}`);
}
}
}
function levenshteinDistance(a, b) {
const matrix = Array(b.length + 1).fill(null).map(() => Array(a.length + 1).fill(null));
for (let i = 0; i <= a.length; i++) matrix[0][i] = i;
for (let j = 0; j <= b.length; j++) matrix[j][0] = j;
for (let j = 1; j <= b.length; j++) {
for (let i = 1; i <= a.length; i++) {
const cost = a[i - 1] === b[j - 1] ? 0 : 1;
matrix[j][i] = Math.min(
matrix[j][i - 1] + 1,
matrix[j - 1][i] + 1,
matrix[j - 1][i - 1] + cost
);
}
}
return matrix[b.length][a.length];
}
// windows-desktop-automation helper
// Usage:
// node scripts/send-command.cjs '{"action":"explore"}'
// echo '{"action":"explore"}' | node scripts/send-command.cjs
//
// This script communicates with MainAgentService HTTP server and sends one JSON command.
// If server is not running, it starts it automatically.
const http = require('http');
const { spawn } = require('child_process');
const path = require('path');
// Resolve agent path relative to this script (using Release build)
const AGENT_PATH = path.resolve(__dirname, '..', 'src', 'src', 'MainAgentService', 'bin', 'Release', 'net9.0-windows', 'MainAgentService.exe');
function readStdin() {
return new Promise((resolve) => {
let data = '';
process.stdin.setEncoding('utf8');
process.stdin.on('data', (chunk) => (data += chunk));
process.stdin.on('end', () => resolve(data.trim()));
if (process.stdin.isTTY) resolve(null);
});
}
// Function to check if service is running
function isServiceRunning() {
return new Promise((resolve) => {
const req = http.get('http://localhost:5000/health', (res) => {
resolve(res.statusCode === 200);
});
req.on('error', () => resolve(false));
req.setTimeout(1000, () => {
req.destroy();
resolve(false);
});
});
}
// Function to start service if not running
function startService() {
return new Promise((resolve, reject) => {
console.error('Starting MainAgentService...');
const serviceProcess = spawn(AGENT_PATH, ['--http-server'], {
detached: true,
stdio: 'ignore',
windowsHide: true
});
serviceProcess.unref();
// Wait for service to start
let attempts = 0;
const checkInterval = setInterval(async () => {
if (await isServiceRunning()) {
clearInterval(checkInterval);
resolve();
} else if (attempts++ > 10) {
clearInterval(checkInterval);
reject(new Error('Failed to start service'));
}
}, 500);
});
}
(async () => {
const arg = process.argv.slice(2).join(' ').trim();
const stdinData = await readStdin();
const payload = arg || stdinData;
if (!payload) {
console.error('No JSON payload provided');
process.exit(1);
}
try {
// Ensure service is running
if (!(await isServiceRunning())) {
await startService();
}
// Send command via HTTP
console.error('Sending command...');
const postData = payload;
const options = {
hostname: 'localhost',
port: 5000,
path: '/command',
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Content-Length': Buffer.byteLength(postData)
}
};
const response = await new Promise((resolve, reject) => {
const req = http.request(options, (res) => {
let body = '';
res.on('data', (chunk) => body += chunk);
res.on('end', () => resolve(body));
});
req.on('error', reject);
req.write(postData);
req.end();
});
process.stdout.write(response);
process.exit(0);
} catch (err) {
console.error('Error:', err.message);
process.exit(1);
}
})();
param(
[string]$PythonDir = ""
)
$ErrorActionPreference = "Stop"
Write-Host "[windows-desktop-automation] Starting bridges and agent..." -ForegroundColor Cyan
# Roots
$root = Split-Path -Parent (Split-Path -Parent $MyInvocation.MyCommand.Path)
$agentProj = Join-Path $root "src\src\MainAgentService\MainAgentService.csproj"
$pyBridge = Join-Path $root "src\bridge_python\bridge.py"
# Resolve Python from system install (preferred)
function Resolve-Python {
# Prefer explicit -PythonDir if user passes one
param([string]$Dir)
if ($Dir -and (Test-Path (Join-Path $Dir "python.exe"))) {
return (Join-Path $Dir "python.exe")
}
# Prefer Windows 'py' launcher if available (stable + versionable)
if (Get-Command py -ErrorAction SilentlyContinue) {
return "py"
}
return "python"
}
$PythonExe = Resolve-Python -Dir $PythonDir
Write-Host "Using Python: $PythonExe" -ForegroundColor Yellow
# If we're using the Windows launcher, target Python 3 explicitly
$PythonArgsPrefix = @()
if ($PythonExe -eq "py") { $PythonArgsPrefix = @("-3") }
# Start Python bridge (includes pywinauto, vision, WinRT OCR, context manager)
Write-Host "[python] launching bridge on :5001 (pywinauto + vision + WinRT OCR)" -ForegroundColor Yellow
$pyArgs = $PythonArgsPrefix + @("`"$pyBridge`"")
$py = Start-Process -FilePath $PythonExe -ArgumentList $pyArgs -WorkingDirectory (Split-Path $pyBridge) -PassThru -WindowStyle Hidden
# Start MainAgentService
Write-Host "[dotnet] launching MainAgentService (FlaUI)" -ForegroundColor Yellow
$agent = Start-Process -FilePath "dotnet" -ArgumentList "run --project `"$agentProj`"" -WorkingDirectory (Split-Path $agentProj) -PassThru -WindowStyle Hidden
Write-Host "All processes started." -ForegroundColor Green
Write-Host "PIDs => python: $($py.Id) | dotnet: $($agent.Id)" -ForegroundColor Green
Write-Host "Use stop-all.ps1 to terminate them."
Write-Host "Stopping windows-desktop-automation processes..." -ForegroundColor Cyan
$procNames = @("python", "python.exe", "dotnet", "dotnet.exe")
foreach ($p in Get-Process | Where-Object { $procNames -contains $_.Name }) {
try {
if ($p.Path -like "*python*" -or $p.Path -like "*dotnet*") {
Write-Host "Killing $($p.Name) PID=$($p.Id)" -ForegroundColor Yellow
$p.Kill()
}
} catch {}
}
Write-Host "Done." -ForegroundColor Green
/**
* Windows Desktop Automation - Test Suite
*
* Automated testing for all commands (v3.3)
*
* Usage:
* node scripts/test-commands.cjs # Run all tests
* node scripts/test-commands.cjs --category discovery # Run category
* node scripts/test-commands.cjs --test explore # Run single test
* node scripts/test-commands.cjs --list # List all tests
*
* Prerequisites:
* - MainAgentService must be built
* - Run from the skill root directory
*/
const { spawn } = require('child_process');
const path = require('path');
const fs = require('fs');
// ==================== CONFIGURATION ====================
const CONFIG = {
agentPath: path.join(__dirname, '../src/src/MainAgentService/bin/Debug/net9.0-windows/MainAgentService.exe'),
timeout: 15000,
startupDelay: 1500,
testApp: 'notepad.exe',
testWindowTitle: 'Notepad',
screenshotDir: path.join(__dirname, '../test-output'),
};
// ==================== COLORS ====================
const colors = {
reset: '\x1b[0m',
bright: '\x1b[1m',
red: '\x1b[31m',
green: '\x1b[32m',
yellow: '\x1b[33m',
blue: '\x1b[34m',
cyan: '\x1b[36m',
gray: '\x1b[90m',
};
const log = {
info: (msg) => console.log(`${colors.blue}[INFO]${colors.reset} ${msg}`),
success: (msg) => console.log(`${colors.green}[PASS]${colors.reset} ${msg}`),
fail: (msg) => console.log(`${colors.red}[FAIL]${colors.reset} ${msg}`),
warn: (msg) => console.log(`${colors.yellow}[WARN]${colors.reset} ${msg}`),
header: (msg) => console.log(`\n${colors.cyan}${colors.bright}=== ${msg} ===${colors.reset}\n`),
detail: (msg) => console.log(`${colors.gray} ${msg}${colors.reset}`),
};
// ==================== TEST RUNNER ====================
class TestRunner {
constructor() {
this.results = { passed: 0, failed: 0, skipped: 0, tests: [] };
this.agentProcess = null;
this.testAppPid = null;
}
/**
* Sends a command to the agent and returns the parsed response
*/
async sendCommand(command, options = {}) {
return new Promise((resolve, reject) => {
const timeout = options.timeout || CONFIG.timeout;
const startupDelay = options.startupDelay || CONFIG.startupDelay;
const child = spawn(CONFIG.agentPath, [], {
stdio: ['pipe', 'pipe', 'pipe'],
windowsHide: true,
});
let stdout = '';
let stderr = '';
let timer = null;
child.stdout.on('data', (data) => { stdout += data.toString(); });
child.stderr.on('data', (data) => { stderr += data.toString(); });
timer = setTimeout(() => {
child.kill();
reject(new Error(`Timeout after ${timeout}ms`));
}, timeout);
setTimeout(() => {
child.stdin.write(JSON.stringify(command) + '\n');
child.stdin.write('{"action":"exit"}\n');
child.stdin.end();
}, startupDelay);
child.on('close', (code) => {
clearTimeout(timer);
try {
// Find JSON in output - look for command response (skip warnings)
const lines = stdout.trim().split('\n');
let response = null;
let lastJson = null;
for (const line of lines) {
if (line.startsWith('{')) {
try {
const parsed = JSON.parse(line);
lastJson = parsed;
// Skip bridge warnings - look for actual command response
if (parsed.status !== 'warn') {
response = parsed;
// Don't break - prefer later responses (command result comes after warnings)
}
} catch (e) { /* not valid JSON, continue */ }
}
}
// If no non-warn response, use the last JSON we found
if (!response && lastJson) {
response = lastJson;
}
if (response) {
resolve(response);
} else {
reject(new Error(`No valid JSON in response: ${stdout.substring(0, 200)}`));
}
} catch (e) {
reject(new Error(`Parse error: ${e.message}`));
}
});
child.on('error', (err) => {
clearTimeout(timer);
reject(err);
});
});
}
/**
* Runs a single test case
*/
async runTest(test) {
const startTime = Date.now();
try {
// Run setup if defined
if (test.setup) {
await test.setup(this);
}
// Send the command
const response = await this.sendCommand(test.command, test.options);
// Validate the response
let passed = true;
let failReason = '';
if (test.validate) {
const result = test.validate(response);
if (result !== true) {
passed = false;
failReason = typeof result === 'string' ? result : 'Validation failed';
}
} else {
// Default validation: check for success status
if (response.status !== 'success') {
passed = false;
failReason = response.message || response.code || 'Non-success status';
}
}
// Run teardown if defined
if (test.teardown) {
await test.teardown(this);
}
const duration = Date.now() - startTime;
if (passed) {
this.results.passed++;
log.success(`${test.name} (${duration}ms)`);
if (test.showResponse) {
log.detail(JSON.stringify(response, null, 2).split('\n').slice(0, 5).join('\n'));
}
} else {
this.results.failed++;
log.fail(`${test.name} - ${failReason}`);
log.detail(`Response: ${JSON.stringify(response).substring(0, 200)}`);
}
this.results.tests.push({ name: test.name, passed, duration, failReason });
return passed;
} catch (error) {
this.results.failed++;
log.fail(`${test.name} - ${error.message}`);
this.results.tests.push({ name: test.name, passed: false, failReason: error.message });
return false;
}
}
/**
* Runs all tests in a category
*/
async runCategory(category, tests) {
log.header(category);
for (const test of tests) {
await this.runTest(test);
}
}
/**
* Prints the final summary
*/
printSummary() {
log.header('TEST SUMMARY');
console.log(` ${colors.green}Passed:${colors.reset} ${this.results.passed}`);
console.log(` ${colors.red}Failed:${colors.reset} ${this.results.failed}`);
console.log(` ${colors.yellow}Skipped:${colors.reset} ${this.results.skipped}`);
console.log(` ${colors.blue}Total:${colors.reset} ${this.results.passed + this.results.failed + this.results.skipped}`);
if (this.results.failed > 0) {
console.log(`\n${colors.red}Failed Tests:${colors.reset}`);
this.results.tests
.filter(t => !t.passed)
.forEach(t => console.log(` - ${t.name}: ${t.failReason}`));
}
return this.results.failed === 0;
}
/**
* Helper to launch test app (Notepad)
*/
async launchTestApp() {
try {
const response = await this.sendCommand({
action: 'launch_app',
path: CONFIG.testApp,
wait_for_window: CONFIG.testWindowTitle
});
if (response.status === 'success' && response.pid) {
this.testAppPid = response.pid;
}
// Give window time to fully render
await this.sleep(500);
return response;
} catch (e) {
log.warn(`Failed to launch test app: ${e.message}`);
return null;
}
}
/**
* Helper to close test app
*/
async closeTestApp() {
try {
// Try to close gracefully first
await this.sendCommand({ action: 'close_window', selector: CONFIG.testWindowTitle });
await this.sleep(200);
// Handle "Save?" dialog if it appears
await this.sendCommand({ action: 'key_press', key: 'n' });
} catch (e) {
// If close fails, try kill
if (this.testAppPid) {
await this.sendCommand({ action: 'kill_process', pid: this.testAppPid, force: true });
}
}
this.testAppPid = null;
}
sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
}
// ==================== TEST DEFINITIONS ====================
const TESTS = {
// -------------------- DISCOVERY --------------------
discovery: [
{
name: 'explore - list windows',
command: { action: 'explore' },
validate: (r) => r.status === 'success' && Array.isArray(r.data) && r.data.length >= 0,
showResponse: false,
},
{
name: 'explore_window - explore test window',
command: { action: 'explore_window', selector: 'Notepad', max_depth: 2 },
setup: async (runner) => { await runner.launchTestApp(); },
validate: (r) => r.status === 'success' && (Array.isArray(r.elements) || Array.isArray(r.data)),
teardown: async (runner) => { await runner.closeTestApp(); },
},
{
name: 'find_element - find button in Notepad',
command: { action: 'find_element', selector: 'Edit', window: 'Notepad' },
setup: async (runner) => { await runner.launchTestApp(); },
validate: (r) => r.status === 'success' || (r.status === 'error' && r.code === 'ELEMENT_NOT_FOUND'),
teardown: async (runner) => { await runner.closeTestApp(); },
},
{
name: 'get_window_info - get window details',
command: { action: 'get_window_info', selector: 'Notepad' },
setup: async (runner) => { await runner.launchTestApp(); },
validate: (r) => r.status === 'success' && r.data && typeof r.data.title === 'string',
teardown: async (runner) => { await runner.closeTestApp(); },
},
],
// -------------------- WINDOW MANAGEMENT --------------------
windowManagement: [
{
name: 'focus_window - bring window to front',
command: { action: 'focus_window', selector: 'Notepad' },
setup: async (runner) => { await runner.launchTestApp(); },
validate: (r) => r.status === 'success',
teardown: async (runner) => { await runner.closeTestApp(); },
},
{
name: 'wait_for_window - wait for window to appear',
command: { action: 'launch_app', path: 'notepad.exe', wait_for_window: 'Notepad' },
validate: (r) => r.status === 'success' && r.pid > 0,
teardown: async (runner) => { await runner.closeTestApp(); },
},
{
name: 'move_window - move to coordinates',
command: { action: 'move_window', selector: 'Notepad', x: 100, y: 100 },
setup: async (runner) => { await runner.launchTestApp(); },
validate: (r) => r.status === 'success',
teardown: async (runner) => { await runner.closeTestApp(); },
},
{
name: 'resize_window - resize window',
command: { action: 'resize_window', selector: 'Notepad', width: 800, height: 600 },
setup: async (runner) => { await runner.launchTestApp(); },
validate: (r) => r.status === 'success',
teardown: async (runner) => { await runner.closeTestApp(); },
},
{
name: 'minimize_window - minimize window',
command: { action: 'minimize_window', selector: 'Notepad' },
setup: async (runner) => { await runner.launchTestApp(); },
validate: (r) => r.status === 'success',
teardown: async (runner) => { await runner.closeTestApp(); },
},
{
name: 'maximize_window - maximize window',
command: { action: 'maximize_window', selector: 'Notepad' },
setup: async (runner) => { await runner.launchTestApp(); },
validate: (r) => r.status === 'success',
teardown: async (runner) => { await runner.closeTestApp(); },
},
{
name: 'restore_window - restore window',
command: { action: 'restore_window', selector: 'Notepad' },
setup: async (runner) => {
await runner.launchTestApp();
await runner.sendCommand({ action: 'maximize_window', selector: 'Notepad' });
await runner.sleep(200);
},
validate: (r) => r.status === 'success',
teardown: async (runner) => { await runner.closeTestApp(); },
},
{
name: 'close_window - close window',
command: { action: 'close_window', selector: 'Notepad' },
setup: async (runner) => { await runner.launchTestApp(); },
validate: (r) => r.status === 'success',
},
],
// -------------------- APP/PROCESS CONTROL --------------------
appProcess: [
{
name: 'launch_app - launch notepad',
command: { action: 'launch_app', path: 'notepad.exe', wait_for_window: 'Notepad' },
validate: (r) => r.status === 'success' && r.pid > 0,
teardown: async (runner) => { await runner.closeTestApp(); },
},
{
name: 'list_processes - list all',
command: { action: 'list_processes' },
validate: (r) => r.status === 'success' && Array.isArray(r.data) && r.data.length > 0,
},
{
name: 'list_processes - filter by name',
command: { action: 'list_processes', filter: 'notepad' },
setup: async (runner) => { await runner.launchTestApp(); },
validate: (r) => r.status === 'success' && Array.isArray(r.data),
teardown: async (runner) => { await runner.closeTestApp(); },
},
{
name: 'kill_process - kill by name',
command: { action: 'kill_process', name: 'notepad' },
setup: async (runner) => { await runner.launchTestApp(); },
validate: (r) => r.status === 'success',
},
],
// -------------------- MOUSE ACTIONS --------------------
mouse: [
{
name: 'click_at - click at coordinates',
command: { action: 'click_at', x: 500, y: 300 },
validate: (r) => r.status === 'success',
},
{
name: 'click_at - right click',
command: { action: 'click_at', x: 500, y: 300, button: 'right' },
validate: (r) => r.status === 'success',
},
{
name: 'mouse_move - move to coordinates',
command: { action: 'mouse_move', x: 400, y: 400 },
validate: (r) => r.status === 'success',
},
{
name: 'scroll - scroll down',
command: { action: 'scroll', direction: 'down', amount: 100 },
validate: (r) => r.status === 'success',
},
{
name: 'scroll - scroll up',
command: { action: 'scroll', direction: 'up', amount: 100 },
validate: (r) => r.status === 'success',
},
{
name: 'drag_and_drop - drag between coordinates',
command: { action: 'drag_and_drop', from_x: 200, from_y: 200, to_x: 400, to_y: 400 },
validate: (r) => r.status === 'success',
},
],
// -------------------- KEYBOARD ACTIONS --------------------
keyboard: [
{
name: 'hotkey - ctrl+a',
command: { action: 'hotkey', keys: 'ctrl+a' },
setup: async (runner) => {
await runner.launchTestApp();
await runner.sendCommand({ action: 'focus_window', selector: 'Notepad' });
},
validate: (r) => r.status === 'success',
teardown: async (runner) => { await runner.closeTestApp(); },
},
{
name: 'key_press - press escape',
command: { action: 'key_press', key: 'escape' },
validate: (r) => r.status === 'success',
},
{
name: 'key_press - press enter',
command: { action: 'key_press', key: 'enter' },
validate: (r) => r.status === 'success',
},
{
name: 'type - type text',
command: { action: 'type', selector: 'Notepad', text: 'Hello Test!' },
setup: async (runner) => {
await runner.launchTestApp();
await runner.sendCommand({ action: 'focus_window', selector: 'Notepad' });
},
validate: (r) => r.status === 'success' || r.code === 'ELEMENT_NOT_FOUND',
teardown: async (runner) => { await runner.closeTestApp(); },
},
],
// -------------------- CLIPBOARD --------------------
clipboard: [
{
name: 'set_clipboard - write text',
command: { action: 'set_clipboard', text: 'Test clipboard content' },
validate: (r) => r.status === 'success',
},
{
name: 'get_clipboard - read text',
command: { action: 'get_clipboard' },
setup: async (runner) => {
await runner.sendCommand({ action: 'set_clipboard', text: 'Verify clipboard' });
},
validate: (r) => r.status === 'success' && typeof r.text === 'string',
},
],
// -------------------- TEXT/OCR & SCREENSHOT --------------------
textAndScreenshot: [
{
name: 'read_text - read from element',
command: { action: 'read_text', selector: 'Notepad' },
setup: async (runner) => { await runner.launchTestApp(); },
validate: (r) => r.status === 'success' || r.status === 'error', // May fail if OCR not available
teardown: async (runner) => { await runner.closeTestApp(); },
},
{
name: 'screenshot - capture screen',
command: { action: 'screenshot', filename: 'test-screenshot.png' },
validate: (r) => r.status === 'success',
},
],
// -------------------- ERROR HANDLING --------------------
errorHandling: [
{
name: 'missing_param - returns MISSING_PARAM error',
command: { action: 'focus_window' }, // Missing selector
validate: (r) => r.status === 'error' && r.code === 'MISSING_PARAM',
},
{
name: 'window_not_found - returns WINDOW_NOT_FOUND error',
command: { action: 'focus_window', selector: 'NonExistentWindow12345' },
validate: (r) => r.status === 'error' && r.code === 'WINDOW_NOT_FOUND',
},
{
name: 'unknown_action - returns UNKNOWN_ACTION error',
command: { action: 'nonexistent_action' },
validate: (r) => r.status === 'error' && r.code === 'UNKNOWN_ACTION',
},
],
// -------------------- SYSTEM (v3.2) --------------------
system: [
{
name: 'health - system health check',
command: { action: 'health' },
validate: (r) => r.status === 'success' && r.version && r.bridges,
},
],
// -------------------- MULTI-MONITOR (v3.2) --------------------
multiMonitor: [
{
name: 'list_monitors - enumerate displays',
command: { action: 'list_monitors' },
validate: (r) => r.status === 'success' && Array.isArray(r.data) && r.data.length >= 1,
},
{
name: 'screenshot_monitor - capture primary monitor',
command: { action: 'screenshot_monitor', monitor: 0, filename: 'test-monitor-0.png' },
validate: (r) => r.status === 'success' && r.filename,
},
{
name: 'screenshot_monitor - invalid monitor returns error',
command: { action: 'screenshot_monitor', monitor: 99 },
validate: (r) => r.status === 'error' && r.code === 'INVALID_MONITOR',
},
{
name: 'move_to_monitor - move window to primary monitor',
command: { action: 'move_to_monitor', selector: 'Notepad', monitor: 0, position: 'center' },
setup: async (runner) => { await runner.launchTestApp(); },
validate: (r) => r.status === 'success',
teardown: async (runner) => { await runner.closeTestApp(); },
},
],
// -------------------- FILE DIALOG (v3.2) --------------------
fileDialog: [
{
name: 'file_dialog - detect when no dialog open',
command: { action: 'file_dialog', dialog_action: 'detect' },
validate: (r) => r.status === 'error' && r.code === 'DIALOG_NOT_FOUND',
},
{
name: 'file_dialog - invalid action',
command: { action: 'file_dialog', dialog_action: 'invalid_action' },
validate: (r) => r.status === 'error' && r.code === 'INVALID_DIALOG_ACTION',
},
],
// -------------------- WAIT FOR ELEMENT (v3.2) --------------------
waitForElement: [
{
name: 'wait_for_element - find existing element',
command: { action: 'wait_for_element', selector: 'Edit', window: 'Notepad', timeout: 3000 },
setup: async (runner) => { await runner.launchTestApp(); },
validate: (r) => r.status === 'success' || (r.status === 'error' && r.code === 'TIMEOUT'),
teardown: async (runner) => { await runner.closeTestApp(); },
},
{
name: 'wait_for_element - timeout for non-existent element',
command: { action: 'wait_for_element', selector: 'NonExistentElement99', timeout: 1000, poll_interval: 200 },
validate: (r) => r.status === 'error' && r.code === 'TIMEOUT',
},
],
// -------------------- BATCH (v3.2) --------------------
batch: [
{
name: 'batch - execute simple command sequence',
command: {
action: 'batch',
commands: [
{ action: 'set_clipboard', text: 'batch_test_1' },
{ action: 'get_clipboard' }
],
stop_on_error: true
},
validate: (r) => r.status === 'success' && r.succeeded === 2 && r.failed === 0,
},
{
name: 'batch - stop on error',
command: {
action: 'batch',
commands: [
{ action: 'set_clipboard', text: 'before_error' },
{ action: 'focus_window', selector: 'NonExistentWindow12345' },
{ action: 'set_clipboard', text: 'after_error' }
],
stop_on_error: true
},
validate: (r) => r.status === 'error' && r.succeeded === 1 && r.failed === 1,
},
{
name: 'batch - continue on error',
command: {
action: 'batch',
commands: [
{ action: 'set_clipboard', text: 'before_error' },
{ action: 'focus_window', selector: 'NonExistentWindow12345' },
{ action: 'set_clipboard', text: 'after_error' }
],
stop_on_error: false
},
validate: (r) => r.succeeded === 2 && r.failed === 1,
},
],
// -------------------- ADVANCED MOUSE (v3.3) --------------------
advancedMouse: [
{
name: 'mouse_path - move along waypoints',
command: { action: 'mouse_path', points: [[100, 100], [200, 150], [300, 100]], duration: 300 },
validate: (r) => r.status === 'success',
},
{
name: 'mouse_bezier - move along bezier curve',
command: { action: 'mouse_bezier', start: [100, 100], control1: [150, 50], control2: [250, 50], end: [300, 100], steps: 30, duration: 300 },
validate: (r) => r.status === 'success',
},
{
name: 'draw - draw path',
command: { action: 'draw', points: [[100, 100], [150, 120], [200, 100]], button: 'left', duration: 300 },
validate: (r) => r.status === 'success',
},
{
name: 'mouse_down - press left button',
command: { action: 'mouse_down', button: 'left', x: 400, y: 400 },
validate: (r) => r.status === 'success',
},
{
name: 'mouse_up - release left button',
command: { action: 'mouse_up', button: 'left' },
validate: (r) => r.status === 'success',
},
{
name: 'click_relative - click relative to element',
command: { action: 'click_relative', selector: 'Notepad', anchor: 'center', offset_x: 10, offset_y: 10 },
setup: async (runner) => { await runner.launchTestApp(); },
validate: (r) => r.status === 'success',
teardown: async (runner) => { await runner.closeTestApp(); },
},
],
// -------------------- ADVANCED KEYBOARD (v3.3) --------------------
advancedKeyboard: [
{
name: 'key_down - press shift',
command: { action: 'key_down', key: 'shift' },
validate: (r) => r.status === 'success',
},
{
name: 'key_up - release shift',
command: { action: 'key_up', key: 'shift' },
validate: (r) => r.status === 'success',
},
{
name: 'type_here - type at cursor',
command: { action: 'type_here', text: 'Test typing' },
setup: async (runner) => {
await runner.launchTestApp();
await runner.sendCommand({ action: 'focus_window', selector: 'Notepad' });
},
validate: (r) => r.status === 'success',
teardown: async (runner) => { await runner.closeTestApp(); },
},
{
name: 'type_at_cursor - alias for type_here',
command: { action: 'type_at_cursor', text: 'More text' },
setup: async (runner) => {
await runner.launchTestApp();
await runner.sendCommand({ action: 'focus_window', selector: 'Notepad' });
},
validate: (r) => r.status === 'success',
teardown: async (runner) => { await runner.closeTestApp(); },
},
],
// -------------------- WAIT FOR STATE (v3.3) --------------------
waitForState: [
{
name: 'wait_for_state - element exists',
command: { action: 'wait_for_state', selector: 'Edit', state: 'exists', window: 'Notepad', timeout: 3000 },
setup: async (runner) => { await runner.launchTestApp(); },
validate: (r) => r.status === 'success' || (r.status === 'error' && r.code === 'TIMEOUT'),
teardown: async (runner) => { await runner.closeTestApp(); },
},
{
name: 'wait_for_state - element not_exists (timeout expected)',
command: { action: 'wait_for_state', selector: 'NonExistent99', state: 'not_exists', timeout: 1000 },
validate: (r) => r.status === 'success',
},
{
name: 'wait_for_state - invalid state returns error',
command: { action: 'wait_for_state', selector: 'Button', state: 'invalid_state', timeout: 1000 },
validate: (r) => r.status === 'error',
},
],
// -------------------- OCR REGION (v3.3) --------------------
ocrRegion: [
{
name: 'ocr_region - read screen region',
command: { action: 'ocr_region', x: 100, y: 100, width: 200, height: 50 },
// OCR may fail if Java bridge is not running, which is acceptable
validate: (r) => r.status === 'success' || (r.status === 'error' && (r.code === 'OCR_FAILED' || r.code === 'OCR_REGION_FAILED')),
},
{
name: 'ocr_region - missing parameters',
command: { action: 'ocr_region', x: 100, y: 100 }, // missing width/height
validate: (r) => r.status === 'error' && r.code === 'MISSING_PARAM',
},
],
};
// ==================== MAIN EXECUTION ====================
async function main() {
const args = process.argv.slice(2);
const runner = new TestRunner();
// Parse arguments
let categoryFilter = null;
let testFilter = null;
let listOnly = false;
for (let i = 0; i < args.length; i++) {
if (args[i] === '--category' && args[i + 1]) {
categoryFilter = args[i + 1].toLowerCase();
i++;
} else if (args[i] === '--test' && args[i + 1]) {
testFilter = args[i + 1].toLowerCase();
i++;
} else if (args[i] === '--list') {
listOnly = true;
} else if (args[i] === '--help') {
console.log(`
Windows Desktop Automation Test Suite
Usage:
node test-commands.cjs Run all tests
node test-commands.cjs --category discovery Run category
node test-commands.cjs --test explore Run single test
node test-commands.cjs --list List all tests
node test-commands.cjs --help Show this help
Categories: ${Object.keys(TESTS).join(', ')}
`);
process.exit(0);
}
}
// List tests
if (listOnly) {
console.log('\nAvailable Tests:\n');
for (const [category, tests] of Object.entries(TESTS)) {
console.log(`${colors.cyan}${category}${colors.reset}`);
tests.forEach(t => console.log(` - ${t.name}`));
}
process.exit(0);
}
// Verify agent exists
if (!fs.existsSync(CONFIG.agentPath)) {
log.fail(`Agent not found at: ${CONFIG.agentPath}`);
log.info('Run: dotnet build (in MainAgentService directory)');
process.exit(1);
}
// Create test output directory
if (!fs.existsSync(CONFIG.screenshotDir)) {
fs.mkdirSync(CONFIG.screenshotDir, { recursive: true });
}
log.header('WINDOWS DESKTOP AUTOMATION TEST SUITE');
log.info(`Agent: ${CONFIG.agentPath}`);
log.info(`Timeout: ${CONFIG.timeout}ms`);
// Run tests
for (const [category, tests] of Object.entries(TESTS)) {
if (categoryFilter && category.toLowerCase() !== categoryFilter) continue;
let filteredTests = tests;
if (testFilter) {
filteredTests = tests.filter(t => t.name.toLowerCase().includes(testFilter));
}
if (filteredTests.length > 0) {
await runner.runCategory(category, filteredTests);
}
}
// Print summary
const success = runner.printSummary();
process.exit(success ? 0 : 1);
}
main().catch(err => {
console.error('Fatal error:', err);
process.exit(1);
});
/**
* Windows Desktop Automation - Vision Stream Test
*
* Tests WebSocket-based real-time screenshot streaming.
*
* Usage:
* node scripts/test-stream.cjs [options]
*
* Options:
* --fps <number> Frames per second (default: 5)
* --quality <number> JPEG quality 1-100 (default: 70)
* --duration <number> Test duration in seconds (default: 5)
* --save Save received frames to disk
* --help Show this help
*/
const path = require('path');
const fs = require('fs');
const http = require('http');
// ==================== CONFIGURATION ====================
const DEFAULTS = {
wsUrl: 'ws://localhost:5001/vision/stream',
fps: 5,
quality: 70,
duration: 5,
saveFrames: false,
outputDir: path.join(__dirname, '../test-output/stream'),
};
// ==================== LOGGING ====================
const c = { reset: '\x1b[0m', bold: '\x1b[1m', red: '\x1b[31m', green: '\x1b[32m', yellow: '\x1b[33m', blue: '\x1b[34m', cyan: '\x1b[36m', gray: '\x1b[90m' };
const log = {
info: (msg) => console.log(`${c.blue}[INFO]${c.reset} ${msg}`),
ok: (msg) => console.log(`${c.green}[OK]${c.reset} ${msg}`),
fail: (msg) => console.log(`${c.red}[FAIL]${c.reset} ${msg}`),
warn: (msg) => console.log(`${c.yellow}[WARN]${c.reset} ${msg}`),
header: (msg) => console.log(`\n${c.cyan}${c.bold}=== ${msg} ===${c.reset}\n`),
detail: (msg) => console.log(`${c.gray} ${msg}${c.reset}`),
};
// ==================== HELPERS ====================
function calcStats(frameTimes) {
if (!frameTimes.length) return { avg: 0, min: 0, max: 0 };
const sum = frameTimes.reduce((a, b) => a + b, 0);
return {
avg: sum / frameTimes.length,
min: Math.min(...frameTimes),
max: Math.max(...frameTimes),
};
}
function getWebSocket() {
if (typeof globalThis.WebSocket !== 'undefined') return globalThis.WebSocket;
try { return require('ws'); } catch { return null; }
}
function checkBridge() {
return new Promise((resolve) => {
const req = http.get('http://localhost:5001/health', (res) => {
let data = '';
res.on('data', chunk => data += chunk);
res.on('end', () => {
try { resolve(['ok', 'success'].includes(JSON.parse(data).status)); }
catch { resolve(false); }
});
});
req.on('error', () => resolve(false));
req.setTimeout(3000, () => { req.destroy(); resolve(false); });
});
}
function parseArgs(args) {
const config = { ...DEFAULTS };
for (let i = 0; i < args.length; i++) {
const arg = args[i];
if (arg === '--fps') config.fps = parseInt(args[++i]) || DEFAULTS.fps;
else if (arg === '--quality') config.quality = parseInt(args[++i]) || DEFAULTS.quality;
else if (arg === '--duration') config.duration = parseInt(args[++i]) || DEFAULTS.duration;
else if (arg === '--save') config.saveFrames = true;
else if (arg === '--help') {
console.log(`
Vision Stream Test - Tests WebSocket screenshot streaming
Usage: node test-stream.cjs [options]
Options:
--fps <n> Frames per second (default: ${DEFAULTS.fps})
--quality <n> JPEG quality 1-100 (default: ${DEFAULTS.quality})
--duration <n> Duration in seconds (default: ${DEFAULTS.duration})
--save Save frames to disk
--help Show this help
`);
process.exit(0);
}
}
return config;
}
// ==================== TEST ====================
async function testVisionStream(config) {
log.header('VISION STREAM TEST');
log.info(`URL: ${config.wsUrl} | FPS: ${config.fps} | Quality: ${config.quality} | Duration: ${config.duration}s`);
const WebSocket = getWebSocket();
if (!WebSocket) {
log.fail('WebSocket not available. Install: npm install ws');
return { success: false };
}
if (config.saveFrames) {
fs.mkdirSync(config.outputDir, { recursive: true });
log.info(`Saving frames to: ${config.outputDir}`);
}
return new Promise((resolve) => {
let frameCount = 0, totalBytes = 0, startTime = null, lastFrameTime = null;
const frameTimes = [];
const url = `${config.wsUrl}?fps=${config.fps}&quality=${config.quality}`;
log.info('Connecting...');
const ws = new WebSocket(url);
ws.on('open', () => {
log.ok('Connected');
startTime = lastFrameTime = Date.now();
setTimeout(() => { log.info('Closing...'); ws.close(); }, config.duration * 1000);
});
ws.on('message', (data) => {
const now = Date.now();
frameCount++;
const frameData = data.toString();
totalBytes += frameData.length;
if (lastFrameTime) frameTimes.push(now - lastFrameTime);
lastFrameTime = now;
// Progress every 5 frames
if (frameCount % 5 === 1) {
const elapsed = (now - startTime) / 1000;
log.detail(`Frame ${frameCount}: ${(frameData.length / 1024).toFixed(1)} KB | FPS: ${(frameCount / elapsed).toFixed(1)}`);
}
// Save if requested
if (config.saveFrames) {
const filename = path.join(config.outputDir, `frame_${String(frameCount).padStart(4, '0')}.jpg`);
try { fs.writeFileSync(filename, Buffer.from(frameData, 'base64')); }
catch (e) { log.warn(`Save failed: ${e.message}`); }
}
});
ws.on('error', (err) => {
log.fail(`Error: ${err.message}`);
resolve({ success: false, error: err.message });
});
ws.on('close', () => {
const totalTime = (Date.now() - (startTime || Date.now())) / 1000;
const avgFps = frameCount / totalTime;
const avgFrameSize = frameCount ? totalBytes / frameCount : 0;
const timing = calcStats(frameTimes);
log.header('RESULTS');
console.log(` Frames: ${frameCount} | Time: ${totalTime.toFixed(2)}s | FPS: ${avgFps.toFixed(2)} (target: ${config.fps})`);
console.log(` Data: ${(totalBytes / 1024 / 1024).toFixed(2)} MB | Avg frame: ${(avgFrameSize / 1024).toFixed(1)} KB`);
console.log(` Timing: avg=${timing.avg.toFixed(1)}ms, min=${timing.min}ms, max=${timing.max}ms`);
const success = frameCount > 0 && avgFps > config.fps * 0.5;
(success ? log.ok : log.fail)(`Test ${success ? 'PASSED' : 'FAILED'}`);
resolve({ success, frameCount, totalTime, avgFps, avgFrameSize, timing });
});
});
}
// ==================== MAIN ====================
async function main() {
const config = parseArgs(process.argv.slice(2));
log.info('Checking bridge...');
if (!await checkBridge()) {
log.fail('Bridge not responding at localhost:5001');
log.info('Start with: scripts/start-all.ps1');
process.exit(1);
}
log.ok('Bridge running');
const result = await testVisionStream(config);
process.exit(result.success ? 0 : 1);
}
main().catch(err => { log.fail(`Fatal: ${err.message}`); process.exit(1); });
"""
Context Manager for Windows Desktop Automation v4.0
Provides intelligent caching and response compression to reduce token consumption
for AI agents. Acts as a proxy layer that:
1. Caches exploration results with window-aware invalidation
2. Compresses verbose responses to essential information
3. Implements progressive disclosure (minimal context first)
4. Tracks window state changes for cache invalidation
Author: AI Engineering Team
Version: 4.0
"""
import hashlib
import json
import time
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Tuple
from collections import OrderedDict
@dataclass
class CacheEntry:
"""A cached response with metadata for invalidation."""
key: str
value: Any
window_hash: str
created_at: float
ttl_seconds: float
access_count: int = 0
last_accessed: float = field(default_factory=time.time)
def is_expired(self) -> bool:
return time.time() - self.created_at > self.ttl_seconds
def touch(self):
self.access_count += 1
self.last_accessed = time.time()
class SemanticCache:
"""
Cache with TTL and window-aware invalidation.
Features:
- TTL-based expiration
- Window hash invalidation (cache invalidates when UI changes)
- LRU eviction for memory management
- Semantic key normalization
"""
DEFAULT_TTL = 30.0 # 30 seconds default TTL
MAX_ENTRIES = 100 # Maximum cache entries
# TTL by command type
TTL_BY_COMMAND = {
"explore": 60.0, # Windows list changes slowly
"explore_window": 30.0, # Window contents change moderately
"get_window_summary": 30.0,
"get_interactive_elements": 20.0,
"find_element": 10.0, # Element positions may change
"element_exists": 5.0, # Quick checks, low TTL
"get_element_brief": 10.0,
}
def __init__(self, max_entries: int = MAX_ENTRIES):
self.max_entries = max_entries
self._cache: OrderedDict[str, CacheEntry] = OrderedDict()
self._window_hashes: Dict[str, str] = {} # window_selector -> hash
self._stats = {"hits": 0, "misses": 0, "evictions": 0, "invalidations": 0}
def _normalize_key(self, action: str, params: Dict[str, Any]) -> str:
"""Create a normalized cache key from action and params."""
# Sort params for consistent hashing
sorted_params = json.dumps(params, sort_keys=True)
key_str = f"{action}:{sorted_params}"
return hashlib.md5(key_str.encode()).hexdigest()
def _get_window_hash(self, window_selector: Optional[str]) -> str:
"""Get or compute window state hash."""
if not window_selector:
return "desktop"
return self._window_hashes.get(window_selector, "unknown")
def get(self, action: str, params: Dict[str, Any]) -> Tuple[bool, Any]:
"""
Try to get a cached response.
Returns:
Tuple of (hit: bool, value: Any or None)
"""
key = self._normalize_key(action, params)
if key not in self._cache:
self._stats["misses"] += 1
return (False, None)
entry = self._cache[key]
# Check expiration
if entry.is_expired():
del self._cache[key]
self._stats["misses"] += 1
return (False, None)
# Check window hash validity
window_selector = params.get("window") or params.get("selector")
current_hash = self._get_window_hash(window_selector)
if entry.window_hash != current_hash and entry.window_hash != "desktop":
del self._cache[key]
self._stats["invalidations"] += 1
return (False, None)
# Cache hit
entry.touch()
self._cache.move_to_end(key) # LRU: move to end
self._stats["hits"] += 1
return (True, entry.value)
def set(self, action: str, params: Dict[str, Any], value: Any):
"""Store a response in cache."""
key = self._normalize_key(action, params)
ttl = self.TTL_BY_COMMAND.get(action, self.DEFAULT_TTL)
window_selector = params.get("window") or params.get("selector")
window_hash = self._get_window_hash(window_selector)
# Evict if at capacity
while len(self._cache) >= self.max_entries:
oldest_key = next(iter(self._cache))
del self._cache[oldest_key]
self._stats["evictions"] += 1
self._cache[key] = CacheEntry(
key=key,
value=value,
window_hash=window_hash,
created_at=time.time(),
ttl_seconds=ttl,
)
def update_window_hash(self, window_selector: str, new_hash: str):
"""Update window hash (invalidates related cache entries)."""
old_hash = self._window_hashes.get(window_selector)
if old_hash != new_hash:
self._window_hashes[window_selector] = new_hash
# Invalidate entries for this window
keys_to_remove = []
for key, entry in self._cache.items():
if entry.window_hash == old_hash:
keys_to_remove.append(key)
for key in keys_to_remove:
del self._cache[key]
self._stats["invalidations"] += 1
def invalidate_window(self, window_selector: str):
"""Explicitly invalidate all cache entries for a window."""
window_hash = self._window_hashes.get(window_selector, "unknown")
keys_to_remove = [
k for k, v in self._cache.items() if v.window_hash == window_hash
]
for key in keys_to_remove:
del self._cache[key]
self._stats["invalidations"] += 1
def clear(self):
"""Clear all cache entries."""
self._cache.clear()
self._window_hashes.clear()
def get_stats(self) -> Dict[str, Any]:
"""Get cache statistics."""
total = self._stats["hits"] + self._stats["misses"]
hit_rate = self._stats["hits"] / total if total > 0 else 0.0
return {
**self._stats,
"entries": len(self._cache),
"hit_rate": round(hit_rate, 3),
}
class ResponseCompressor:
"""
Compresses verbose responses to reduce token consumption.
Strategies:
- Remove empty/null fields
- Truncate long text content
- Summarize large element lists
- Extract only essential properties
"""
MAX_TEXT_LENGTH = 100
MAX_ELEMENTS_FULL = 10
MAX_ELEMENTS_SUMMARY = 50
# Essential fields by response type
ESSENTIAL_FIELDS = {
"element": ["name", "type", "id", "x", "y", "w", "h"],
"window": ["title", "bounds", "is_enabled"],
"summary": ["title", "element_count", "key_elements"],
}
@classmethod
def compress(
cls, response: Dict[str, Any], context: str = "default"
) -> Dict[str, Any]:
"""
Compress a response based on context.
Args:
response: The raw response from MainAgentService
context: Hint about what kind of response this is
Returns:
Compressed response
"""
if not isinstance(response, dict):
return response
# Don't compress errors
if response.get("status") == "error":
return response
compressed = {}
for key, value in response.items():
if value is None or value == "" or value == []:
continue # Skip empty values
if isinstance(value, str) and len(value) > cls.MAX_TEXT_LENGTH:
compressed[key] = value[: cls.MAX_TEXT_LENGTH] + "..."
elif isinstance(value, list) and len(value) > cls.MAX_ELEMENTS_FULL:
# Summarize large lists
compressed[key] = cls._compress_element_list(value)
elif isinstance(value, dict):
compressed[key] = cls._compress_dict(value)
else:
compressed[key] = value
return compressed
@classmethod
def _compress_element_list(cls, elements: List[Any]) -> Dict[str, Any]:
"""Compress a list of elements to a summary."""
if not elements:
return {"count": 0, "elements": []}
# Take first N full elements
full_elements = elements[: cls.MAX_ELEMENTS_FULL]
# Compress each element
compressed_elements = [
cls._compress_element(e) if isinstance(e, dict) else e
for e in full_elements
]
return {
"count": len(elements),
"showing": len(compressed_elements),
"elements": compressed_elements,
}
@classmethod
def _compress_element(cls, element: Dict[str, Any]) -> Dict[str, Any]:
"""Extract essential fields from an element."""
essential = {}
for field in cls.ESSENTIAL_FIELDS.get("element", []):
if field in element and element[field]:
value = element[field]
if isinstance(value, str) and len(value) > 50:
value = value[:50] + "..."
essential[field] = value
# Add automation_id if present and different from name
if element.get("automation_id") and element.get("automation_id") != element.get(
"name"
):
essential["id"] = element["automation_id"]
return essential if essential else element
@classmethod
def _compress_dict(cls, d: Dict[str, Any]) -> Dict[str, Any]:
"""Recursively compress a dictionary."""
result = {}
for key, value in d.items():
if value is None or value == "" or value == []:
continue
if isinstance(value, dict):
result[key] = cls._compress_dict(value)
elif isinstance(value, list) and len(value) > 10:
result[key] = cls._compress_element_list(value)
else:
result[key] = value
return result
class ProgressiveDisclosure:
"""
Implements progressive disclosure pattern for AI agents.
Strategy:
1. Return minimal context first (summary/counts)
2. If agent requests more detail, provide fuller response
3. Track what the agent has already seen to avoid repetition
"""
DISCLOSURE_LEVELS = {
"minimal": 0, # Just counts and key elements
"standard": 1, # Interactive elements with essential fields
"detailed": 2, # Full element tree with all properties
}
def __init__(self):
self._agent_context: Dict[str, int] = {} # window -> disclosure level seen
self._failure_count: Dict[str, int] = {} # track failures to auto-escalate
def get_disclosure_level(self, window_selector: str) -> str:
"""Get current disclosure level for a window."""
level = self._agent_context.get(window_selector, 0)
failures = self._failure_count.get(window_selector, 0)
# Auto-escalate after failures
effective_level = min(level + failures, 2)
if effective_level == 0:
return "minimal"
elif effective_level == 1:
return "standard"
else:
return "detailed"
def record_request(
self, window_selector: str, explicit_level: Optional[str] = None
):
"""Record that agent made a request (auto-escalates disclosure)."""
if explicit_level and explicit_level in self.DISCLOSURE_LEVELS:
self._agent_context[window_selector] = self.DISCLOSURE_LEVELS[
explicit_level
]
else:
current = self._agent_context.get(window_selector, 0)
self._agent_context[window_selector] = min(current + 1, 2)
def record_failure(self, window_selector: str):
"""Record that an operation failed (triggers more detail on next request)."""
current = self._failure_count.get(window_selector, 0)
self._failure_count[window_selector] = current + 1
def reset(self, window_selector: Optional[str] = None):
"""Reset disclosure state."""
if window_selector:
self._agent_context.pop(window_selector, None)
self._failure_count.pop(window_selector, None)
else:
self._agent_context.clear()
self._failure_count.clear()
class WindowStateTracker:
"""
Tracks window state changes for cache invalidation.
Uses a lightweight hash of:
- Window bounds
- Element count
- Key element names
"""
def __init__(self):
self._state_hashes: Dict[str, str] = {}
self._last_check: Dict[str, float] = {}
self._check_interval = 5.0 # Seconds between state checks
def compute_hash(self, window_info: Dict[str, Any]) -> str:
"""Compute a state hash from window info."""
# Extract key state indicators
state_parts = [
str(window_info.get("bounds", "")),
str(window_info.get("element_count", 0)),
str(window_info.get("title", "")),
]
# Include key element names if available
key_elements = window_info.get("key_elements", [])
if key_elements:
element_names = [e.get("name", "") for e in key_elements[:5]]
state_parts.extend(element_names)
state_str = "|".join(state_parts)
return hashlib.md5(state_str.encode()).hexdigest()[:12]
def check_changed(self, window_selector: str, new_info: Dict[str, Any]) -> bool:
"""Check if window state has changed since last check."""
new_hash = self.compute_hash(new_info)
old_hash = self._state_hashes.get(window_selector)
if old_hash is None:
self._state_hashes[window_selector] = new_hash
self._last_check[window_selector] = time.time()
return False # First check, not a change
changed = new_hash != old_hash
self._state_hashes[window_selector] = new_hash
self._last_check[window_selector] = time.time()
return changed
def should_recheck(self, window_selector: str) -> bool:
"""Check if enough time has passed to recheck window state."""
last = self._last_check.get(window_selector, 0)
return time.time() - last > self._check_interval
def get_hash(self, window_selector: str) -> Optional[str]:
"""Get current hash for a window."""
return self._state_hashes.get(window_selector)
class ContextManager:
"""
Main context manager that orchestrates caching, compression, and disclosure.
Acts as an intelligent proxy layer between AI agents and MainAgentService.
Usage:
ctx = ContextManager()
# Before calling MainAgentService:
cached, response = ctx.get_cached("explore_window", {"selector": "Calculator"})
if cached:
return response
# After getting response from MainAgentService:
compressed = ctx.process_response("explore_window", params, raw_response)
return compressed
"""
# Commands that should be cached
CACHEABLE_COMMANDS = {
"explore",
"explore_window",
"get_window_summary",
"get_interactive_elements",
"find_element",
"element_exists",
"get_element_brief",
"get_window_info",
}
# Commands that modify state (invalidate cache)
STATE_MODIFYING_COMMANDS = {
"click",
"double_click",
"right_click",
"type",
"hotkey",
"key_press",
"smart_click",
"smart_type",
"vision_click",
"close_window",
"minimize_window",
"maximize_window",
}
def __init__(self):
self.cache = SemanticCache()
self.compressor = ResponseCompressor
self.disclosure = ProgressiveDisclosure()
self.state_tracker = WindowStateTracker()
self._enabled = True
def is_enabled(self) -> bool:
return self._enabled
def enable(self):
self._enabled = True
def disable(self):
self._enabled = False
def get_cached(self, action: str, params: Dict[str, Any]) -> Tuple[bool, Any]:
"""
Try to get a cached response for the command.
Returns:
Tuple of (cache_hit: bool, response: Any or None)
"""
if not self._enabled:
return (False, None)
if action not in self.CACHEABLE_COMMANDS:
return (False, None)
return self.cache.get(action, params)
def process_response(
self,
action: str,
params: Dict[str, Any],
response: Dict[str, Any],
compress: bool = True,
) -> Dict[str, Any]:
"""
Process and optionally cache a response.
This should be called after getting a response from MainAgentService.
Args:
action: The command that was executed
params: The parameters used
response: The raw response from MainAgentService
compress: Whether to compress the response
Returns:
Processed (possibly compressed) response
"""
if not self._enabled:
return response
# Check for state changes
if action in self.STATE_MODIFYING_COMMANDS:
window_selector = params.get("window") or params.get("selector")
if window_selector:
self.cache.invalidate_window(window_selector)
# Cache if appropriate
if action in self.CACHEABLE_COMMANDS and response.get("status") == "success":
self.cache.set(action, params, response)
# Update window state hash if we have summary info
if action in ("get_window_summary", "explore_window"):
window_selector = params.get("selector") or params.get("window")
if window_selector and "data" in response:
new_hash = self.state_tracker.compute_hash(response.get("data", {}))
self.cache.update_window_hash(window_selector, new_hash)
# Track failures for progressive disclosure
if response.get("status") == "error":
window_selector = params.get("window") or params.get("selector")
if window_selector:
self.disclosure.record_failure(window_selector)
# Compress if requested
if compress:
return self.compressor.compress(response, context=action)
return response
def get_stats(self) -> Dict[str, Any]:
"""Get context manager statistics."""
return {"enabled": self._enabled, "cache": self.cache.get_stats()}
def clear(self):
"""Clear all caches and state."""
self.cache.clear()
self.disclosure.reset()
# Singleton instance for use across the bridge
_context_manager: Optional[ContextManager] = None
def get_context_manager() -> ContextManager:
"""Get or create the singleton ContextManager instance."""
global _context_manager
if _context_manager is None:
_context_manager = ContextManager()
return _context_manager
"""
Vision module for Windows Desktop Automation v4.0
Provides AI-powered UI element detection using OmniParser + Windows OCR
"""
from .detector import VisionDetector
from .ocr import WindowsOCR
from .vision_service import VisionService
__all__ = ["VisionDetector", "WindowsOCR", "VisionService"]
__version__ = "4.0.0"
Add-Type -AssemblyName System.Runtime.WindowsRuntime
$null = [Windows.Media.Ocr.OcrEngine, Windows.Foundation, ContentType = WindowsRuntime]
$null = [Windows.Graphics.Imaging.BitmapDecoder, Windows.Foundation, ContentType = WindowsRuntime]
$null = [Windows.Storage.Streams.RandomAccessStream, Windows.Foundation, ContentType = WindowsRuntime]
function Await($WinRtTask, $ResultType) {
$asTaskGeneric = ([System.WindowsRuntimeSystemExtensions].GetMethods() |
Where-Object { $_.Name -eq 'AsTask' -and $_.GetParameters().Count -eq 1 -and
$_.GetParameters()[0].ParameterType.Name -eq 'IAsyncOperation`1' })[0]
$asTask = $asTaskGeneric.MakeGenericMethod($ResultType)
$netTask = $asTask.Invoke($null, @($WinRtTask))
$netTask.Wait(-1) | Out-Null
$netTask.Result
}
$imagePath = $args[0]
$stream = [System.IO.File]::OpenRead($imagePath)
$randomAccessStream = [System.IO.WindowsRuntimeStreamExtensions]::AsRandomAccessStream($stream)
$decoder = Await ([Windows.Graphics.Imaging.BitmapDecoder]::CreateAsync($randomAccessStream)) ([Windows.Graphics.Imaging.BitmapDecoder])
$softwareBitmap = Await ($decoder.GetSoftwareBitmapAsync()) ([Windows.Graphics.Imaging.SoftwareBitmap])
$ocrEngine = [Windows.Media.Ocr.OcrEngine]::TryCreateFromUserProfileLanguages()
if ($ocrEngine -eq $null) {
$ocrEngine = [Windows.Media.Ocr.OcrEngine]::TryCreateFromLanguage("en-US")
}
$ocrResult = Await ($ocrEngine.RecognizeAsync($softwareBitmap)) ([Windows.Media.Ocr.OcrResult])
$results = @()
foreach ($line in $ocrResult.Lines) {
foreach ($word in $line.Words) {
$rect = $word.BoundingRect
$results += @{
text = $word.Text
left = [int]$rect.X
top = [int]$rect.Y
width = [int]$rect.Width
height = [int]$rect.Height
}
}
}
$stream.Close()
ConvertTo-Json -InputObject $results -Compress
"""
Diagnostic script for OmniParser performance analysis
Run this to get detailed timing and raw output statistics
"""
import time
import numpy as np
from PIL import ImageGrab, Image
import sys
import os
# Add parent to path
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from vision.detector import VisionDetector, Detection
def run_diagnostic():
results = {}
print("=" * 60)
print("OmniParser Vision Layer Diagnostic")
print("=" * 60)
# 1. Screenshot capture timing
print("\n[1] Screenshot Capture...")
t0 = time.perf_counter()
screenshot = ImageGrab.grab()
t_screenshot = (time.perf_counter() - t0) * 1000
print(f" Time: {t_screenshot:.1f}ms")
print(f" Size: {screenshot.size[0]}x{screenshot.size[1]}")
print(f" Mode: {screenshot.mode}")
results["screenshot_ms"] = t_screenshot
results["screen_size"] = screenshot.size
# 2. Model loading timing
print("\n[2] Model Loading...")
t0 = time.perf_counter()
detector = VisionDetector(
confidence_threshold=0.01
) # Very low threshold for diagnostic
_ = detector.session # Force load
t_load = (time.perf_counter() - t0) * 1000
print(f" Time: {t_load:.1f}ms")
print(f" Model: {os.path.basename(detector.model_path)}")
results["model_load_ms"] = t_load
# 3. Preprocessing timing
print("\n[3] Image Preprocessing...")
if screenshot.mode != "RGB":
screenshot = screenshot.convert("RGB")
t0 = time.perf_counter()
input_tensor, scale_x, scale_y, pad_x, pad_y = detector._preprocess_image(
screenshot
)
t_preprocess = (time.perf_counter() - t0) * 1000
print(f" Time: {t_preprocess:.1f}ms")
print(f" Input shape: {input_tensor.shape}")
print(f" Scale: {scale_x:.4f}")
print(f" Padding: ({pad_x}, {pad_y})")
print(f" Value range: [{input_tensor.min():.3f}, {input_tensor.max():.3f}]")
results["preprocess_ms"] = t_preprocess
# 4. ONNX Inference timing
print("\n[4] ONNX Inference...")
input_name = detector.session.get_inputs()[0].name
t0 = time.perf_counter()
outputs = detector.session.run(None, {input_name: input_tensor})
t_inference = (time.perf_counter() - t0) * 1000
print(f" Time: {t_inference:.1f}ms")
print(f" Output shape: {outputs[0].shape}")
results["inference_ms"] = t_inference
# 5. Raw output analysis
print("\n[5] Raw Model Output Analysis...")
raw_output = outputs[0]
predictions = raw_output[0].T # [N, 5]
print(f" Total predictions: {len(predictions)}")
# Confidence distribution
confidences = predictions[:, 4]
print(f" Confidence stats:")
print(f" Min: {confidences.min():.4f}")
print(f" Max: {confidences.max():.4f}")
print(f" Mean: {confidences.mean():.4f}")
print(f" Std: {confidences.std():.4f}")
# Count by threshold
thresholds = [0.01, 0.05, 0.1, 0.2, 0.3, 0.5, 0.7]
print(f"\n Detections by threshold:")
for th in thresholds:
count = np.sum(confidences >= th)
print(f" >= {th}: {count}")
results["raw_predictions"] = len(predictions)
results["max_confidence"] = float(confidences.max())
# 6. Postprocessing with different thresholds
print("\n[6] Postprocessing Analysis...")
for th in [0.01, 0.1, 0.3]:
detector.confidence_threshold = th
t0 = time.perf_counter()
detections = detector._postprocess_output(outputs[0], scale_x, pad_x, pad_y)
t_post = (time.perf_counter() - t0) * 1000
print(f" Threshold {th}: {len(detections)} detections ({t_post:.1f}ms)")
if detections:
print(
f" Top 3 confidences: {[round(d.confidence, 3) for d in detections[:3]]}"
)
# 7. Full detection timing (threshold 0.1)
print("\n[7] Full Detection Pipeline (threshold=0.1)...")
detector.confidence_threshold = 0.1
t0 = time.perf_counter()
detections = detector.detect(screenshot)
t_full = (time.perf_counter() - t0) * 1000
print(f" Time: {t_full:.1f}ms")
print(f" Detections: {len(detections)}")
results["full_pipeline_ms"] = t_full
results["detections_th01"] = len(detections)
if detections:
print("\n Detected elements:")
for i, d in enumerate(detections[:10]):
print(
f" [{i}] pos=({d.x}, {d.y}) size={d.width}x{d.height} conf={d.confidence:.3f}"
)
# 8. Summary
print("\n" + "=" * 60)
print("PERFORMANCE SUMMARY")
print("=" * 60)
print(f" Screenshot: {results['screenshot_ms']:6.1f}ms")
print(f" Preprocess: {results['preprocess_ms']:6.1f}ms")
print(f" Inference: {results['inference_ms']:6.1f}ms")
print(f" Full pipeline: {results['full_pipeline_ms']:6.1f}ms")
print(f"\n Raw predictions: {results['raw_predictions']}")
print(f" Max confidence: {results['max_confidence']:.4f}")
print(f" Final detections (th=0.1): {results['detections_th01']}")
return results
if __name__ == "__main__":
run_diagnostic()
"""
Vision Streaming - WebSocket-based real-time screenshot streaming
"""
import time
import logging
from PIL import ImageGrab
from .vision_service import VisionConfig, optimize_screenshot, get_screenshot_cache
class VisionStreamer:
"""Handles real-time screenshot streaming over WebSockets."""
def stream_screenshots(
self, ws, fps=5, quality=None, max_width=None, max_height=None
):
"""
Stream screenshots over WebSocket.
Args:
ws: WebSocket connection
fps: Frames per second (1-30)
quality: JPEG quality override
max_width: Max width override
max_height: Max height override
"""
# Apply defaults and clamp FPS
fps = max(1, min(30, fps))
interval = 1.0 / fps
quality = quality or VisionConfig.jpeg_quality
max_width = max_width or VisionConfig.max_width
max_height = max_height or VisionConfig.max_height
cache = get_screenshot_cache()
logging.info(f"Starting screenshot stream: {fps} FPS, quality={quality}")
try:
while True:
frame_start = time.time()
# Try cache first, capture if miss
optimized = cache.get(
quality=quality, max_width=max_width, max_height=max_height
)
if not optimized:
screenshot = ImageGrab.grab()
optimized = optimize_screenshot(
screenshot,
max_width=max_width,
max_height=max_height,
jpeg_quality=quality,
)
cache.put(
optimized,
quality=quality,
max_width=max_width,
max_height=max_height,
)
ws.send(optimized.data)
# Maintain target FPS
sleep_time = interval - (time.time() - frame_start)
if sleep_time > 0:
time.sleep(sleep_time)
except Exception as e:
logging.info(f"Stream closed: {e}")
_streamer = VisionStreamer()
def get_vision_streamer():
return _streamer
Related skills
FAQ
What apps can it automate?
Windows GUI apps, including Flutter, Electron, and Canvas apps handled by its vision layer.
How many commands does it expose?
98 JSON commands plus 18 HTTP endpoints across discovery, window, mouse, keyboard, OCR, and vision categories.