web-infra-dev/midscene-skills
7 skills15k installs2k starsGitHub
Install
npx skills add https://github.com/web-infra-dev/midscene-skillsSkills in this repo
1Browser AutomationMidscene is a vision-driven browser automation tool that operates entirely from screenshots, eliminating the need for DOM or accessibility labels. It runs in headless Puppeteer by default and supports CDP and Bridge modes to control existing Chrome browsers with preserved login state. Developers use it to automate multi-step web workflows, scrape data, fill forms, click buttons, and validate UI behavior by giving natural language instructions. Key workflows include connecting to a URL, taking screenshots, executing actions via `act` commands, verifying page state with assertions, and reporting results. The tool persists the browser session across CLI calls, enabling chained interactions without session loss. Vision-based automation from screenshots - no DOM selectors or accessibility labels required Three connection modes: Puppeteer (headless), CDP (user's Chrome with login preserved), Bridge (Chrome Extension) Natural language prompts for complex interactions - handles clicking, typing, scrolling, dragging within single `act` command Assertion framework to verify page state and validate UI behavior post-action Browser session persists across CLI calls with built-in support for re.4.3kinstalls2Android Device AutomationBrowser Automation runs vision-driven web workflows with Midscene.js using screenshots instead of DOM or accessibility labels. Default Puppeteer mode launches a persistent headless Chrome; CDP mode connects to the user's Chrome with login state preserved; Bridge mode uses the Midscene Chrome Extension. Critical rules require synchronous one-command-at-a-time execution, roughly one minute per step, and a final results summary with screenshots and findings. Commands include connect, take_screenshot, act with natural language prompts, assert for QA checks, tap with locate JSON for reference images, disconnect, and close. The workflow connects, screenshots state, executes high-level act tasks, asserts outcomes, then reports. Model API keys must be configured via MIDSCENE_MODEL variables for Gemini, Qwen, or Doubao providers. Best practices batch related form steps into one act, use assert for verification, and prefer tap locate when the user supplies reference images. Deep locate and deep think flags improve hard targeting and multi-step planning.3.1kinstalls3Ios Device AutomationThe ios-device-automation skill runs vision-driven iOS device automation through npx @midscene/ios, controlling iPhones and iPads via WebDriverAgent without DOM or accessibility labels. Critical rules forbid background midscene commands, require one synchronous command at a time, allow about one minute per action for AI inference, and mandate a final user-facing results summary. Prerequisites configure MIDSCENE_MODEL_API_KEY, NAME, BASE_URL, and FAMILY environment variables with examples for Gemini and Qwen providers. The act command supports tap, double-tap, long-press, typing, scrolling, drag, zoom, and system navigation from screenshots of the current screen. Agents act as the brain choosing sequential CLI actions after reading each screenshot output. Typical triggers include iOS app testing, visual verification on iPhone, and end-to-end QA on physical or simulators where visible UI automation is required. Powered by Midscene.js, the skill complements browser automation skills while focusing on native iOS surfaces.2.3kinstalls4Harmonyos Device AutomationThe harmonyos-device-automation skill | It covers browse, navigate, or open web pages. Key workflows include scrape, extract, or collect data from websites. This skill has three modes. Choose based on the user's intent: Mode Selection Guide | Mode | When to use | How it works | |------|------------|-------------| | Puppeteer default | User wants to browse a URL, scrape data, test UI - no need for their own browser | Launches a new headless Chrome, isolated from user's browser | | CDP mode | User says "connect to my Chrome", "control my browser", "CD Developers invoke harmonyos-device-automation when the task matches the triggers and reference files in SKILL.md for grounded, stepwise execution. Reference files and progressive disclosure keep context focused while preserving concrete commands, configuration fields, and validation checks copied from the upstream documentation.1.9kinstalls5Vitest Midscene E2eVitest Midscene E2E integrates Vitest with Midscene's AI-driven UI automation to scaffold and test multi-platform applications (Web via Playwright, Android via ADB, iOS via WDA). Developers describe UI actions and assertions in natural language; the `aiAct` API translates these into executable interactions without fragile selectors. Provides project scaffolding from a boilerplate reference, supports test creation/update/debugging/execution, and includes platform-specific context classes (`WebTest`, `AndroidTest`, `IOSTest`) that expose agent instances and lifecycle hooks. Mandatory rule: pass user intent directly to `aiAct` rather than decomposing into fine-grained APIs.1.7kinstalls6Computer AutomationThe computer-automation skill is designed for vision-driven desktop automation using Midscene. Control your local desktop (macOS, Windows, Linux) or a remote Windows desktop over RDP with natural language commands. Desktop Computer Automation > CRITICAL RULES — VIOLATIONS WILL BREAK THE WORKFLOW: > > 1. Each command must run synchronously so you can read its output (especially screenshots) before deciding the next action. Invoke when the user asks about computer automation or related SKILL.md workflows.1.6kinstalls7Harmonyos Device Automationharmonyos-device-automation is a Claude Code skill in the Automation & Workflows category.91installs