SuperAPI AI Directory

browser-automation

General & other

View repository

Updated 10-16 09:45

About this skill

Automation Operations (Desktop & Browser Automation)

ACTION REQUIRED (execute immediately after reading)

  1. NOW: Confirm whether the current task falls within the applicable scope of this skill
  2. NOW: Read ../tool-index.md, verify tool availability and actual paths
  3. NEXT: When a tool is missing, call bootstrap; do not guess paths
  4. ACT: Enter the first step of the "workflow" and execute it; do not stop at the confirmation state

Applicable Scope

Use this skill when the task belongs to the following scenarios:

Browser scenarios (Playwright / agent-browser)

  • Open a web page and operate page elements (click, fill forms, submit)
  • Scrape page content or take screenshots
  • Automated login flows
  • Interact with Web pages during penetration testing (submit payloads, trigger XSS)
  • Automated handling of CAPTCHA pages
  • Batch form submission

Desktop application scenarios (OpenReverse)

  • Operate Windows desktop applications (IDA Pro, x64dbg, Wireshark, etc.)
  • Requires vision-driven interaction (CUA mode)
  • Requires structured UI operations (UIA mode)
  • Network traffic observation for desktop applications (built-in mitmproxy)
  • GUI operations for automated reverse engineering tools
  • Black-box testing of desktop software

Division of labor with other tools

Scenario What to use
Operate web pages (inside browser) Playwright / agent-browser
Operate desktop applications (Windows GUI) OpenReverse
Packet capture analysis, HTTP request capture anything-analyzer or OpenReverse network lane
JS breakpoints, Hook, CDP debugging jshookmcp
Locating signature algorithms, reproducing with environment patching js-reverse

Simple judgment:

  • Target is a web page → Playwright
  • Target is a Windows desktop application → OpenReverse
  • Both are needed → use in combination

Part 1: Browser automation (Playwright / agent-browser)

Core workflow

# 1. 打开页面
agent-browser open <url>

# 2. 获取可交互元素(返回 @e1, @e2... 引用)
agent-browser snapshot -i

# 3. 用引用操作元素
agent-browser click @e1
agent-browser fill @e2 "text"

# 4. 完成后关闭
agent-browser close

Command reference

# 导航
agent-browser open <url>
agent-browser close

# 页面快照
agent-browser snapshot        # 完整无障碍树
agent-browser snapshot -i     # 仅可交互元素(推荐)

# 交互操作
agent-browser click @e1
agent-browser fill @e2 "text"
agent-browser type @e2 "text"
agent-browser press Enter
agent-browser scroll down 500

# 获取信息
agent-browser get text @e1
agent-browser get title
agent-browser get url

# 等待
agent-browser wait @e1
agent-browser wait 2000
agent-browser wait --load networkidle

Notes

  • Must execute agent-browser close, otherwise process leakage occurs
  • Take a snapshot before operating; do not guess element references
  • After submitting the form, use wait --load networkidle to wait for page stabilization, etc.

Part 2: Desktop application automation (OpenReverse)

Overview

OpenReverse is a desktop interaction and evidence collection framework for AI Agents, supporting:

  • UIA mode: Windows UI Automation, structured desktop control operations
  • CUA mode: vision-driven interaction (Computer Use Agent), suitable for complex GUIs
  • Network observation: built-in mitmproxy proxy + local capture

Interaction mode selection

Mode Suitable scenarios Underlying
UIA The target application has standard Windows controls (buttons, text boxes, lists) Windows UI Automation API
CUA Target application UI is complex or has non-standard controls (IDA's disassembly view, custom-rendered interfaces) Visual recognition + mouse and keyboard

Network observation mode

Mode Suitable scenarios
Proxy Lane Target application can be configured with a proxy (recommended)
Local Lane Target application cannot go through a proxy and requires local capture

Installation and configuration

# 1. Clone 项目
git clone https://github.com/zhexulong/openreverse.git
cd openreverse

# 2. 安装依赖
npm install

# 3. 接入 Agent 宿主(Claude Code / Codex / Zed)
npm run init:agents -- --target=all /path/to/project

# 4. 安装 CUA runtime(如果需要视觉驱动模式)
npm run install:cua-runtime
npm run doctor:cua-runtime

# 5. 安装网络观察依赖(如果需要抓包)
npm run install:mitmproxy
npm run doctor:network

Common combinations

Requirement Configuration
Operate desktop applications only UIA or CUA, without connecting to the network lane
Operate desktop applications + packet capture UIA/CUA + proxy lane
Operate desktop applications + local capture UIA/CUA + local lane

Reverse engineering scenario examples

场景:自动化操作 IDA Pro 进行批量分析

1. 用 OpenReverse CUA 模式打开 IDA Pro
2. 自动加载目标二进制
3. 等待分析完成
4. 通过 UI 操作导出函数列表
5. 同时用 network lane 观察 IDA 的网络行为(如 Lumina 请求)
场景:自动化操作 x64dbg 调试

1. 用 OpenReverse UIA 模式启动 x64dbg
2. 加载目标程序
3. 设置断点
4. 运行并观察寄存器/内存变化
5. 截图保存证据

On-Demand Bootstrap

Automation capability boundaries

Tool Can be installed automatically Installation method Description
Playwright ✓ npm + npx playwright install Browser automation engine
agent-browser CLI ✓ npm install -g agent-browser Browser operation CLI
Node.js ✓ winget Prerequisite dependencies
OpenReverse ✗ Manual clone + npm install Experimental stage, heavy dependencies
mitmproxy ✗ Manual installation OpenReverse network observation dependency

Bootstrap trigger

  • Browser operation missing Playwright → automatic bootstrap
  • Desktop operation requires OpenReverse → guide the user to install manually (provide complete steps)

OpenReverse manual installation guide

If the AI detects that desktop application automation is needed but OpenReverse is not installed:

⚠️ **需要 OpenReverse 进行桌面应用自动化**

**安装步骤**:
1. `git clone https://github.com/zhexulong/openreverse.git`
2. `cd openreverse && npm install`
3. `npm run init:agents -- --target=all <你的项目路径>`
4. 如需视觉模式:`npm run install:cua-runtime`
5. 如需网络观察:`npm run install:mitmproxy`

**验证**:`npm run doctor:cua-runtime` 和 `npm run doctor:network`

Routing context

Upstream entry: skills/SKILL.md(master control),routing.md Applicable scenarios: Any task that requires automated operation of a browser or desktop application Downstream exit:

  • Captured requests need analysis → anything-analyzer or js-reverse
  • JS debugging/Hook needed → jshookmcp
  • Signature algorithm restoration needed → js-reverse
  • The desktop application is a reverse engineering tool → ida-reverse/

Sibling related modules: js-reverse(after browser operations, JS analysis may be needed),ida-reverse(OpenReverse can automate IDA GUI operations)

Task completion self-check (MUST pass before claiming completion)

  • Did I execute every step in the workflow (rather than just reading it)?
  • Am I based on tool-index and used real tool paths?
  • Did I produce reproducible evidence (commands/scripts/screenshots/reports)?
  • Did I complete and write back the Checklist items required by RULES?