computer-use

Computer Use（macOS）

Tool combination：screencapture + cliclick + osascript + Swift CGEvent (scrolling)

First confirm whether this skill is really needed:

• Operating a web page/browser → use the agent-browser skill, which is more stable and saves more tokens
• Reading/writing files, calling APIs, executing scripts → use built-in tools directly
• Only when the target is alocal GUI app with no available API(native apps such as WeChat, Feishu, Figma) should you continue further

---

Step 1: Run the initialization script (execute once at the start of each task)

bash <skill_dir>/scripts/init.sh

Initialization outputs an independent tool directory. Record the path in the output as CU_DIR; each task uses its own directory, and only that task directory is cleaned up afterward. If dependency or screen geometry reading fails, stop; never guess the scaling factor.

Initialization does three things:

1. Detect the Retina scaling factor and write it to "$CU_DIR"/_cu_scale
2. Compile the Swift CGEvent scrolling tool to "$CU_DIR"/_cu_scroll
3. Install the screenshot helper script "$CU_DIR"/_cu_snap.sh, and print the currently visible process names

Commands available after initialization:

Command	Purpose
"$CU_DIR"/_cu_snap.sh out.png	Capture the full screen (automatically scaled to logical resolution)
"$CU_DIR"/_cu_snap.sh out.png W H x y	After screenshotting, crop to a W×H region (saves tokens)
"$CU_DIR"/_cu_scroll x y amount	Scroll (amount < 0 scrolls down, > 0 scrolls up)
cat "$CU_DIR"/_cu_scale	Read the scaling factor

The temporary directory may be cleaned up by the system; reinitialize when the tools stop working. Before multi-monitor cross-screen operations, verify the coordinate mapping for each screen separately; the current helper script does not prove that arbitrary cross-screen coordinates are correct.

---

Perception-action loop

1. _cu_snap 截图 → Read 读取
   └─ 优先裁剪到目标区域：只截需要看的部分，大幅减少 token 消耗
2. 定位目标
   └─ 优先：AX 元素名（osascript）
   └─ 其次：截图坐标（cu_snap 输出已是逻辑坐标，直接用于 cliclick）
3. 执行操作
4. sleep 0.3~1.5s 等渲染 → 再截图确认
5. 重复直到完成

---

Screenshot

"$CU_DIR"/_cu_snap.sh /tmp/s.png              # 全屏
"$CU_DIR"/_cu_snap.sh /tmp/s.png 900 600 0 300  # 裁剪：宽900 高600 从(x=0,y=300)开始

Uncropped, additionally unscaled main-screen screenshot coordinates can be used directly as logical coordinates. After cropping, the crop origin must be added back:屏幕坐标 = 图内坐标 + (x_offset, y_offset). For example, crop origin (0, 300), target in image (40, 50), the click should be (40, 350). For multiple screens or when the host scales the image again, first verify the display origin and scaling ratio; main-screen coordinates cannot be applied directly.

---

Mouse operations

cliclick c:960,490     # 单击
cliclick dc:960,490    # 双击
cliclick rc:960,490    # 右键
cliclick p             # 打印当前坐标（调试）

⚠️ cliclick dd:x,y is the start of a drag, not a scroll.

---

Scrolling

# 向下滚到底部
for i in $(seq 1 8); do "$CU_DIR"/_cu_scroll 750 400 -20; sleep 0.05; done

# 向上滚
"$CU_DIR"/_cu_scroll 750 400 5

⚠️ The input box of chat apps (Feishu, WeChat, Slack) will seize keyboard focus, and the End / PageDown keys will type into the input box. You must use _cu_scroll, not the keyboard.

---

Entering text

# 英文
osascript -e 'tell application "System Events" to tell process "AppName" to keystroke "hello"'

# 中文（直接 keystroke 会乱码，必须走剪贴板）
echo -n "你好世界" | pbcopy
osascript -e 'tell application "System Events" to tell process "AppName" to keystroke "v" using command down'

---

Key press

cliclick kp:return   kp:esc   kp:tab   kp:delete   kp:arrow-down   kp:page-down

---

Activate app

osascript -e 'tell application "Feishu" to activate'
sleep 0.8

The process name must use the system name; the process list was printed during initialization. Common error-prone pairs:

• Feishu → Feishu(not Lark）
• WeChat → WeChat

⚠️ Do not use set frontmost to true—it will report error -10006. activate is sufficient; if you need to bring it to the front, you can add set bounds of front window to {…}。

---

Click UI elements (more accurate than coordinates)

A more reliable way than estimating coordinates from screenshots: click directly via the Accessibility element name.

# 按名称点击按钮
osascript -e 'tell application "System Events" to tell process "App" to click button "OK" of window 1'

# 先查有哪些可点元素
osascript -e 'tell application "System Events" to tell process "App" to get every UI element of window 1'

---

Wait for the window to be ready

osascript << 'EOF'
tell application "App" to activate
tell application "System Events"
    tell process "App"
        set w to 0
        repeat until (count of windows) > 0 or w > 10
            delay 0.3
            set w to w + 0.3
        end repeat
    end tell
end tell
EOF