SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsdesktop

desktop

Office & productivity

View repository

Updated 10-15 17:20

About this skill

Desktop — CC Desktop Control

Core concept: haptic feedback

Instead of capturing the full screen and sending it to AI (slow, expensive), use macOS native OCR as "touch". Wherever the mouse moves → OCR there → the Agent knows where its hand is and what it's touching.

CLI tool stack

All operations are performed by calling CLI tools via Bash:

# ===== 眼（感知） =====

# OCR 工具路径（ocr.py 在本 skill 目录下，venv 在 ~/tools/ocr-env/）
SKILL_DIR=".claude/skills/desktop"
OCR="$HOME/tools/ocr-env/bin/python3 $SKILL_DIR/ocr.py"
Full-screen OCR (fast mode, ~240ms, note: Chinese must use precise mode)

$OCR --screen --fast

Full-screen OCR + JSON bounding boxes (precise mode, ~870ms)

$OCR --screen --bbox

OCR near the mouse (haptic, ~250ms, default 300x200)

$OCR --cursor

OCR near the mouse + JSON (with clickable coordinates)

$OCR --cursor --bbox

OCR near the mouse (custom region size)

$OCR --cursor --size 200x100

OCR a specified image

$OCR /path/to/image.png

===== Hand (Control) ===== Get mouse position

cliclick p:.

Move mouse

cliclick m:500,300

Click

cliclick c:500,300

Double-click

cliclick dc:500,300

Right-click

cliclick rc:500,300

Type text

cliclick t:"Hello World"

Press key (Return, Tab, etc.) Note: cliclick kp:return does not work in some apps such as WeChat; prefer osascript

osascript -e 'tell application "System Events" to key code 36' # Return (recommended) cliclick kp:return # Return (alternative) cliclick kp:tab cliclick kp:escape

Keyboard shortcuts

osascript -e 'tell application "System Events" to keystroke "c" using command down' osascript -e 'tell application "System Events" to keystroke "v" using command down' osascript -e 'tell application "System Events" to keystroke "s" using command down' osascript -e 'tell application "System Events" to keystroke "a" using command down'

===== Perception (window/app state) ===== List of currently visible apps

osascript -e 'tell application "System Events" to get name of every process whose visible is true'

Current foreground window information

osascript -e 'tell application "System Events" to get {name, position, size} of every window of (first process whose frontmost is true)'

Switch to a specified app

osascript -e 'tell application "APP_NAME" to activate'

Click a menu

osascript -e 'tell application "System Events" to click menu item "ITEM" of menu "MENU" of menu bar 1 of process "APP"'

Full-screen screenshot (used when passing to Claude vision as a fallback)

screencapture -x /tmp/desktop-screenshot.png

Partial screenshot

screencapture -x -R "x,y,w,h" /tmp/desktop-region.png


## 操作流程

### 标准流程：点击按钮

  1. Full-screen OCR to find the target $OCR --screen --fast → Found "Save" @ (550, 300)

  2. Move mouse + haptic confirmation cliclick m:550,300 $OCR --cursor --size 200x100 → Confirm "Save" is under the mouse ✓

  3. Click cliclick c:550,300

  4. Haptic verification of the result $OCR --cursor → Confirm the operation succeeded


### 快捷流程：已知坐标直接点

cliclick c:550,300


### 文字输入

  1. Click the input box cliclick c:400,200

  2. Clear (select all + delete) osascript -e 'tell application "System Events" to keystroke "a" using command down' cliclick kp:delete

  3. Type text cliclick t:"content to type"


### OCR 找不到时的兜底

  1. Full-screen screenshot screencapture -x /tmp/desktop-screenshot.png

  2. Use the Read tool to read the screenshot (Claude vision) → Claude automatically analyzes the screenshot content

  3. Decide the operation based on Claude's analysis result


## 安全规则

| 级别 | 操作 | 策略 |
|------|------|------|
| 绿色 | 截图、OCR、读剪贴板、列窗口、移动鼠标 | 自动执行 |
| 黄色 | 点击、输入文字、切换窗口 | 执行并告知用户 |
| 红色 | 涉及密码、支付、删除、发消息 | 先问用户 |

**禁区**：
- 不操控 Terminal/iTerm（防 GUI 绕过权限控制）
- 不输入密码
- 不操作支付页面
- 不修改系统安全设置

## OCR 输出格式

### 简洁模式（默认）

cursor: (516,703) ocr: 277ms [0.50] 上线的时候记得把密码改强一点 @ (665,678) [0.50] 己置那个上游的api 呢？ @ (508,744)


### JSON 模式（--bbox）
```json
{
  "cursor": [516, 703],
  "elapsed_ms": 243,
  "items": [
    {
      "text": "保存",
      "confidence": 0.98,
      "bbox": [540, 290, 60, 30],
      "center": [570, 305]
    }
  ]
}

center is the text center coordinate, which can be passed directly to cliclick c: to click.

Performance reference

Operation Time
Mouse move/click < 10ms
Local OCR (300x200) ~250ms
Full-screen OCR (fast) ~240ms
Full-screen OCR (precise) ~870ms
Get window list < 50ms

WeChat control script

There are three WeChat control scripts under this skill directory, encapsulating the complete read/send/verify workflow:

SKILL_DIR=".claude/skills/desktop"

# 读取当前聊天内容（OCR 聊天区域，排除标题栏和输入框）
$SKILL_DIR/wechat-read.sh              # 简洁文字
$SKILL_DIR/wechat-read.sh --bbox       # JSON 包围盒

# 发送文字消息（自动验证：OCR 输入框 count==0 = 成功）
$SKILL_DIR/wechat-send.sh "消息内容"

# 发送文件/图片（剪贴板 POSIX file → 粘贴 → 回车 → OCR 验证）
$SKILL_DIR/wechat-send-file.sh /path/to/file

Prerequisites

  • WeChat is already open and on the target chat window
  • Requires cliclick (brew install cliclick）
  • Requires an OCR environment (~/tools/ocr-env/，pyobjc + Vision Framework）

Key parameters (fixed in the scripts)

  • SIDEBAR=310: fixed width of the WeChat sidebar (icon 60 + list 250)
  • TITLE_H=50: height of the top title bar
  • INPUT_H=110: height of the bottom input box + toolbar

Notes

  • Do not use --fast for Chinese OCR: fast mode garbles Chinese, the scripts use precise mode by default
  • Coordinates obtained dynamically: before each operation osascript get {position, size}, not hardcoded
  • Focus grabbing prevention: activate + delay are written in the same osascript block