claude-computer-use

Claude Computer Use

Anthropic's official desktop automation API (beta). Throughscreenshots + Claude visual understanding + coordinate operationscontrol any desktop application.

When to Use

This is theultimate fallback, used only in the following situations:

1. peekaboo (macOS) / pywinauto (Windows) / browser (web pages) cannot operate the target application
2. Visual judgment is needed (looking at screen content to decide the next step)
3. The target application does not expose an Accessibility API
4. The user explicitly requests precise coordinate control

Prefer lightweight solutions (lower cost, faster):

• Web pages → browser tool
• macOS desktop → peekaboo
• Windows desktop → pywinauto

Supported Operations

Operation	Description
screenshot	Capture the current screen
left_click	Click at coordinates [x, y]
right_click	Right-click
double_click	Double-click
triple_click	Triple-click (select all text)
middle_click	Middle-click
mouse_move	Move the mouse to coordinates
left_click_drag	Drag (from the current position to the target coordinates)
scroll	Scroll (direction + amount)
type	Type text
key	Key/key combination (e.g. "ctrl+s")
hold_key	Hold a key for a specified duration
wait	Wait for a specified number of seconds

Workflow

循环:
  1. 截屏 (screencapture / pyautogui.screenshot)
  2. 发送截图给 Claude Computer Use API
  3. Claude 分析截图，返回 tool_use (action + coordinates)
  4. 在本地执行操作 (cliclick/xdotool/pyautogui)
  5. 如果任务未完成，回到步骤 1

Complete Example

macOS Implementation

import anthropic
import subprocess
import base64
import json

client = anthropic.Anthropic()

def screenshot():
    """截取 macOS 屏幕"""
    subprocess.run(["screencapture", "-x", "/tmp/screen.png"], check=True)
    with open("/tmp/screen.png", "rb") as f:
        return base64.b64encode(f.read()).decode()

def execute_action(action_type, **kwargs):
    """执行鼠标/键盘操作 (macOS 使用 cliclick 或 AppleScript)"""
    if action_type == "left_click":
        x, y = kwargs["coordinate"]
        subprocess.run(["cliclick", f"c:{x},{y}"])
    elif action_type == "type":
        text = kwargs["text"]
        subprocess.run(["cliclick", f"t:{text}"])
    elif action_type == "key":
        key = kwargs["key"]
        # 通过 AppleScript 发送按键
        subprocess.run(["osascript", "-e",
            f'tell application "System Events" to keystroke "{key}"'])
    elif action_type == "scroll":
        direction = kwargs.get("direction", "down")
        amount = kwargs.get("amount", 3)
        delta = -amount if direction == "down" else amount
        # AppleScript 模拟滚动
        subprocess.run(["osascript", "-e",
            f'tell application "System Events" to scroll area 1 by {delta}'])
    elif action_type == "mouse_move":
        x, y = kwargs["coordinate"]
        subprocess.run(["cliclick", f"m:{x},{y}"])
    elif action_type == "screenshot":
        pass  # 下一轮循环自动截屏

def computer_use_loop(task: str, max_iterations: int = 10):
    """Computer Use 主循环"""
    messages = [{"role": "user", "content": [
        {"type": "text", "text": task}
    ]}]

    for i in range(max_iterations):
        # 1. 截屏
        img_b64 = screenshot()

        # 2. 追加截图到消息
        if i > 0:
            messages.append({"role": "user", "content": [
                {"type": "tool_result", "tool_use_id": tool_use_id,
                 "content": [{"type": "image", "source": {
                     "type": "base64", "media_type": "image/png", "data": img_b64
                 }}]}
            ]})
        else:
            messages[0]["content"].append({
                "type": "image", "source": {
                    "type": "base64", "media_type": "image/png", "data": img_b64
                }
            })

        # 3. 调用 Claude Computer Use API
        response = client.beta.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=1024,
            tools=[{
                "type": "computer_20250124",
                "name": "computer",
                "display_width_px": 1920,
                "display_height_px": 1080,
            }],
            messages=messages,
            betas=["computer-use-2025-01-24"],
        )

        # 4. 解析响应
        messages.append({"role": "assistant", "content": response.content})

        # 检查是否完成
        if response.stop_reason == "end_turn":
            print("任务完成")
            break

        # 5. 执行操作
        for block in response.content:
            if block.type == "tool_use":
                tool_use_id = block.id
                action = block.input.get("action")
                print(f"  执行: {action} {block.input}")
                execute_action(action, **block.input)

    return messages

Windows Implementation

import pyautogui
import time

def screenshot_win():
    """截取 Windows 屏幕"""
    img = pyautogui.screenshot()
    img.save("/tmp/screen.png")
    with open("/tmp/screen.png", "rb") as f:
        return base64.b64encode(f.read()).decode()

def execute_action_win(action_type, **kwargs):
    """执行操作 (Windows 使用 pyautogui)"""
    if action_type == "left_click":
        x, y = kwargs["coordinate"]
        pyautogui.click(x, y)
    elif action_type == "right_click":
        x, y = kwargs["coordinate"]
        pyautogui.rightClick(x, y)
    elif action_type == "double_click":
        x, y = kwargs["coordinate"]
        pyautogui.doubleClick(x, y)
    elif action_type == "type":
        pyautogui.typewrite(kwargs["text"], interval=0.02)
    elif action_type == "key":
        pyautogui.hotkey(*kwargs["key"].split("+"))
    elif action_type == "scroll":
        amount = kwargs.get("amount", 3)
        direction = kwargs.get("direction", "down")
        clicks = -amount if direction == "down" else amount
        x, y = kwargs.get("coordinate", (960, 540))
        pyautogui.scroll(clicks, x, y)
    elif action_type == "mouse_move":
        x, y = kwargs["coordinate"]
        pyautogui.moveTo(x, y)
    elif action_type == "left_click_drag":
        sx, sy = kwargs.get("start_coordinate", pyautogui.position())
        x, y = kwargs["coordinate"]
        pyautogui.moveTo(sx, sy)
        pyautogui.drag(x - sx, y - sy, duration=0.5)

API Parameters

tools=[{
    "type": "computer_20250124",  # 或 computer_20251124 (Opus 4.6)
    "name": "computer",
    "display_width_px": 1920,     # 屏幕宽度
    "display_height_px": 1080,    # 屏幕高度
    "display_number": 1,          # X11 显示号（可选）
}]
betas=["computer-use-2025-01-24"]  # 或 computer-use-2025-11-24 (Opus 4.6)

Cost Estimation

• Each screenshot is about 1000-3000 tokens (depending on resolution)
• Typical task (5-10 screenshot loops): about 10,000-30,000 tokens
• Suggestion: lowering the screenshot resolution (1280x800 works best) can significantly reduce cost

Safety Rules

• HITL confirmation before each operation: inform the user what operation is about to be performed on the screen
• Do not enter passwords automatically: when login is required, prompt the user to do it manually
• Do not operate financial applications: when payment/transfer is involved, user confirmation is required
• Limit the number of loops: max_iterations prevents infinite screenshot loops
• Least privilege: do not give Claude administrator privileges