SuperAPI AI Directory
Partner中文AI newsFavorites
Skillscomputer-use-mcp

computer-use-mcp

Agents & models

View repository

Updated 10-17 10:19

About this skill

[Historical author source dated 2026-08-14 (UTC). The checked current repository snapshot 9dccb2d0dc06 no longer contains this file path. Current installation and compatibility are unverified.]

Source: https://raw.githubusercontent.com/dmmulroy/.dotfiles/f1c9a48fd9445383488a6e87751bcd5de90e2ab3/home/.agents/skills/computer-use-mcp/SKILL.md

Computer Use MCP

Use the most efficient available interface for the task. Prefer a purpose-built tool, API, CLI, or MCP server when it can complete the work more directly or reliably. Use Computer only for auth-bound, desktop-only, or genuinely visual work, including UI verification after a deterministic change.

When Computer is the right interface, drive the UI with a tight loop: choose the right identity, snapshot once, chain only against fresh state, and verify the outcome.

1. Choose the identity

For Helium, resolve the browser profile before navigation:

  • work account, company domain, or internal service → work
  • personal account, finance, shopping, or personal service → personal
  • ambiguous identity or mixed accounts → ask which profile

Target Helium as net.imput.helium. Inspect its profile control or Profiles menu and switch when needed. Confirm the selected profile from the profile control or signed-in account marker. Profile selection is complete only when the intended profile is visible.

For another app, use the name or bundle identifier already known from the current session. Call computer_list_apps only when the app identity is unknown or stale, then keep the returned identifier stable.

Pi exposes the server tools with the computer_ prefix. Pass mcp.args as a serialized JSON object, as in the example below. List the computer server only when its tools are unavailable; describe only an unfamiliar tool, then reuse that schema for the session.

2. Start the turn with fresh state

Begin each assistant turn that interacts with an app by calling computer_get_app_state. Start with its defaults. The snapshot's element indices belong only to that state.

mcp({ tool: "computer_get_app_state", args: "{\"app\":\"net.imput.helium\"}" })

Use the refreshed state returned by each action to choose the next action. Call get_app_state again only after navigation, reload, modal or window changes, a failed action, or evidence that the returned tree is incomplete.

Keep snapshots compact:

  • raise text_limit only when truncated semantic text is required; prefer a bounded integer before "max"
  • raise max_tree_nodes or max_tree_depth only when a visible long page, list, or table is missing from the tree after scrolling
  • retain only the few element indices and state facts needed for the next chain

The state is fresh when it identifies the intended app/window/profile and exposes the next target or proves that the target is absent.

3. Act in short stable chains

Prefer semantic element actions over coordinates.

  • Click: use computer_click with element_index; omit click_method so auto applies.
  • Fill: when the element is marked settable, use computer_set_value. Otherwise click the editable element, confirm focus in the refreshed state, then use computer_type_text for literal text.
  • Keys: use computer_press_key for named keys and combinations, not literal prose.
  • Coordinates: use them only when the rendered tree has no target. Keep the default auto method unless a specific fallback is justified.

Chain multiple calls in one assistant turn only while every next target is present in the latest action result and the window has not changed. Stop the chain at navigation, submission, modal transitions, downloads/uploads, or uncertainty; inspect before continuing.

4. Recover by changing the precondition

One failed call ends that strategy:

  • stale element or changed page → refresh state and choose a current index
  • no focused editable element → click the field, inspect focus, then type; use set_value when the field is settable
  • non-settable element → focus it and type rather than repeating set_value
  • app or window not found → call list_apps once, adopt its canonical identifier, then refresh state
  • unsupported key or click method → use a supported key name or return to auto
  • tool/catalog or connection error → reconnect or reload once, then rediscover the server surface
  • permission error → report the required OS permission and pause for the user

A retry is valid only when the state, target, arguments, or method changed.

5. Verify

Use the latest action result when it proves the requested outcome; otherwise refresh state once. Completion requires visible evidence of the outcome, not merely a successful tool response.

Reference branches

Read REFERENCE.md only when overriding snapshot budgets or selecting a non-default macOS click method.