computer-use

Background computer use

Use the installed computer-use CLI for macOS UI work that must leave the user's cursor and keyboard alone. Read computer-use help and computer-use browser help for the current command syntax. Results and errors from operational commands are JSON on stdout; errors exit nonzero.

Check availability

1. Run command -v computer-use. If missing, install from this repository with cargo install --path . --locked at its root, or pass the repository's absolute path from another directory.
2. For native apps, run computer-use doctor. Accessibility is required for inspection and actions; Screen Recording is required for window capture. The host app running the CLI needs those macOS permissions.
3. For web work in the isolated browser, use browser start; it does not need those macOS permissions. It starts a separate headless Brave or Chromium profile, so it does not share the user's open tabs or cookies.

Native app loop

1. Find the target with computer-use apps. When a separate background instance is appropriate, use computer-use launch --app '/path/to/Target.app' and keep its returned PID. Launch requests no activation, but an app can activate itself later.
2. Run computer-use windows --pid PID, then computer-use observe --pid PID --output /tmp/target.png. Use computer-use inspect --pid PID --depth N --nodes N when the default tree misses a control.
3. Choose a path from the latest tree. Use computer-use describe --pid PID --path PATH if you need exact role, identifier, title, value, or advertised actions.
4. Act with a live identity precondition. Prefer --expect-id; use --expect-title or --expect-value when there is no stable identifier. Add --expect-role when useful. For example:

computer-use press --pid 1234 --path 0/2/1 --expect-id save-button
printf '%s' 'Draft text' | computer-use set --pid 1234 --path 0/3 --expect-id draft-field --stdin

1. Observe again after each UI change and verify the effect. Paths are child indexes from the app root and can change; an expected attribute is compared with the live element immediately before acting. If a path or precondition fails, inspect again before retrying.

Leave out --unchecked and --allow-frontmost during background work. The CLI rejects actions on the current foreground app by default. perform accepts only advertised AXPress, AXIncrement, AXDecrement, AXConfirm, AXCancel, and AXScrollToVisible actions. Prefer paths for covered windows; describe-at and press-at hit-test global screen points and may miss a covered control.

Isolated browser loop

computer-use browser start
computer-use browser navigate 'https://example.com'
computer-use browser snapshot
computer-use browser screenshot --output /tmp/page.png
computer-use browser click --selector 'button[type=submit]'
computer-use browser focus --selector 'input[name=q]'
printf '%s' 'Search term' | computer-use browser type --stdin
computer-use browser snapshot
computer-use browser stop

Prefer CSS selectors to coordinate clicks. Coordinates are CSS pixels in the 1280×800 headless viewport. Check the result after navigation; loaded:false does not prove the page is ready. Snapshot output is limited to 500 nodes at depth 12 and redacts common sensitive fields heuristically. Screenshots contain the rendered page. browser stop ends the process but keeps the private profile data for later sessions.

Limits

Native actions require the app to expose accessible controls. Arbitrary background dragging and keyboard shortcuts are unsupported. Browser selectors operate in the main document, not cross-origin frames or shadow roots. If a requested action cannot be done through these supported operations, report the specific limit instead of sending global input that would disturb the user.