SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsdym-agent-desktop-drive

dym-agent-desktop-drive

Agents & models

View repository

Updated 10-18 01:23

About this skill

dym-agent-desktop-drive

When to use

  • You need to read state or perform actions in a macOS desktop app (native or Electron with AX enabled), across multiple steps, and you want refs that don't die midway.
  • You need evidence of "which element was actually clicked" rather than blind coordinates.
  • Pure web in the browser → see /dym-agent-desktop route first, usually playwright or CDP is much cheaper.

Steps

Set beforehand: AD=<đường dẫn binary>, export AGENT_DESKTOP_HOME=<abs>.

  1. Pin the window before snapshotting. Multi-window app → snapshot --app returns AMBIGUOUS_TARGET along with a list of candidates (this is a feature, not a bug):
$AD list-windows --app "Microsoft Teams"
$AD snapshot --window-id w-9249 --skeleton -i --compact
  1. Skeleton first, drill after — only when the app is dense:
$AD snapshot --app "System Settings" --skeleton -i --compact   # đo: 4630 B, 27 ref
$AD snapshot --root "@s23de9t04j589e:e5" -i --compact           # bung đúng vùng cần

Sparse apps (Finder, Calculator, TextEdit) → skip the skeleton, snapshot directly. Already know the name/role → find is cheapest, skip the snapshot:

$AD find --app "System Settings" --role button --name "Sound" --exact --limit 2
  1. Act by full ref @<snapshot_id>:e<N> — always wrap in quotes because of : in the shell:
$AD click "@s23de9t04j589e:e12"
$AD set-value "@s23de9t04j589e:e2" "text"
$AD toggle "@s23de9t04j589e:e6"      # hoặc check/uncheck (idempotent)
$AD select "@s23de9t04j589e:e4" "Option B"

Default headless: takes the semantic AX path, doesn't steal focus, doesn't move the mouse, doesn't synthesize keys. If you need real typing/dragging, add --headed globally: $AD --headed drag --from "@s..:e1" --to "@s..:e5".

  1. Wait for state instead of sleep:
$AD wait --element "@s..:e5" --predicate actionable --timeout 5000
$AD wait --text "Done" --app "App" --timeout 5000
$AD wait --menu --app "App"        # chờ menu bung

  1. Verify by drilling the same region again that changed. Refs of other regions remain alive (scoped invalidation) — no need to re-snapshot the whole app.

  2. Batch multiple commands in one go when the steps don't depend on intermediate observations:

$AD batch '[{"command":"list-surfaces","args":{"app":"System Settings"}},{"command":"clipboard-get","args":{}}]' --stop-on-error

Measured: 2 commands in 629 ms, max_entries: 64, max_input_bytes: 1 MiB.

Rules

  • Exit codes: 0 ok · 1 structured error · 2 parameter error (all three measured for real). Branch on error.code, NOT on error.message — the repo self-declares that message/suggestion may change between versions.
  • Refs persist across processes: a snapshot taken in a previous command still resolves in a later command thanks to the refmap written to $AGENT_DESKTOP_HOME/snapshots/<id>/refmap.json (measured: a ref taken that way minutes ago, in a different process, still get works). This is a platform difference compared to orca computer element-index (which only lives within a single return round).
  • Wrong ref index → STALE_REF (rc=1) with recovery.strategy; snapshot_id doesn't exist → SNAPSHOT_NOT_FOUND. Never guess the next index.
  • Common error codes: PERM_DENIED (grant permission + reopen the terminal) · AMBIGUOUS_TARGET (pin --window-id or add --exact) · ACTION_NOT_SUPPORTED (switch to a different command) · POLICY_DENIED (currently headless but need --headed) · TIMEOUT (read error.details.kind before loosening the timeout).
  • Click was receives_events + occluder: {...} = has an overlapping element. Bring the window to the front or dismiss the overlay, don't retry blindly.
  • If Electron doesn't enable AX, both tools are blind. Measured on Microsoft Teams: orca computer sees 5 nodes, agent-desktop sees ref_count: 4 — both stop at the "Web content" boundary. The way out: launch --cdp (only when you accept restarting the app) then use the CDP client.
  • The entire original doc is taken from the binary itself when you need details: $AD skills get desktop --full (in SKILL + 5 reference files). Don't copy them into the repo.