SuperAPI AI Directory
Partner中文AI newsFavorites
Skillsagent-s

agent-s

Agents & models

View repository

Updated 11-07 04:01

About this skill

Agent S operator

Operate Agent S3 from the gui-agents package as a supervised tool, never as an unattended one. This skill was audited against commit 0fff0e634869dab72027435b4586e62300185d41 (2026-10-21) and the published gui-agents 0.3.2 wheel; audited facts, hazards and the pin are in references/upstream.md.

When to use this skill

  • Fit-check, install, configure, preflight, run, or troubleshoot Agent S3 and its agent_s CLI.
  • Choose providers, set up a grounding model (UI-TARS), or explain --ground_* and --grounding_* flags.
  • Decide whether --enable_local_env is acceptable (it runs arbitrary Python and Bash).
  • Not for Playwright or Selenium test automation, tasks with a CLI or API (browser-harness, stagehand), or Agent S4's AWS-VM evolution loop.

Instructions

1. Freeze the surface

Record one mode (orient, install, configure, run, troubleshoot), one concrete task, and the machine it will run on. Prefer a disposable VM, a separate OS user, or a cleared desktop. The model sees the whole screen on every step.

2. Run the offline gate first

The helper reads host facts and, with --repo, a checkout. It never starts Agent S, pip, a model call or a screenshot, opens no sockets, and writes nothing:

python3 .agent-skills/agent-s/scripts/preflight.py --json
python3 .agent-skills/agent-s/scripts/preflight.py --repo /path/to/Agent-S --expect-commit 0fff0e634869dab72027435b4586e62300185d41 --json

An installed skill uses python3 "$SKILLS_ROOT/agent-s/scripts/preflight.py". Read the python check carefully: pip refuses gui-agents 0.3.2 on Python 3.12.1 and later, so the environment needs 3.9 to 3.11 (or exactly 3.12.0). audit:* lines report what the checkout's cli_app.py actually does.

3. Gate every side effect

Require separate explicit approval for each:

  1. pip install gui-agents (or pip install -e .), which pulls heavy dependencies such as paddleocr and paddlepaddle, plus brew install tesseract.
  2. Creating or using a grounding endpoint (UI-TARS on Hugging Face Inference Endpoints or another provider; paid, and it receives every screenshot).
  3. Exporting provider keys; never paste them into chat or logs.
  4. The first run: the agent clicks and types on the real desktop and executes model-generated code.
  5. --enable_local_env: arbitrary Python and Bash with the user's permissions. Allow it only inside a disposable VM or container.
  6. Any task that sends messages, spends money, deletes data, or touches credentials: require a human confirmation step the tool itself does not provide.

4. Follow the matching reference branch

  • Install, Python range, dependencies → Install in references/upstream.md.
  • Models, grounding, flags, coordinates → Models and grounding.
  • What the run loop does, logs, pause keys → Run loop.
  • Safety and the unused permission dialog → Safety findings.
  • Agent S4, OSWorld, OpenClaw wrapper → Other upstream surfaces.

5. Verify narrowly

Start with a harmless task such as opening a calculator or a blank text editor, with --task "<one sentence>" and a person watching. Ctrl+C pauses the run, Esc resumes, a second Ctrl+C quits. Report what the agent did from the screen and the run's logs/ files, not from the model's claim of success.

Examples

Fit check before installing

Run the preflight, report the Python verdict and the tesseract result, list the model and grounding endpoints the user must supply, and state that screenshots go to them. Do not install.

"Let it clean my downloads folder"

Decline an unattended run on the personal machine. Offer a VM, a one-folder task with --task, no --enable_local_env, and a human watching with Ctrl+C ready.

Best practices

  • Treat the generated code as untrusted input: this version executes it directly with exec().
  • Do not assume a confirmation dialog exists: show_permission_dialog is defined in cli_app.py but never called.
  • Match --grounding_width and --grounding_height to the grounding model's output space (1920x1080 for UI-TARS-1.5-7B, 1000x1000 for UI-TARS-72B).
  • Close password managers, chats and mail before a run; the model reads whatever is on screen.
  • Keep the helper offline and honest: a tool on PATH is not a working model endpoint or screen permission.

References