Agent S is an open-source computer use agent framework developed by Simular. It receives natural language tasks, observes the screen, and completes tasks in ordinary desktop and web applications through operations such as clicking, typing, and scrolling, without API integration and without writing separate scripts for each application.
Main capabilities:
- Operates a real GUI like a human: uses the mouse, keyboard, and screen to perform tasks in desktop and web applications.
- Cross-platform support: can run on macOS, Windows, and Linux.
- Model compatibility: supports models such as OpenAI, Anthropic, Gemini, Open Router, vLLM, and requires a grounding model (UI-TARS-1.5-7B recommended).
- Performance: Agent S3 reaches 72.60% on the OSWorld benchmark, surpassing human level (about 72%) for the first time; it also demonstrates strong zero-shot generalization ability on WindowsAgentArena and AndroidWorld.
- Local code execution environment (optional): can execute Python and Bash code for tasks such as data processing, file operations, and system automation.
Usage:
- Installation: can be installed via pip (without cloning the repository), or by cloning the repository and then installing. An additional brew install tesseract is required to support Pytesseract.
- API configuration: add environment variables in .bashrc (Linux) or .zshrc (macOS).
- Running: use command-line arguments to specify the main model, grounding model and its URL, coordinate resolution, etc. For example, UI-TARS-1.5-7B requires setting --grounding_width 1920 --grounding_height 1080.
- Code integration: can import the AgentS3 and OSWorldACI classes, define engine parameters, and then query the agent.
- Local coding environment: enabled via --enable_local_env, but it will execute arbitrary code and must be used with caution.
Applicable scenarios:
- Research on operating system agents.
- Automating personal desktop tasks.
- Participating in open-source computer use projects.
- Repetitive workflow automation (can be implemented through the Simular cloud platform or API).
Limitations and notes:
- Only single-monitor screens are supported.
- The agent runs Python code to control the computer and must be used with caution.
- The local coding environment executes arbitrary code with user permissions and should only be used in trusted environments.
- Bash script execution has a 30-second timeout limit.
- Some advanced features (such as the Simular cloud platform and API) are hosted services, and specific pricing and availability are not yet provided.
