What OSWorld is OSWorld is a benchmark evaluation project for multimodal agents, aiming to evaluate agents' ability to complete open-ended tasks in real computer environments. The project paper was accepted to NeurIPS 2024, and the repository provides environments, benchmark tasks, evaluation scripts, and baseline agents to facilitate researchers in reproducing and comparing results.
Main capabilities Provides real computer environments (virtual machines) as evaluation sandboxes; includes open-ended task benchmarks covering multiple domains; provides runnable baseline agents, supporting settings such as screenshots only; supports single-threaded and parallel execution; can output screenshots, action records, and videos, and calculate success rates and total scores by domain and category; provides a manual task inspection tool for manually verifying task correctness and evaluation metrics.
Version updates On 2025-07-28, OSWorld-Verified was released, fixing multiple issues reported by the community, enhancing AWS support, supporting parallelization to compress evaluation time to within 1 hour, and making benchmark signals more effective, while updating the latest model results on the official website. On 2025-05-01, pre-downloaded files for initializing state were provided. On 2024-10-22, Docker hosting of virtual machines was supported. On 2024-06-15, the environment code was refactored, the VMware integration was decomposed, and support began for VirtualBox, AWS, Azure, and other platforms. On 2024-04-11, the paper, environment, and benchmark were released.
Run modes On non-virtualized systems (desktops, laptops, bare metal), you can clone the repository and install the dependencies in requirements.txt. It is recommended to use Conda to manage the environment, and the Python version must be no lower than 3.10; you can install only the environment without installing the benchmark tasks; you need to install VMware Workstation Pro (on Apple chip systems, install VMware Fusion) and configure the vmrun command. VirtualBox is also supported (but parallelism and macOS support on Apple chips may be poor). On servers that support KVM, Docker support is recommended: check KVM support on Linux; macOS hosts usually do not support KVM, so it is recommended to use VMware instead; when initializing DesktopEnv, set provider_name to docker and os_type to Ubuntu or Windows. If the experiment is abnormally interrupted, you can run docker stop $(docker ps -q) && docker rm $(docker ps -a -q) to clean up leftover containers. Modal (VM sandbox, use --provider_name modal when running evaluation) and Daytona (the sandbox itself acts as the desktop, no host KVM or Docker qcow2 workflow required) are also supported. Using cloud services for parallel evaluation can significantly speed things up; the project provides AWS support based on a Host-Client architecture for large-scale parallel evaluation, and can even serve as training infrastructure.
Usage Running the minimal example allows interacting with the environment and observing logs for environment creation, setup completion, and action execution. Running the paper baseline agents (such as GPT-4o with the pure screenshot setup) requires setting OPENAI_API_KEY, and optionally setting OPENAI_BASE_URL to use a compatible endpoint. Both single-threaded (deprecated, using the vmware provider as an example) and parallel execution (with the example switching the provider to docker) are supported. Host-side credentials or keys required by tasks should be injected at startup rather than baked into the virtual machine image. Results (including screenshots, actions, and videos) are saved to ./results or a specified result_dir. Commands can be used to obtain per-domain success rates, category statistics (Office, Daily, Professional), overall success rate, and total score; using --detailed displays a compact "score/total" format for each domain. For local evaluation, you need to read the agent interface and environment interface, implement and import a custom agent into run.py (single-threaded) or scripts/python/run_multienv.py / run_multienv_xxx.py (parallel). For public evaluation, you need to schedule with the maintainers, who will run the agent code and report the results; you need to upload and allow disclosure of the agent implementation under the OSWorld framework (you may choose not to disclose the model API), and provide an explanatory report; trusted institutions may also share monitoring data and trajectories.
Use cases Suitable for researchers evaluating multimodal computer-use agents performing open-ended tasks in real desktop environments; for comparing different models or agent approaches; for infrastructure for training or large-scale parallel evaluation; and for manually verifying tasks and metrics.
Limitations and caveats Default virtual machine credentials: under vmware, virtualbox, and docker, Ubuntu is user / password; cloud services such as aws default to osworld-public-evaluation, and after changing it you need to set the client_password variable and pass it to DesktopEnv and Agent. Some tasks require a Google account and OAuth2.0 configuration, and some tasks require proxy settings; otherwise task execution fails and leads to lower scores. macOS hosts generally do not support KVM. VirtualBox may not work well in parallel and on Apple silicon macOS. Support for some platforms (such as Azure) is under development.
Project address:https://github.com/xlang-ai/OSWorld
