ExploitGym is a large-scale, realistic-environment benchmark built on real-world vulnerabilities, designed to evaluate the ability of AI agents to develop exploit programs. The project comes from the Sunblaze lab at UC Berkeley and is licensed under Apache-2.0.
Purpose This benchmark is used to measure the ability of AI agents to develop working exploits in real-world vulnerability scenarios, covering targets such as user-space programs, Google's V8 engine, and the Linux kernel.
Main capabilities and content
- Built on real-world vulnerabilities (CVEs), emphasizing realism and scale.
- The current release is v1.0 and contains 869 instances.
- The canonical task list for the current version is located at data/task_ids/v1.txt.
- The repository does not embed any secrets: the controller generates the token salt, flag seed, and API key at startup (or reads them from the corresponding environment variables), and both ends must use the same values.
Usage
- Detailed setup steps (system dependencies, GDB, static node, agent CLI) are in docs/setup.md.
- Setup: Python dependencies, GDB, socat/nc, node, and the agent CLI.
- Docker images: pull target images by task family.
- Evaluation: controller / firewall / LLM proxy, as well as examples/run_agent.py.
- Defense: disable system defense mechanisms (such as ASLR, etc.).
- Firewall: provides outbound network isolation for the agent container.
- Submission: the submission format and requirements for the benchmark leaderboard.
Applicable scenarios Suitable for researching the evaluation of AI agents' capabilities in exploit development, as well as related benchmarking and leaderboard submissions.
Limitations and notes
- The repository does not embed secrets; they must be configured yourself or generated by the controller.
- Task data is located in data/tasks/, comes from external upstream sources, and retains their respective licenses; see DATA_LICENSE.md for details.
- The benchmark is continuously maintained; see CHANGELOG.md for the complete version history.
- If you use ExploitGym in your research, please cite it as required by the repository.
