SWE-bench is a benchmark for evaluating the ability of large language models on real-world software engineering tasks, with the full title "SWE-bench: Can Language Models Resolve Real-world GitHub Issues?". The project collects real-world software issues from GitHub, and the task setting is: given a codebase and an issue, the language model needs to generate a patch that can resolve the described problem. The related paper has been accepted as an oral presentation at ICLR 2024, and another SWE-bench Multimodal paper has been accepted at ICLR 2025.
Main capabilities and datasets: The project provides code and data, and supports evaluating language models on real GitHub issues. The currently provided dataset aliases include full, verified, multimodal, and multilingual, and it also supports HuggingFace ids or local paths. SWE-bench Verified is a subset released on August 13, 2024, containing 500 problems confirmed as solvable by real software engineers. SWE-bench Multimodal was integrated into this repository on January 13, 2025, and its v2 version was fully open-sourced on September 1, 2026, providing 480 tasks for local evaluation.
Usage: SWE-bench uses Docker for reproducible evaluation, and Docker must first be installed according to the Docker setup guide. On M-series Macs or other ARM systems, the --task-repo parameter must be used so that images are built locally through Docker Buildx. The evaluation command is: python -m swebench.harness.run_evaluation. The DATASET parameter accepts aliases (full, verified, multimodal, multilingual), a HuggingFace id, or a local path. The older command form python -m swebench.harness.run_evaluation ... is still usable, with the same parameters as before. This command generates Docker build logs (logs/build_images) and evaluation results in the current directory.Test logs (logs/evaluation), run summaries are written to logs/evaluation/<run_id>/results.json. Note that results are cached only by run_id and instance_id; if you want to re-evaluate the same instance with a different prediction patch, you must change the run_id. To view the full list of parameters of the evaluation framework, run the corresponding command. In addition, you can automatically run evaluations on AWS via sb-cli, or use Modal to run evaluations in the cloud.
Resource requirements and limits: SWE-bench evaluation can be very resource-intensive. It is recommended to run on an x86_64 machine, with at least 120GB of available storage, 16GB of memory, and 8 CPU cores. --max_workers is recommended not to exceed min(0.75 * os.cpu_count(), 24). When using Docker Desktop, you need to increase the virtual disk space to about 120GB available. Support for arm64 machines is experimental.
Other related features: You can train your own model on the preprocessed dataset provided by the project; you can run inference on existing models (local and API models), where the inference step is to provide the model with the repository and issue and have it generate a fix; you can also use SWE-bench's data collection pipeline to create new SWE-bench tasks on your own repository, but the project has temporarily suspended support for queries related to creating SWE-bench instances. The project also provides SWE-smith, a dedicated toolkit for creating SWE training data. The project is under the MIT license, and contributions from the NLP, machine learning, and software engineering research communities are welcome.
