Multi-SWE-bench is a multilingual benchmark for evaluating the capabilities of large language models in real-world code issue-solving scenarios, mainly aimed at automated issue resolution and reinforcement learning research.
Purpose: This benchmark addresses the problem that existing evaluation resources are Python-centric and lack multilingual coverage, with the goal of accelerating progress in automated issue resolution and reinforcement learning and narrowing the gap toward AGI.
Main capabilities and datasets: The framework covers 7 languages (Java, TypeScript, JavaScript, Go, Rust, C, C++) and contains 1,632 high-quality instances, which were selected by 68 expert annotators from 2,456 candidates to ensure reliability. The project provides the full benchmark as well as two lightweight versions: Multi-SWE-bench mini (400 instances, covering 8 languages, designed to reduce computational cost and speed up evaluation) and Multi-SWE-bench flash (300 curated multilingual evaluation instances for rapid evaluation and efficient agent rollout). In addition, the Multi-SWE-RL community is an open-source initiative for large-scale reinforcement learning datasets, with an initial release containing 4,723 instances.
Evaluation and framework: The project conducted a comprehensive evaluation of nine models (GPT-4o, OpenAI-o1, OpenAI-o3-mini-high, Claude-3.5-Sonnet, Claude-3.7-Sonnet, DeepSeek-V3, DeepSeek-R1, Qwen2.5-72B-Instruct, Doubao-1.5-Pro) under three agent frameworks (Agentless, SWE-agent, OpenHands), and derived several valuable insights.
Usage: Evaluation uses Docker for reproducible runs, and Docker must be installed by following the Docker installation guide; on Linux, it is also recommended to check the post-installation steps. Building from source requires setting up the development environment, installing development dependencies, and configuring pre-commit hooks. Running the evaluation requires preparing: a patch file in JSONL format, where each item contains org (organization name), repo (repository name), number (Pull Request number), and fix_patch (fix patch content); a dataset file in JSONL format, available on Hugging Face, for example Multi-SWE-bench or Multi-SWE-RL; optional Docker images, downloadable as mini, verified, or RL images via scripts/download_images.ps1 (Windows) or scripts/download_images.sh (Linux/macOS). This step is optional; if the images do not exist locally, they will be automatically built during evaluation. After the evaluation run, final_report.json is generated in the specified output_dir, containing a summary of metrics such as resolved_instances and unresolved_instances; failed instances and their specific error causes can be viewed in the log files under the log_dir directory. If problems occur when using git apply to apply configurations, the corresponding configuration item can be added to replace git apply with patch --batch, thereby improving the success rate of patch application.
Use cases: suitable for researchers studying the ability of large language models to resolve issues in multilingual real-world code repositories, researchers constructing and evaluating reinforcement learning datasets, and developers who wish to extend datasets, tools, and research collaboration based on this benchmark. The community provides a contribution incentive program, a quick start guide, a contribution demo, and a Discord discussion channel.
Limitations: the above materials do not provide information on costs, pricing, platform support scope, performance guarantees, or a complete list of specific operational steps. Some content (such as pretraining hook settings and specific evaluation commands) is not fully shown in the original text. The project uses the Apache License 2.0, see the LICENSE file for details.
