What is Multi-SWE-bench
Multi-SWE-bench is the first multilingual code repair benchmark open-sourced by the ByteDanceDoubaolarge model team. Building on SWE-bench, it covers for the first time 7 mainstream programming languages beyond Python, including Java, TypeScript, JavaScript, Go, Rust, C, and C++, making it a truly "full-stack engineering"-oriented evaluation benchmark. The dataset contains 1,632 real repair tasks, all from GitHub issues, rigorously filtered and manually verified to ensure that each sample has a clear problem description, a correct repair patch, and a reproducible runtime test environment. A task difficulty grading mechanism is introduced, dividing problems into three categories: easy, medium, and hard, covering development challenges ranging from one-line modifications to multi-file, multi-step, multi-semantic-dependency tasks.
Multi-SWE-bench's Main Features
- Multilingual code repair evaluation: As the industry's first multilingual code repair benchmark dataset, Multi-SWE-bench covers for the first time 7 mainstream programming languages beyond Python, including Java, TypeScript, JavaScript, Go, Rust, C, and C++. This enables the dataset to more comprehensively evaluate the automatic code repair capabilities of large models in different programming language environments.
- Task difficulty grading: The dataset introduces a task difficulty grading mechanism, dividing problems into three categories: Easy, Medium, and Hard. This grading approach covers development challenges ranging from one-line modifications to multi-file, multi-step, multi-semantic-dependency tasks, enabling a more systematic measurement of large models' performance at different capability levels.
- Real data support: All 1,632 instances in Multi-SWE-bench come from real open-source repositories (GitHub issues), and have undergone unified testing standards and review and filtering by professional developers. Each sample has a clear problem description, a correct repair patch, and a reproducible runtime test environment, ensuring the quality and practicality of the dataset.
Multi-SWE-bench's Technical Principles
- Data sources and quality control: All 1,632 instances in the dataset come from real open-source repositories (GitHub issues), and have undergone unified testing standards and review and filtering by professional developers. During the construction process, the team adopted a rigorous five-stage data construction workflow:
- Open-source repository screening: Based on public GitHub repositories, high-quality project repositories are screened from multiple dimensions.
- Pull Request crawling: Collect pull requests (PRs) related to issues, and extract key information.
- Docker environment construction: Build a corresponding Docker container for each PR, ensuring that every task in the dataset is fully runnable.
- PR filtering and verification: Identify valid repair behaviors through a three-state testing process (original state, only applying the test patch, and applying both the test and repair patches).
- Manual verification: Introduce a manual double-annotation process to ensure the reliability and accuracy of the data.
- Reinforcement learning support: To support the application of reinforcement learning (RL) in code repair tasks, the team open-sourced Multi-SWE-RL. This community provides 4,723 structured training samples, each equipped with a reproducible Docker environment, supporting one-click startup, automatic evaluation, and rapid integration into RL training frameworks. This "evaluation + training" dual-driven model provides strong support for the continuous optimization of large models.
Multi-SWE-bench project address
- Project official website：https://multi-swe-bench.github.io/#/
- Github repository：https://github.com/multi-swe-bench/multi-swe-bench
- HuggingFace dataset：https://huggingface.co/datasets/ByteDance-Seed/Multi-SWE-bench
- arXiv technical paper：https://arxiv.org/pdf/2504.02605
Application scenarios of Multi-SWE-bench
- Automated code repair: Developers can use models trained with Multi-SWE-bench to automatically identify and fix bugs in code, reducing the time and effort required for manual debugging.
- Model performance evaluation and improvement: The dataset provides a systematic evaluation benchmark for large models, helping developers and researchers assess model performance across different programming languages and task difficulty levels.
- Comparative research on programming languages: By comparing bug repair capabilities across different programming languages, researchers can analyze the advantages and limitations of each language in greater depth.
- Intelligent learning and education: For developers and learners, Multi-SWE-bench is a platform for learning and improvement. By studying and using this dataset, developers can better understand common errors and repair methods in different programming languages, improving their own programming and problem-solving abilities.