What is AutoCodeBench
AutoCodeBench is a benchmark test set launched by Tencent Hunyuan specifically for evaluating the code capabilities of large models. It contains 3,920 problems evenly distributed across 20 programming languages. The dataset is highly challenging, practical, and diverse, and can measure the performance of large models in multilingual programming tasks. The benchmark test set generates data through an automated workflow, ensuring high quality and coverage, and provides a simplified version (AutoCodeBench-Lite) and a version used to evaluate base models (AutoCodeBench-Complete).
Main features of AutoCodeBench
- Multilingual code capability evaluation: Provides 3,920 problems covering 20 programming languages, comprehensively measuring the multilingual code generation capabilities of large models.
- High-difficulty benchmark testing: Supports the design of high-difficulty problems, effectively identifying the shortcomings of large models in complex programming tasks.
- Amplification of performance differences: Builds AutoCodeBench-Lite based on selected problems, amplifying the performance differences between different models for easier comparative analysis.
- Base model evaluation: Uses 3-shot prompting to build AutoCodeBench-Complete, specifically evaluating the code generation performance of base models.
- Automated code data generation: Generates test inputs based on LLMs and obtains outputs with the help of a sandbox, synthesizing high-quality multilingual code generation data.
- Multilingual code execution verification: Provides the MultiLanguageSandbox service, supporting compilation and execution of more than 30 programming languages to verify the correctness of generated code.
Technical principles of AutoCodeBench
- Automated data generation: AutoCodeGen generates test inputs through large language models (LLMs) and passes the test inputs to a sandbox environment. The sandbox environment executes the code and returns test outputs, constructing high-quality test functions. Programming problems are constructed in reverse order, ensuring that the generated problems are highly difficult and diverse. Multiple strategies are used to filter the generated data, further ensuring high quality, high difficulty, and practicality of the data.
- Multilingual support: The 3,920 problems in AutoCodeBench are evenly distributed across 20 programming languages, ensuring that each language has enough problems for evaluation and avoiding the problem of uneven language distribution. MultiLanguageSandbox supports compilation and execution of more than 30 programming languages, and can verify the correctness and performance of generated code in different language environments, ensuring that the model's performance in multiple languages can be accurately evaluated.
- High difficulty and practicality: Based on reverse-order problem construction and strategy filtering, the generated problems are highly difficult and can effectively evaluate the model's performance in complex programming tasks. The generated problems are highly difficult and have practical application value, can reflect complex problems in real programming scenarios, and help models perform better in practical applications.
Project address of AutoCodeBench
- Project official website：https://autocodebench.github.io/
- GitHub repository：https://github.com/Tencent-Hunyuan/AutoCodeBenchmark
- HuggingFace model library：https://huggingface.co/datasets/tencent/AutoCodeBenchmark
- arXiv technical paper：https://arxiv.org/pdf/2508.09101
Application scenarios of AutoCodeBench
- Model performance evaluation: Used to comprehensively measure the code generation capabilities of large models in multilingual programming tasks, helping identify the strengths and weaknesses of models.
- Dataset construction and optimization: Generates high-quality, high-difficulty code generation datasets, supports customized dataset construction, and improves model training results.
- Multilingual capability validation: Validate the performance of large models in different programming languages (including low-resource languages), and promote research on multilingual programming capabilities.
- Model training and validation: As a supplement to training data, improve the model's performance on complex programming tasks, and regularly validate the training effect.
- Academic and industrial applications: Provide standardized benchmarks for academic research and support the development and optimization of code generation tools in industrial scenarios.