What is CodeElo
CodeElo is a benchmark tool for evaluating the programming competition-levelcode generationcapabilities of large language models (LLMs). It measures the programming proficiency of LLMs by comparing them with the Elo rating system of human programmers. The tool selects problems from the CodeForces platform and classifies them by contest division, difficulty level, and algorithm tag to ensure the diversity and representativeness of the problems. CodeElo's evaluation method is very robust. Submitted code is tested directly on the CodeForces platform, and based on a special evaluation mechanism, it ensures accurate judgment of code correctness. It uses the Elo rating system to calculate scores, taking into account problem difficulty and penalizing errors. After testing multiple open-source and proprietary LLMs, OpenAI's o1-mini model performed best, surpassing 90% of human participants. CodeElo was launched to address the limitations of existing benchmarks and provide a more comprehensive and accurate evaluation environment to help researchers and developers better understand and improve LLMs'programmingcapabilities.
Main Features of CodeElo
- Problem Selection and Classification
- Wide Range of Sources: The problems mainly come from the CodeForces platform, an online programming competition website with a large number of high-quality programming problems.
- Detailed Classification: Problems are classified by contest division, difficulty level, and algorithm tag, enabling researchers to select suitable problems for testing according to different needs, such as conducting targeted evaluations of problems for specific algorithms or difficulty levels.
- Code Submission and Testing
- Direct Submission: Researchers can submit code generated by LLMs directly to the CodeForces platform for testing, without additional configuration or environment setup.
- Special Evaluation Mechanism: Based on CodeForces' special evaluation mechanism, it can accurately judge the correctness of code, including exact matching of output results and restrictions on execution efficiency, ensuring the accuracy and reliability of evaluation results.
- Rating Calculation and Comparison
- Elo Rating System: It uses an Elo rating system similar to the rating of human chess players to calculate the programming ability scores of LLMs, taking into account problem difficulty and code correctness, and can more fairly reflect the model's actual level.
- Comparison with Humans: By comparing with the Elo ratings of human programmers, it is possible to intuitively understand the performance of LLMs in programming competitions and the gap between them and human programmers, providing a reference for model improvement.
- Analysis of Evaluation Results
- Performance Comparison: CodeElo provides detailed performance comparison analysis, which can show the performance of different LLMs on various types of problems, helping researchers understand the strengths and weaknesses of models in different scenarios.
- Error Analysis: It analyzes erroneous submissions to identify the common types and causes of errors in the model's code generation process, providing a basis for model debugging and optimization.
- Addressing the Limitations of Existing Benchmarks
- Making Up for Shortcomings: In response to the limitations of existing benchmarks such as LiveCodeBench and USACO, such as the lack of private test cases, no support for special judge mechanisms, and inconsistent execution environments, CodeElo provides a more comprehensive, accurate, and consistent evaluation environment, making the evaluation of LLMs' programming capabilities more reliable and effective.
CodeElo's project address
- Project official website：https://codeelo-bench.github.io
- HuggingFace model library：https://huggingface.co/datasets/Qwen/CodeElo
- arXiv technical paper：https://arxiv.org/pdf/2501.01257
CodeElo's test results
- After testing 30 open-source LLMs and 3 proprietary LLMs, OpenAI's o1-mini model performed the best, with an Elo rating of 1578, surpassing 90% of human participants.
- Among open-source models, QwQ-32B-Preview ranked first with 1261 points.
- Many models still struggle to solve simple problems and usually rank in the bottom 20% of human participants.
CodeElo's application scenarios
- Evaluating model programming capabilities: By comparing with the Elo rating system for human programmers, CodeElo can effectively evaluate the programming capabilities of large language models (LLMs) in programming competitions.
- Assisting programming teaching: CodeElo provides a standardized programming capability evaluation tool that can be used to assist programming teaching and learning. Teachers can use CodeElo to understand students' performance on different programming problems, identify students' weak areas, and provide more targeted teaching guidance.
- Student self-assessment: Students can also use CodeElo to conduct self-assessment of their programming capabilities, understand their level in programming competitions, and clarify their learning goals and direction of effort.
- Model optimization and improvement: Researchers can use CodeElo to test and evaluate newly developed LLM models, understand the models' performance in programming competitions, and guide model optimization and improvement.
- Code generation and completion: Enterprises can use CodeElo to evaluate and select suitable LLM models for development tasks such as code generation and completion, improving development efficiency and code quality.