InfiBench

Purpose

InfiBench is a benchmark for comprehensively evaluating the ability of Code LLMs to answer free-form real-world coding-domain questions. Many existing benchmarks (such as HumanEval, DS-1000, MBPP, etc.) mainly focus on code generation tasks, but they are insufficient to cover the full expected capabilities of Code LLMs—in addition to generating code, they also need to answer various coding-related questions. InfiBench aims to fill this gap. It is the first large-scale free-form question-answering (QA) benchmark (to the best of the R&D team's knowledge), containing 234 carefully selected high-quality Stack Overflow questions and covering 15 programming languages.

Main Capabilities

• Evaluate the ability of Code LLMs to answer real-world coding questions, rather than being limited to code generation.
• Use four model-free automatic evaluation metrics: keyword matching, fill-in-the-blank, unit tests, and dialogue similarity.
• The scoring criteria for each question are carefully specified by domain experts to ensure evaluation correctness.
• Support systematic comparison of a large number of models; more than 100 recent Code LLMs have already been evaluated on InfiBench.
• The benchmark is fully open source and continuously expanded to promote more scientific and systematic Code LLM evaluation practices.

Usage

Currently InfiBench only supports the Linux environment. The steps are as follows:

1. Convert or save the model weights into Hugging Face Transformers format.
2. Clone the code repository.
3. Follow the short tutorial to generate answers and evaluate them on InfiBench.

During evaluation, best@10 is used as the main metric: sample 10 answers for each question and evaluate them, record the best score for each question, and sum them. The sampling temperature T is set to 0.2, and the top p truncation threshold is set to 0.9. Each question has the same weight, each worth 1 point, for a total score of 234 points, and the default reported score is a percentage score (score obtained divided by the full score of 234). The 1 point for each question can be further decomposed into multiple scoring points. For example, if a question contains four keywords with weights of 2, 1, 1, and 1 respectively, then matching each keyword contributes 0.4, 0.2, 0.2, and 0.2 points respectively.

Model running settings: the maximum number of generated tokens is 1024. For models with more than 30B parameters, due to resource constraints, they are evaluated only once; the remaining models are evaluated three times and the mean and standard deviation are reported.

Applicable Scenarios

• Researchers and developers need to comprehensively evaluate the real-world question-answering ability of Code LLMs.
• Compare the performance of different Code LLMs on Stack Overflow-style questions.
• Promote the scientific and systematic development of Code LLM evaluation methods.

Limitations

• Currently only supports the Linux environment.
• Model weights need to be converted into Hugging Face Transformers format.
• Exploration of evaluation hyperparameters (such as other temperature and top p settings) is left for future work.
• The benchmark has 234 questions. Although it covers 15 languages, there is still room for expansion.

Statistics and Examples

InfiBench contains 234 carefully selected high-quality Stack Overflow questions, covering 15 programming languages, broadly following the natural question distribution of Stack Overflow. The R&D team recruited five domain experts to create the benchmark and annotate the correctness evaluation criteria. The benchmark integrates four model-free metrics for evaluating correctness.

More Leaderboards

In addition to InfiBench, it is recommended to use diverse benchmarks and leaderboards to comprehensively understand LLM coding ability, for example: Big Code Models Leaderboard, Chatbot Arena Leaderboard, HumanEval.jl (the Julia version of HumanEval, including EvalPlus test cases), NaturalCodeBench, SWE-bench Verified, TabbyML Leaderboard.