C-Eval

What is C-Eval

C-Eval is a multi-level, multi-disciplinary Chinese evaluation suite for large language models, jointly launched by researchers from Shanghai Jiao Tong University, Tsinghua University, and the University of Edinburgh in May 2023. It contains 13,948 multiple-choice questions covering 52 different subjects and four difficulty levels, and is used to evaluate the Chinese understanding ability of large models. Through zero-shot and few-shot testing, C-Eval can evaluate a model's adaptability and generalization ability on unseen tasks.

Main Features of C-Eval

• Multi-disciplinary coverage: C-Eval includes questions from 52 different subjects, covering multiple fields such as STEM, social sciences, and humanities, comprehensively evaluating the knowledge reserve of language models.
• Multi-level difficulty grading: It has four difficulty levels, from basic to advanced, carefully evaluating the model's reasoning and generalization ability at different difficulty levels.
• Quantitative evaluation and standardized testing: It includes 13,948 multiple-choice questions and provides quantitative performance metrics through a standardized scoring system, supporting horizontal comparison of different models.

How to use C-Eval

• Data download：
  •  Hugging Face download:

from datasets import load_dataset
dataset = load_dataset("ceval/ceval-exam", name="computer_network")

• • Or directly download the ZIP file and unzip it:

wget https://huggingface.co/datasets/ceval/ceval-exam/resolve/main/ceval-exam.zip
unzip ceval-exam.zip

• Choose evaluation mode
  • Zero-shot: The model answers questions directly without any examples.
  • Few-shot: The model answers questions with prompts containing a small number of examples (such as 5).
• Prepare the model:Ensure the model has been loaded and is ready for inference. If it is a Hugging Face-based model, load it with the following code:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "your-model-name"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

• Build a prompt
  • Zero-shot prompt：

以下是中国关于{科目}考试的单项选择题，请选出其中的正确答案。
{测试题目}
A. {选项A}
B. {选项B}
C. {选项C}
D. {选项D}
答案：

• • Few-shot prompt：

以下是中国关于{科目}考试的单项选择题，请选出其中的正确答案。
{题目1}
A. {选项A}
B. {选项B}
C. {选项C}
D. {选项D}
答案：A
[k-shot 示例]
{测试题目}
A. {选项A}
B. {选项B}
C. {选项C}
D. {选项D}
答案：

• Generate answers:Use the model to generate answers. Extract the answer option (A, B, C, D) from the generated text, or calculate the probability of each option and choose the answer with the highest probability.

inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
answer = extract_answer(response)  # 自定义函数，提取答案选项

•  Evaluate the model
  • For the validation set (val), directly calculate the accuracy.
  • For the test set (test), you need to submit the results to the C-Eval official platform to obtain a score.

from sklearn.metrics import accuracy_score

# 假设 `predictions` 是模型的预测结果，`labels` 是真实答案
accuracy = accuracy_score(labels, predictions)
print(f"Validation Accuracy: {accuracy:.2f}")

• Submit results:Prepare a JSON file containing the prediction results for all test questions:

{
  "chinese_language_and_literature": {
    "0": "A",
    "1": "B",
    ...
  },
  ...
}

• • Log in to the C-Eval official platform and submit the results to obtain the final score.

Application scenarios of C-Eval

• Language model performance evaluation: Comprehensively measure the knowledge level and reasoning ability of language models, helping developers optimize model performance.
• Academic research and model comparison: Provide researchers with a standardized testing platform to analyze and compare the performance of different language models across subjects, promoting academic research and technological progress.
• Application development in the education field: Help develop intelligent tutoring systems and educational assessment tools, use models to generate practice questions and automatically grade, and improve the level of intelligence in the education field.
• Industry application optimization: In industries such as finance, healthcare, and customer service, evaluate and optimize the domain knowledge and application capabilities of language models, improving the effectiveness of intelligent industry solutions.
• Community collaboration and technical evaluation: As an open platform, it promotes communication and cooperation within the developer community, and provides fair benchmarking tools for model competitions and technical evaluations.