CMMLU

What is CMMLU

CMMLU is a comprehensive Chinese evaluation benchmark specifically designed to evaluate the knowledge and reasoning abilities of language models in Chinese contexts, covering 67 topics ranging from basic disciplines to advanced professional levels. It includes natural sciences that require calculation and reasoning, humanities and social sciences that require knowledge, and Chinese driving rules that require everyday common sense. Many tasks in CMMLU have China-specific answers that may not be generally applicable in other regions or languages. CMMLU provides rich test data and a leaderboard, supports multiple evaluation methods such as five-shot and zero-shot testing, and is an important tool for measuring the performance of Chinese language models.

Main Features of CMMLU

• Leaderboard: Displays the performance of different language models under five-shot and zero-shot testing, helping compare model performance.
• Dataset: Provides development and test data, supporting quick use and evaluation.
• Preprocessing Code: Provides prompt generation methods, facilitating model training and testing.
• Evaluation Tools: Supports multiple evaluation methods, making it convenient for researchers and developers to test model capabilities.

How to Use CMMLU

• Obtain the Dataset：
  • Download from GitHub: Visit the CMMLU GitHub page:https://github.com/haonan-li/CMMLU/，在dataFind the development and test datasets in the directory.
  • Obtain via Hugging Face: Visit the Hugging Face platform:https://huggingface.co/datasets/haonan-li/cmmlu，直接加载CMMLU数据集。
• Prepare the Test Environment：
  • Install Dependencies: Ensure that the necessary Python libraries are installed, such astransformers、datasets, etc.
  • Clone the Code Repository: Clone the CMMLU GitHub repository to obtain the test code and preprocessing tools.

git clone https://github.com/haonan-li/CMMLU.git
cd CMMLU

• Preprocess the Data: In thesrc/mp_utilsdirectory, use the provided scripts to preprocess the data and generate a format suitable for model input.

python src/mp_utils/preprocess.py

• Run the Evaluation Code
  • Select a Model: Load the model and tokenizer according to the language model to be evaluated.
  • Run the Test Script: In thescriptdirectory, run the test script to evaluate the model's performance on different tasks.

python script/evaluate.py --model <model_name> --data_path <data_path>

• Submit Test Results：
  • Open-Source Models: Directly submit a pull request (PR) to update the test code and results.
  • Closed Models: Send the test code and results to the specified email (such ashaonan.li@librai.tech), and wait for verification before updating the leaderboard.
• Analyze the Results: In the leaderboard section of the GitHub page, view the model's performance on different tasks and analyze the model's strengths and weaknesses.

Application Scenarios of CMMLU

• Language Model Performance Evaluation: Used to test and compare the knowledge and reasoning abilities of language models in Chinese multi-task scenarios, helping optimize model architecture.
• Intelligent Tutoring in Education: Develop intelligent tutoring systems to provide students with multidisciplinary exercises and learning suggestions, improving learning outcomes.
• Intelligent Customer Service Optimization: Evaluate the model's knowledge comprehension ability in specific domains, optimize intelligent customer service systems, and improve customer service quality.
• Cultural Knowledge Dissemination: Use the CMMLU dataset to develop a cultural question-answering system, disseminate Chinese cultural knowledge, and promote cultural inheritance.
• Medical and Health Knowledge Assessment: Evaluate the knowledge comprehension ability of language models in the medical field, assist in developing medical consultation tools, and provide health advice.