ACLUE

Purpose

ACLUE (Ancient Chinese Language Understanding Evaluation) is an evaluation benchmark focused on ancient Chinese language understanding, used to assess the capabilities of large language models in ancient Chinese. It aims to help researchers test and improve models' understanding of ancient Chinese.

Main Capabilities

ACLUE consists of 15 tasks, covering multiple domains:

• Vocabulary: T1 Ancient Chinese single-character polysemy, T2 interchangeable characters, T3 Ancient Chinese named entity recognition
• Syntax: T4 Ancient Chinese sentence segmentation
• Semantics: T5 couplets, T6 prediction of the preceding and following lines of ancient poetry
• Reasoning: T7 ancient poetry quality assessment, T8 ancient Chinese reading comprehension, T9 appreciation of ancient poetry, ci, and qu, T10 poetry sentiment classification
• Knowledge: T11 ancient Chinese knowledge, T12 traditional Chinese studies common knowledge, T13 medical ancient Chinese, T14 ancient literature knowledge, T15 ancient phonology

All tasks are multiple-choice questions, with 4 options per question and only one correct answer. The questions cover a broad time range from the Xia Dynasty (2070 BCE) to the Ming Dynasty (1368 CE).

Usage

• The data is provided as comma-separated .csv files, located in the data directory, divided into the data/dev and data/test directories.
• The preprocessing code is located in the src/utils directory.
• The evaluation code for each model is located in the src directory, and the run scripts are located in the script directory.
• Examples of the data after adding direct-answer prompts are provided.
• Researchers can use ACLUE to test models and can contribute model results by contacting or submitting a pull request.

Applicable Scenarios

Suitable for evaluating the performance of large language models in ancient Chinese understanding, and suitable for academic research in fields such as natural language processing, ancient Chinese studies, and language model evaluation.

Limitations

• All tasks are in multiple-choice question format.
• The data time range is limited (Xia Dynasty to Ming Dynasty).
• The project uses the MIT License, and the dataset uses the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
• Other specific limitations are not yet provided.