LLM-eval-survey is the official GitHub page of the survey paper "A Survey on Evaluation of Large Language Models," jointly maintained by researchers from institutions such as Jilin University, Microsoft Research, the Institute of Automation of the Chinese Academy of Sciences, Carnegie Mellon University, Westlake University, Peking University, the University of Illinois, and the Hong Kong University of Science and Technology. The repository is positioned as a compilation of papers and resources in the field of large language model evaluation, and the papers are organized according to the chapter structure of the survey.
In terms of purpose, the repository serves readers who wish to systematically understand the current state of large language model evaluation research, including researchers, developers, and practitioners interested in LLM evaluation. Its core purpose is to provide an index of evaluation literature classified by topic and to serve as a continuously updated vehicle for the survey paper. The repository description notes that because arXiv papers cannot be updated in real time, the latest updates are subject to this repository, and the paper may be updated in subsequent revisions.
Its main capabilities are reflected in literature organization and resource aggregation. The repository is divided according to the survey structure into sections such as "What to evaluate" and "Where to evaluate." Among them, "What to evaluate" covers directions such as natural language processing, robustness/ethics/bias and trustworthiness, social sciences, natural sciences and engineering, medical applications, agent applications, and other applications. The repository collects a large number of evaluation-related papers, such as multi-task, multilingual, and multimodal evaluation in the natural language processing direction, sentiment analysis evaluation, topic classification evaluation, and so on; the robustness direction involves out-of-distribution generalization, adversarial prompt robustness, the prompt benchmark PromptBench, and so on; the ethics and trustworthiness direction includes toxicity, bias, fairness evaluation, and so on; in addition, it also involves mathematical reasoning, logical reasoning, factual consistency, hallucination detection, multilingual evaluation, legal and social science applications, and so on.
In terms of usage, users can directly browse the folders and files in the GitHub repository and find paper lists according to the survey chapters; each paper entry usually includes authors, publication venue, and paper link. The repository also provides links to related projects, including PromptBench and LLM-eval for large language model robustness evaluation. The repository welcomes contributions through pull requests or issues, and contributors will be listed in the acknowledgments.
Applicable scenarios include: literature investigation when conducting research related to large language model evaluation; understanding representative work in specific evaluation dimensions such as robustness, bias, hallucination, and reasoning ability; tracking updates and newly added papers in the survey; and serving as an entry point for evaluation methodologystudy.
In terms of limitations, the materials do not provide details such as the repository's license information, specific update frequency, size of the maintenance team, or whether contributions in a specific format are accepted. The repository itself is a list of papers and resources and does not provide directly runnable evaluation code or tools; the availability of some evaluation benchmarks and datasets needs to be checked in the corresponding papers. In addition, because arXiv cannot be updated in real time, there may be a time lag between the repository and the paper version. Information such as fees, prices, and platform support is not yet provided in the materials.
