SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

CL-bench - A model learning capability evaluation benchmark launched by Tencent's Yao Shunyu team

Author: AI小集 · Original source: AI工具集

What is CL-bench

CL-bench is a Context learning capability evaluation benchmark jointly launched by Tencent Hunyuan and Fudan University. It can measure the ability of large language models to learn and apply knowledge in real time from entirely new context information. The CL-bench benchmark contains 500 carefully constructed complex scenarios by experts and 1,899 tasks, covering four major categories: domain knowledge reasoning, rule system application, procedural task execution, and experience discovery and simulation. Tests show that the strongest modelGPT-5.1has a solve rate of only 23.7%, revealing the core bottleneck that current AI "cannot learn on the spot" and pointing out a new direction for model optimization.

CL-bench

CL-bench's main features

  • Real-time learning capability evaluation: Supports evaluating the ability of large language models to learn and apply knowledge in real time from entirely new context information.
  • Large-scale test set construction: Constructs a test set containing 500 complex scenarios, 1,899 tasks, and 31,607 validation criteria, covering four types of real-world scenarios: domain knowledge reasoning, rule system application, procedural task execution, and experience discovery and simulation.
  • Contamination-free data design: Adopts a contamination-free design to ensure data novelty, using fictional creation, modification of existing content, and integration of niche emerging content to prevent models from relying on memorization rather than learning to solve problems.
  • Sequence-dependent validation: Validates the model's multi-round reasoning ability in sequence-dependent tasks; 51.1% of tasks require subsequent reasoning based on the results of prior interactions.
  • Multi-dimensional evaluation system: Provides a multi-dimensional evaluation system, with an average of 16.6 evaluation criteria per task, comprehensively examining the model's understanding and application accuracy of Context from multiple angles.

CL-bench's technical principles

  • Self-contained Context environment: The core technical principle of CL-bench is to construct a fully self-contained Context environment, ensuring that all information needed to solve the task is explicitly provided within the Context itself, without requiring external retrieval and without supporting hidden assumptions. It forces the model to draw nourishment from the new information in the current input rather than invoking internal knowledge sealed during the pretraining stage, truly reflecting the model's Context learning ability rather than memorization ability.
  • Triple contamination-free strategy: To achieve contamination-free evaluation, CL-bench adopts a triple technical strategy: expert creation of completely fictional content, such as designing a complete legal system for a fictional country or creating a new programming language with unique grammar; systematic modification of real-world content to create variants, including changing historical events, altering scientific definitions, or modifying technical documents; and incorporating niche or recently emerging content with extremely low representation in pretraining datasets, such as cutting-edge research findings and newly released product manuals.
  • Complexity and verifiability design: In task design, CL-bench emphasizes high complexity and sequence dependency. 51.1% of tasks are designed with multi-round interaction mechanisms, where the solutions to subsequent tasks depend on the results of prior interactions, significantly increasing task difficulty and simulating real work scenarios. At the same time, each task is set with fully verifiable evaluation criteria, with an average of 63.2 validation criteria associated with each Context, ensuring a comprehensive examination of model performance through multi-angle evaluation and avoiding evaluation bias caused by a single metric.

CL-bench's project address

  • Project official website：https://www.clbench.com/
  • GitHub repository：https://github.com/Tencent-Hunyuan/CL-bench
  • HuggingFace model library：https://huggingface.co/datasets/tencent/CL-bench

CL-bench's application scenarios

  • AI model capability evaluation: Provide research institutions and enterprises with a standardized Context learning capability evaluation tool, accurately identify the capability shortcomings of models in real-world application scenarios, and guide the direction of model optimization.
  • New Model R&D Validation: In the large language model R&D process, serve as a core testing step to verify whether a new version of a model truly has a breakthrough in learning from dynamic information, rather than merely improving parameter memory capability.
  • Industry Solution Selection: Help enterprise users evaluate the Context learning performance of different commercial models in specific business scenarios, and provide an objective basis for choosing AI solutions suitable for their own needs.
  • Education and Training: As a teaching case and experimental platform, help AI practitioners understand the essential difference between Context learning and parameter learning, and cultivate model design and tuning capabilities for real application scenarios.
  • Academic Research Benchmark: Provide academia with a unified research benchmark for Context learning, promote the formation of comparable and reproducible research results in related fields, and accelerate the overall progress of Context learning theory and technology.