SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

UNO-Bench - The Omni-modal Large Model Evaluation Benchmark Launched by Meituan LongCat

Author: AI小集 · Original source: AI工具集

What is UNO-Bench

UNO-Bench is an omni-modal large model evaluation benchmark launched by the Meituan LongCat team. UNO-Bench addresses the shortcomings of existing evaluation systems by constructing high-quality, diverse data to precisely measure models' uni-modal and omni-modal capabilities. The benchmark is the first to validate the "composition law" of omni-modal large models, revealing the complex relationship between uni-modal and omni-modal capabilities. UNO-Bench's innovative multi-step open-ended questions and efficient data compression algorithms improve evaluation discrimination and efficiency, providing a scientific assessment tool for advancing the development of omni-modal large models.

UNO-Bench

Main Features of UNO-Bench

  • Accurate Evaluation of Model Capabilities: Through high-quality, diverse datasets, it simultaneously measures model performance on uni-modal and omni-modal tasks involving images, audio, video, and text.
  • Revealing Capability Composition Laws: It is the first to validate the "composition law" of omni-modal large models, revealing the complex relationship between uni-modal and omni-modal capabilities and providing theoretical support for model optimization.
  • Innovative Evaluation Methods: It introduces multi-step open-ended questions (MO), which can effectively assess capability decay in complex reasoning tasks and precisely distinguish models' reasoning depth.
  • Efficient Data Management: Through clustering-guided stratified sampling, it significantly reduces evaluation costs while maintaining high consistency in model rankings.
  • Support for Multimodal Fusion Research: It provides researchers with a unified evaluation framework, promotes the development of omni-modal large models, and reserves space for the emergence of stronger future models.

Technical Principles of UNO-Bench

  • Unified Capability System: Model capabilities are deconstructed into a perception layer and a reasoning layer. The perception layer covers basic recognition, cross-modal alignment, and other capabilities, while the reasoning layer includes high-order tasks such as spatial reasoning and temporal reasoning. This two-dimensional framework provides a clear blueprint for data construction and model evaluation.
  • High-Quality Data Construction：
    • Data Collection and Annotation: Through manual annotation and multiple rounds of quality inspection, it ensures the high quality and diversity of the data. More than 90% of the data is privately original, avoiding data contamination.
    • Cross-Modal Solvability: Through modality ablation experiments, it ensures that more than 98% of questions must rely on multimodal information to be answered, avoiding redundancy in uni-modal information.
    • Separation and Recombination of Audio and Visual: Audio content is designed independently and manually combined with visual materials, breaking information redundancy and forcing models to perform genuine cross-modal fusion.
    • Data Optimization and Compression: It adopts clustering-guided stratified sampling to select representative samples from large-scale data, reducing evaluation costs while maintaining consistency in model rankings.
  • Innovative Evaluation Methods: Complex reasoning tasks are decomposed into multiple sub-questions, requiring models to provide open-ended text answers, and expert-weighted scoring is used to precisely evaluate models' reasoning capabilities. Through refined question types and multiple rounds of annotation iteration, automatic scoring is achieved for various question types, with scoring accuracy reaching 95%.
  • Validation of the Composition Law: Through regression analysis and ablation experiments, it reveals that omni-modal performance is not a simple linear superposition of uni-modal capabilities but follows a power-law synergy pattern. This nonlinear relationship provides a new analytical paradigm for evaluating model fusion efficiency.

UNO-Bench Project Address

  • Project Website：https://meituan-longcat.github.io/UNO-Bench/
  • GitHub Repository：https://github.com/meituan-longcat/UNO-Bench
  • HuggingFace Model Hub：https://huggingface.co/datasets/meituan-longcat/UNO-Bench
  • arXiv Technical Paper：https://arxiv.org/pdf/2510.18915

Application Scenarios of UNO-Bench

  • Model Development and Optimization: Provides standardized evaluation tools for developers, helping optimize model architecture and enhance multimodal fusion capabilities.
  • Industry Application Evaluation: Used in fields such as intelligent customer service and autonomous driving to evaluate model performance in multimodal interaction scenarios and optimize user experience.
  • Academic Research and Competitions: Serves as a unified academic evaluation benchmark, supporting model performance comparison and multimodal competitions, and driving technological breakthroughs.
  • Product Development and Market Evaluation: Helps enterprises evaluate product features and market competitiveness, providing a scientific basis for multimodal product development.
  • Cross-Modal Application Development: Supports fields such as multimedia content creation and intelligent security, improving the performance and reliability of multimodal applications.