What is MME-CoT
MME-CoT is a benchmark testing framework jointly launched by The Chinese University of Hong Kong, Shenzhen, The Chinese University of Hong Kong, ByteDance, Nanjing University, Shanghai Artificial Intelligence Laboratory, the University of Pennsylvania, Tsinghua University and other institutions for evaluating the Chain-of-Thought (CoT) reasoning capabilities of Large Multimodal Models (LMMs). It covers six domains, including mathematics, science, OCR, logic, spatiotemporal, and general scenes, and contains 1,130 questions, each annotated with key reasoning steps and reference image descriptions. The MME-CoT benchmark comprehensively evaluates models' reasoning capabilities based on three novel evaluation metrics—reasoning quality (logical soundness), robustness (interference with perception tasks), and efficiency (relevance of reasoning steps). Experimental results reveal some key issues in the CoT reasoning of current multimodal models, such as the inefficiency of reflection mechanisms and negative impacts on perception tasks.
Main features of MME-CoT
- Multi-domain reasoning capability evaluation: Covers six main domains (mathematics, science, OCR, logic, spatiotemporal, and general scenes), comprehensively evaluating models' reasoning capabilities in different scenarios.
- Fine-grained reasoning quality evaluation: Based on annotated key reasoning steps and reference image descriptions, evaluates the logical soundness (quality), robustness (interference with perception tasks), and efficiency (relevance of reasoning steps) of model reasoning.
- Revealing model reasoning issues: Reveals problems existing in the CoT reasoning of current multimodal models, such as the inefficiency of reflection mechanisms and interference with perception tasks.
- Providing references for model optimization: The evaluation results and analysis provided offer important references for the design and optimization of multimodal models, helping researchers improve models' reasoning capabilities.
Technical principles of MME-CoT
- Multimodal dataset construction: Constructs a high-quality multimodal dataset containing 1,130 questions, covering six domains and 17 subcategories. Each question is annotated with key reasoning steps and reference image descriptions, used to evaluate the model's reasoning process.
- Fine-grained evaluation metrics：
- Reasoning quality: Evaluates the logical soundness and accuracy of reasoning steps based on Recall and Precision.
- Reasoning robustness: Evaluates the impact of CoT on perception tasks and reasoning tasks based on Stability and Efficacy.
- Reasoning efficiency: Evaluates the relevance of reasoning steps and the effectiveness of reflection based on Relevance Rate and Reflection Quality.
- Reasoning step parsing and evaluation: Uses models such as GPT-4o to parse model outputs into steps such as logical reasoning, image descriptions, and background information, and evaluates the steps one by one.
MME-CoT project addresses
- Project official website：https://mmecot.github.io/
- GitHub repository：https://github.com/CaraJ7/MME-CoT
- HuggingFace model library：https://huggingface.co/datasets/CaraJ/MME-CoT
- arXiv technical paper：https://arxiv.org/pdf/2502.09621
Application scenarios of MME-CoT
- Model evaluation and comparison: As a standardized benchmark, used to evaluate and compare the performance of different multimodal models in terms of reasoning quality, robustness, and efficiency.
- Model optimization: Based on fine-grained evaluation metrics, reveal problems in the model's reasoning process and provide direction for model optimization.
- Multimodal Research: Provide tools for multimodal reasoning research, helping explore new model architectures and training methods.
- Education and Training: Used in the education field to help students and researchers understand the reasoning logic of multimodal models.
- Industry Applications: In fields such as intelligent education, autonomous driving, and medical imaging, evaluate and improve the model's performance in practical applications.