What is MV-MATH
MV-MATH is a new benchmark dataset proposed by the Institute of Automation, Chinese Academy of Sciences, to evaluate the mathematical reasoning ability of multimodal large language models (MLLMs) in multi-visual scenarios. The dataset contains 2,009 high-quality mathematical problems, each combining multiple images and text to form interleaved text-image multi-visual scenarios. The problems are divided into three types: multiple-choice, fill-in-the-blank, and multi-step question-answering, covering 11 mathematical domains, including analytic geometry, algebra, metric geometry, combinatorics, transformation geometry, logic, solid geometry, arithmetic, combinatorial geometry, descriptive geometry, and statistics, and are divided into three difficulty levels.
Main features of MV-MATH
- Multi-visual scenario reasoning: Each problem contains multiple images (2-8), interwoven with text to form complex scenarios, closer to real-world mathematical problems, and can comprehensively evaluate the model's reasoning ability in handling multi-visual information.
- Diverse mathematical domain coverage: Covers 11 mathematical domains (such as analytic geometry, algebra, solid geometry, etc.) and 3 difficulty levels, enabling comprehensive evaluation of the model's reasoning performance across different domains.
- Image correlation analysis: Introduces image correlation labels for the first time, dividing the dataset into a mutually dependent set (MD) and an independent set (ID), allowing separate evaluation of the model's reasoning ability when handling related and independent images.
- Educational applications: Derived from real K-12 educational scenarios, it can be used to develop intelligent tutoring systems to help students solve complex mathematical problems through a combination of text and images.
- Research tool: Provides a standardized evaluation tool for multimodal learning research, helping researchers identify and improve performance gaps of models in mathematical reasoning.
- High-quality annotation: Each sample is cross-validated by at least two annotators, including questions, answers, detailed analysis, and image correlation annotations, providing detailed information for model evaluation.
- Real problem collection: All problems are sourced from real scenarios, ensuring the practicality and reliability of the dataset.
Technical principles of MV-MATH
- Mutually Dependent Set (MD): Images are interrelated, and understanding one image requires referring to other images.
- Independent Set (ID): Images are mutually independent and can be interpreted individually.
MV-MATH project address
- Project official website：https://eternal8080.github.io/MV-MATH.github.io/
- Github repository：https://github.com/eternal8080/MV-MATH
- arXiv technical paper：https://arxiv.org/pdf/2502.20808
- HuggingFace dataset：https://huggingface.co/datasets/PeijieWang/MV-MATH
Application scenarios of MV-MATH
- Intelligent tutoring systems: The MV-MATH dataset can be used to develop intelligent tutoring systems to help students solve complex mathematical problems through a combination of text and images.
- Multimodal learning research: MV-MATH provides a standardized evaluation tool for multimodal learning research. Researchers can use the dataset to evaluate the mathematical reasoning ability of multimodal large language models (MLLMs) in multi-visual scenarios, promoting the development of multimodal learning technology.
- Performance gap analysis: Through extensive experiments, researchers can identify and improve performance gaps of models in mathematical reasoning.
- Multi-image reasoning tasks: The dataset can be used to develop and optimize solutions for multi-image reasoning tasks, handling multiple images and text information in complex mathematical problems.
- Automated assessment system: The dataset can be used to evaluate and optimize automated examination systems, ensuring their accuracy and reliability when processing multimodal inputs.