We-Math is a benchmark specifically designed to explore the problem-solving principles of large multimodal models (LMMs) in visual mathematical reasoning, with the core goal of evaluating whether models achieve human-like mathematical reasoning. Unlike existing benchmarks, We-Math is built around textbook knowledge units, decomposes composite problems into sub-problems based on the required knowledge concepts, and adopts methods such as a hierarchical knowledge structure, knowledge-based reasoning evaluation, and knowledge concept augmentation.
Purpose We-Math is mainly used to evaluate the performance of LMMs in visual mathematical reasoning, focusing not only on the correctness of the final answer but also deeply analyzing the intrinsic problems in the model's problem-solving process, such as insufficient knowledge mastery, lack of generalization ability, complete mastery, or rote memorization. It aims to provide a new evaluation path for research on visual mathematical reasoning in LMMs.
Main Capabilities
- Hierarchical knowledge structure: strictly follows the knowledge in mathematics textbooks, has a rigorous hierarchical and multi-category architecture, ensures the independence of knowledge concepts within the same level, and establishes logical relationships among concepts at different levels.
- Knowledge-based reasoning evaluation: decomposes complex mathematical problems into more manageable sub-problems and uses diverse measurement dimensions for detailed evaluation.
- Knowledge concept augmentation: introduces descriptions of 67 knowledge concepts from Wikipedia and textbooks to provide necessary knowledge support for the reasoning process of LMMs.
- Four-dimensional evaluation metrics: including insufficient knowledge (IK), insufficient generalization (IG), complete mastery (CM), and rote memorization (RM), used to hierarchically evaluate problems in the reasoning process of LMMs.
Usage We-Math provides a benchmark dataset (the testmini subset contains 1,740 examples), and recommends using VLMEvalKit for evaluation, with Strict Score as the reported metric. More results can be viewed on OpenCompass. Users can submit results to the leaderboard by sending the results JSON file to the specified email address.
Applicable Scenarios We-Math is suitable for research institutions, AI developers, and researchers interested in the mathematical reasoning capabilities of multimodal models, for evaluating and improving the performance of LMMs in visual mathematical reasoning, especially exploring the model's performance in knowledge mastery, generalization ability, and reasoning reliability.
Limitations Currently, the materials do not provide specific information about We-Math's fees, pricing, platform support, etc. Its evaluation mainly targets visual mathematical reasoning and may not be applicable to other types of reasoning tasks. In addition, the construction of the benchmark relies on textbook knowledge and may not cover all mathematicaldomains.
