PerceptionBench - Moonshot AI's Open-Source Visual Perception Diagnostic Benchmark

What is PerceptionBench

PerceptionBench is Moonshot AI's open-source visual perception diagnostic benchmark. PerceptionBench derives 10 atomic visual perception capabilities from the failure cases of frontier models across 42 existing evaluation sets, and constructs 3,000 isolated verification questions, each testing only a single perception capability without requiring reasoning or external knowledge. Tests show that none of the current 16 mainstream multimodal models reach 60% accuracy, revealing that atomic-level visual perception remains an unsolved challenge for the industry.

Main Features of PerceptionBench

• Atomic Capability Diagnosis: Decomposes visual perception into 10 atomic capabilities, including visual relations, counting, attributes, depth, localization, composition, fine-grained recognition, context, OCR, and hallucination, precisely locating model breakdown points.
• Isolated Evaluation: Each question tests only a single atomic capability, avoiding interference from reasoning and knowledge, ensuring scores reflect true perception levels.
• Failure Case Attribution: Based on failure cases from 42 existing benchmarks, 1,800 atomic sub-questions are decomposed, extracting evaluation dimensions from real errors.
• Comprehensive Capability Profile: The Leaderboard displays detailed scores of models across 10 capabilities, revealing vastly different capability distributions under similar total scores.
• Open Source and Reproducible: Provides complete code, Hugging Face dataset, and paper, supporting community reproduction and extension.

Technical Principles of PerceptionBench

• Bottom-Up Error Taxonomy: By diagnosing the earliest failure points of frontier MLLMs across 42 existing benchmarks, an error classification tree is constructed, extracting 10 atomic capabilities of the perception branch.
• Difficulty Stratification and Capability Balance: From an internal pool of 17,000+ validation samples, 3,000 questions are sampled with balanced capability dimensions and difficulty stratification, of which 60% are sub-questions decomposed from failure cases and 40% are newly composed questions based on supplementary images.
• Short-Answer Unambiguous Design: All questions require short, clear answers, reducing evaluation uncertainty and making automated evaluation more reliable.
• Perception-Reasoning Decoupling: Question design ensures challenges stem from visual perception itself, rather than reasoning chains or domain knowledge, thereby isolating and measuring perception boundaries.

Follow on WeChat and reply "Open Source" to join theAI Open Source Project Exchange Group

How to Use PerceptionBench

• Clone Repository: Pull the MoonshotAI/PerceptionBench open-source code from GitHub.
• Download Dataset: Obtain 3,000 verification questions and annotation data from Hugging Face.
• Integrate Model: Connect the multimodal model to be evaluated to the evaluation framework, outputting answers in standard format.
• Run Evaluation: Execute the evaluation script to automatically calculate the model's accuracy on 10 atomic capabilities and overall.
• View Profile: Analyze model capability weaknesses against the Leaderboard to guide subsequent training optimization directions.

Core Advantages of PerceptionBench

• Precise Localization Rather Than Vague Scoring: Unlike traditional benchmarks that conflate perception, reasoning, and knowledge, PerceptionBench can precisely diagnose which perceptual stage a model fails at.
• Growing from Real Failures: Distilled from real failure cases across 42 mainstream benchmarks, ensuring evaluation stays close to actual model weaknesses.
• Exposing Hidden Capability Gaps: Models with similar total scores may differ vastly in atomic capabilities; PerceptionBench's capability profiles can reveal these gaps masked by total scores.
• Driving Perception-First Development: Shifting industry focus from score-chasing back to perceptual fundamentals, promoting more faithful and more consistent visual understanding capabilities.