VSI-Bench - A Visual-Spatial Intelligence Benchmark Launched by Fei-Fei Li and Saining Xie's Team

What is VSI-Bench

VSI-Bench (Visual-Spatial Intelligence Benchmark) is a visual-spatial intelligence benchmark launched by Fei-Fei Li, Saining Xie, and their research team. Researchers built it to evaluate the capabilities of Multimodal Large Language Models (MLLMs) in spatial cognition and understanding. VSI-Bench contains over 5,000 question-answer pairs, covering nearly 290 real indoor scene videos, involving various environments such as residences, offices, and factories. VSI-Bench tasks are divided into configurational tasks (such as object counting, relative distance, etc.), measurement estimation (such as object size, room size, etc.), and spatiotemporal tasks (such as object appearance order), which can systematically test and improve the performance of MLLMs in visual-spatial intelligence.

Main Features of VSI-Bench

• Evaluating Visual-Spatial Intelligence: Quantitatively evaluates the visual-spatial intelligence of Multimodal Large Language Models (MLLMs), including the ability to perceive, understand, and remember spatial relationships.
• Benchmarking: Provides a standardized test set containing over 5,000 question-answer pairs, used for benchmarking and comparing the performance of different MLLMs on visual-spatial tasks.
• Task Diversity: Includes configurational tasks (object counting, relative distance, relative direction, route planning), measurement estimation (object size, room size, absolute distance), and spatiotemporal tasks (object appearance order), comprehensively covering multiple aspects of visual-spatial intelligence.
• Video Understanding: Based on video input, VSI-Bench tests MLLMs' understanding of continuous, temporal input, which is closer to the way humans observe the world than static images.
• Data Quality and Control: Manual review ensures data quality, eliminates ambiguity and incorrect annotations, and improves the reliability of test results..

Technical Principles of VSI-Bench

• Dataset Construction: Based on multiple public indoor 3D scene reconstruction datasets (such as ScanNet, ScanNet++, and ARKitScenes), the dataset provides high-fidelity video scans and object-level 3D annotations.
• Question-Answer Pair Generation: Automatically generates question-answer pairs based on metadata in the dataset (such as object categories, bounding boxes) and question templates, while manually annotating route planning tasks.
• Quality Control: Implements a manual review process to ensure questions are clear and unambiguous, and traces and corrects erroneous or ambiguous questions.
• Model Evaluation: Evaluates various video-supported MLLMs under zero-shot settings, tests with default prompts, and uses exact matching and fuzzy matching as the main evaluation metrics.
• Performance Metrics: Uses accuracy (ACC) for multiple-choice question (MCA) tasks, and introduces a new metric for numerical answer (NA) tasks—mean relative accuracy (MRA).
• Cognitive Map Generation: Prompts MLLMs to predict the center positions of objects in videos, generate cognitive maps, and evaluate the model's internal spatial representation and memory capabilities.

VSI-Bench Project Address

• Project Official Website：vision-x-nyu.github.io/thinking-in-space
• GitHub Repository：https://github.com/vision-x-nyu/thinking-in-space
• HuggingFace Model Library：https://huggingface.co/datasets/nyu-visionx/VSI-Bench
• arXiv Technical Paper：https://arxiv.org/pdf/2412.14171

Application Scenarios of VSI-Bench

• Robot Navigation and Interaction: In the field of robotics, evaluates and trains MLLMs to better understand spatial layouts and improve robots' navigation and obstacle avoidance capabilities in unknown environments.
• Augmented Reality (AR)/Virtual Reality (VR): In AR/VR applications, it helps MLLMs better understand the user's physical space environment and provide a more natural interactive experience.
• Autonomous Vehicles: Autonomous driving systems require precise spatial understanding capabilities to handle complex traffic scenarios, and are used in the development and testing of visual-spatial intelligence systems that support autonomous driving.
• Smart Home Systems: Smart home systems need to understand the layout of living spaces and the positions of objects. VSI-Bench helps train MLLMs so that they can better serve smart home control and automation.
• Interior Design and Architectural Planning: Evaluate the application of MLLMs in spatial planning and design, helping systems provide more reasonable layout suggestions.