What is WorldSense
WorldSense is a benchmark launched by Xiaohongshu and Shanghai Jiao Tong University for evaluating the comprehensive understanding ability of multimodal large language models (MLLMs) toward visual, auditory, and textual inputs in real-world scenarios. WorldSense contains 1,662 audio-video synchronized diverse videos, covering 8 major domains and 67 fine-grained subcategories, as well as 3,172 multiple-choice question-answer pairs, involving 26 different cognitive tasks. WorldSense emphasizes the tight coupling of audio and video information, and all questions require both modalities to derive the correct answer. WorldSense's high-quality annotations were manually completed by 80 expert annotators and underwent multiple rounds of verification to ensure annotation accuracy and reliability.
Main Features of WorldSense
- Multimodal Collaborative Evaluation: Emphasizes the tight coupling of audio and video information, and designs questions that require both visual and auditory information to answer correctly. It rigorously tests the model's understanding ability under multimodal inputs, ensuring that the model can effectively integrate information from different modalities to achieve accurate understanding.
- Diverse Video and Task Coverage: WorldSense contains 1,662 audio-video synchronized diverse videos, covering 8 major domains and 67 fine-grained subcategories, as well as 3,172 multiple-choice question-answer pairs, covering 26 different cognitive tasks.
- High-Quality Annotation and Verification: All question-answer pairs were manually annotated by 80 expert annotators and underwent multiple rounds of verification, including manual review and automatic model verification, ensuring annotation accuracy and reliability.
Technical Principles of WorldSense
- Multimodal Input Processing: WorldSense requires the model to simultaneously process video, audio, and text inputs. The synchronization of video and audio ensures that the model can capture the association between visual and auditory information and understand the scene more comprehensively. Multimodal input processing capability is key to evaluating whether a model can handle complex environments like humans.
- Task Design and Annotation:Based on carefully designed question-answer pairs, it ensures that each question requires the integration of multimodal information to derive the correct answer. The annotation process involves multiple rounds of manual review and automatic verification to ensure the reasonableness of the questions and the accuracy of the annotations.
- Multimodal Fusion and Reasoning: Based on diverse task designs, it evaluates the model's multimodal understanding ability at different levels, including basic perception (such as detection of audio and visual elements), understanding (grasping multimodal relationships), and reasoning (such as causal inference and abstract thinking). The multi-level evaluation method can comprehensively test the model's multimodal fusion and reasoning abilities.
- Data Collection and Filtering: WorldSense's data collection process includes filtering video clips with strong audio-visual associations from large-scale video datasets, and ensuring the quality and diversity of video content based on manual review, ensuring that the benchmark covers a wide range of real-world scenarios.
WorldSense Project Address
- Project Official Website：https://jaaackhongggg.github.io/WorldSense/
- GitHub Repository：https://github.com/JaaackHongggg/WorldSense
- HuggingFace Model Library：https://huggingface.co/datasets/honglyhly/WorldSense
- arXiv Technical Paper：https://arxiv.org/pdf/2502.04326
Application Scenarios of WorldSense
- Autonomous Driving: Helps autonomous driving systems better understand visual and auditory information in traffic environments and improve decision-making accuracy.
- Intelligent Education: Evaluates and improves educational tools' understanding of teaching video content, assisting personalized learning.
- Intelligent Surveillance: Enhance the monitoring system's ability to perceive and understand visual and audio information in videos, and improve security detection effectiveness.
- Intelligent customer service: Evaluate the intelligent customer service system's ability to understand users' voice, facial expressions, and text input, and optimize the interaction experience.
- Content creation: Help multimedia content creation and analysis systems understand video content more intelligently, and improve creation and recommendation efficiency.