What is HourVideo
HourVideo is a long video understanding benchmark dataset launched by the team of Fei-Fei Li and Jiajun Wu at Stanford University. It contains 500 first-person perspective videos, ranging from 20 to 120 minutes in length, covering 77 types of daily activities, and can evaluate the long video understanding capabilities of multimodal models. The dataset is based on a series of tasks, such as summarization, perception, visual reasoning, and navigation, testing the model's ability to recognize and synthesize information from multiple temporal segments in videos, and promoting the development of long video understanding technology.
Main features of HourVideo
- Long video understanding evaluation: Based on videos containing up to one hour, HourVideo can test a model's ability to understand long-duration visual data streams.
- Multi-task test suite: The dataset contains multiple tasks, such as summarization, perception, visual reasoning, and navigation, comprehensively evaluating the model's performance in different aspects of video-language understanding.
- High-quality question generation: Based on 12,976 multiple-choice questions generated by human annotators and large language models (LLMs), providing standardized test questions.
- Model performance comparison: Comparison with other multimodal models, evaluating the performance of different models on long video understanding tasks.
Technical principles of HourVideo
- Video dataset construction: HourVideo selects 500 first-person perspective videos from the Ego4D dataset, covering daily activities, with video durations ranging from 20 to 120 minutes.
- Task suite design: Design a task suite containing multiple subtasks, each requiring the model to understand and reason about long-term dependencies in the video content.
- Question prototype development: Design question prototypes for each task, ensuring that correctly answering the questions requires recognizing and synthesizing information from multiple temporal segments of the video.
- Data generation process: Based on a multi-stage data generation process, including video filtering, question generation, human feedback optimization, blind screening, and expert optimization, to generate high-quality multiple-choice questions.
HourVideo project address
- arXiv technical paper：https://arxiv.org/pdf/2411.04998v1
Application scenarios of HourVideo
- Multimodal artificial intelligence research: Research and development of multimodal models that understand long-duration continuous video content.
- Autonomous agents and assistant systems: Help develop autonomous agents and virtual assistants that understand long-duration visual information and make decisions.
- Augmented reality (AR) and virtual reality (VR): Provide a technical foundation for creating immersive AR/VR experiences that can understand and adapt to user behavior.
- Video content analysis: Analyze and understand video content, such as surveillance videos, news reports, educational videos, etc., and extract key information and insights.
- Robot vision: Enable robots to understand long-sequence visual information and improve their navigation and operation capabilities in complex environments.