What is VideoChat-Flash
VideoChat-Flash is a multimodal large language model (MLLM) for long video modeling jointly developed by Shanghai Artificial Intelligence Laboratory, Nanjing University, and other institutions. The model efficiently processes long videos through hierarchical compression technology (HiCo), significantly reducing computation while retaining key information. It adopts a multi-stage short-to-long learning scheme and combines it with the real-world long video dataset LongVid to further enhance its understanding of long videos.
Main Features of VideoChat-Flash
- Long video understanding capability: VideoChat-Flash effectively processes long videos through hierarchical compression technology (HiCo), and can handle video content lasting up to several hours. In the "Needle In A Haystack" (NIAH) task, it achieved 99.1% accuracy on 10,000 frames (about 3 hours of video) for the first time among open-source models.
- Efficient model architecture: By encoding each video frame into only 16 tokens, the model significantly reduces computation, with inference speed 5-10 times faster than previous-generation models. The multi-stage short-to-long learning scheme, combined with the real-world long video dataset LongVid, further improves the model's performance.
- Powerful video understanding capability: VideoChat-Flash performs excellently on multiple long-video and short-video benchmarks, surpassing other open-source MLLM models and even exceeding larger-scale models in some tasks.
- Multi-hop context understanding: VideoChat-Flash supports multi-hop NIAH tasks, and can track multiple related image sequences in long videos, further enhancing its understanding of complex contexts.
Technical Principles of VideoChat-Flash
- Hierarchical compression technology (HiCo): HiCo is one of the core innovations of VideoChat-Flash, designed to efficiently handle redundant visual information in long videos.
- Segment-level compression: Divide long videos into shorter segments and encode each segment independently.
- Video-level compression: Based on segment encoding, further compress the contextual information of the entire video to reduce the number of tokens that need to be processed.
- Semantic association optimization: Combine the semantic information of the user query to further reduce unnecessary video tokens, thereby lowering computation.
- Multi-stage learning scheme: VideoChat-Flash adopts a multi-stage learning scheme from short videos to long videos, gradually improving the model's ability to understand long contexts.
- Initial stage: Use short videos and their annotations for supervised fine-tuning to establish the model's basic understanding capability.
- Expansion stage: Gradually introduce long video data to train the model to handle more complex contexts.
- Mixed corpus training: Finally, train on a mixed corpus containing short videos and long videos to achieve comprehensive understanding of videos of different lengths.
- Real-world long video dataset LongVid: To support model training, the research team constructed the LongVid dataset, containing 300,000 hours of real-world long videos and 200 million words of annotations. This dataset provides rich training material for the model, enabling it to better adapt to long video understanding tasks.
- Model architecture: The architecture of VideoChat-Flash consists of three main parts: a visual encoder, a vision-language connector, and a large language model (LLM). Through this hierarchical architecture, the model can efficiently encode video content into compact token sequences and perform long-context modeling through the LLM.
Project address of VideoChat-Flash
- GitHub repository：https://github.com/OpenGVLab/VideoChat-Flash
- arXiv technical paper：https://arxiv.org/pdf/2501.00574
Application scenarios of VideoChat-Flash
- Video captioning and translation: The model can generate detailed and accurate video captions, suitable for multilingual translation and accessible caption generation, helping viewers better understand video content.
- Video question answering and interaction: VideoChat-Flash supports natural language question answering based on video content. Users can obtain key information in videos by asking questions, such as movie plot analysis, knowledge points in documentaries, and more.
- Embodied AI and robot learning: In the field of embodied AI, VideoChat-Flash can help robots learn complex tasks through long-duration egocentric videos, such as making coffee, by analyzing key events in the videos to guide the robot in completing tasks.
- Sports video analysis and highlight generation: The model can analyze sports match videos, extract key events, and generate highlights, helping viewers quickly understand the exciting moments of the match.
- Surveillance video analysis: VideoChat-Flash can process long-duration surveillance videos, identify and track key events, and improve the efficiency and accuracy of surveillance systems.