What is LongVU
LongVU is a long video understanding model launched by the Meta AI team, based on a spatiotemporal adaptive compression mechanism. It addresses the challenge of processing long videos being limited by the context size of large language models (LLMs). LongVU is based on cross-modal queries and inter-frame dependencies, and it can reduce the number of video tokens while preserving the visual details of long videos. LongVU uses DINOv2 features to remove highly similar redundant frames, uses text-guided cross-modal queries for selective frame feature reduction, and performs spatial token compression based on temporal dependencies when necessary. LongVU can effectively process a large number of frames with little loss of visual information within a given context length.
Main Features of LongVU
- Spatiotemporal Adaptive Compression: LongVU processes long videos by reducing the number of video tokens, preserves key visual details in the video, and can process very long video content within a limited context length.
- Cross-Modal Queries: Based on text-guided cross-modal queries to selectively reduce the features of video frames, it can preserve the detailed information of the frames most relevant to the text query and reduce other frames to low-resolution token representations.
- Utilization of Inter-Frame Dependencies: Based on analyzing the temporal dependencies between video frames, LongVU can perform spatial token compression based on the dependencies, reducing the model's context length requirements.
- Long Video Understanding: LongVU can effectively process video input sampled at 1 fps, and can adaptively reduce the average number of tokens per frame for hour-long videos to 2, adapting to multimodal large language models (MLLMs) with an 8k context length.
Technical Principles of LongVU
- Temporal Compression Strategy: Uses DINOv2 features to identify and remove highly similar redundant video frames, reducing redundancy in the temporal dimension of the video.
- Selective Feature Reduction: Based on text-guided cross-modal queries, it retains complete tokens for frames relevant to the text query and applies spatial pooling to other frames, reducing redundancy in the spatial dimension.
- Spatial Token Compression: For extremely long videos, LongVU further compresses spatial tokens based on the temporal dependencies between frames. It calculates the similarity of spatial tokens between frames, removes the spatial tokens of subsequent frames that are too similar to the first frame, and reduces the amount of data the model needs to process.
- Multimodal Training: LongVU combines image-language pretraining and video-language fine-tuning, and uses large-scale video-text pairs for training to improve the model's performance in video understanding tasks.
Project Address of LongVU
- Project Official Website：vision-cair.github.io/LongVU
- GitHub Repository：https://github.com/Vision-CAIR/LongVU
- HuggingFace Model Library：https://huggingface.co/collections/Vision-CAIR/longvu-67181d2debabfc1eb050c21d
- arXiv Technical Paper：https://arxiv.org/pdf/2410.17434
- Online Experience Demo：https://huggingface.co/spaces/Vision-CAIR/LongVU
Application Scenarios of LongVU
- Video Content Analysis: LongVU is used to analyze long video content and extract key information, for example, identifying important events and scenes in surveillance videos, news reports, or documentaries.
- Video Search and Indexing: Based on understanding video content, LongVU helps build video search engines, allowing users to quickly locate relevant clips in videos with text queries.
- Video Content Generation: LongVU is used to generate descriptions, summaries, or subtitles for video content, improving the accessibility and inclusiveness of video content.
- Video Question Answering System: LongVU supports building video question-answering systems, where users ask questions about video content, and the system can understand the questions and provide accurate answers.
- Education and Training: In the field of education, LongVU is used to analyze instructional videos, extract key teaching points, and help students better understand and master the course content.