What is SlowFast-LLaVA-1.5
SlowFast-LLaVA-1.5 (abbreviated as SF-LLaVA-1.5) is an efficient video large language model designed for long video understanding. Based on a dual-stream (SlowFast) mechanism, it balances processing more input frames with reducing the number of tokens per frame, can capture detailed spatial features, and can efficiently process long temporal motion information. The model includes models ranging from 1B to 7B parameters, trained on a simplified two-stage training pipeline and a high-quality mixture of public datasets. The model performs excellently in long video understanding tasks, can maintain strong performance in image understanding tasks, shows significant advantages on small-scale models, and provides strong support for lightweight and mobile-friendly video understanding applications.
Main Features of SlowFast-LLaVA-1.5
- Efficient Long Video Understanding: Can efficiently process complex spatiotemporal information in long videos, capture long temporal context, and is suitable for understanding and analyzing long video content.
- Multimodal Fusion: Combines video and image inputs, provides comprehensive visual understanding capabilities, supports joint training of video and image tasks, and improves the model's performance across various visual tasks.
- Lightweight and Mobile-Friendly: The model design focuses on being lightweight, suitable for deployment in resource-constrained environments such as mobile devices, and meets the needs of edge computing and real-time applications.
- Powerful Reasoning Capability: Based on a large language model (LLM) architecture, it has powerful natural language processing capabilities, can generate detailed descriptions of video content, answer video-related questions, and more.
- Scalability: Provides models ranging from 1B to 7B parameters. Users can choose models of appropriate size according to specific needs to achieve a balance between performance and resources.
Technical Principles of SlowFast-LLaVA-1.5
- Dual-Stream Mechanism (SlowFast)：
- Slow stream: Processes video at a lower frame rate, captures detailed static spatial features, and is suitable for processing key frame information in videos.
- Fast stream: Processes video at a higher frame rate, but with fewer features per frame, focuses on capturing motion information, and is suitable for processing dynamic changes in videos.
- Two-Stage Training Pipeline：
- First Stage (Image Understanding): Uses image data for supervised fine-tuning (SFT), provides the model with general knowledge and reasoning capabilities, and ensures that the model has a good foundational performance on image tasks.
- Second Stage (Joint Video and Image Training): Builds on the first stage by combining image and video data for joint training, further improving the model's performance in video understanding tasks while maintaining strong understanding capability for image tasks.
- High-Quality Data Mixture：
- Image data: Includes general, text-rich, and knowledge-based datasets, such as LLaVA Complex Reasoning, ShareGPT-4v, Coco Caption, and others.
- Video data: Covers large-scale video data and long video understanding tasks, such as LLaVA-Hound, ShareGPT4Video, ActivityNet-QA, and others, ensuring that the model performs excellently across various video tasks.
- Model Architecture: Uses Oryx-ViT as the vision encoder,Qwen2.5series as the language model (LLM), and designs different projectors for video and image inputs to adapt to the input characteristics of different modalities.
SlowFast-LLaVA-1.5 project address
- GitHub repository：https://github.com/apple/ml-slowfast-llava
- arXiv technical paper：https://arxiv.org/html/2503.18943v1
Application scenarios of SlowFast-LLaVA-1.5
- Long video content understanding and summarization: Automatically generate summaries of long videos, helping users quickly grasp the core content of the video and save time.
- Video question answering system: Users ask questions in natural language, and the model generates accurate answers based on the long video content, improving the interactive experience.
- Video editing and creation: Automatically edit key segments from long videos to generate short videos, improving creation efficiency.
- Video surveillance and analysis: Identify abnormal behavior in surveillance videos in real time, such as gatherings of people, improving the level of surveillance intelligence.
- Multimedia content recommendation: Recommend related long video content based on users' historical viewing records, enhancing user stickiness.