What is Apollo
Apollo is a large multimodal model (LMMs) jointly launched by Meta and Stanford University, focusing on video understanding. Based on systematic research, Apollo reveals the key drivers of video understanding in LMMs and introduces the "Scaling Consistency" phenomenon, whereby design decisions on smaller models can effectively scale to larger models. The Apollo project introduces ApolloBench, an efficient video understanding evaluation benchmark, and a series of advanced Apollo models that demonstrate excellent performance across different scales, especially in processing long videos, efficiently understanding videos up to several hours long. The Apollo-3B and Apollo-7B models surpass models with more parameters in multiple benchmarks, marking new progress in video LMMs research.
Main Features of Apollo
- Improved Video Understanding Capability: Apollo improves the understanding of video content based on large multimodal models (LMMs), including the capture and processing of spatiotemporal features.
- Design Space Exploration: Systematically explores the design space of video LMMs, including video sampling, architecture, data composition, training schedule, etc.
- Performance Optimization: Based on the discovery of the "Scaling Consistency" phenomenon, Apollo can make design decisions on smaller models and effectively transfer these decisions to larger models, reducing computational costs.
- Efficient Evaluation: Introduces ApolloBench, an efficient video understanding evaluation benchmark, to quickly and accurately evaluate model performance.
- Model Family: Introduces the Apollo model family, which demonstrates excellent performance across different scales, especially in processing long videos.
Technical Principles of Apollo
- Scaling Consistency: Finds that effective design and training decisions on smaller models can transfer to larger models, called "Scaling Consistency".
- Video Sampling Strategy: Apollo finds that frame rate (fps) sampling outperforms uniform sampling during training and inference.
- Encoder Selection: Determines the best single encoder and encoder combination based on experiments to achieve optimal video representation.
- Token Resampling: Uses Perceiver Resampler for resampling visual tokens, reducing the number of tokens per frame and improving model efficiency.
- Data Mixing: Studies the mixing ratios of different text, image, and video data, finding that including an appropriate amount of text data and maintaining a slightly video-heavy mix yields the best performance.
- Training Schedule: Based on a multi-stage training schedule, gradually unfreezes different components to optimize model training dynamics.
Apollo Project Address
- Project Website：apollo-lmms
- GitHub Repository：https://github.com/Apollo-LMMs/Apollo
- HuggingFace Model Library：https://huggingface.co/Apollo-LMMs
- arXiv Technical Paper：https://arxiv.org/pdf/2412.10360
Application Scenarios of Apollo
- Video Content Analysis: Analyzes video content, identifying objects, scenes, and events in videos, providing support for automatic annotation and indexing of video content.
- Video Search and Recommendation: Based on understanding video content, improves video search engines, providing more accurate search results and personalized video recommendations.
- Intelligent Surveillance: In the field of security surveillance, identifies abnormal behavior and provides real-time analysis and response.
- Autonomous Driving: Apollo's video understanding capability is used in autonomous driving systems to help vehicles better understand their surroundings.
- Education and Training: In the field of education, analyze instructional videos and provide personalized learning suggestions and feedback.