What is Long-VITA
Long-VITA is a multimodal model open-sourced by Tencent Youtu Lab, Nanjing University, and Xiamen University. It can handle long text inputs of over 1 million tokens and performs excellently on short-text tasks. Long-VITA is based on staged training, progressively expanding the contextual understanding capabilities of vision and language, and supports multimodal inputs of images, videos, and text. Long-VITA uses a dynamic chunked encoder to process high-resolution images, and achieves support for unlimited-length inputs based on context-parallel distributed inference. Long-VITA is trained on open-source datasets, including long-text data such as comic summaries and movie plots, and achieves new state-of-the-art performance on multiple multimodal benchmarks.
Main features of Long-VITA
- Long-text processing capability: It can handle inputs of over 1 million tokens and supports multimodal tasks involving long text, long videos, and high-resolution images.
- Multimodal understanding: It supports inputs of images, videos, and text, and is suitable for tasks such as video understanding, high-resolution image analysis, and long-text generation.
- Context extension capability: Based on staged training, it progressively expands the model's context window while maintaining high performance on short-text tasks.
- Open-source data training: It is trained on open-source datasets without the need for internal data, lowering the development barrier.
- Scalability: It supports context-parallel distributed inference, can handle inputs of unlimited length, and is suitable for large-scale deployment.
Technical principles of Long-VITA
- Staged training：
- Vision-language alignment: The language model and vision encoder are frozen, and only the projector is trained to establish the initial connection between visual and language features.
- General knowledge learning: Multitask learning is performed using image-text data to improve the model's general knowledge understanding capability.
- Long-sequence fine-tuning: The context length is progressively expanded (from 128K to 1M), and long-text and video understanding data is added to optimize the model's ability to understand long content.
- Context-parallel distributed inference: Based on tensor parallelism and context parallelism technologies, it supports inference for unlimited-length inputs and solves the memory bottleneck in long-text processing.
- Dynamic chunked encoder: It uses a dynamic chunking strategy to efficiently process high-resolution images and supports inputs with different aspect ratios.
- Masked language modeling head: During the inference stage, it outputs logits based on the mask, significantly reducing memory usage and supporting large-scale long-text generation.
Project addresses of Long-VITA
- GitHub repository：https://github.com/VITA-MLLM/Long-VITA
- HuggingFace model library：https://huggingface.co/VITA-MLLM
- arXiv technical paper：https://arxiv.org/pdf/2502.05177v1
Application scenarios of Long-VITA
- Video content generation: Automatically generate video summaries, subtitles, or answer video-related questions.
- Image analysis: Assist in artistic creation, medical imaging diagnosis, or satellite image analysis.
- Long-text processing: Generate novels, academic reports, or document summaries.
- Intelligent dialogue: In customer service, education, or smart homes, interact with users through text, images, and videos.
- Real-time meeting assistance: Provide real-time translation, subtitles, and meeting minutes generation.