What is LinGen
LinGen is a new text-to-video generationframework jointly launched by Princeton University and Meta. The framework is based on the linear-complexity MATE module (including the MA-branch and TE-branch), replacing the quadratic-complexity self-attention module in traditional Diffusion Transformers, enabling efficient generation of high-resolution, minute-level videos on a single GPU. LinGen significantly reduces computational cost while maintaining high-quality video output, outperforming existing state-of-the-art models in both video quality and generation efficiency, opening new paths for long video generation and real-time interactive video applications.
Main Features of LinGen
- High-resolution video generation: Supports generating high-resolution videos (such as 512p, 1024p), meeting the needs of high-quality content creation.
- Long-duration video generation: Supports generating minute-level videos, breaking through the limitation of traditional models that can only generate short videos (10-20 seconds).
- Linear computational complexity: Based on the MATE module with linear complexity, it significantly reduces computational cost, making video generation more efficient and suitable for running on a single GPU.
- High-quality video output: The generated videos are comparable to existing state-of-the-art models in visual quality and text alignment, while maintaining inter-frame consistency.
- Real-time interactive video generation: LinGen provides the potential for real-time interactive video generation and editing, suitable for various dynamic content creation scenarios.
Technical Principles of LinGen
- MA-branch (Multi-scale Attention Branch)：
- Bidirectional Mamba2 module: Mamba2 is an efficient linear-complexity sequence model; based on a bidirectional design, it captures bidirectional dependencies in the sequence.
- Rotary Major Scan（RMS）: Rearranges the 3D video token tensor based on different scanning methods (such as spatial row-major, spatial column-major, temporal row-major, temporal column-major), enhancing short-range correlations while reducing computational latency.
- Review Tokens: Adds an average-pooled token sequence before sequence processing to provide a global overview of the entire sequence, enhancing long-range correlations.
- TE-branch (Temporal Attention Branch): Divides the 3D video token tensor into small windows and computes self-attention within the windows. TESA can capture correlations between spatially adjacent and temporally medium-distance tokens. The windows shift alternately between different layers, expanding the receptive field and enhancing video consistency.
- Linear complexity: Based on the design of the MATE module, LinGen's computational complexity has a linear relationship with the number of pixels in the generated video, rather than the traditional quadratic relationship. This enables LinGen to significantly reduce computational cost and improve generation efficiency while maintaining high-quality output.
- Training strategy: LinGen uses a progressive training strategy, first pre-training on low-resolution text-to-image tasks, then gradually increasing video resolution and length for pre-training. In the text-to-video pre-training stage, text-image pairs are combined for mixed training to improve the consistency of generated videos. Based on fine-tuning on high-quality video datasets, the quality of generated videos is further improved.
LinGen Project Address
- Project official website：https://lineargen.github.io/
- GitHub repository：https://github.com/jha-lab/LinGen
- arXiv technical paper：https://arxiv.org/pdf/2412.09856
Application Scenarios of LinGen
- Content creation: Rapidly generate high-quality video content, such as advertisements, movies, TV series, etc., significantly shortening the creation cycle and cost.
- Entertainment Industry: Generate cutscenes and background videos in games, enhancing the game's visual effects and immersion.
- Education and Training: Generate educational videos, such as course explanations and experiment demonstrations, improving the fun and interactivity of teaching; generate training videos to help employees quickly understand and master knowledge, enhancing training effectiveness.
- Advertising Videos: Rapidly generate advertising videos to meet the needs of different advertising scenarios, improving the efficiency and effectiveness of advertising production.
- Artistic Creation: Generate artistic videos, providing artists with new creative tools and inspiring creativity.