What is Snap Video
Snap Video is an AI video generation model developed by the research team at Snap (the company behind the social media platform Snapchat). Its goal is to synthesize videos from text descriptions, meaning users can input a descriptive text and the model can generate corresponding video content based on that text. Because video content is highly redundant and complex, Snap Video aims to address the challenge of generating high-quality, temporally coherent videos with high motion fidelity from text descriptions. User studies show that Snap Video is comparable to or even better than video generation models such as Pika and Runway Gen-2.
Snap Video's official website entrance
- Official project homepage:https://snap-research.github.io/snapvideo/
- arXiv research paper:https://arxiv.org/abs/2402.14797
Snap Video's main features
- Video-first model: Snap Video is a model specifically designed for video content generation, with special optimization for handling video temporal continuity and motion. This differs from many other models, which are usually extended from image generation models and may have limitations when handling video.
- Extended EDM framework: This model extends theEDM framework, enabling it to better handle the spatial and temporal redundancy of video data, thereby improving the quality of video generation.
- Efficient Transformer architecture: Snap Video adopts a Transformer-based FIT architecture, which is very effective at processing sequence data and capturing long-term dependencies, especially in video generation tasks and when handling high-resolution videos.
- Fast training and inference: Compared with traditional architectures such as U-Net, Snap Video trains and performs inference faster, and can generate videos more efficiently.
- High-resolution video generation: Snap Video can generate high-resolution video content, which was a challenge in previous text-to-video generation models because more details and complex motion need to be handled.
- Joint spatial-temporal modeling: Through joint modeling of spatial and temporal dimensions, Snap Video can generate videos with complex motion and high temporal consistency.
Snap Video's technical architecture
- Extended EDM framework: Snap Video extends the EDM framework to adapt to high-resolution video generation. The EDM framework was originally used for image generation, generating new data samples by simulating the data diffusion process. Snap Video adjusts the EDM forward diffusion process by introducing an input scaling factor (σin) to maintain the signal-to-noise ratio (SNR) at the original resolution, thereby maintaining performance in high-resolution video generation.
- Transformer architecture: Snap Video adopts a Transformer-based architecture, which is an efficient deep learning model particularly suitable for processing sequence data. Transformers capture global dependencies in input data through Self-Attention and Cross-Attention mechanisms. In Snap Video, Transformers are used to process video frames to generate videos with coherent motion.
- FIT architecture: Snap Video usesFIT（Far-reaching Interleaved Transformers）architecture, a transformer architecture specifically designed for high-resolution image and video synthesis. FIT learns a compressed representation of the input data and performs computation in a learnable latent space, thereby allowing the input dimensionality to grow without significantly affecting performance. This compressed representation enables Snap Video to efficiently process high-resolution video data.
- Joint spatial-temporal modeling: When generating videos, Snap Video considers both the spatial and temporal dimensions simultaneously, which helps capture the dynamic changes and motion in the video. This joint modeling strategy improves the temporal consistency and motion quality of the video.
- Two-stage cascaded model: To generate high-resolution videos, Snap Video adopts a two-stage cascaded model. The first-stage model generates low-resolution videos, and the second-stage model upsamples these videos to a higher resolution. This cascaded approach allows the model to generate high-quality videos while maintaining performance.
- Training and inference process: Snap Video uses the LAMB optimizer during training and adopts a cosine learning rate schedule. During inference, the model uses a deterministic sampler to generate video samples from Gaussian noise and applies Classifier-Free Guidance to improve text-video alignment.
- Conditioning information: During generation, Snap Video utilizes conditioning information such as text descriptions, noise levels, frame rates, and resolutions. This information is provided to the model through additional read operations to control the generation process.