ExVideo - An AI Video Length Extension Tuning Technique Launched by Alibaba and East China Normal University

What is ExVideo

ExVideo is a post-tuning technique for video synthesis models launched by researchers from Alibaba and East China Normal University. It can extend the temporal scale of existing video synthesis models to generate longer videos with more frames. Based on the Stable Video Diffusion model, the team trained an extended model capable of generating coherent videos up to 128 frames long while preserving the original model's generative capabilities. By optimizing temporal modules such as 3D convolutions, temporal attention, and positional embeddings, ExVideo enables the model to handle content over longer time spans. While maintaining the original model's generative capabilities, it significantly increases the number of video frames, and the training cost is low, making it particularly suitable for situations with limited computational resources.

ExVideo's Features

• Temporal Scale Extension: One of ExVideo's core functions is to extend the temporal scale of video synthesis models, enabling them to process and generate longer video sequences than the original models were designed for. Through this extension, ExVideo can generate videos with more frames, thereby telling more complete stories or displaying dynamic scenes over longer periods of time.
• Post-Tuning Strategy: ExVideo's post-tuning strategy is a key part of its technology. By retraining specific parts of models such as Stable Video Diffusion, ExVideo can enable these models to generate longer videos, reaching 128 frames or more. It not only increases video length but also maintains the model's generalization ability for various inputs, making the generated videos diverse and highly adaptable.
• Parameter Efficient: Compared with traditional training methods, ExVideo adopts a post-tuning strategy, eliminating the need to train a completely new model from scratch and instead optimizing based on an existing model. This significantly reduces the number of required parameters and computational resources, making model extension more efficient and practical.
• Maintains Generative Capability: While extending video length, ExVideo focuses on maintaining video quality. The generated videos are not only extended in time but also meet high standards in visual coherence, clarity, and overall quality.
• Compatibility and Generality: ExVideo's design takes into account compatibility with a variety of video synthesis models, enabling it to be widely applied to different video generation tasks. Whether it is 3D convolutions, temporal attention, or positional embeddings, ExVideo can provide corresponding extension strategies to adapt to different model architectures.

ExVideo's Official Website Entry

• Official project homepage:https://ecnu-cilab.github.io/ExVideoProjectPage/
• GitHub code repository:https://github.com/modelscope/DiffSynth-Studio
• Hugging Face model download:https://huggingface.co/ECNU-CILab/ExVideo-SVD-128f-v1
• ModelScope model download:https://www.modelscope.cn/models/ECNU-CILab/ExVideo-SVD-128f-v1/summary
• arXiv technical paper:https://arxiv.org/abs/2406.14130

ExVideo's Technical Principles

• Parameter Post-Tuning: ExVideo adopts a parameter post-tuning method to improve existing video synthesis models. This includes retraining specific parts of the model rather than retraining the entire model, thereby improving efficiency.
• Temporal Module Extension: For the temporal modules in video synthesis models, ExVideo proposes extension strategies. These strategies include optimization of 3D convolution layers, temporal attention mechanisms, and positional embedding layers to adapt to longer video sequences.
  
• 3D convolution layer: 3D convolutional layers are used in video synthesis to capture features along the temporal dimension. ExVideo retains the 3D convolutional layers in the original model because they can adapt to different time scales without additional fine-tuning.
• Temporal Attention Mechanism: To improve the model's ability to handle long sequences, ExVideo fine-tunes the temporal attention module. This helps the model better understand the temporal coherence of video content.
• Positional Embedding: Traditional video synthesis models may use static or trainable positional embeddings to represent the order of frames in a video. ExVideo adapts to longer video sequences by introducing trainable positional embeddings initialized in a cyclic pattern.
• Identity 3D Convolution Layer: After the positional embedding layer, ExVideo introduces an additional identity 3D convolution layer to learn long-term video features. This layer is initialized as an identity matrix before training to ensure it does not alter the video representation and maintains consistency with the original model.
• Engineering Optimization: To train effectively under limited computational resources, ExVideo adopts multiple engineering optimization techniques, such as parameter freezing, mixed-precision training, gradient checkpointing, and Flash Attention, as well as using the DeepSpeed library to shard optimizer states and gradients.
• Training Process: ExVideo is trained using a publicly available dataset, OpenSoraPlan2, which contains a large number of videos, thereby enhancing the model's ability to generate diverse videos.
• Loss Function and Noise Schedule: During training, ExVideo keeps the loss function and noise schedule strategy consistent with the original model, ensuring the stability and efficiency of model training.