What is Step-Video-T2V
Step-Video-T2V is an open-sourcetext-to-videopre-trained model launched by the StepFun team, with 30 billion parameters, capable of generating high-quality videos up to 204 frames. The model is based on a deeply compressed variational autoencoder (Video-VAE), achieving 16×16 spatial compression and 8× temporal compression, significantly improving training and inference efficiency. Step-Video-T2V is equipped with a bilingual text encoder, supports Chinese and English prompt input, and further improves video quality through direct preference optimization (DPO). The model is based on a diffusion-based Transformer (DiT) architecture and 3D full attention mechanism, and performs excellently in generating videos with strong motion dynamics and high aesthetic quality.
Main Features of Step-Video-T2V
- High-Quality Video Generation: Step-Video-T2V has 30 billion parameters, can generate high-quality videos up to 204 frames, and supports 544×992 resolution.
- Bilingual Text Support: Equipped with a bilingual text encoder, it supports direct input of Chinese and English prompts, and can understand and generate videos consistent with text descriptions.
- Dynamics and Aesthetic Optimization: Through the 3D full attention DiT architecture and Flow Matching training method, it generates videos with strong dynamic effects and high aesthetic quality.
Technical Principles of Step-Video-T2V
- Deeply Compressed Variational Autoencoder (Video-VAE): Step-Video-T2V uses a deeply compressed variational autoencoder (Video-VAE), achieving 16×16 spatial compression and 8× temporal compression. It significantly reduces the computational complexity of video generation tasks while maintaining excellent video reconstruction quality.
- Bilingual Text Encoder: The model is equipped with two pre-trained bilingual text encoders, capable of processing Chinese and English prompts. Step-Video-T2V can directly understand Chinese and English input and generate videos consistent with text descriptions.
- Diffusion-Based Transformer (DiT) Architecture: Step-Video-T2V is based on a diffusion-based Transformer (DiT) architecture, including a 3D full attention mechanism. Through Flow Matching training, input noise is gradually denoised into latent frames, using text embeddings and time steps as conditioning factors. It performs excellently in generating videos with strong motion dynamics and high aesthetic quality.
- Direct Preference Optimization (DPO): To further improve the quality of generated videos, Step-Video-T2V introduces the video direct preference optimization (Video-DPO) method. DPO fine-tunes the model through human preference data, reduces artifacts and enhances visual effects, making the generated videos smoother and more realistic.
- Cascaded Training Strategy: The model adopts a cascaded training process, including text-to-image (T2I) pre-training, text-to-video/image (T2VI) pre-training, text-to-video (T2V) fine-tuning, and direct preference optimization (DPO) training. It accelerates model convergence and fully utilizes video data of different qualities.
- System Optimization: Step-Video-T2V has been optimized at the system level, including tensor parallelism, sequence parallelism, and Zero1 optimization, achieving efficient distributed training. It introduces the high-performance communication framework StepRPC and the two-layer monitoring system StepTelemetry to optimize data transmission efficiency and identify performance bottlenecks.
Project Address of Step-Video-T2V
- Github Repository：https://github.com/stepfun-ai/Step-Video-T2V
- HuggingFace Model Library：https://huggingface.co/stepfun-ai/stepvideo-t2v
- arXiv Technical Paper：https://arxiv.org/pdf/2502.10248
Application Scenarios of Step-Video-T2V
- Video Content Creation: Step-Video-T2V can quickly generate creative videos based on text prompts, helping creators save time and effort and lowering the barrier to video production.
- Advertising Production: It can generate personalized video advertising content for brands and advertisers, enhancing the appeal and reach of advertisements.
- Education and Training: Step-Video-T2V can generate instructional videos to help students better understand and remember knowledge.
- Entertainment and Film/Television: It provides creative materials for film and television production, assists in generating special effects, animations, or short drama clips, and accelerates the creative process.
- Social Media: Step-Video-T2V provides users with a personalized video generation tool, enriches the content ecosystem of social platforms, and enhances user interactivity. The generated videos can be used for sharing creative content on social media.