What is Step-Video-TI2V
Step-Video-TI2V is an open-source Image-to-Video generation model launched by StepFun, with 30 billion parameters, capable of generating videos up to 102 frames based on text descriptions and image inputs. The model is based on a deeply compressed variational autoencoder (Video-VAE), achieving 16×16 spatial compression and 8× temporal compression, significantly improving training and inference efficiency. Users can balance the dynamics and stability of the video by setting the motion score. It supports various camera movements such as push, pull, pan, tilt, rotate, and follow.
Main Features of Step-Video-TI2V
- Image-to-Video Generation: Users can provide an image and related text description, and the model will generate a coherent video based on these inputs.
- High-Quality Video Output: Supports generating videos up to 102 frames, 5 seconds, and 540P resolution, meeting various creative needs.
- Dynamic Adjustment: Users can control the dynamics of the video by setting the motion score. For example, when the motion score is 2, the video is more stable but less dynamic; when the motion score is 10 or 20, the video is more dynamic.
- Balancing Dynamics and Stability: By adjusting the motion score, users can find the best balance between dynamic effects and stability.
- Camera Movement Control: Supports various camera movements, including fixed camera, up/down/left/right movement, up/down/left/right pan, zoom in/out, push in/pull out, rotation, orbit, and focus shift, etc.
- Cinematic Camera Effects: Capable of generating complex camera effects similar to cinematic levels, meeting professional creative needs.
- Anime Effect Optimization: Performs excellently in anime-style video generation, capable of generating videos with effects such as blurred backgrounds and dynamic actions. Suitable for animation creation, short video production, and other application scenarios.
- Flexible Video Sizes: Supports video generation in various sizes, including landscape, portrait, and square, allowing users to choose the appropriate video size based on different creative needs and platform characteristics.
- Multilingual Support: Equipped with a bilingual text encoder, supports Chinese and English prompt input, convenient for users with different language backgrounds.
- Special Effects Generation Capability: Initially possesses special effects generation capability, and will further improve the special effects generation effect through technical optimization in the future.
Technical Principles of Step-Video-TI2V
- Deeply Compressed Variational Autoencoder (Video-VAE): Step-Video-TI2V uses a deeply compressed variational autoencoder (Video-VAE), achieving 16×16 spatial compression and 8× temporal compression. This significantly reduces the computational complexity of video generation tasks while maintaining excellent video reconstruction quality. Video-VAE adopts a dual-path architecture, which can effectively separate high and low frequency information, further optimizing the video generation effect.
- Diffusion-Based Transformer (DiT) Architecture: The model is based on the diffusion-based Transformer (DiT) architecture, containing a 3D full attention mechanism. Through the Flow Matching training method, the input noise is gradually denoised into latent frames, with text embeddings and timesteps as conditioning factors. This architecture performs excellently in generating videos with strong motion dynamics and high aesthetic quality.
- Bilingual Text Encoder: Step-Video-TI2V is equipped with a bilingual text encoder that can handle Chinese and English prompts. This allows the model to directly understand Chinese or English input and generate videos that match the text description.
- Direct Preference Optimization (DPO): To further improve the quality of generated videos, Step-Video-TI2V introduces the Video Direct Preference Optimization (Video-DPO) method. DPO fine-tunes the model with human preference data, reducing artifacts and enhancing visual effects, making the generated videos smoother and more realistic.
- Cascaded Training Strategy: The model adopts a cascaded training process, including text-to-image (T2I) pretraining, text-to-video/image (T2VI) pretraining, text-to-video (T2V) fine-tuning, and Direct Preference Optimization (DPO) training. This accelerates model convergence and fully utilizes video data of different qualities.
- System Optimization: Step-Video-TI2V has been optimized at the system level, including tensor parallelism, sequence parallelism, and Zero1 optimization, enabling efficient distributed training. It introduces the high-performance communication framework StepRPC and the two-layer monitoring system StepTelemetry to optimize data transmission efficiency and identify performance bottlenecks.
Step-Video-TI2V Project Address
- Github Repository：https://github.com/stepfun-ai/Step-Video-TI2V
- HuggingFace Model Library：https://huggingface.co/stepfun-ai/stepvideo-ti2v
- arXiv Technical Paper：https://arxiv.org/pdf/2503.11251
How to Use Step-Video-TI2V
- Visit Yuewen Video: VisitYuewen Video's official website or App.
- Steps: Click to upload an image and enter a text description. Adjust parameters (such as motion score, etc.). Click the generate button to download or share the video.
Application Scenarios of Step-Video-TI2V
- Animation Production: Step-Video-TI2V is particularly good at generating anime-style videos and can generate smooth animations based on input images and text descriptions.
- Short Video Production: The model supports multiple camera movement methods, such as dolly in/out, pan, tilt, rotation, orbit, etc., and can generate short videos with cinematic effects.
- Action Teaching: Step-Video-TI2V can generate complex dynamic scenes, such as sports action teaching, dance teaching, etc.
- Special Effects Production: The model can generate videos with aesthetic appeal and realism, suitable for special effects production in movies, TV series, and games.
- Product Display: Step-Video-TI2V can generate attractive advertising videos to showcase product features or brand stories.