What is Ruyi
Ruyi is an image-to-video large model launched by TuSimple, designed specifically to run on consumer-grade graphics cards. It supports multi-resolution and multi-duration video generation, and features first-frame control, first-and-last-frame control, motion amplitude control, and camera control. Ruyi is based on the DiT architecture and consists of a Casual VAE module and a Diffusion Transformer, used for video data compression and generation. Ruyi can reduce the development cycle and cost of anime and game content, making it an ideal tool for ACG enthusiasts and creators. Currently, TuSimple has officially open-sourced the Ruyi-Mini-7B version.
Ruyi's main features
- Multi-resolution, multi-duration generation: Ruyi supports video generation from a minimum of 384×384 to a maximum of 1024×1024 resolution, can handle any aspect ratio, and can generate videos up to 120 frames/5 seconds.
- First-frame and first-and-last-frame controlled generation: Generates videos based on up to 5 starting frames and up to 5 ending frames, using cyclic overlay to generate videos of arbitrary length.
- Motion amplitude control: Provides 4 levels of motion amplitude control, making it convenient for users to control the degree of change in the overall picture.
- Camera control: Provides 5 types of camera control: up, down, left, right, and static, increasing the flexibility of video generation.
Ruyi's technical principles
- Model architecture: Ruyi is based on the DiT (Diffusion Model with Transformers) architecture and consists of two parts:
- Casual VAE module: Responsible for compression and decompression of video data.
- Diffusion Transformer: Responsible for generating the compressed video.
- Compression and encoding: The Casual VAE module compresses the spatial resolution to 1/8 and the temporal resolution to 1/4. After compression, each pixel is represented by 16-bit BF16.
- Positional encoding: The DiT part uses 3D full attention, using 2D RoPE (Rotary Positional Encoding) for spatial positional encoding and sin_cos for temporal positional encoding.
- Training loss function: The final loss is trained using DDPM (Denoising Diffusion Probabilistic Models).
- Parameter count and training data: The total parameter count of the model is about 7.1B, and it was trained on about 200M video clips.
- Training stages: The entire training process is divided into four stages, from low-resolution pretraining to high-resolution fine-tuning, gradually improving model performance.
Ruyi's project address
- GitHub repository：https://github.com/IamCreateAI/Ruyi-Models
- HuggingFace model library：https://huggingface.co/IamCreateAI/Ruyi-Mini-7B
Ruyi's application scenarios
- Animation pre-production: Quickly generate dynamic demonstrations of animated characters and scenes to evaluate animation design and story flow.
- Game CG generation: In game development, automatically generate game promotional videos or dynamic in-game backgrounds.
- Movie special effects preview: In film production, help directors and special effects teams preview special effects scenes and optimize post-production special effects work.
- Virtual streamers: Generate dynamic videos of virtual streamers for live streaming, news broadcasting, or online education.
- Social media content: Create attractive dynamic video content for social media platforms to increase user engagement and brand exposure.