What is Next-Frame Diffusion
Next-Frame Diffusion (NFD) is an autoregressivevideo generationmodel jointly launched by Peking University and Microsoft Research, combining the high-fidelity generation capability of diffusion models with the causality and controllability of autoregressive models. Next-Frame Diffusion achieves efficient frame-level generation based on Block-wise Causal Attention and Diffusion Transformer. While maintaining video quality and coherence, the model achieves real-time video generation at over 30 FPS. The model introduces techniques such as Consistency Distillation and Speculative Sampling to further improve sampling efficiency. The model performs excellently in large-scale action-conditioned video generation tasks, significantly outperforming existing methods.
Main Features of Next-Frame Diffusion
- Real-time video generation: Supports real-time video generation at over 30 FPS on high-performance GPUs, making it highly suitable for interactive applications requiring rapid response, such as gaming, virtual reality, and real-time video editing scenarios.
- High-fidelity video generation: Generates high-fidelity video content in continuous space, better capturing details and textures compared to traditional autoregressive models.
- Action-conditioned generation: Generates corresponding video content based on the user's real-time operations, offering extremely high flexibility and controllability in interactive applications.
- Long-term video generation: Supports generating video content of arbitrary length, suitable for applications requiring long-term coherence, such as storytelling or simulated environments.
Technical Principles of Next-Frame Diffusion
- Block-wise Causal Attention: The core of the model is the block-wise causal attention mechanism, combining bidirectional attention within frames and causal dependencies between frames. Within each frame, the model performs bidirectional self-attention operations to capture spatial dependencies within the frame. Between frames, the model maintains causality, where each frame can only depend on previous frames, ensuring coherence and causal consistency of generation.
- Diffusion Models and Diffusion Transformer: The model is based on the principles of diffusion models, generating video frames in a progressive denoising process. The Diffusion Transformer is a key component of NFD, leveraging the powerful modeling capability of the Transformer architecture to handle the spatiotemporal dependencies of video.
- Consistency Distillation: To accelerate the sampling process, consistency distillation technology is introduced, extending the sCM (Simplified Consistency Model) from the image domain to the video domain, significantly improving generation speed while maintaining high quality of the generated content.
- Speculative Sampling: Utilizes the consistency of adjacent frame action inputs to generate future frames in advance. If subsequent detection finds changes in action input, the speculatively generated frames are discarded, and generation restarts from the last verified frame, significantly reducing inference time and improving the efficiency of real-time generation.
- Action Conditioning: NFD controls the direction and content of video generation based on action-conditioned inputs. Action inputs can be user operation commands, control signals, or other forms of conditional information, and the model generates corresponding video frames based on the input.
Project Address of Next-Frame Diffusion
- Project Website：https://nextframed.github.io/
- arXiv Technical Paper：https://arxiv.org/pdf/2506.01380
Application Scenarios of Next-Frame Diffusion
- Game DevelopmentThe model generates dynamic game environments in real time based on player actions, enhancing the gaming experience.
- Virtual Reality (VR) and Augmented Reality (AR)The model generates virtual scenes in real time, providing an immersive experience for VR and AR applications.
- Video Content CreationThe model generates high-quality video content, suitable for advertising, film, and television production.
- Autonomous Driving and RoboticsIt generates the behaviors and scenarios of autonomous vehicles or robots in different environments, used for training and testing.
- Education and TrainingIt generates virtual laboratory environments, helping students conduct scientific experiments and learning.