What is Self Forcing
Self Forcing is a new autoregressive video generation algorithm jointly launched by Adobe Research and the University of Texas at Austin. It solves the exposure bias problem of traditional generative models during training and testing. By simulating the self-generation process during training and generating subsequent frames conditioned on previously generated frames rather than relying on real frames, it bridges the difference between training and testing distributions. Self Forcing introduces a rolling KV cache mechanism, supports theoretically infinitely long video generation, and achieves real-time generation capability of 17 FPS on a single H100 GPU with latency below one second. The breakthrough provides new possibilities for live streaming, gaming, and real-time interactive applications, such as real-time generation of virtual backgrounds or special effects. The efficiency and low latency of Self Forcing make it an important tool for future multimodal content creation.
Main features of Self Forcing
- Efficient real-time video generation: Self Forcing can achieve efficient real-time video generation on a single GPU, with a frame rate of 17 FPS and latency below one second.
- Infinitely long video generation: Through the rolling KV cache mechanism, Self Forcing supports theoretically infinitely long video generation. It can continuously generate video content without interruption due to length limits, providing strong support for dynamic video creation.
- Bridging the gap between training and testing: Self Forcing simulates the self-generation process during training and generates subsequent frames conditioned on generated frames rather than relying on real frames. It effectively solves the exposure bias problem in autoregressive generation, bridges the distribution difference between the training and testing stages, and improves the quality and stability of generated videos.
- Low resource requirements: Self Forcing optimizes the use of computational resources and can achieve streaming video generation on a single RTX 4090 graphics card, reducing dependence on hardware resources and making it easier to deploy and use on ordinary devices.
- Supports multimodal content creation: The efficiency and real-time nature of Self Forcing enable it to support multimodal content creation, such as generating backgrounds or special effects in real time during game live streaming, or dynamically generating visual content in virtual reality experiences, providing creators with broader application space.
Technical principles of Self Forcing
-
Autoregressive unrolling and overall loss supervision: Self Forcing simulates the autoregressive generation process during inference in the training stage, that is, the generation of each frame is based on frames previously generated by the model itself rather than real frames. It supervises the entire generated sequence through a video-level overall distribution matching loss function, not merely frame-by-frame evaluation. The model can learn directly from its own prediction errors, effectively alleviating exposure bias.
-
Rolling KV cache mechanism: To support long video generation, Self Forcing introduces a rolling key-value (KV) cache mechanism. This mechanism maintains a fixed-size cache area that stores the KV embeddings of the most recent frames. When a new frame is generated, the cache area removes the oldest entry and adds the new embedding.
-
Few-step diffusion model and gradient truncation strategy: To improve training efficiency, Self Forcing adopts a few-step diffusion model combined with a stochastic gradient truncation strategy. Specifically, during training the model randomly selects the number of denoising steps and performs backpropagation only on the final denoising step.
-
Dynamic conditional generation mechanism: When generating each frame, Self Forcing dynamically combines two types of conditional inputs: the clear frames already generated at past time steps and the noisy frame at the current time step. Generation is completed through iterative denoising, ensuring the coherence and naturalness of the generation process.
Self Forcing project address
- Project website：https://self-forcing.github.io/
- Github repository：https://github.com/guandeh17/Self-Forcing
- arXiv technical paper：https://arxiv.org/pdf/2506.08009
Application scenarios of Self Forcing
- Live streaming and real-time video streams: Self Forcing can achieve real-time video generation at 17 FPS on a single GPU, with latency below one second. It is suitable for live streaming scenarios, such as generating virtual backgrounds, special effects, or dynamic scenes in real time during a live stream, bringing viewers a brand-new visual experience.
- Game development: In game development, Self Forcing can generate game scenes and special effects in real time, without the need to pre-produce a large amount of video assets. It generates dynamic environmental changes or special effects based on the player's real-time operations, enhancing the game's immersion and interactivity.
- Virtual reality and augmented reality: Self Forcing's low latency and efficient generation capabilities can provide real-time visual content for virtual reality (VR) and augmented reality (AR) applications. It can generate realistic virtual scenes in real time in VR experiences, or overlay virtual elements in real time in AR applications.
- Content creation and video editing: Self Forcing can be used in short video creation tools to help creators quickly generate high-quality video content.
- World simulation and training: Self Forcing can be used in world simulation and training scenarios, such as generating realistic natural environments or urban landscapes for military training, urban planning, or environmental simulation.