What is DiffuEraser
DiffuEraser is a video restoration model based on stable diffusion models, filling masked regions in videos with richer details and more coherent structures. The model provides initialization and weak conditions by combining prior information, which helps reduce noise artifacts and suppress hallucinations. To improve temporal consistency during long-sequence inference, DiffuEraser extends the temporal receptive field of both the prior model and itself, further enhancing consistency based on the temporal smoothing properties of video diffusion models. The network architecture of DiffuEraser is inspired by AnimateDiff, integrating motion modules into the image restoration model. It mainly consists of a main denoising UNet and an auxiliary BrushNet. The BrushNet branch receives conditional latent inputs composed of masked images, masks, and noise latents. The features extracted by BrushNet are integrated layer by layer into the denoising UNet after passing through zero convolution blocks. The denoising UNet processes the noise latents. To enhance temporal consistency, the model introduces temporal attention mechanisms after the self-attention and cross-attention layers. After denoising, the generated images are fused with the input masked images using a blurred mask.
Main features of DiffuEraser
- Generation of unknown pixels: Based on the powerful generative capabilities of stable diffusion models, DiffuEraser can generate reasonable content with rich details and textures for pixels that have never appeared in the video, effectively solving the blurring and mosaic problems commonly encountered by traditional Transformer models when handling large masks.
- Propagation of known pixels: Through the enhanced propagation capabilities of the motion module and the prior model, DiffuEraser ensures that known pixels (pixels that have appeared in certain masked frames) can be fully and consistently propagated across different frames. It prevents conflicts between the restored content and the unmasked regions, improving the correctness and stability of the results.
- Maintenance of temporal consistency: During long-sequence inference, DiffuEraser enhances the temporal consistency of the completed content across all frames by extending the temporal receptive field of both the prior model and itself, based on the temporal smoothing properties of video diffusion models.
- Injection of prior information: DiffuEraser provides initialization and weak conditions by injecting prior information, which helps reduce noise artifacts, suppress visual hallucinations common in diffusion models, and generate more accurate and realistic restoration results.
- Network architecture optimization: The network architecture of DiffuEraser is inspired by AnimateDiff, integrating motion modules into the image restoration model BrushNet. By introducing temporal attention mechanisms after the self-attention and cross-attention layers, it further enhances temporal consistency.
Technical principles of DiffuEraser
- Network architecture: The network architecture of DiffuEraser is inspired by AnimateDiff, integrating motion modules into the image restoration model BrushNet. The architecture mainly consists of a main denoising UNet and an auxiliary BrushNet. The BrushNet branch receives conditional latent inputs composed of masked images, masks, and noise latents, and the extracted features are integrated layer by layer into the denoising UNet after passing through zero convolution blocks. The denoising UNet is responsible for processing the noise latents. To enhance temporal consistency, the model introduces temporal attention mechanisms after the self-attention and cross-attention layers. After denoising, the generated images are fused with the input masked images using a blurred mask.
- Decomposition of the video restoration problem: DiffuEraser decomposes the video restoration problem into three sub-problems: propagation of known pixels, generation of unknown pixels, and maintenance of temporal consistency of the completed content.
- Temporal consistency optimization: To improve temporal consistency during long-sequence inference, DiffuEraser adopts the following strategies:
- Expand the temporal receptive field of the prior model: Through pre-propagation and pre-inference, the prior model can propagate known pixels across the entire temporal domain, ensuring consistency between the completed content and the unmasked regions and stabilizing the results.
- Expand the temporal receptive field of DiffuEraser: Through pre-inference, DiffuEraser can process a wider range of video frames, ensuring consistency of the generated content during long-sequence inference.
DiffuEraser Project Address
- Project Official Website：https://lixiaowen-xw.github.io/DiffuEraser-page/
- GitHub Repository：https://github.com/lixiaowen-xw/DiffuEraser
- arXiv Technical Paper：https://arxiv.org/pdf/2501.10018
DiffuEraser Application Scenarios
- Post-production for films and TV series: In the post-production of films or TV series, DiffuEraser can be used to repair masked regions in videos, improve video quality, perform deblurring and super-resolution processing, and adapt to playback requirements of different resolutions.
- Old film restoration: For the digital restoration of old films, DiffuEraser can remove scratches, dust, and other degradation from film, improve resolution, and give old films a new lease on life.
- Surveillance video enhancement: In the field of security surveillance, DiffuEraser can enhance the clarity of surveillance video, help identify details, and improve surveillance efficiency.
- Video content conversion: Content creators can use DiffuEraser to convert standard-definition (SD) video content to high-definition (HD) or 4K, meeting the needs of modern display devices.
- Live sports events: In live sports events, DiffuEraser can be used for real-time video stream enhancement, providing a clearer viewing experience.