What is RelightVid
RelightVid is a temporally consistent diffusion model for video relighting launched by Shanghai AI Lab, Fudan University, Shanghai Jiao Tong University, Zhejiang University, Stanford University, and The Chinese University of Hong Kong. It supports fine-grained and consistent scene editing of input videos based on text prompts, background videos, or HDR environment maps, and supports full-scene relighting and foreground-preserving relighting. The model generates high-quality video relighting data pairs based on a custom enhancement pipeline, combines real videos and 3D rendered data, and inserts trainable temporal layers into the pretrained image illumination editing diffusion framework (IC-Light) to enhance the effect of video relighting. RelightVid has significant advantages in maintaining temporal consistency and illumination details, providing new possibilities for the fields of video editing and generation.
Main Features of RelightVid
- Text-conditioned relighting: Relights a video according to the text description input by the user, such as "sunlight filtering through leaves, creating dappled light and shadow" or "soft morning light, golden hour."
- Background video-conditioned relighting: Uses a background video as the lighting condition to dynamically adjust the lighting of foreground objects, making the foreground lighting consistent with the background lighting.
- HDR environment map-conditioned relighting: Uses HDR environment maps to precisely control lighting and achieve high-quality relighting effects.
- Full-scene relighting: Relights both the foreground and the background so that the entire scene matches the lighting conditions.
- Foreground-preserving relighting: Relights the foreground while preserving the background, suitable for scenes where the foreground object needs to be highlighted.
Technical Principles of RelightVid
- Diffusion model extension: RelightVid is based on a pretrained image relighting diffusion model (such as IC-Light), extends the architecture to support video input, and introduces temporal layers to capture temporal dependencies between frames, ensuring the temporal consistency of relighting.
- Joint training with multimodal conditions:The model can simultaneously use background videos, text prompts, and HDR environment maps as conditions, embeds the conditions into the model based on encoders, and achieves collaborative editing through the cross-attention mechanism.
- Illumination-Invariant Ensemble (IIE): Uses brightness to enhance the input video and averages the predicted noise, improving the model's robustness under different lighting conditions and preventing albedo changes.
- Data enhancement pipeline (LightAtlas): Generates high-quality relighting data pairs based on real videos and 3D rendered data, providing the model with rich prior knowledge of lighting and enhancing its adaptability to complex lighting scenes.
Project Address of RelightVid
- Project website：https://aleafy.github.io/relightvid/
- GitHub repository：https://github.com/Aleafy/RelightVid
- arXiv technical paper：https://arxiv.org/pdf/2501.16330
- Online experience Demo：https://huggingface.co/spaces/aleafy/RelightVid
Application Scenarios of RelightVid
- Film and television production: Adjusts the lighting effects of scenes in movies or TV series to match different plot requirements or the director's creative vision, without the need for reshoots.
- Game development: Dynamically changes scene lighting in games, enhancing the game's immersion and visual effects, and adapting to different time and weather conditions.
- Augmented reality (AR): Adjusts the lighting of virtual elements in real time in AR applications to be consistent with the lighting of the real environment, improving the user experience.
- Video advertising and marketing: Quickly generate multiple lighting styles for advertising videos to meet the promotional needs of different brands or campaigns and enhance visual appeal.
- Video content creation: Help video bloggers or content creators easily change the lighting atmosphere of videos, such as simulating scenes of different weather or times, to enrich creative content.