What is Light-A-Video
Light-A-Video is a training-free video relighting method jointly launched by Shanghai Jiao Tong University, the University of Science and Technology of China, the Chinese University of Hong Kong, the Hong Kong University of Science and Technology, Stanford University, and Shanghai AI Lab. It is based on progressive light fusion technology to achieve high-quality, temporally consistent video lighting control. Light-A-Video is implemented through two key modules: Consistent Light Attention (CLA) and Progressive Light Fusion (PLF). CLA is based on enhanced inter-frame interaction to stabilize the generation of background lighting sources; PLF is based on the motion prior of video diffusion models to progressively integrate the relighting effect into the video, ensuring smooth lighting transitions. Light-A-Video supports relighting the entire video and relighting the foreground sequence to generate the background.
Main features of Light-A-Video
- Temporal consistency: Based on enhanced inter-frame lighting consistency, it avoids video flickering and lighting discontinuity issues.
- High-quality relighting: Uses a pretrained image relighting model to adjust the lighting of each frame in the video while maintaining image quality.
- Separate processing of foreground and background: Supports relighting the video foreground and automatically generates a background consistent with the lighting conditions.
- Zero-shot generation: Requires no additional training or optimization, and directly generates videos that match the lighting conditions according to text prompts.
- Strong compatibility: Compatible with a variety of popular video generation models (such as AnimateDiff, CogVideoX, etc.), with broad applicability.
Technical principles of Light-A-Video
- Consistent Light Attention (CLA)：
- The CLA module stabilizes the generation of lighting sources based on enhanced inter-frame interaction. It introduces cross-frame information into the self-attention layers of the image relighting model and suppresses high-frequency flicker in lighting based on temporally averaged features, generating stable background lighting.
- CLA uses a dual-stream attention fusion strategy: one stream processes the original frame information to preserve details; the other stream processes based on temporal averaging to suppress flicker. Finally, the outputs of the two streams are fused through weighted averaging.
- Progressive Light Fusion (PLF)：
- The PLF module uses the motion prior of a video diffusion model (VDM) to progressively integrate the relighting effect into the video. Based on linear fusion, it combines the relit image appearance with the original video appearance to ensure smooth lighting transitions.
- PLF progressively adjusts the lighting target during the denoising process of the video diffusion model. Based on dynamically adjusted fusion weights, it gradually guides the video denoising direction to achieve temporally consistent relighting effects.
- Overall process: The input video is first encoded into a noise signal in latent space and progressively restored through the denoising process of the VDM. At each denoising step, the CLA and PLF modules work together to progressively inject relighting information into the video.
Light-A-Video project address
- Project official website：https://bujiazi.github.io/light-a-video.github.io/
- GitHub repository：https://github.com/bcmi/Light-A-Video/
- arXiv technical paper：https://arxiv.org/pdf/2502.08590
Application scenarios of Light-A-Video
- Film and television post-production: Quickly adjust lighting conditions, such as changing day to night, saving cost and time.
- Game development: Dynamically change scene lighting to enhance immersion.
- Video creation: Quickly change video style to meet creative needs.
- VR/AR: Adjust lighting in real time to improve the fusion effect of virtual and reality.
- Video conferencing: Optimize video lighting to improve the remote collaboration experience.