What is LCVD
LCVD (Lighting Controllable Video Diffusion Model) is a high-fidelity, lighting-controllable portrait animation generation framework introduced by Sichuan University. LCVD separates the intrinsic features of a portrait (such as identity and appearance) from extrinsic features (such as pose and lighting), and the Reference Adapter and Shading Adapter map the features into different subspaces. During animation generation, LCVD combines the feature subspaces and finely regulates the lighting effects based on a multi-condition classifier-free guidance mechanism, preserving the identity and appearance of the portrait. The model is based on the Stable Video Diffusion model (SVD) and generates high-quality portrait animations that are consistent with the pose of the driving video and conform to the target lighting conditions. LCVD significantly outperforms existing methods in lighting realism, image quality, and video consistency, providing strong technical support for fields such as virtual reality, video conferencing, and film and television production.
Main Features of LCVD
- Portrait Animation: Converts a static portrait into a dynamic video, matching the head movements and expressions in the driving video.
- Lighting Control: During animation generation, relights the portrait according to the lighting conditions specified by the user or in the reference image.
- Identity and Appearance Preservation: During animation and relighting, maintains the identity and appearance features of the portrait, avoiding loss of identity information.
- High-Quality Video Generation: The generated videos perform excellently in lighting realism, image quality, and video consistency, making them suitable for scenarios such as virtual reality, video conferencing, and film and television production.
Technical Principles of LCVD
- Feature Separation: The Reference Adapter maps the intrinsic features of the reference portrait (identity and appearance) into the feature space. The Shading Adapter maps the extrinsic features (lighting and pose) into the feature space. Based on the separation of intrinsic and extrinsic features, the model independently controls lighting and pose during the animation process.
- Lighting-Controllable Diffusion Model: Based on the Stable Video Diffusion Model, it uses multi-condition classifier-free guidance to adjust lighting effects. By modifying the guidance strength (such as the weight ω), the influence of the lighting prompt is enhanced or weakened, achieving fine-grained lighting control.
- Motion Alignment and Long Video Generation: Based on the motion alignment module, it ensures that the generated portrait is consistent with the pose in the driving video. Using diffusion model sampling methods, it generates videos of arbitrary length, and based on an overlap strategy, it ensures smooth transitions between video segments.
- Training and Optimization: In the training phase, self-supervised learning optimizes the adapters and the diffusion model, ensuring consistency in lighting, pose, and identity of the generated videos. Loss functions (such as LPIPS, FID, etc.) are used to evaluate and optimize the quality of the generated videos.
LCVD Project Address
- arXiv Technical Paper：https://arxiv.org/pdf/2502.19894
Application Scenarios of LCVD
- Virtual Reality (VR) and Augmented Reality (AR): Creates realistic virtual characters that blend naturally with virtual or real scenes.
- Video Conferencing: Generates high-quality portrait animations in real time, reducing bandwidth requirements and improving user experience.
- Film and Television Production: Quickly generates portrait animations that conform to different lighting conditions, for use in special effects and virtual scenes.
- Game Development: Generate lifelike virtual character animations to enhance the realism and immersion of games.
- Social Media and Content Creation: Support users in generating personalized animated avatars or short videos, enriching the forms of content creation.