What is LivePortrait
LivePortrait is an open-source portrait animation generation framework launched by Kuaishou, focusing on efficiently and controllably transferring the expressions and poses of a driving video to static or dynamic portraits to create expressive videos. This technology is implemented through an implicit keypoint framework, utilizing large-scale high-quality training data and a hybrid training strategy to enhance the model's generalization ability and motion control accuracy. LivePortrait achieves extremely fast single-frame generation speed on an RTX 4090 GPU, reaching 12.8 milliseconds, and has potential for further optimization. The open-source community has responded enthusiastically, and LivePortrait's GitHub page provides detailed usage guides and resources.
Main Features of LivePortrait
- Expression and Pose Transfer: LivePortrait can transfer expressions and poses from a driving video to static or dynamic portraits in real time, generating videos with rich expressions.
- High Efficiency: The framework achieves a single-frame generation speed of up to 12.8 milliseconds on an RTX 4090 GPU, demonstrating extremely high processing speed.
- Generalization Ability: Through a video-image hybrid training strategy and large-scale high-quality training data, LivePortrait has good generalization and can adapt to portraits of different styles and identities.
- Controllability: Utilizing implicit keypoints and a lightweight MLP network, LivePortrait enhances control over the animation generation process.
- Multi-Style Support: LivePortrait can handle portraits in various styles, including real people and stylized (such as anime) portraits.
- High-Resolution Animation Generation: LivePortrait supports generating high-resolution animations, providing clearer visual effects.
- Stitching and Retargeting Modules: Stitching modules and eye and mouth retargeting modules are designed to adapt to complex scenarios such as cropping and group photos, avoiding pixel misalignment.
Technical Principles of LivePortrait
- Base Model Training: In the first stage, the appearance extractor, motion extractor, warping module, and decoder are mainly optimized, and the models are trained from scratch.
- Stitching and Retargeting Module Training: After the base model is trained in the first stage, the appearance extractor, motion extractor, warping module, and decoder are frozen and remain unchanged in the second stage. In the second stage, only the stitching module and retargeting module are optimized.
- Video-Image Hybrid Training: LivePortrait treats each image as a frame of a video clip and trains the model on both videos and images simultaneously, improving the model's generalization ability.
- Upgraded Network Structure: LivePortrait unifies the canonical implicit keypoint estimation network, head pose estimation network, and expression deformation estimation network into a single model, and adopts ConvNeXt-V2-Tiny as its structure to directly estimate the canonical implicit keypoints, head pose, and expression deformation of the input image.
- Keypoint-Guided Implicit Keypoint Optimization: 2D keypoints are introduced to capture micro-expressions, and keypoint-guided loss is used as guidance for implicit keypoint optimization.
- Cascaded Loss Functions: Adopts face vid2vid's implicit keypoint invariance loss, keypoint prior loss, head pose loss, and deformation prior loss, and applies perceptual and GAN losses to improve texture quality.
LivePortrait Project Address
- Project Website：https://liveportrait.github.io/
- GitHub Repository：https://github.com/KwaiVGI/LivePortrait
- Hugging Face Model Library：https://huggingface.co/spaces/KwaiVGI/LivePortrait
- arXiv Technical Paper：https://arxiv.org/pdf/2407.03168
Application Scenarios of LivePortrait
- Social Media Content Creation: Users can convert their photos or videos into dynamic content for social media platforms, increasing interactivity and appeal.
- Virtual Streamers and Live Streaming: With LivePortrait technology, virtual avatars can be created for live streaming or video production without the need for a real person to appear on camera, making it suitable for scenarios such as game streaming and educational presentations.
- Film and Animation Production: In film and television post-production, LivePortrait can be used for character facial expression capture and animation generation, improving production efficiency and reducing costs.
- Advertising and Marketing: Enterprises can use LivePortrait technology to create engaging advertising videos, using dynamic character images to attract the attention of potential customers.