What Is PortraitGen
PortraitGen is an AI portrait video editing tool launched by a research team at the University of Science and Technology of China. Based on 3D Gaussian Splatting technology and a neural Gaussian texture mechanism, it converts 2D portrait videos into 4D Gaussian fields, enabling high-quality 3D and temporally consistent editing. The tool supports multimodal editing, including text-driven editing, image-driven editing, and relighting, and can quickly and efficiently perform operations such as stylization, clothing changes, and lighting adjustments on people in videos. Through face-aware editing and expression similarity guidance, PortraitGen ensures that edited portraits remain natural and consistent with the original video frames.
Main Features of PortraitGen
- Multimodal Portrait Editing: Supports two editing modes: text-driven and image-driven. Users can specify changes in a person's actions, expressions, and scene by entering text descriptions, or use reference images to achieve style transfer or perform operations such as virtual try-on.
- Relighting: Based on IC-Light technology, dynamically adjusts the lighting effects in the video according to text descriptions, so that the lighting blends naturally with the scene.
- Face-Aware Editing: Through the face-aware editing module, ensures that during video editing, the person's facial structure and personalized features are preserved, and the naturalness and consistency of the person's expressions and facial movements are optimized.
- Style Transfer and Virtual Try-On: Supports style transfer and virtual try-on. Users can use reference images to achieve global style transfer (such as converting to an animation style), or add items such as virtual clothing to people in the video.
- Multi-Shot and Complex Scene Processing: Can process multi-shot videos while maintaining style and character consistency, and can use Gaussian texture technology to render videos in complex styles, such as LEGO style or pixel art style.
- Fast Generation and High Frame Rate Output: Can complete editing in a short time, and supports rendering speeds of up to 100 frames per second (FPS) when generating videos, making it suitable for efficient video production scenarios.
Technical Principles of PortraitGen
- 3D Gaussian Splatting (3DGS): Represents the scene based on 3D Gaussians, defining the center point, direction, and size of each Gaussian, as well as opacity and color attributes, through a 3D covariance matrix, enabling the construction of dynamic 3D fields.
- Neural Gaussian Texture Mechanism: Maintains a 3D Gaussian field in the UV space of the SMPL-X model, deforms the Gaussians according to the underlying mesh deformation tracked from the input video, and uses UV mapping and a 2D neural renderer to convert feature maps into RGB signals.
- Face-Aware Editing Module: Edits the head region twice to enhance facial structure perception and improve editing quality.
- Expression Similarity Guidance: Maps the rendered image and the input source image to EMOCA's latent expression space, and uses a loss function to ensure expression similarity.
- Multimodal Editing Technology: Combines knowledge from large-scale 2D generative models to enable text-driven editing, image-driven editing, and relighting.
PortraitGen Project Address
- Project Website：ustc3dv.github.io/PortraitGen
- GitHub Repository：https://github.com/USTC3DV/PortraitGen-code
- arXiv Technical Paper：https://arxiv.org/pdf/2409.13591
Application Scenarios of PortraitGen
- Film and Television Production: In the production of films, TV series, or short films, create or modify characters' appearances to achieve special effects makeup or stylized scene transformations.
- Artistic creation: Artists and illustrators use PortraitGen to create portrait artworks with specific styles, such as converting portraits into pixel art or oil painting styles.
- Advertising and promotion: In the advertising industry, customized editing of portraits is carried out based on brand image or product characteristics to attract the target audience.
- Fashion industry: Fashion designers and retailers use virtual try-on features to display clothing and accessories in a virtual environment, providing customers with a new shopping experience.
- Social media and short videos: Content creators and influencers use PortraitGen to edit their own portrait videos, adding creative effects and improving the appeal and interactivity of the content.
- Game development: In game character design, quickly generate or edit the appearance of characters to improve the personalization and richness of the game.