What is PersonaCraft
PersonaCraft is a personalized full-body image synthesis technology launched by Seoul National University in South Korea. It combines diffusion models and 3D human modeling to generate realistic, personalized full-body images of multiple people from a single reference image. PersonaCraft can effectively handle occlusion between people, supports user-defined body shape adjustments, and enables more flexible personalization. Based on precise control of body pose and shape, PersonaCraft performs excellently in generating high-quality images in complex scenes, setting a new standard for multi-person image synthesis.
Main Features of PersonaCraft
- Multi-person image synthesis: Generates realistic images containing multiple people from a single reference image.
- Occlusion handling: Effectively manages occlusion between people, ensuring that each person's body parts are accurately displayed in the image.
- Full-body personalization: Not only focuses on personalizing facial identity, but also accurately personalizes each person's full-body shape.
- User-defined body shape control: Users can adjust the body proportions and shapes of generated people according to personal preferences.
- 3D-aware pose conditioning control: Uses SMPLx-ControlNet (SCNet) for 3D-aware pose conditioning control, improving the accuracy of body shape and pose.
Technical Principles of PersonaCraft
- Combination of 3D human modeling and diffusion models: PersonaCraft integrates 3D human modeling (especially the SMPLx model) and diffusion models to enhance control over human body shape and pose.
- SMPLx-ControlNet（SCNet）: Uses the depth map generated by the SMPLx model as a conditioning signal to accurately capture body shape and pose, effectively handling complex occlusion problems.
- Facial and body identity extraction: Extracts facial identity embeddings from reference images based on technologies such as InsightFace, and extracts body shape parameters using the SMPLx fitting method.
- 3D-aware pose conditioning: Unlike traditional 2D skeleton pose conditioning, PersonaCraft uses 3D pose information to provide a more comprehensive representation of human pose, including depth information.
- Multi-person personalized image synthesis: Combines SCNet and IdentityNet, uses facial masks to accurately locate identities, and achieves personalized image synthesis of multiple identities.
PersonaCraft Project Address
- Project official website：gwang-kim.github.io/persona_craft
- GitHub repository：https://github.com/gwang-kim/PersonaCraft
- arXiv technical paper：https://arxiv.org/pdf/2411.18068
Application Scenarios of PersonaCraft
- Social media: Users customize virtual avatars of themselves or with friends according to their preferences, for use in social media profile pictures, representative images in virtual spaces, etc.
- Advertising and marketing: Brands create personalized advertising images that match the preferences of target audiences, improving the appeal of advertisements and user resonance.
- Fashion and clothing: In the fashion industry, display how clothing looks on people with different body types and identities, providing a more personalized shopping experience.
- Games and entertainment: In game design, create game characters with personalized appearances, enhancing players' immersion and personalized experience.
- Film and animation production: In film or animation production, quickly generate or modify character appearances, improving production efficiency and reducing costs.