ExAvatar - Meta's 3D digital avatar generation model

What is ExAvatar

ExAvatar is a 3D digital avatar generation model jointly launched by DGIST and Meta's Codec Avatars Lab. It clones a person from a short video and converts them into a 3D digital avatar. It supports mobile phone scanning to quickly capture full-body movements and expressions and convert them into realistic 3D models. ExAvatar combines the SMPL-X model and 3D Gaussian technology to achieve high-quality dynamic rendering, supports full-body animation, reduces artifacts, and provides a natural and smooth experience. It is suitable for fields such as animation, gaming, and VR/AR, simplifies the 3D avatar creation process, and broadens application scenarios.

ExAvatar's main features

• Full-body 3D driving: Supports animation production for full-body movements and expressions, including the body, hands, and face.
• Hybrid representation: Combines 3D Gaussians and surface mesh technology to improve consistency in geometry and appearance and reduce artifacts in animation.
• Convenient capture: Quickly create a 3D avatar using only a mobile phone scan, with simple operation.
• High-quality rendering: Uses advanced algorithms to achieve high-quality rendering of dynamic performance and visual effects.
• Technical optimization: Through SMPL-X model and multilayer perceptron (MLP) processing, optimizes the expressiveness and naturalness of the hands and face.

ExAvatar's technical principles

• SMPL-X full-body parametric mesh model: ExAvatar uses the SMPL-X model to capture human posture and movement. SMPL-X is a highly parametric 3D human body model that can accurately simulate various human postures.
• 3D Gaussian Splatting (3DGS): Used to improve the realism and efficiency of rendering. By simulating the scattering of light on surfaces, 3D models look more realistic.
• Hybrid representation: ExAvatar combines 3D Gaussians and surface mesh to ensure consistency in geometry and appearance under different poses and reduce artifacts that may appear in new poses.
• Multilayer perceptron (MLP): Used to process the features of each Gaussian, extract key information, and combine this information with the standard mesh to form a 3D avatar that can be animated in standard space.
• Linear blend skinning (LBS) algorithm: Used for animation processing. In this way, ExAvatar can generate smooth and natural transition movements.
• Joint offsets and facial offsets: Before training ExAvatar, the research team jointly registered the body, hands, and face and introduced these offsets to optimize hand bone lengths and facial region shapes, improving the avatar's expressiveness and naturalness.
• Connectivity-based regularizer: Reduces artifacts in new expressions and poses. Through predefined connectivity, it ensures that the 3D Gaussians match the mesh topology of SMPL-X.

ExAvatar's project address

ExAvatar's application scenarios

• 3D animation production: Animators can use ExAvatar to quickly create realistic 3D characters for the production of animated films, TV series, or short films.
• Game development: Game developers can use ExAvatar technology to design dynamic full-body movements and expressions for game characters, improving the gaming experience.
• Virtual reality (VR): In a VR environment, users can generate their own 3D avatars through ExAvatar for an immersive experience.
• Augmented reality (AR): ExAvatar can be used in AR applications, for example, to increase interactivity in social media, advertising, or educational applications.
• Film Visual Effects: Film visual effects artists can use ExAvatar technology to capture actors' movements and expressions for later visual effects production.