What is TaoAvatar
TaoAvatar is a high-fidelity, lightweight 3D full-body conversational virtual human technology launched by Alibaba Group's research team. Based on 3D Gaussian Splatting technology, it can generate photo-realistic 3D full-body avatars, supporting high-resolution rendering with low storage requirements. TaoAvatar can run in real time at a high frame rate of 90FPS on a variety of mobile and AR devices, driven by multiple signals such as speech, expressions, gestures, and body poses, achieving natural synchronization of lip movements, expressions, and motions.
Main Features of TaoAvatar
- High-Fidelity Full-Body Dynamic Avatar Generation: It can generate realistic, topologically consistent 3D full-body avatars from multi-view image sequences, supporting fine-grained control over poses, gestures, and expressions.
- Real-Time Rendering with Low Storage Requirements: It can run in real time at a high frame rate of 90FPS on a variety of mobile and AR devices, supporting high-resolution rendering while maintaining low storage requirements.
- Multi-Signal Driven: It can be driven by multiple signals such as speech, expressions, gestures, and body poses, achieving natural synchronization of lip movements, expressions, and motions.
- Lightweight Architecture: By "baking" complex non-rigid deformations into a lightweight MLP network, combined with blendshape compensation details, it significantly improves operational efficiency.
Technical Principles of TaoAvatar
- 3D Gaussian Splatting (3DGS) Technology:: 3DGS represents points in a scene using 3D Gaussian functions and projects these Gaussian functions onto a 2D image plane for rendering. Each 3D Gaussian is described by parameters such as position, covariance, color, and opacity. Structure from Motion (SfM) technology is used to estimate a 3D point cloud from multi-view images, then each point is converted into a Gaussian function and trained using stochastic gradient descent.
- Pose-Dependent Non-Rigid Deformation Processing: TaoAvatar decomposes complex non-rigid deformations into two parts: rigid deformation and shape deformation, and "bakes" the shape deformation into a lightweight MLP network through knowledge distillation technology. It can efficiently handle complex pose-dependent non-rigid deformations while maintaining the realism and controllability of the avatar.
- Learnable Gaussian Blendshapes: To further enhance the appearance details of the avatar, TaoAvatar introduces learnable Gaussian blendshapes. By training a neural network to learn the Gaussian blendshape parameters under different poses and expressions, these parameters are then applied to the avatar. This allows the avatar to maintain extremely high fidelity under different poses and expressions.
- Real-Time Rendering and Optimization: TaoAvatar adopts a variety of optimization techniques, such as GPU acceleration, reducing unnecessary computations, and optimizing model structure and parameters, to achieve high-quality real-time rendering. On high-definition stereoscopic display devices such as Apple Vision Pro, it can maintain smooth operation at 90 frames per second.
TaoAvatar Project Address
- Project Official Website：https://pixelai-team.github.io/TaoAvatar/
- arXiv Technical Paper：https://arxiv.org/pdf/2503.17032
Application Scenarios of TaoAvatar
- E-Commerce Live Streaming: Create realistic virtual streamers, enhancing user experience and reducing labor costs.
- Holographic Communication: Generate realistic virtual avatars in remote communication, enhancing immersion.
- Virtual Meetings: Participants can communicate using personalized virtual avatars, enhancing interactivity.
- Online Education: Use virtual humans for online course teaching to increase engagement.
- Virtual Entertainment: Create personalized virtual characters in games and virtual reality applications.