What is HRAvatar
HRAvatar is a monocular video reconstruction technology launched by Tsinghua University in collaboration with the IDEA team, supporting the generation of high-quality, relightable 3D avatars from ordinary monocular videos. HRAvatar uses learnable deformation bases and linear blend skinning, and based on a precise expression encoder, reduces tracking errors and improves reconstruction quality. HRAvatar decomposes avatar appearance into attributes such as albedo, roughness, and Fresnel reflection, and combines them with a physically-based rendering model to achieve realistic relighting effects. HRAvatar outperforms existing methods on multiple metrics and supports real-time rendering (about 155 FPS), providing a new technical solution for digital humans, virtual anchors, AR/VR, and other fields.
Main features of HRAvatar
- High-quality reconstruction: Generates detailed and expressive 3D avatars from ordinary monocular videos.
- Real-time performance: Supports real-time rendering with a speed of about 155 FPS, suitable for real-time applications.
- Relightable: Performs real-time relighting on generated avatars to adapt to different lighting conditions.
- Animation: Supports driving avatars for animation of expressions and movements.
- Material editing: Supports editing the avatar's material attributes (such as albedo, roughness, and reflectivity) to achieve different visual effects.
- Cross-view rendering: Supports rendering avatars from different viewpoints, providing 3D consistency and multi-view support.
Technical principles of HRAvatar
- Precise expression tracking: Uses an end-to-end expression encoder to jointly optimize and extract more accurate expression parameters, reducing the impact of pre-tracking parameter errors on reconstruction. The expression encoder is jointly trained with 3D avatar reconstruction and supervised by a Gaussian reconstruction loss, improving reconstruction accuracy.
- Geometric deformation model: Based on learnable linear blendshapes, introduces shape bases, expression bases, and pose bases for each Gaussian point, enabling flexible deformation from canonical space to pose space. Uses linear blend skinning to transform Gaussian points into pose space, adapting to individual pose deformations.
- Appearance modeling: Decomposes avatar appearance into material attributes such as albedo, roughness, and Fresnel reflectivity. Uses the BRDF physically-based rendering model for shading, combined with a simplified SplitSum approximation technique, to achieve high-quality, relightable real-time rendering. Introduces an albedo pseudo-prior to better disentangle material information and avoid mistakenly mixing local lighting into albedo.
- Normal estimation and material priors: Uses the shortest axis of each Gaussian point as its normal, and supervises the rendered normal map based on the normal map obtained from depth derivatives to ensure geometric consistency. Uses existing models to extract pseudo-ground-truth albedo to supervise the rendered albedo, and constrains roughness and base reflectivity within predefined ranges to obtain more realistic materials.
HRAvatar project address
- Project website：https://eastbeanzhang.github.io/HRAvatar/
- GitHub repository：https://github.com/Pixel-Talk/HRAvatar
- arXiv technical paper：https://arxiv.org/pdf/2503.08224
Application scenarios of HRAvatar
- Digital humans and virtual anchors: Creates realistic digital humans and virtual anchors, supports real-time expressions and movements, and enhances interactivity.
- AR/VR: Generates 3D avatars that can be relit in real time, enhancing the immersion of virtual environments.
- Immersive meetings: Provides high-quality 3D avatars, making remote communication more natural and realistic.
- Game development: Quickly generate high-quality 3D character avatars to enhance the game's visual effects.
- Film and Television Production: Used in special effects production to quickly generate high-quality avatars and improve production efficiency.