What is AvatarGO
AvatarGO is a novel framework jointly launched by Nanyang Technological University S-Lab, Shanghai AI Lab, and the University of Hong Kong. It is used to directly generate animatable 4D human-object interaction scenes from text input. Through a zero-shot method based on pre-trained diffusion models, it solves the problem that traditional methods are limited by the lack of large-scale interaction data when generating daily HOI scenes. The core of AvatarGO includes: LLM-guided contact relocation, which identifies contact parts from text prompts based on Lang-SAM, ensuring accurate representation of the spatial relationship between the human and the object; correspondence-aware motion optimization, which uses the linear blend skinning function of SMPL-X to construct a motion field, optimizes the animation of the human and the object, and reduces penetration issues. The AvatarGO framework demonstrates superior generation and animation capabilities under various human-object combinations and diverse poses.
Main features of AvatarGO
- Generate 4D interaction scenes from text: Directly generate 4D animations containing human-object interactions based on simple text descriptions.
- Accurate human-object contact representation: Accurately identify the contact parts between the human and the object (such as hands, feet, etc.), ensuring that in the generated 3D and 4D scenes, the spatial relationship between the human and the object is reasonable.
- Solve penetration issues in animation: During the animation generation process, effectively avoid penetration between the human and the object.
- Diverse 4D animation generation: Generate dynamic 4D animations, supporting a variety of human actions and object interactions.
- Support multiple human and object combinations: Handle various combinations of humans and objects, including virtual characters (such as anime characters, superheroes) and real people, as well as various everyday objects (such as weapons, tools, musical instruments, etc.).
Technical principles of AvatarGO
- LLM-guided contact relocation: Lang-SAM (Language Segment Anything Model) extracts contact parts (such as "hand") from text descriptions. Based on rendering the 3D human model into a 2D image, combined with text prompts, it generates segmentation masks for the human contact parts. The masks are back-projected into the 3D model to initialize the position of the object, ensuring that the contact parts between the object and the human are accurate.
- Spatially aware score distillation sampling: Introduces SSDS to enhance the attention weights of text tokens related to human-object interaction (such as "holding"), helping the diffusion model understand the spatial relationship between the human and the object.
- Correspondence-aware motion optimization: The SMPL-X model serves as an intermediary to construct motion fields for the human and the object. Based on the linear blend skinning (LBS) function, the motion of the object is optimized synchronously with the motion of the human. A new training objective—correspondence-aware loss—is introduced to minimize the spatial deviation between the human and the object, ensuring that penetration does not occur between the two during animation.
- 3D and 4D generation based on diffusion models：
- 3D generation: Use the DreamGaussian method to generate high-quality 3D human and object models. The scene is represented based on 3D Gaussian splatting, and score distillation sampling (SDS) optimizes the generation results.
- 4D animation generation: On the basis of the 3D models, dynamic 4D animations are generated based on HexPlane features and the SMPL-X model. The global parameters of the object (such as rotation, translation) and the human motion sequence are optimized to generate coherent and realistic 4D animations.
AvatarGO project address
- Project website：https://yukangcao.github.io/AvatarGO
- GitHub repository：https://github.com/yukangcao/AvatarGO
- arXiv technical paper：https://arxiv.org/pdf/2410.07164
Application scenarios of AvatarGO
- Virtual shopping guide: Provides customers with product information and shopping suggestions in stores.
- Exhibition hall docent: Introduces exhibits and product information in museums or exhibition halls.
- Digital lobby manager: Provides consultation and guidance services in banks or business halls.
- In-car virtual assistant: Acts as an intelligent assistant in cars, providing companionship and interactive experiences.
- VR/AR content creation: Generates 4D animations for character and interaction design in virtual reality and augmented reality.