What is GAS
GAS (Generative Avatar Synthesis from a Single Image) is a framework proposed by researchers from Carnegie Mellon University, Shanghai Artificial Intelligence Laboratory, and Stanford University for generating high-quality, view-consistent, and temporally coherent avatars from a single image. The core of GAS lies in combining the strengths of regression-based 3D human reconstruction models and diffusion models. Based on the 3D human reconstruction model, an intermediate view or pose is generated from a single image and used as a conditioning input to a video diffusion model, achieving high-quality view consistency and temporal coherence. The framework introduces a "mode switcher" to distinguish between view synthesis and pose synthesis tasks, further improving generation quality.
Main features of GAS
- View-consistent multi-view synthesis: Generates high-quality multi-view renderings from a single image, ensuring consistency in appearance and structure across different views.
- Temporally coherent dynamic pose animation: Through a given pose sequence, generates smooth and realistic non-rigid deformation animations, achieving coherence in dynamic poses.
- Unified framework and generalization capability: Unifies view synthesis and pose synthesis tasks. By sharing model parameters and leveraging large-scale real-world data (such as web videos) for training, it improves the model's generalization ability to real-world scenarios.
- Dense appearance prompts: Dense information generated by the 3D reconstruction model is used as conditioning input, ensuring high fidelity in appearance and structure of the generated results.
Technical principles of GAS
- 3D human reconstruction and dense conditioning signals: GAS first uses a regression-based 3D human reconstruction model (such as single-view generalizable human NeRF) to generate an intermediate view or pose from the input image. By mapping the input image to canonical space and repositioning it, dense appearance prompts are generated. These dense representations are used as conditioning input, providing rich detail and structural information for the subsequent diffusion model, ensuring high quality and consistency of the generated results.
- Video diffusion model and unified framework: The generated intermediate view or pose is used as conditioning input to a video diffusion model, generating high-quality view-consistent and temporally coherent animations through the diffusion model. GAS proposes a unified framework that merges view synthesis and pose synthesis tasks, shares model parameters, and achieves natural generalization from pose synthesis to view synthesis.
- Mode switcher: To distinguish between view synthesis and pose synthesis tasks, GAS introduces a mode switcher module. The module enables the network to focus on consistency when generating views, and on realistic deformation when generating poses.
- Generalization capability on real-world data: By combining large-scale real-world videos (such as web videos) for training, GAS significantly improves generalization ability to real-world scenarios. The diversity of data sources enables the model to adapt to various lighting, clothing, and motion conditions.
- Training and inference: GAS training is divided into two stages: first train the 3D human reconstruction model, then freeze that model and train the video diffusion model. During inference, different classifier-free guidance (CFG) strategies are used depending on the nature of the task (view synthesis or pose synthesis).
Project address of GAS
- Project website：https://humansensinglab.github.io/GAS/
- arXiv technical paper：https://arxiv.org/pdf/2502.06957
Application scenarios of GAS
- Games and virtual reality (VR): GAS can generate high-quality avatars from a single image, supporting coherent synthesis across multiple views and dynamic poses.
- Film and television production: In film and television special effects and animation production, GAS can quickly generate realistic virtual characters, reducing the time and cost of traditional modeling and animation production.
- Sports and fitness: By generating dynamic virtual avatars from a single image, GAS can be used to create personalized sports animations, helping athletes analyze movements or for use in fitness applications.
- Fashion and apparel design: GAS can generate virtual avatars in different poses and views, helping designers quickly preview clothing effects and improve design efficiency.