What is SOLAMI
SOLAMI is an innovative VR 3D role-playing AI system launched by a research team at Nanyang Technological University. It supports users in engaging in immersive interactions with virtual characters using voice and body language, and is based on a social vision-language-action model, providing a natural communication experience that goes beyond traditional text and voice interaction. SOLAMI is driven by an end-to-end VLA model, can recognize users' body language and respond, and supports interaction with multiple characters, such as dancing and playing games. SOLAMI brings a new immersive experience to AI role-playing games.
Main Features of SOLAMI
- Immersive Interaction: Users interact naturally with 3D virtual characters in a VR environment using voice and body language.
- Multimodal Response: The system can generate corresponding character voice and action responses based on the user's voice and motion input.
- Character Diversity: Supports multiple characters, including superheroes, robots, anime characters, etc., providing a rich interactive experience.
- Interactive Games: Supports simple interactive games with characters, such as rock-paper-scissors.
Technical Principles of SOLAMI
- Social Vision-Language-Action Model (Social VLA): Uses an end-to-end VLA model to process the user's voice and motion input and generate the character's response.
- Multimodal Input Processing: Based on Motion Tokenizer and Speech Tokenizer, the user's voice and motions are converted into tokens that the model can understand.
- LLM Foundation: Uses a large language model (LLM) as the foundation to process the input tokens and autoregressively output the character's voice and motion tokens.
- Motion Representation: The user's motions are represented by SMPL-X 3D rotations and encoded using VQ-VAE.
- Speech Processing: The user's speech is encoded using an RVQ-VAE structure and decoded using SoundStorm to achieve voice cloning.
- Training Process: Includes multi-task pretraining and instruction fine-tuning training, allowing the model to learn the associations between motions, speech, and text and handle multi-turn multimodal dialogues.
Project Address of SOLAMI
- Project Official Website：solami-ai.github.io
- arXiv Technical Paper：https://arxiv.org/pdf/2412.00174
Application Scenarios of SOLAMI
- Virtual Social Interaction: Users engage in social interaction with AI characters in a virtual environment, simulating real conversations and nonverbal communication.
- Game Interaction: In VR games, acts as an NPC (non-player character), interacting with players more naturally and enhancing the gaming experience.
- Education and Training: Simulates the role of a teacher or student, providing educational scenarios such as language learning and social skills training.
- Psychotherapy: Simulates the role of a therapist in virtual reality, helping users with psychotherapy and exposure therapy for social phobia.
- Entertainment and Performance: Users interact with virtual singers, dancers, or actors and enjoy an immersive entertainment experience.