What is MoCha
MoCha AI is an end-to-end conversational charactervideo generationmodel jointly developed by Meta and the University of Waterloo. It can generate complete character animations with synchronized speech and natural movements based on text or speech input. MoCha adopts a speech-video window attention mechanism, solving the problems of audio resolution mismatch and lip movement misalignment during video compression. It supports multi-character turn-taking conversations and can generate character animations with emotional expression and full-body movements.
MoCha's main features
- Speech-driven character animation generation: Users input speech, and MoCha can generate character lip movements, facial expressions, gestures, and body movements synchronized with the speech content.
- Text-driven character animation generation: Users input only a text script, and MoCha first automatically synthesizes speech, then drives the character to perform complete lip movements and actions.
- Full-body animation generation: Unlike traditional models that generate only facial expressions or lip movements, MoCha can generate natural full-body motion, covering lip synchronization, gestures, and interactions between multiple characters.
- Multi-character turn-taking dialogue generation: MoCha provides structured prompt templates and character tags, can automatically identify dialogue turns, and enables natural back-and-forth dialogue presentation between characters. In multi-character scenes, users only need to define character information once and can reference these characters in different scenes through simple tags (such as "Character 1" and "Character 2") without repeated descriptions.
MoCha's technical principles
- Diffusion Transformer (DiT) architecture: MoCha is based on the Diffusion Transformer architecture, effectively capturing semantics and temporal dynamics by sequentially incorporating text and speech conditions into the model through cross-attention mechanisms. It can generate realistic and expressive full-body movements while ensuring precise synchronization between character animation and input speech.
- Speech-video window attention mechanism: To solve the speech-video alignment problems caused by video compression and parallel generation, MoCha introduces a speech-video window attention mechanism. It restricts each video token to attending only to audio tokens temporally adjacent to it, improving the accuracy of lip synchronization and the effectiveness of speech-video alignment. It simulates how human speech works, enabling the character's lip movements to precisely match the dialogue content.
- Joint training strategy: MoCha adopts a joint training strategy, training simultaneously on video data annotated with speech and text annotations. It enhances the model's generalization ability across diverse character movements and enables fine-grained control over character expressions, actions, interactions, and environments through natural language prompts.
- Structured prompt templates: To simplify the text descriptions of multi-character dialogues, MoCha designs structured prompt templates, assigning a unique tag to each character and using these tags in the text to describe the characters' actions and interactions. This reduces redundancy and improves the model's generation performance in multi-character scenes.
- Multi-stage training framework: MoCha adopts a multi-stage training framework, classifying data according to shot types (such as close-up shots and medium shots) and gradually introducing more complex tasks. This ensures the model's performance on tasks of varying difficulty while improving training efficiency.
MoCha's project address
- Project official website：https://congwei1230.github.io/MoCha/
- arXiv technical paper：https://arxiv.org/pdf/2503.23307
MoCha's application scenarios
- Virtual Streamer: MoCha can automatically generate content such as daily vlogs and character Q&As. Through voice or text input, it generates character lip sync, facial expressions, gestures, and body movements synchronized with the voice content, making virtual streamers more vivid and natural.
- Animation and Film Creation: MoCha supports AI automatic dubbing and automatic animation generation, which can reduce the production cost of animation and film creation. It can generate full-body animation, making character movements more natural and approaching cinematic-level digital human performances.
- Educational Content Creation: MoCha can serve as an AI teacher character to give lectures or interact. Through text-driven methods, it generates character animations that match the teaching content, improving the fun and appeal of educational content.
- Digital Human Customer Service: MoCha can be used for anthropomorphic enterprise customer service and consultation roles. Through voice or text input, it generates natural and smooth customer service dialogue animations, enhancing the user experience.