What is TANGO
TANGO is an open-source framework jointly launched by the University of Tokyo and CyberAgent AI Lab, focused on generating full-body gesture videos synchronized with target speech. Based on hierarchical audio-motion embedding and a diffusion interpolation network, it perfectly matches target speech audio with motions from a reference video library, ensuring the production of high-fidelity, motion-synchronized videos. TANGO's technological breakthrough greatly reduces the cost of video content production, including fields such as news broadcasting, virtual human narration, and virtual YouTube content creation, providing users with an efficient and economical solution.
TANGO's Main Features
- Speech-Driven Full-Body Gesture Generation: Generates full-body gesture videos synchronized with the target speech audio.
- High-Fidelity Video Production: Ensures the generated videos have high fidelity, with natural motions that precisely match the speech content.
- Cross-Modal Alignment: Based on hierarchical audio-motion embedding technology, achieves precise alignment between audio signals and video motions.
- Transition Frame Generation: Uses a diffusion interpolation network to generate high-quality transition frames, ensuring the coherence of video motions.
- Appearance Consistency Preservation: Maintains the same character appearance and background as the reference video in the generated videos, avoiding visual inconsistency issues.
TANGO's Technical Principles
- Hierarchical Audio-Motion Embedding (AuMoCLIP): An implicit hierarchical audio-motion joint embedding space, used for encoding paired audio and motion data. Based on contrastive learning, it maps speech audio and motion data into a common latent space, making matched audio and motion closer in the space, achieving precise motion retrieval.
- Diffusion Interpolation Network (ACInterp): The network is based on existing video generation diffusion models, used to generate high-quality transition frames. It includes a reference motion module and a homography background flow, maintaining appearance consistency between the generated video and the reference video. It effectively eliminates the blur and ghosting artifacts commonly seen in traditional flow-based interpolation methods.
- Motion Graph Retrieval Method: TANGO is based on a learning-based method, rather than simple audio onset features and keyword matching, to retrieve the motion path that best matches the target speech audio. It can better handle situations where different speakers' motions are not synchronized with audio onsets, and the problem of missing specific keywords in the reference video.
- Graph Structure: TANGO uses a directed graph structure to represent video frames (nodes) and valid transitions between them (edges). Given the target audio, the system extracts temporal features and uses them to retrieve a subset of the video playback path. When transition edges do not exist in the original reference video, ACInterp is used to generate smooth transition frames.
TANGO's Project Address
- Project Website：pantomatrix.github.io/TANGO
- arXiv Technical Paper：https://arxiv.org/pdf/2410.04221
- Online Experience Demo：https://huggingface.co/spaces/H-Liu1997/TANGO
TANGO's Application Scenarios
- News Broadcasting: Generates full-body gesture videos synchronized with news scripts, improving the naturalness of news broadcasting and the audience's viewing experience.
- Virtual YouTuber: Creates full-body motion videos synchronized with speech for virtual YouTubers, enhancing fan interaction and engagement.
- Online Education: When producing educational content, TANGO is used to generate teachers' full-body gesture videos, making remote teaching more vivid and effective.
- Corporate Training: Add gestures synchronized with the narration in corporate training videos to improve the appeal of learning materials and the efficiency of information delivery.
- Video Conferencing: In video conferencing, use gesture videos generated by TANGO to enhance participants' communication experience, especially during remote collaboration.