What is Large Motion Model
Large Motion Model (LMM) is a unified multimodal motion generation model jointly launched by the S-Lab at Nanyang Technological University, Singapore, and the research team at SenseTime. LMM can handle a variety of motion generation tasks, including text-to-motion, music-to-dance, and more, and demonstrates performance comparable to expert models across multiple benchmarks. The model is based on the comprehensive MotionVerse dataset created by integrating datasets of different modalities, formats, and tasks, and adopts the innovative ArtAttention mechanism and pretraining strategies to achieve precise control over body parts and broad knowledge generalization. LMM shows strong generalization ability when handling unseen tasks, providing a new perspective for future research on large motion models.
Main Features of Large Motion Model
- Multi-task motion generation: Can perform a variety of motion generation tasks, such as text-to-motion, music-to-dance, action-to-motion, etc.
- Dataset integration: Creates the MotionVerse dataset, based on integrating datasets of different modalities, formats, and tasks, to achieve unified motion representation.
- Precise control: Adopts the ArtAttention mechanism to support precise control over different body parts, improving the fineness of motion generation.
- Generalization ability: Demonstrates strong generalization ability, effectively generating motion on a variety of unseen tasks.
- Multimodal input processing: Simultaneously processes multiple modal inputs, such as text, music, video, etc., to generate corresponding motion outputs.
Technical Principles of Large Motion Model
- Unified dataset (MotionVerse): Based on the unified dataset MotionVerse, the dataset contains motion data from multiple tasks and modalities, and unifies motion data in different formats based on the TOMATO representation.
- Diffusion Transformer backbone network: A Diffusion model based on the Transformer architecture, using the denoising diffusion probabilistic model (DDPM) for high-quality motion sequence generation.
- ArtAttention mechanism: Designs a novel attention mechanism, ArtAttention, which combines body part-aware modeling and supports the model in independently controlling and learning different body parts.
- Pretraining strategy: Adopts a pretraining strategy with random frame rates and multiple masking techniques to enhance the model's learning ability and generalization ability for different data sources.
- Zero-shot learning: Uses a zero-shot method to generate long-sequence motion, allowing the model to generate motion without additional samples.
Project Address of Large Motion Model
- Project official website：https://mingyuan-zhang.github.io/projects/LMM
- GitHub repository：https://github.com/mingyuan-zhang/LMM
- arXiv technical paper：https://arxiv.org/pdf/2404.01284
- Online experience Demo：https://huggingface.co/spaces/mingyuan/LMM
Application Scenarios of Large Motion Model
- Animation and game production: Generates realistic character animations, reducing the time and cost of manual animation production and improving the efficiency of animation production.
- Virtual reality (VR) and augmented reality (AR): In VR and AR applications, generates virtual character actions that match user movements, enhancing immersion.
- Film and video production: Generates special effects in films, such as simulating complex fight scenes or dance movements, improving production efficiency.
- Motion analysis and training: Analyze athletes' movements, provide training suggestions, or generate standard movement templates.
- Robotics: Teach robots to perform complex human movements and improve robots' application capabilities in the service, medical, or industrial fields.