What is MotionCLR
MotionCLR is an attention-based human motion generation and editing model that can generate motions based on text prompts and supports interactive user editing, such as motion emphasis, weakening, replacement, erasure, and style transfer. MotionCLR understands and edits motions based on self-attention and cross-attention mechanisms, supporting diverse motion editing tasks such as motion emphasis/weakening, in-place motion replacement, and example-based motion generation. The model performs well on the HumanML3D dataset, generating high-quality motions with precise alignment to text descriptions, and achieves state-of-the-art levels in diversity, interpretability, and editing capability.
Main Features of MotionCLR
- Text-driven motion generation: Generate corresponding motion sequences based on text descriptions provided by users.
- Motion emphasis and weakening: Adjust the weights of motion keywords in the text to enhance or weaken the generated motions.
- In-place motion replacement: Support users in replacing specific motions while keeping other motions unchanged.
- Example-based motion generation: Generate new motion sequences similar to a given example motion.
- Motion style transfer: Apply the style of one motion to another motion to generate motion sequences with new style characteristics.
- Motion sequence editing: Support more complex editing of motion sequences, such as adjusting motion order, changing motion duration, etc.
Technical Principles of MotionCLR
- Self-attention mechanism: Capture the temporal relationships between frames within a motion sequence to ensure the coherence and naturalness of the motion.
- Cross-attention mechanism: Establish fine-grained correspondences between text descriptions and motion sequences, so that the generated motions accurately reflect the content of the text.
- Diffusion model: Use a diffusion model to progressively improve the generated motions, making the model more refined and realistic.
- Training-free editing: The model supports motion editing without retraining, improving the convenience and practicality of editing.
- Attention map manipulation:Achieve motion editing based on directly manipulating attention maps, such as adjusting attention weights to change the emphasis or weakening of motions.
Project Address of MotionCLR
- Project Website：lhchen.top/MotionCLR
- GitHub Repository：https://github.com/IDEA-Research/MotionCLR
- arXiv Technical Paper：https://arxiv.org/pdf/2410.18977
- Online Experience Demo：https://huggingface.co/spaces/EvanTHU/MotionCLR
Application Scenarios of MotionCLR
- Game development: Generate game character animations and provide natural and diverse character motions.
- Animation production: Assist animators in quickly generating or modifying character motions, improving animation production efficiency.
- Virtual Reality (VR) and Augmented Reality (AR): Generate realistic user motions for use in VR/AR interactive experiences.
- Film and television production: Use for pre-visualizing motion scenes or providing motion references for post-production.
- Human-computer interaction: In interactive systems that require human motion as input, such as motion capture and game control.