What is MIMO
MIMO is a novel AI framework for controllable character video synthesis launched by the Intelligent Computing Research Institute of Alibaba Group. Based on spatial decomposition modeling technology, it converts 2D videos into 3D spatial codes, achieving precise control over characters, actions, and scenes. MIMO can handle the synthesis of arbitrary characters, adapt to novel 3D actions, and interact with real-world scenes. The core of MIMO lies in decomposing the video into three parts: the main character, the underlying scene, and floating occlusions, which are then respectively encoded into identity codes, motion codes, and scene codes, serving as control signals for the synthesis process. It not only improves the realism of the synthesized video but also enhances the user's ability to control the video content.
Main Features of MIMO
- Controllable character synthesis: Users control the appearance of the character in the video by providing simple inputs.
- Motion control: MIMO can synthesize the character's actions based on the provided pose sequence, including complex 3D actions.
- Scene interaction: Naturally integrates the character into real-world scenes, including handling occlusions and object interactions.
- Spatial decomposition modeling: Decomposes the video into different spatial components, including the main character, the underlying scene, and floating occlusions.
- 3D-aware synthesis: Improves the realism and depth perception of the synthesized video based on 3D representations.
- Flexible user control: Users freely combine different latent codes to control various aspects of video synthesis.
- Scalability for arbitrary characters: MIMO can synthesize arbitrary characters, not limited to characters in the training dataset.
Technical Principles of MIMO
- 3D depth estimation: Uses a monocular depth estimator to convert 2D video frames into representations in 3D space.
- Spatial decomposition: Based on the estimated 3D depth information, decomposes the video into three main spatial components:
- Main character: The core object in the video.
- Underlying scene: The background part of the video.
- Floating occlusions: Foreground objects in the video, such as objects that occlude the subject.
- Component encoding: Further encodes the above spatial components into different codes:
- Identity code: Represents the visual identity of the character.
- Motion code: Represents the character's actions and poses.
- Scene code: Represents the background scene information of the video.
- Structured motion representation: Uses a deformable human body model (such as SMPL) to represent and encode the character's motions, allowing more precise capture of complex 3D movements.
- Canonical identity representation: Transforms the character into a canonical pose (such as A-pose) to decouple identity and motion, making the identity representation independent of motion.
- Scene and occlusion encoding: Uses a shared variational autoencoder (VAE) to encode the scene and occlusion components, which are then reorganized into a complete scene code.
MIMO Project Address
- Project official website：menyifang.github.io/projects/MIMO/index.html
- GitHub repository：https://github.com/menyifang/MIMO
- arXiv technical paper：https://arxiv.org/pdf/2409.16160
Application Scenarios of MIMO
- Film and video production: Rapidly generate performances of animated characters, reducing the cost and time of special effects and animation production.
- Game Development: Create realistic movements and interactions for game characters, enhancing the gaming experience.
- Virtual Reality (VR): Create dynamic characters that interact with users in virtual environments, enhancing immersion.
- Augmented Reality (AR): Overlay virtual characters and objects in the real world for education, entertainment, or navigation.
- Social Media and Entertainment: Allow users to create and share virtual character videos with personalized movements and scenes.
- Advertising and Marketing: Produce engaging dynamic advertisements in which characters are customized according to the target audience.
- Education and Training: Create simulated scenes and characters for educational purposes, such as historical reenactments or language learning.