What is Perception-as-Control
Perception-as-Control is an image animation framework launched by Alibaba Tongyi Lab, capable of achieving fine-grained motion control according to user intent. Perception-as-Control is based on constructing 3D-aware motion representations, transforming camera and object motion into intuitive and consistent visual changes, using perception results as motion control signals, and supporting a variety of motion-related video synthesis tasks. The Perception-as-Control framework is based on a U-Net architecture diffusion model, combining the appearance information of a reference image with the motion information of motion control signals to generate controllable image animations. Perception-as-Control introduces a three-stage training strategy, balancing camera and object motion control, promoting fine-grained motion control, and demonstrating superior performance in a variety of application scenarios.
Main features of Perception-as-Control
- Fine-grained collaborative motion control: Achieves collaborative control of camera and object motion, supporting users in controlling each element in the animation in a fine-grained manner. Allows users to precisely adjust the motion of objects in the scene and changes in camera viewpoint.
- Multiple motion-related video synthesis tasks: Supports a variety of motion-related video synthesis tasks, including image-based motion generation, video-based motion cloning, motion transfer, and motion editing.
- Motion generation: Generates animations based on a user-input reference image and 2D trajectories.
- Motion cloning: Imitates the camera and object motion in a source video.
- Motion transfer: Transfers local motion from a source video to objects at different positions and scales in a reference image.
- Motion editing: The user provides a segmentation mask to edit the motion within the segmentation mask.
- 3D-aware motion representation: Based on constructing 3D-aware motion representations, simplifying complex 3D scenes into key object parts (represented by unit spheres) and a world envelope, achieving intuitive and consistent visual changes.
Technical principles of Perception-as-Control
- 3D-aware motion representation: Simplifies 3D scenes into key object parts (represented by unit spheres) and a world envelope, capturing local object motion and global camera motion based on 3D point tracking and visual odometry technology.
- Network architecture: Based on a denoising U-Net architecture, using two lightweight encoders to separately encode camera and object control signals, avoiding RGB-level interference, with a fusion module merging the encoded signals.
- Reference image injection: ReferenceNet injects the appearance information of the reference image into the generation process, ensuring that the generated video maintains the specific appearance of the reference image.
- Three-stage training strategy：
- First stage: Trains only the camera encoder, processing video clips containing only camera motion.
- Second stage: Adds the object encoder and fusion module, processing video clips containing both camera and object motion.
- Third stage: Achieves fine-grained object motion control based on sparse unit spheres, adaptively determining the control range of each rendered point.
- Diffusion model: Uses the basic principles of image diffusion probabilistic models, using a denoising U-Net architecture to generate videos, with the optimization objective of minimizing the difference between predicted noise and actual noise, and conditional inputs including the reference image and motion control signals.
Project address for Perception-as-Control
- Project official website：https://chen-yingjie.github.io/projects/Perception-as-Control/
- GitHub repository：https://github.com/chen-yingjie/Perception-as-Control
- arXiv technical paper：https://arxiv.org/pdf/2501.05020
Application scenarios of Perception-as-Control
- Film and video effects: Generate animated scenes with specific motions, imitate complex motions in existing videos, or transfer the motion of one character to another character.
- Game development: Generate natural and controllable animations for game characters and objects, enhancing the immersion and dynamic effects of games.
- Virtual reality (VR) and augmented reality (AR): Generate real-time animation feedback in VR environments to enhance user interaction experience; integrate virtual object animations with real scenes in AR applications.
- Advertising and marketing: Generate attractive dynamic advertisements and brand promotion animations to enhance brand image and memorability.
- Education and training: Generate animations of scientific experiments and simulation animations for skills training to help students and trainees better understand and master knowledge and skills.