What is Sketch2Anim
Sketch2Anim is an automated framework jointly launched by the University of Edinburgh, Snap Research, and Northeastern University, capable of directly converting 2D sketch storyboards into high-quality 3D animations. Based on conditional motion synthesis technology, it uses 3D key poses, joint trajectories, and action words to precisely control animation generation. The framework includes two core modules: a multi-condition motion generator and 2D-3D neural mappers. Sketch2Anim can generate natural and smooth 3D animations and supports interactive editing, greatly improving the efficiency and flexibility of animation production.
Main features of Sketch2Anim
- Automatic conversion: Quickly converts 2D sketch storyboards into 3D animations, reducing the cumbersome steps of manual production.
- High-quality animation: Generates smooth and natural 3D animations, supporting seamless transitions between multiple segments.
- Interactive editing: Users can adjust the generated animation in real time, such as modifying key poses or trajectories, enhancing design flexibility.
- Improved efficiency: The automated process significantly improves animation production efficiency and supports rapid design iteration.
Technical principles of Sketch2Anim
- Multi-condition motion generator: Based on a motion diffusion model, it combines trajectory ControlNet and a keypose adapter. Trajectory ControlNet is responsible for embedding joint trajectories into the motion diffusion model to control global dynamic motion. On the basis of trajectory ControlNet, the keypose adapter further refines local static poses to ensure that the generated animation accurately matches the input key poses.
- 2D-3D neural mappers: Includes 2D and 3D key pose and trajectory encoders. It aligns 2D sketches and 3D motion constraints in a shared embedding space to achieve direct 2D-to-3D animation control.
- Motion fusion and optimization: Based on the deterministic DDIM inversion process and guided denoising, it smoothly transitions multiple animation segments to generate a complete animation. Classifier-free guidance and second-order optimization methods are used to further optimize the generated animation so that it better matches the input 2D conditions.
- Data processing and augmentation: Trained using the HumanML3D dataset, which contains 14,646 motions and 44,970 corresponding motion annotations. Based on camera view augmentation, joint perturbation, and body proportion perturbation, it generates diverse 2D key poses and trajectories, improving the model's generalization ability.
Project address of Sketch2Anim
- Project website：https://zhongleilz.github.io/Sketch2Anim/
- arXiv technical paper：https://arxiv.org/pdf/2504.19189
Application scenarios of Sketch2Anim
- Film and television animation: Used for pre-production previews in film and television production to improve efficiency.
- Game development: Accelerates game action design and story animation production.
- Advertising and promotion: Turns creative sketches into 3D animated advertisements, enhancing visual effects and attracting audiences.
- Education and training: Helps students better understand complex knowledge.
- VR/AR applications: Directly converts 2D sketches into immersive 3D animations, enriching virtual scenes.