SCAIL-2 - Zhipu and Tsinghua Open-Source Character Animation Model

What is SCAIL-2

SCAIL-2 is the second generation of the film-grade character animation framework open-sourced by Zhipu in collaboration with Professor Liu Yongjin's research group at Tsinghua University. Based on the Diffusion Transformer (DiT) architecture, it adopts end-to-end training to directly achieve high-quality animation transfer, without the traditional intermediate pose estimation step, avoiding skeleton graph ambiguity. It supports dual modes of animation generation and character replacement, achieving precise matching through SAM3 mask enhancement. It can drive human bodies and can handle non-human targets such as animals and hand-drawn characters, supports multi-person interaction, and provides an efficient animation solution for fields such as film and television, gaming, and virtual idols.

Main Features of SCAIL-2

• End-to-End Animation Generation: Based on the DiT Diffusion Transformer architecture, it directly bypasses the traditional intermediate pose estimation step, achieving high-quality end-to-end transfer from reference images to driving video motions, avoiding the skeleton graph ambiguity problem.
• Animation Mode: Fully transfers the appearance of the reference character onto the motion sequence of the driving video, achieving character motion replication.
• Replacement Mode: Precisely replaces the specified character in the driving video while preserving the original motion trajectory and background environment.
• SAM3 Mask Enhancement: Uses SAM3 to extract explicit masks for the reference image and driving sequence as conditional input, significantly improving the matching accuracy between character and motion.
• Diverse Driving Source Support: It not only supports human motion driving, but can also handle non-human targets such as animals, hand-drawn characters, and robots, and can even control quadruped animals without animal training data.
• Multi-Person Interaction Handling: Supports character motion coordination and relative position relationship handling in multi-person scenes, achieving complex interactive animation.
• 3D-Consistent Pose Representation: Preserves depth information and occlusion relationships, ensuring the spatial plausibility of the generated animation.
• Full-Context Pose Injection: Combines P-RoPE (Pose-shifted RoPE) to achieve joint spatiotemporal reasoning, improving the coherence of long-sequence animation.

Technical Principles of SCAIL-2

• Diffusion Transformer (DiT) Foundation Architecture: Builds a generative model based on the Diffusion Transformer, leveraging its powerful sequence modeling capability to handle long-duration animation generation tasks.
• In-Context Learning Driven: Directly utilizes the inherent in-context learning capability of DiT for motion transfer, without relying on pose estimation such as NLFPose as an intermediate representation, avoiding the ambiguity problem of skeleton graphs in complex scenes.
• 3D-Consistent Pose Representation: Preserves depth information and occlusion relationships inside the model, ensuring the plausibility and consistency of the generated animation in the spatial dimension.
• Full-Context Pose Injection: Through the P-RoPE (Pose-shifted RoPE) mechanism, pose information is injected into the full-sequence context, achieving joint spatiotemporal reasoning and improving the coherence of long-sequence animation.
• SAM3 Explicit Mask Conditions: Uses SAM3 to extract precise masks for the reference image and driving video sequence as explicit conditional input, enhancing the precise alignment between character appearance and motion driving.
• End-to-End Unified Training: Abandoning the traditional "pose estimation → animation generation" two-stage pipeline, it adopts an end-to-end unified training strategy, directly learning motion from the driving video and transferring it to the reference character.
• High-quality data filtering pipeline: Build a high-quality training dataset that has undergone rigorous filtering and manual review, covering diverse targets such as humans, animals, and hand-drawn characters, to ensure the model's generalization ability and generation quality.

How to use SCAIL-2

• Clone the repository and configure the environment: Pull the code from the GitHub repository https://github.com/zai-org/SCAIL-2 , and install the dependency environment according to the README (usually including PyTorch, Diffusers, and related vision libraries).
• Download the pretrained model: Download the SCAIL-2 model weight files from HuggingFace https://huggingface.co/zai-org/SCAIL-2 , and place them in the specified local directory.
• Prepare input materials: Prepare a reference character image (Reference Image) and a driving video (Driving Video); the driving video contains the target motion sequence. The system will automatically extract explicit masks from the reference image and driving sequence via SAM3 as conditional input.
• Select the generation mode: Choose according to your needs Animation Mode(transfer the reference character's appearance onto the driving video's motion) or Replacement Mode(replace the specified character in the driving video while preserving the original motion and background).
• Run the animation generation command: Run the script for the corresponding mode in the terminal, for example, use the end-to-end animation mode:python NLFPoseExtract/process_animation_aio.py --subdir <example_dir> --e2e_mode; or use the replacement mode:python NLFPoseExtract/process_replacement.py --subdir <example_dir>。
• Obtain the generation results: After the script finishes running, obtain the generated character animation video in the output directory, which can be used directly for post-production editing or further processing.

SCAIL-2 project address

• Project official website：https://teal024.github.io/SCAIL-2/
• Github repository：https://github.com/zai-org/SCAIL-2
• HuggingFace model library：https://huggingface.co/zai-org/SCAIL-2

SCAIL-2's core advantages

• End-to-end with no intermediate state: Directly implements animation transfer based on the DiT diffusion transformer, without requiring traditional pose estimation (such as NLFPose) as an intermediate representation, avoiding skeleton diagram ambiguity, making the process simpler and the results more reliable.
• Film-grade generation quality: Adopts a high-quality data filtering pipeline and a 3D-consistent pose representation, preserving depth information and occlusion relationships, with output results reaching film-grade standards.
• Flexible dual-mode application: Supports Animation Mode (appearance transfer) and Replacement Mode (character replacement), meeting the two core needs of animation generation and video editing.
• Diversified driving capability: It not only supports human motion driving, but can also handle non-human targets such as animals, hand-drawn characters, and robots, and can control quadruped animals without corresponding training data.
• Precise mask alignment: Integrates SAM3 to extract explicit masks as conditional input, significantly improving the matching accuracy between character appearance and driving motion.
• Multi-person interaction support: Has the ability to coordinate character motions and handle relative position relationships in multi-person scenes, adapting to complex interaction scenarios.
• Spatiotemporal coherent inference: Achieves joint spatiotemporal inference through the full-context pose injection (P-RoPE) mechanism, ensuring the temporal coherence and motion consistency of long-sequence animations.

Comparison of SCAIL-2 with similar competitors

Application Scenarios of SCAIL-2

• Film and Animation Production: Replaces the traditional motion capture workflow, quickly transfers character designs onto preset motions, and reduces animation production cost and cycle.
• Game Development: Generates animation assets such as walking, combat, and interaction for game characters, or is used for efficient production of cutscenes (CG).
• Virtual Idols and Digital Humans: Drives virtual streamers and digital employees for live interactive broadcasts or short video content creation, enabling real-time or batch animation generation.
• Advertising and Brand Marketing: Embeds brand IP characters or virtual spokespersons into dynamic advertising videos, and supports the character replacement mode to quickly adapt to multiple versions of materials.
• Social Media Content Creation: Converts hand-drawn figures, emojis, or real-person photos into dynamic videos, used for content production and distribution on short video platforms.
• Education and Training: Produces animated content such as historical figure recreations and teaching character demonstrations, enhancing the fun and expressiveness of online courses.