What is Diffutoon
Diffutoon is an AI framework launched by researchers from Alibaba and East China Normal University for convertingvideos into cartoon and anime style. Based on an editable cartoon shading technology using diffusion models, it can convert realistic videos into anime style. By decomposing the task into subtasks such as stylization, consistency enhancement, structure guidance, and shading, this technology achieves high-resolution and long-duration rendering of videos. Diffutoon also features content editing capabilities, allowing video details to be adjusted based on text prompts. It maintains a high level of visual quality and consistency when processing videos, achieving efficient, high-quality video animation processing.
Diffutoon's features
- Cartoon video rendering: Diffutoon uses diffusion models to convert realistic-style videos into videos with a cartoon or anime style, achieving a visually flattened and stylized effect. The conversion includes not only changes in color and texture but also artistic processing of elements such as lighting and contours to imitate the visual effect of hand-drawn animation.
- High-resolution support: Diffutoon can process high-resolution videos and supports resolutions of at least 1536×1536 pixels, ensuring clarity and detail even on enlarged or high-definition display devices, making it suitable for high-quality video production and presentation needs.
- Video editing: Users can edit video content through text prompts, and Diffutoon can recognize and adjust specific parts of the video based on these prompts. The editing function supports modifying the appearance and attributes of characters and scene elements, such as changing clothing colors and adjusting character expressions.
- Inter-frame consistency: Through specific algorithms and techniques, Diffutoon ensures that each frame in a video sequence remains consistent in style and content, avoiding issues such as flickering, sudden color changes, or content discontinuity that may occur during video playback, thereby improving the viewing experience.
- Structure preservation: During video stylization, Diffutoon can identify and preserve key structural information in the video, such as character contours and object edges, ensuring that even after stylization, the main content and shapes of the video remain clearly recognizable.
- Automatic shading: Diffutoon has an automatic shading function that can automatically select suitable colors for filling based on video content and style requirements. Automatic shading not only improves production efficiency but also ensures color coordination and visual appeal, making the final video more harmonious in color.
Diffutoon official website entry
- Official project homepage:https://ecnu-cilab.github.io/DiffutoonProjectPage/
- GitHub code repository:https://github.com/modelscope/DiffSynth-Studio
- arXiv technical paper:https://arxiv.org/abs/2401.16224
Diffutoon's technical principles
- Application of diffusion models: Diffutoon uses diffusion models as the core technology for image synthesis, learning the distribution characteristics of images and videos in datasets to achieve conversion from a high-dimensional latent space to image data.
- Multi-module denoising: Diffutoon builds a multi-module denoising model that combines technologies such as ControlNet and AnimateDiff to address controllability and consistency issues in videos.
- Stylization, consistency enhancement, structure guidance, and shading: Diffutoon decomposes the cartoon shading problem into four sub-problems, each solved by a specific model:
- Stylization: Achieving anime stylization using a personalized Stable Diffusion model.
- Consistency Enhancement: Maintaining content consistency between video frames by inserting AnimateDiff-based motion modules into the UNet.
- Structural Guidance: Using a ControlNet model to extract and preserve the structural information of the video, such as contours.
- Coloring: Another ControlNet model is used for coloring to improve video quality, even when the input video resolution is low.
- Sliding Window Method: A sliding window method is adopted to iteratively update the latent embedding of each frame; this method helps process long videos and maintain coherence between frames.
- Editing Branch: In addition to the main cartoon coloring pipeline, Diffutoon also includes an editing branch for generating editing signals based on text prompts, which are provided to the main pipeline in the form of colored video.
- Efficient Attention Mechanism: By introducing Flash Attention, GPU memory usage is reduced and the efficiency of processing high-resolution video is improved.
- Classifier-Free Guidance: A classifier-free guidance mechanism is used to optimize visual quality through text prompts.
- DDIM Scheduler: The DDIM (Denoising Diffusion Implicit Models) scheduler is used to control the video generation process, balancing generation quality and speed.
- Post-Processing Methods: Post-processing techniques such as FastBlend are used to further enhance the long-term consistency and visual effects of the video.