What is ColorFlow
ColorFlow is an image sequence colorization model jointly launched by Tsinghua University and Tencent ARC Lab. It can perform colorization while finely preserving the individual identity in an image sequence. Based on retrieval augmentation, in-context learning, and super-resolution technology, ColorFlow ensures that the colorization of black-and-white image sequences is color-consistent with reference images, making it suitable for industrial applications such as comics and animation production. In the field of image sequence colorization, ColorFlow demonstrates outstanding performance beyond existing technologies based on improving CLIP-IS, reducing FID, increasing PSNR and SSIM scores, and improving AS, providing a new colorization standard for the art industry.
ColorFlow's main features
- Retrieval-Augmented Pipeline (RAP): Extracts the color image patches most relevant to the input black-and-white image from a reference image pool to guide the colorization process.
- In-context Colorization Pipeline (ICP): Accurately retrieves color identity based on powerful in-context learning, and uses a dual-branch design for colorization.
- Guided Super-Resolution Pipeline (GSRP): Upsamples low-resolution colorization output to produce high-resolution color images, enhancing detail recovery and improving output quality.
ColorFlow's technical principles
- Retrieval-Augmented: Generates embeddings of the input image and reference images based on a pretrained CLIP image encoder, calculates cosine similarity to identify the most similar reference image patches, and uses them in subsequent colorization training.
- In-context Colorization: Introduces an auxiliary branch, the "Colorization Guider," to integrate conditional information, and gradually integrates features based on a U-Net diffusion model to achieve pixel-level conditional embedding. Uses a lightweight LoRA (Low-Rank Adaptation) method to fine-tune the pretrained diffusion model while preserving its colorization capability.
- Super-Resolution Enhancement: Handles the downsampling problem in the colorization process and reduces structural distortion. Combines high-resolution black-and-white images with low-resolution color output to enhance detail recovery and improve output quality.
- Self-Attention Mechanism: Uses a self-attention mechanism in the diffusion model to place the reference image and grayscale image on the same canvas, extract features, and input them layer by layer into the diffusion model for colorization.
- Timestep Shifted Sampling: Adjusts the sampling strategy to emphasize sampling at higher timesteps, enhancing the effectiveness of the colorization process.
- Screenstyle Augmentation: Randomly linearly interpolates grayscale images and ScreenVAE outputs to enhance the style adaptability of the input image and improve colorization performance.
ColorFlow's project address
- Project official website：zhuang2002.github.io/ColorFlow
- GitHub repository：https://github.com/TencentARC/ColorFlow
- HuggingFace model library：https://huggingface.co/TencentARC/ColorFlow
- arXiv technical paper：https://arxiv.org/pdf/2412.11815
- Online experience Demo：https://huggingface.co/spaces/TencentARC/ColorFlow
ColorFlow's application scenarios
- Comic and cartoon colorization: Converts black-and-white comic or cartoon images into color versions, suitable for comic artists and animators to quickly color line art during the creation process.
- Old photo restoration: Provides color for historical black-and-white photos, helping restore the original appearance of photos from old times and increasing the viewing value of historical images.
- Film and video post-production: Used for colorizing black-and-white films, or providing specific color grading effects for modern film production.
- Artistic creation: Artists add color to black-and-white artworks, or explore different color schemes during the creative process.
- Education and learning: In the field of education, as a tool to help students understand the impact of color on images, and learn the basic concepts of image processing and computer vision.