What is MT-Color
MT-Color is a controllable image colorization framework based on diffusion models, jointly launched by Shanghai Jiao Tong University and Bilibili. It achieves precise instance-level image colorization based on user-provided instance-aware text and masks. The framework uses a pixel-level mask attention mechanism to prevent color overflow, employs instance mask and text guidance modules to address color binding errors, and uses a multi-instance sampling strategy to enhance instance-aware effects. MT-Color has constructed the GPT-Color dataset, providing high-quality instance-level annotations and supporting finer-grained image colorization tasks. MT-Color outperforms existing methods in color accuracy and visual quality, and the generated images are more consistent with human visual perception.
Main Features of MT-Color
- Precise instance-level colorization: Based on user-provided instance masks and text descriptions, it precisely colorizes different objects in the image, ensuring that each object's color is consistent with its description.
- Prevention of color overflow: Based on a pixel-level mask attention mechanism, it effectively avoids erroneous diffusion of colors between different objects and keeps color boundaries clear.
- High-quality color generation: The generated color images are rich and natural in color, with high resolution (512×512) and clearer details.
- Flexible user control: It supports users in finely controlling the image colorization process with text descriptions and masks, meeting different colorization needs.
- Dataset support: It constructs a dedicated GPT-Color dataset, providing high-quality instance-level annotations and supporting finer-grained image colorization tasks.
Technical Principles of MT-Color
- Pixel-level mask attention mechanism: It aligns the grayscale image features extracted by ControlNet with the latent features of Stable Diffusion through a pixel-level cross-attention mechanism, preventing pixel information exchange between different instances and alleviating the color overflow problem. Segmentation masks are used to construct cross-attention masks, ensuring that pixel information does not interfere with each other between different instances.
- Instance mask and text guidance module: It extracts the mask and text representation of each instance and fuses them with latent features based on a self-attention mechanism. Based on the instance mask, a self-attention mask is formed to prevent instance text from guiding the colorization of other regions, reducing color binding errors.
- Multi-instance sampling strategy: During the sampling process, each instance region is sampled separately, and the results are fused to further enhance instance-aware colorization effects.
- GPT-Color dataset: Based on pre-trained vision-language models (such as GPT-4 and BLIP-2), it automatically generates high-quality instance-level annotations on existing image datasets, including fine-grained text descriptions and corresponding segmentation masks, providing rich training data for model training.
Project Address of MT-Color
- arXiv technical paper：https://arxiv.org/pdf/2505.08705
Application Scenarios of MT-Color
- Historical photo restoration: Add vivid colors to black-and-white historical photos, enhancing their观赏性 and sense of history.
- Film and television post-production: Restore and enhance black-and-white or low-quality color footage, improving visual effects.
- Artistic creation: Help artists add color to black-and-white sketches or illustrations and realize their creativity.
- Medical image processing: Add color to medical images to improve diagnostic value.
- Education field: Add color to black-and-white images in teaching materials to enhance the learning experience.