What is Jodi
Jodi is a diffusion model framework launched by the Institute of Computing Technology, Chinese Academy of Sciences, and the University of Chinese Academy of Sciences. Based on joint modeling of the image domain and multiple label domains, it unifies visual generation and understanding. Jodi is based on a linear diffusion Transformer and a role-switching mechanism, and performs three tasks: joint generation (simultaneously generating images and multiple labels), controllable generation (generating images based on label combinations), and image perception (predicting multiple labels from images). Jodi is trained on the Joint-1.6M dataset, which contains 200,000 high-quality images and 7 visual domain labels. Jodi performs excellently in both generation and understanding tasks, demonstrating strong scalability and cross-domain consistency.
Jodi's Main Features
- Joint Generation: Simultaneously generates images and multiple labels, such as depth maps, normal maps, edge maps, etc. The generated images and labels remain consistent in semantics and space.
- Controllable Generation: Generates images according to a given label combination. Users specify certain labels as conditional inputs to control specific attributes or features of the generated image.
- Image Perception: Simultaneously predicts multiple labels from a given image, enabling multi-dimensional understanding and analysis of the image, such as simultaneously performing depth estimation, edge detection, semantic segmentation, and other tasks.
Jodi's Technical Principles
- Joint Modeling: Models based on the joint distribution of the image domain and multiple label domains. By learning the joint distribution p(x, y1, y2, ..., yM), it derives the marginal and conditional distributions required for generation tasks and understanding tasks.
- Role-Switching Mechanism: During training, each domain is randomly assigned one of three roles: generation target ([G]), conditional input ([C]), or ignored ([X]). The model can simultaneously learn different types of probability distributions, including joint generation, controllable generation, and image perception.
- Linear Diffusion Transformer: Uses a linear diffusion Transformer as the backbone network. Based on a linear attention mechanism, it effectively reduces computational complexity, allowing the model to efficiently process multiple visual domains. Masked linear attention and domain-invariant positional embeddings are introduced to enhance consistency and alignment across different visual domains.
- Dataset Construction: To support joint modeling of multiple visual domains, Jodi constructs the Joint-1.6M dataset, which contains 200,000 high-quality images and corresponding automatic labels for 7 visual domains, providing rich data support for model training.
Jodi's Project Address
- Project Website：https://vipl-genun.github.io/Project-Jodi/
- GitHub Repository：https://github.com/VIPL-GENUN/Jodi
- HuggingFace Model Library：https://huggingface.co/VIPL-GENUN/Jodi
- arXiv Technical Paper：https://arxiv.org/pdf/2505.19084
Jodi's Application Scenarios
- Creative Content Generation: Provides inspiration for artists and designers, quickly generating images with specific styles or elements.
- Multimodal Data Augmentation: Generates high-quality multimodal data to enhance the training datasets of machine learning models.
- Image Editing and Restoration: Repairs or edits images according to user input, generating missing parts or adjusting style.
- Visual Understanding and Analysis: Predicts multiple visual labels to assist image understanding tasks, such as medical image analysis.
- Virtual Reality and Augmented Reality: Generates realistic virtual scenes and annotation information, improving the experience of VR and AR applications.