What is D-DiT
D-DiT (Dual Diffusion Transformer) is a multimodal diffusion model launched by Carnegie Mellon University, Yale University, and ByteDance Seed Lab, capable of unifying image generation and understanding tasks. The model combines continuous image diffusion (flow matching) and discrete text diffusion (masked diffusion), and simultaneously trains image and text modalities based on a bidirectional attention mechanism. D-DiT can achieve bidirectional tasks of text-to-image generation and image-to-text generation, supporting various applications such as visual question answering and image caption generation. Based on a multimodal diffusion Transformer architecture and trained with a joint diffusion objective, the model demonstrates multimodal understanding and generation capabilities comparable to autoregressive models, providing a new direction for the development of vision-language models.
Main features of D-DiT
- Text-to-image generation: Generates high-quality images based on input text descriptions.
- Image-to-text generation: Generates descriptive text based on input images, such as image captions, titles, or answers to visual questions.
- Visual question answering: Combines images and question text to generate accurate answers.
- Multimodal understanding: Supports various vision-language tasks, such as image captioning, visual instruction understanding, and long-text generation.
- Bidirectional generation capability: Simultaneously supports generation tasks from text to image and from image to text, with a high degree of flexibility.
Technical principles of D-DiT
- Dual-branch diffusion model: D-DiT combines continuous image diffusion (Continuous Image Diffusion) and discrete text diffusion (Discrete Text Diffusion). Continuous image diffusion uses flow matching (Flow Matching) technology to generate images through a reverse diffusion process. Discrete text diffusion uses masked diffusion (Masked Diffusion) technology to gradually denoise and generate text.
- Multimodal Transformer architecture：
- Image branch: Processes image data and outputs the diffusion objective for images.
- Text branch: Processes text data and outputs the diffusion objective for text.
- Joint training objective: Simultaneously trains image and text modalities based on a joint diffusion objective. The image diffusion loss is based on the flow matching loss, optimizing the reverse diffusion process of image generation. The text diffusion loss is based on the masked diffusion loss, optimizing the reverse diffusion process of text generation. Based on joint training, the model can learn the joint distribution between images and text.
- Bidirectional attention mechanism: D-DiT uses a bidirectional attention mechanism, supporting the model to flexibly switch between images and text and supporting unordered processing of input modalities. This allows the model to fully utilize information from images and text during the generation process, improving performance on multimodal tasks.
D-DiT project addresses
- Project website：https://zijieli-jlee.github.io/dualdiff.github.io/
- GitHub repository：https://github.com/zijieli-Jlee/Dual-Diffusion
- arXiv technical paper：https://arxiv.org/pdf/2501.00289
Application scenarios of D-DiT
- Text-to-image generation: Generates high-quality images based on text descriptions, suitable for creative design, game development, advertising production, and education.
- Image-to-text generation: Generates descriptive text for images, assisting visually impaired individuals, content recommendation, smart photo albums, and more.
- Visual question answering: Combines images and questions to generate accurate answers, used in intelligent assistants, educational tools, and customer support.
- Multimodal dialogue system: Combines images in dialogue to generate detailed responses, suitable for intelligent customer service, virtual assistants, and educational tutoring.
- Image editing and enhancement: Repairs, transforms, or enhances images based on text descriptions, used in image restoration, style transfer, and image enhancement.