What is DoraCycle
DoraCycle is a unified generative model for multimodal domain adaptation introduced by the Show Lab at the National University of Singapore. It achieves information conversion and alignment between different modalities through two multimodal cycles (text-to-image-to-text and image-to-text-to-image). Based on the bidirectional mapping learned by the unified generative model, it performs domain adaptation training with unpaired data, without the need for large amounts of annotated data. The model is optimized through the cross-entropy loss at the cycle endpoints, promoting self-evolution and adaptation to specific domains.
Main Features of DoraCycle
- Domain adaptation without paired data: Through cycle consistency learning, DoraCycle is the first to achieve domain adaptation of generative models using unpaired data, significantly reducing data acquisition costs.
- Flexible task adaptability: DoraCycle can handle tasks that do not require paired knowledge (such as stylization), and can effectively combine a small amount of paired data to complete tasks that require new knowledge (such as identity generation).
Technical Principles of DoraCycle
- Multimodal cycle consistency learning: DoraCycle integrates two multimodal cycles: text-to-image-to-text (T cycle) and image-to-text-to-image (I cycle). These two cycles use a pre-trained unified generative model (such as a vision-language alignment model) for cross-modal mapping.
- T cycle: Starting from an input text sequence, the model first converts it into an image representation, then converts the generated image back into a text sequence, and optimizes the model by calculating the cross-entropy loss between the generated text and the original text.
- I cycle: Starting from an input image, it first converts it into a text description, then converts the text description back into an image, and optimizes the model by calculating the cross-entropy loss between the generated image and the original image.
- Cross-modal alignment self-supervised learning: DoraCycle is based on the bidirectional mapping between vision and language learned by the unified generative model. Through these two cycles, data can be maintained within the same modality, imposing constraints on biases introduced during the process. This enables the model to achieve cross-modal alignment between vision and language through self-supervised learning.
- Enhanced training stability: In the multi-step inference process, to avoid the gradient explosion problem, DoraCycle adopts the following techniques:
- Gradient clipping: Avoids conflicts between the optimization directions of the two cycles, thereby improving training stability.
- EMA model: Maintains a slowly updated exponential moving average (EMA) model for inference to generate pseudo data, enhancing the stability of pseudo data generation.
DoraCycle Project Address
- Github repository：https://github.com/showlab/DoraCycle
- arXiv technical paper：https://arxiv.org/pdf/2503.03651
Application Scenarios of DoraCycle
- Stylized design: DoraCycle can be used to generate images and text content that conform to a specific style.
- Virtual character generation: In virtual character design, DoraCycle can combine a small amount of paired data and large-scale unpaired data to generate virtual characters with specific identities and styles.
- Personalized advertising content: DoraCycle can generate personalized advertising images and copy based on brand style and target audience.
- Personalized learning materials: DoraCycle can generate personalized learning materials based on students' learning styles and preferences.