What is DreamO
DreamO is a unified framework for image customization generation jointly launched by the ByteDance creative team and the School of Electronic and Computer Engineering at Peking University Shenzhen Graduate School, based on a pre-trained Diffusion Transformer (DiT) model to achieve flexible customization for various image generation tasks. DreamO supports seamless integration of multiple conditions such as identity, subject, style, and background, and improves the consistency of generation results and condition decoupling capability based on feature routing constraints and placeholder strategies. DreamO uses a staged training strategy to ensure that the model converges efficiently in complex tasks while maintaining high-quality generation. The framework is widely applicable to scenarios such as virtual try-on, style transfer, and subject-driven generation, providing powerful customization capabilities for image generation.
Main Features of DreamO
- Multi-condition integration: Supports customization of multiple conditions such as identity, subject, style, and background, seamlessly integrating conditions into image generation.
- High-quality generation: Based on a staged training strategy, it ensures the high quality of generated images and corrects the bias introduced by low-quality data
- Flexible condition control: Supports users in precisely controlling the position and layout of conditions in generated images.
- Broad applicability: Supports handling complex multi-condition scenarios and is suitable for various image generation tasks such as virtual try-on, style transfer, and subject-driven generation.
Technical Principles of DreamO
- Diffusion Transformer (DiT) framework: Uses the Diffusion Transformer as the core architecture and achieves image customization based on unified processing of different types of inputs (such as text, images, conditions, etc.). The diffusion model generates images by gradually removing noise, and the Transformer architecture improves the model's ability to understand and process input conditions.
- Feature routing constraints: To improve the consistency between generated results and reference images, feature routing constraints are introduced. The constraints optimize the attention mechanism between conditional images and generated images to ensure that specific regions of the generated image correspond to the conditional image and avoid coupling between conditions.
- Placeholder strategy: Based on adding placeholders (such as [ref#1]) in text descriptions, conditional images are associated with specific objects in the text description, enabling precise control of condition positions in generated images.
- Staged training strategy: Based on a staged training method, including an initial stage (simple tasks), a comprehensive training stage (multiple tasks), and a quality alignment stage (correcting bias). This helps the model converge smoothly under complex data distributions and maintain high-quality generation.
- Large-scale training data: To achieve broad generalization ability, a large-scale training dataset covering multiple tasks (such as identity customization, subject-driven generation, virtual try-on, style transfer, etc.) is constructed to ensure that the model can learn generation capabilities under different conditions.
DreamO Project Address
- Project official website：https://mc-e.github.io/project/DreamO/
- GitHub repository：https://github.com/bytedance/DreamO
- arXiv technical paper：https://arxiv.org/pdf/2504.16915
Application Scenarios of DreamO
- Virtual try-on: Users upload their own photos and clothing images to generate try-on effects.
- Style transfer: Converts ordinary photos into artistic style images, or generates visual effects in different styles based on design sketches, suitable for artistic creation and design inspiration exploration.
- Subject-driven generation: Generate personalized avatars or virtual characters based on user-uploaded photos, supporting multi-subject fusion, used in social media, games, and animation production.
- Identity customization: Generate images containing specific character likenesses, supporting the preservation and fusion of identity features, suitable for virtual socializing and personalized content creation.
- Creative content generation: Generate creative advertising, film and television special effects, or educational scene images based on text descriptions and conditional images, supporting a variety of complex customization tasks to meet creative needs.