What is PhotoDoodle
PhotoDoodle is an artistic image editing framework jointly launched by the National University of Singapore, Shanghai Jiao Tong University, Beijing University of Posts and Telecommunications, ByteDance, and the Tiamat team. It learns an artist's unique style from a small number of samples to achieve photo doodling. PhotoDoodle uses a two-stage training strategy: pretraining a general image editing model, OmniEditor, on large-scale data, then fine-tuning it with a small number of artist-curated before-and-after image pairs to capture a specific editing style. PhotoDoodle introduces a positional encoding reuse mechanism and a noise-free conditioning paradigm to ensure seamless integration and consistency between generated results and the background. PhotoDoodle releases a high-quality dataset containing 6 styles and over 300 samples, providing a benchmark for related research.
Main Features of PhotoDoodle
- Art style learning and reproduction: Learns a unique editing style from a small number of samples provided by an artist and applies it to new image editing tasks.
- Decorative element generation: Supports adding decorative elements to photos (such as hand-drawn lines, color blocks, decorative patterns, etc.), ensuring these elements blend seamlessly with the background.
- Maintaining background consistency: During editing, strictly preserves the background content of the original photo, avoiding background distortion or destruction of the style.
- Instruction-driven editing: Controls image editing content based on natural language instructions, achieving precise local modifications and stylized processing.
- Efficient style customization: With the help of low-rank adaptation (LoRA) technology, only 30-50 pairs of samples are needed to quickly adapt to different artists' styles, reducing training costs.
Technical Principles of PhotoDoodle
- OmniEditor pretraining: Fine-tunes a pretrained DiT model on a large-scale image editing dataset, converting it into a general image editor (OmniEditor). Introduces a Positional Encoding Cloning mechanism and a Noise-free Conditioning paradigm to ensure spatial consistency and background preservation during editing. Based on multimodal attention (MMA), it combines text instructions and image conditions to achieve precise image editing.
- EditLoRA fine-tuning: On the basis of the pretrained OmniEditor, performs low-rank adaptation (LoRA) fine-tuning with a small number of before-and-after image pairs provided by an artist. Adaptively adjusts model weights based on low-rank decomposition matrices to capture a specific artist's editing style while preserving the general capabilities of the pretrained model.
- Positional encoding cloning mechanism: Shares the same positional encoding between the source image and the target image, ensuring spatial consistency of the generated results and avoiding misalignment between the background and decorative elements.
- Noise-free conditioning paradigm: Preserves the noise-free condition of the source image during generation, preventing background content from being destroyed during iterative denoising and maintaining the details and texture of the original image.
- Conditional flow matching loss function: Optimizes the conditional flow matching loss function, guiding the model to learn the generation path from noise to the target image and further improving the editing effect.
Project Address of PhotoDoodle
- GitHub repository：https://github.com/showlab/PhotoDoodle
- HuggingFace model library：https://huggingface.co/nicolaus-huang/PhotoDoodle
- arXiv technical paper：https://arxiv.org/pdf/2502.14397
Application Scenarios of PhotoDoodle
- Digital art creation: Adds artistic styles to photos and quickly generates creative works.
- Commercial Design: Quickly generate design images that match the brand style, improving design efficiency.
- Social Media: Add decorative effects to personal photos, enhancing the appeal of shared content.
- Art Education: Assist teaching, helping students understand and practice different art styles.
- Entertainment and Interaction: Generate artistic images in real time, enhancing the interactive experience.