What is D-Edit
D-Edit is a multi-functional image editing framework based on images and text. It leverages pre-trained diffusion models and unique prompts to achieve precise control and editing of specific items in images. The framework can handle a variety of editing tasks, including image-based editing, text-based editing, mask-based editing, and item removal. D-Edit segments an image into multiple items, associates a unique prompt with each item, and enables decoupled control over the items. Users can achieve diverse editing effects by modifying prompts, masks, or the associations between items and prompts. D-Edit is the first framework capable of item editing based on mask editing and combining image and text editing.
Main Features of D-Edit
- Text-Based Editing: Users change the text prompt associated with a specific item to replace or edit the item in the image.
- Image-Based Editing: Supports users in replacing items in a target image with items from a reference image.
- Mask-Based Editing: Users can edit the mask of a specific item in the image, including moving, resizing, reshaping, etc., to change the item's appearance.
- Item Removal: Removes specific items from the image by deleting the mask and prompt pair associated with the item, allowing the surrounding area to naturally fill the gap.
- Multi-Functional Image Editing: Implements the aforementioned editing functions within a unified framework, providing flexible and diverse image editing capabilities.
Technical Principles of D-Edit
- Item-Prompt Interaction: D-Edit decomposes an image into multiple items and assigns a unique prompt to each item. The prompts are specifically learned in the pre-trained diffusion model to control specific items.
- Decoupled Cross-Attention Layers: D-Edit introduces a decoupled cross-attention mechanism that enables the model to process the control flow of each item separately from its corresponding prompt, achieving precise item-level editing.
- Two-Step Optimization Process：
- Prompt Injection: Injects new tokens representing each item into the vocabulary of the text encoder, randomly initializing the token embeddings.
- Model Fine-Tuning: Optimizes the embedding matrix of the text encoder and the weights of the UNet model to establish associations between items and prompts, enabling the model to reconstruct the original image using the given item prompts.
- Degrees of Freedom in Editing Operations: After establishing the associations between items and prompts, D-Edit supports changing prompts, masks, or the mapping between items and prompts to implement various editing operations.
- Flexibility and Control: The design of D-Edit provides the ability to precisely control specific items during editing while maintaining a natural harmony of the overall image. By adjusting prompts and masks, users can achieve various editing effects from subtle adjustments to complete replacements.
Project Address of D-Edit
- GitHub Repository：https://github.com/collovlabs/d-edit
- arXiv Technical Paper：https://arxiv.org/pdf/2403.04880
- Online Experience Demo：https://huggingface.co/spaces/Collov-Labs/d-edit
Application Scenarios of D-Edit
- Digital Art Creation: Artists and designers create new digital artworks, editing specific elements in images to achieve unique visual effects.
- Photo Editing: Ordinary users edit personal photos, such as changing the background, adjusting a person's pose, changing clothing styles, etc.
- Advertising and marketing: Marketers change specific elements in advertising images, such as products, logos, or text, to adapt to different marketing campaigns.
- Fashion and apparel: Fashion designers present different versions of clothing designs, exploring different design possibilities by changing the color, texture, or style of the clothing.
- Architecture and urban planning: Architects and urban planners edit architectural design images, adjusting building appearances or urban planning layouts.