What Is ICEdit
ICEdit (In-Context Edit) is an instruction-based image editing framework launched by Zhejiang University and Harvard University. Built on the powerful generation and context-awareness capabilities of large-scale Diffusion Transformers, it uses natural language instructions to precisely edit images. ICEdit requires only 0.1% of the training data and 1% of the trainable parameters, greatly reducing resource requirements compared with traditional methods, and performs excellently in multi-turn editing and multi-task editing. ICEdit offers advantages such as being open source, low cost, and fast processing (about 9 seconds per image), making it suitable for a wide range of application scenarios.
Main Features of ICEdit
- Instruction-Driven Image Editing: Precisely modify images based on natural language instructions, such as changing the background, adding text, or changing a person's clothing.
- Multi-Turn Editing: Supports multiple consecutive edits, with each edit based on the result of the previous one, suitable for complex creative needs.
- Style Transfer: Supports converting images into different artistic styles, such as watercolor painting, comics, and more.
- Object Replacement and Addition: Replace objects in an image or add new elements, such as replacing a person with a cartoon character.
- Efficient Processing: Fast processing speed (about 9 seconds per image), suitable for rapid generation and iteration.
Technical Principles of ICEdit
- In-Context Editing Framework: Based on "In-Context Prompting," editing instructions are embedded into the generation prompt, and the model generates the edited image based on the processed prompt. No modifications to the model architecture are required, avoiding the complex structural adjustments in traditional methods. Based on contextual understanding, the model directly generates editing results that comply with the instructions.
- LoRA-MoE Hybrid Fine-Tuning: Combines parameter-efficient LoRA (Low-Rank Adaptation) adapters with dynamic expert routing (Mixture-of-Experts, MoE). LoRA uses low-rank matrix decomposition to efficiently adjust model parameters and adapt to different editing tasks. MoE dynamically selects the expert modules best suited to the current task, further improving editing quality and flexibility. Only a small amount of data (50K samples) is needed for fine-tuning, significantly improving the editing success rate.
- Early Filter Inference-Time Scaling: During the inference stage, a vision-language model (VLM) is used to evaluate early-generated noise samples and select the initial noise that best matches the editing instruction. A small number of steps (such as 4 steps) is used to evaluate the editing effect, quickly selecting the optimal initial noise and further improving editing quality.
ICEdit Project Addresses
- Project Website：https://river-zhang.github.io/ICEdit-gh-pages/
- GitHub Repository：https://github.com/River-Zhang/ICEdit
- HuggingFace Model Library：https://huggingface.co/sanaka87/ICEdit-MoE-LoRA
- arXiv Technical Paper：https://arxiv.org/pdf/2504.20690
- Online Experience Demo：https://huggingface.co/spaces/RiverZ/ICEdit
Application Scenarios of ICEdit
- Creative Design: Convert photos into artistic styles (such as watercolor painting), or add creative elements, for use in design and advertising.
- Film and Television Production: Quickly generate character designs or scene concept art, assisting in early-stage film and television development.
- Social Media: Edit personal photos (such as changing the background or adding special effects) to create attractive social content.
- Education: Generate images for teaching, such as turning historical figures into comic style, to assist teaching.
- Commercial advertising: Quickly create product promotional images, such as changing the background and adding brand logos.