Diffree - AI image editing framework launched by OpenGVLab

• Text-to-image editing: Users input text descriptions, and Diffree can identify and add corresponding objects in the image.
• Automatic mask prediction: Diffree can automatically predict and determine the position and shape of newly added objects without requiring users to manually draw masks.
• Seamless blending: Newly added objects remain consistent with the original image's background in terms of lighting, tone, color, and other aspects, achieving natural blending.
• Background information preservation: During the process of adding new objects, Diffree preserves the background information of the original image, ensuring harmony between new and old elements.
• High-quality image output: Through gradual generation and refinement, Diffree outputs high-quality images, and newly added objects look as if they were originally in the image.

• Text description input: Users provide text descriptions to the system, clearly specifying the objects they want to add to the image, such as "add a cat".
• Mask prediction: Diffree uses advanced algorithms to predict where new objects should appear and their approximate shape. The process is equivalent to creating a virtual outline on the image to guide the AI in understanding where the object should be placed.
• Gradual generation: Diffree does not simply "paste" objects onto the image, but gradually constructs the new image. The process is similar to solving a jigsaw puzzle, where the AI builds the new image piece by piece while ensuring that each piece blends with the original image's background.
• Background preservation: While generating new objects, Diffree places great emphasis on preserving the background information of the original image. This includes lighting, shadows, and textures, ensuring that new objects can seamlessly blend into the image.
• Multi-stage refinement: Diffree may use a multi-stage refinement process to gradually optimize the blending of new objects with the background, improving the realism of the final image.
• High-quality dataset training: Diffree is trained using the high-quality dataset OABench, which contains a large number of real-world image pairs to help the AI learn how to add new objects while maintaining background consistency.

• Project website：https://opengvlab.github.io/Diffree/
• GitHubRepository：https://github.com/OpenGVLab/Diffree
• Hugging Face model library：https://huggingface.co/spaces/LiruiZhao/Dif_free
• arXiv technical paper：https://arxiv.org/pdf/2407.16982

Diffree's application scenarios