What is BlenderFusion
BlenderFusion is a generative visual compositing framework introduced by Google DeepMind that combines traditional 3D editing software (Blender) with AI models to achieve precise geometric editing and diverse visual composition. The framework is implemented based on three steps: first, extract objects of interest from the source image and convert them into editable 3D elements (object-centric layering); then perform diverse edits on the objects in Blender (Blender-based editing); finally, use a generative compositor to seamlessly blend the edited elements and generate the final photorealistic image (generative compositing). BlenderFusion performs excellently in complex visual compositing tasks, enabling flexible, decoupled, and 3D-aware manipulation of objects, cameras, and backgrounds.
Main Features of BlenderFusion
- Precise 3D Geometric Control: Enables precise 3D editing of objects based on Blender, including transformations such as position, rotation, and scaling, as well as modifications to attributes such as color, material, and shape.
- Flexible Camera Control: Supports adjusting the camera viewpoint independently of object manipulation, enabling complex viewpoint changes.
- Complex Scene Compositing: Seamlessly blends edited objects and backgrounds to generate realistic final images, supporting multi-object manipulation and complex scene editing.
- Decoupled Object and Camera Control: Enables manipulating objects while keeping the camera fixed, or adjusting the camera while keeping objects fixed, achieving highly decoupled control.
- Generalization Ability: Supports application to unseen scenes and objects, and supports editing tasks from simple to complex, including progressive multi-step editing.
Technical Principles of BlenderFusion
- Object-centric Layering: Uses visual foundation models (such as SAM2 for segmentation and Depth Pro for depth estimation) to extract objects from the input image and convert them into editable 3D elements. Optionally uses image-to-3D models (such as Rodin and Hunyuan3D) to generate complete 3D meshes, which are aligned with 2.5D surface meshes to enable more flexible editing at test time.
- Blender-grounded Editing: Imports the 3D objects obtained from the layering step into Blender, and performs diverse editing operations based on Blender's powerful capabilities, including basic object transformations, attribute modifications, non-rigid deformations, etc. Supports camera control and background replacement, providing precise 3D control signals for the generative compositing step.
- Generative Compositing: A diffusion model-based generative compositor blends Blender's rendering results with the background to generate the final photorealistic image. The generative compositor is based on a dual-stream architecture that simultaneously processes information from the original scene (before editing) and the target scene (after editing), and fuses the information from both based on a cross-view attention mechanism. Based on two training strategies, source masking and simulated object jittering, it improves the model's flexibility and decoupling ability in complex editing tasks.
BlenderFusion Project Address
- Project Website：https://blenderfusion.github.io/
- arXiv Technical Paper：https://arxiv.org/pdf/2506.17450
Application Scenarios of BlenderFusion
- Film and Television Production: Used in visual effects (VFX) production for movies and TV series, adding virtual objects, adjusting scene layouts, changing backgrounds, etc., to create realistic composite scenes.
- Game Development: Help game developers quickly design and edit game scenes, add and adjust objects in the game, change the camera perspective, etc., and create realistic game environments.
- Advertising: Help advertising designers create high-quality product display images and highlight product features.
- Architectural design: Architects and interior designers carry out interior design visualization, add and adjust furniture, decorations, etc., and generate realistic interior renderings.
- Artistic creation: Artists use 3D editing and generative synthesis capabilities to create unique digital artworks and realize creative visualization.