What is ObjectMover
ObjectMover is a new image editing model jointly proposed by the University of Hong Kong and Adobe Research, addressing issues such as inconsistent lighting and shadows and object distortion when moving, inserting, and removing objects in images. It treats object movement as a special case of two-frame video, leveraging the cross-frame consistency learning capability of pre-trained video generation models, and transfers the model to image editing tasks through fine-tuning. The model adopts sequence-to-sequence modeling, with inputs including the original image, the target object image, and an instruction map, and outputs a composite image after object movement.
Main Features of ObjectMover
- Object Movement: It can move objects in an image to a specified position, automatically adjust related physical effects such as lighting, shadows, and reflections, while preserving the object's identity features.
- Object Removal: It can realistically fill in the background of removed objects, not generate irrelevant new objects, and accurately remove the light and shadow related to the object.
- Object Insertion: It can accurately preserve the identity features of the inserted object and automatically generate lighting and shadow effects consistent with the environment.
Technical Principles of ObjectMover
- Video Prior Transfer: ObjectMover treats the object movement task as a special case of two-frame video, leveraging the cross-frame consistency learning capability of pre-trained video generation models (such as diffusion models). By fine-tuning the model, it transfers from video generation tasks to image editing tasks. It can fully utilize the physical laws and object correspondences learned during video model pre-training, achieving precise light and shadow synchronization and identity feature preservation in image editing tasks.
- Sequence-to-Sequence Modeling: The model reframes the object movement task as a sequence prediction problem. The inputs include the original image, the target object image, and an instruction map (annotating the movement position and direction), and the output is a composite image after object movement. This enables the model to better understand and handle lighting and shadow changes and occlusion relationships of objects at different positions.
- Synthetic Dataset Construction: Due to the lack of large-scale real data for object movement, the research team used modern game engines (such as Unreal Engine) to generate high-quality synthetic data pairs. The data covers complex lighting, materials, and occlusion scenarios, enhancing the diversity and generalization ability of model training.
- Multi-Task Learning Strategy: ObjectMover combines four sub-tasks: object movement, removal, insertion, and video data insertion, and trains on synthetic data and real video data through a unified framework. This improves the model's generalization ability to real scenes and enables higher adaptability and robustness when handling different image editing tasks.
Project Address of ObjectMover
- Project Official Website：https://xinyu-andy.github.io/ObjMover/
- arXiv Technical Paper：https://arxiv.org/pdf/2503.08037
Application Scenarios of ObjectMover
- Special Effects Production: For some complex special effects scenes, such as the disappearance or appearance of objects, ObjectMover can achieve object removal and insertion while maintaining the realism of the scene.
- Virtual Scene Editing: In virtual reality and game development, flexible adjustments to objects in virtual scenes are needed. ObjectMover can be used to move objects in the scene, such as moving a prop from one position to another, while keeping the object's lighting and shadows consistent with the environment.
- Game Level Design: Developers can use ObjectMover to quickly adjust the layout of objects in a level, improving the efficiency of level design.
- Product Display: For product advertisements, ObjectMover can be used to place products in different scenes, showcasing different usage scenarios for the product.
- Space Planning: In architecture and interior design, ObjectMover can be used to move furniture or decorations to different positions, evaluating different design schemes.