What is InstructMove
InstructMove is an instruction-based image editing model jointly launched by the University of Tokyo and Adobe. It learns how to perform image operations according to instructions by observing changes between frame pairs in videos. The model is based on multimodal large language models (MLLMs) to generate editing instructions describing the changes between frame pairs, and is trained to perform complex non-rigid editing tasks, such as adjusting subject poses, changing expressions, and viewpoints, while maintaining content consistency. InstructMove uses real video frames as the data source, ensuring the naturalness and realism of the content during editing and overcoming the limitations of synthetic datasets in complex editing tasks. InstructMove supports precise local editing based on control mechanisms such as masks, further enhancing its flexibility and practicality in real-world applications.
Main Features of InstructMove
- Non-rigid editing: It can adjust non-rigid features of subjects in images, such as poses and expressions, in accordance with the given editing instructions.
- Viewpoint adjustment: It changes the shooting viewpoint of the image according to instructions, such as moving the camera viewpoint to the left or right, thereby changing the composition and visual effect of the image.
- Element rearrangement: It rearranges or moves elements in the image, such as putting the legs of a toy together or making a bird's tail visible, to meet specific editing needs.
- Precise local editing: Combined with control mechanisms such as masks, it supports precise local editing of specific regions of the image, enabling more detailed modification effects.
Technical Principles of InstructMove
- Dataset construction
- Video frame sampling: Frame pairs are sampled from internet videos to ensure that there are meaningful transformations between the frame pairs, such as subject pose changes, element movement, or camera viewpoint adjustments, thereby obtaining a large number of natural and realistic image transformation samples.
- Instruction generation by multimodal language models: Multimodal large language models (MLLMs), such as GPT-4o or Pixtral-12B, are used to analyze the differences between the sampled frame pairs and generate accurate editing instructions.
- Model architecture and training
- Fine-tuning of pretrained models: Pretrained text-to-image (T2I) models, such as Stable Diffusion, are fine-tuned on the constructed dataset.
- Spatial conditioning strategy: A spatial conditioning strategy is introduced to concatenate the reference image and the noise input along the spatial dimension instead of the traditional channel concatenation.
- Denoising network training: The concatenated input is fed into a denoising U-Net network to predict the noise map. By computing the difference between the predicted noise map and the original noise map, the model parameters are optimized to accurately denoise and reconstruct the target image according to the editing instructions, thereby achieving image editing.
- Control mechanism integration
- Mask guidance: It supports integration with control mechanisms such as masks to achieve precise local editing. During inference, masks are used to control the editing region, and the updated latent representation is fused with the reference latent representation to modify specific parts of the image.
- Other spatial controls: It integrates with controllable diffusion models such as ControlNet and accepts additional visual cues provided by users, such as sketches or skeleton keypoints, to achieve more complex and precise image editing operations.
InstructMove Project Address
- Project official website：ljzycmd.github.io/projects/InstructMove
- arXiv technical paper：https://arxiv.org/pdf/2412.12087v1
Application scenarios of InstructMove
- Film and television post-production: VFX artists adjust the expressions of alien creature characters in a sci-fi movie to make them better fit the angry emotion required by the plot.
- Advertising creative design: Designers use it to adjust the racing perspective and background elements for a car advertisement, highlighting the new model's speed and passion characteristics to attract consumers' attention.
- Interior design: Interior designers adjust the position of a bedside table and the curtain style in a bedroom to meet the client's needs for aesthetics and practicality and create a warm and comfortable sleeping environment.
- Art education: In a painting class, the teacher adjusts a figure's movements to help students understand the relationship between movement and emotion and deepen their understanding of artistic creation.
- Personal photo editing: Individual users adjust expressions in party photos to make them more natural and relaxed, then share them on social platforms and receive likes and positive comments from friends.