What is Migician
Migician is a multimodal large language model (MLLM) jointly launched by research teams from Beijing Jiaotong University, Huazhong University of Science and Technology, and Tsinghua University. It is specifically designed for free-form Multi-Image Grounding (MIG) tasks and has developed a large-scale training dataset MGrounding-630k. Based on free-form queries (such as text descriptions, images, or a combination of both), it identifies and precisely locates relevant visual regions across multiple images. Migician is trained on the large-scale instruction-tuning dataset MGrounding-630k, using a two-stage training method that combines multi-image understanding and single-image grounding capabilities to achieve end-to-end multi-image grounding functionality. Migician's design and training methods provide new ideas for the application of multimodal models in complex visual scenes, promoting the integration of multi-image understanding and fine-grained visual grounding.
Main Features of Migician
- Cross-image grounding: Find objects or regions related to the query across multiple images and provide their precise locations (such as coordinate boxes).
- Flexible input forms: Supports text, images, or a combination of both as queries, for example, "Find an object in Figure 2 similar to Figure 1 but with a different color."
- Multi-task support: Handles various tasks related to multiple images, such as object tracking, difference identification, and common object grounding.
- Efficient inference: Based on an end-to-end model design, it performs inference directly in multi-image scenarios, avoiding the multi-step reasoning and error propagation problems in traditional methods.
Technical Principles of Migician
- End-to-end multi-image grounding framework: Based on an end-to-end model architecture, it directly handles multi-image grounding tasks, avoiding the complexity and efficiency issues of traditional methods that decompose tasks into multiple subtasks (such as first generating text descriptions and then grounding). It simultaneously understands the content of multiple images and directly outputs the locations of target objects based on the query.
- Large-scale instruction-tuning dataset (MGrounding-630k): Contains over 630,000 data entries for multi-image grounding tasks. The dataset covers various task types (such as static difference grounding, common object grounding, and object tracking). Combined with free-form instructions, the model learns diverse grounding capabilities.
- Two-stage training method：
- First stage: The model is trained on various multi-image tasks to learn basic multi-image understanding and grounding capabilities.
- Second stage: Based on free-form instruction tuning, it enhances the model's grounding capability under complex queries while maintaining adaptability to diverse tasks.
- Multimodal fusion and reasoning: Combines information from visual and language modalities. Based on multimodal fusion, it achieves understanding and grounding of complex queries, handling abstract visual semantic information, such as grounding target objects through comparison, similarity, or functional association.
- Model merging technology: Based on model merging technology, it averages weights from different training stages to optimize overall performance.
Migician Project Addresses
- Project homepage：https://migician-vg.github.io/
- GitHub repository：https://github.com/thunlp/Migician
- HuggingFace model library：https://huggingface.co/Michael4933/Migician
- arXiv technical paper：https://arxiv.org/pdf/2501.05767
Application Scenarios of Migician
- Autonomous driving: Quickly locate targets around the vehicle (such as pedestrians and obstacles), supporting multi-view perception and dynamic target tracking.
- Security Monitoring: Multiple cameras work in coordination to identify abnormal behavior or targets, and analyze abnormal situations such as crowd gathering and rapid movement.
- Robot Interaction: Precisely locate target objects, supporting robots in completing tasks such as grasping and navigation in complex environments.
- Image Editing: Analyze the content of multiple images to achieve object replacement, deletion, or creative content generation.
- Medical Imaging: Fuse multimodal images to quickly locate lesion areas or abnormal tissues, supporting dynamic monitoring.