What is BGE-VL
BGE-VL is a multimodal embedding model launched by the Beijing Academy of Artificial Intelligence (BAAI) in collaboration with multiple universities, trained on large-scale synthetic data MegaPairs. BGE-VL focuses on multimodal retrieval tasks, such as image-text retrieval and composed image retrieval, and improves the model's generalization ability and retrieval performance based on an efficient multimodal data synthesis method. The BGE-VL series includes BGE-VL-Base and BGE-VL-Large based on the CLIP architecture, and BGE-VL-MLLM based on a multimodal large model architecture. The models perform excellently in multiple benchmarks, setting new state-of-the-art results in composed image retrieval tasks and significantly improving retrieval accuracy. The core advantages of BGE-VL lie in the scalability and high quality of its data synthesis method, and the model's outstanding generalization ability on multimodal tasks.
Main Features of BGE-VL
- Image-Text Retrieval: Retrieve the most relevant images based on the input text description, or retrieve relevant text information based on the input image.
- Composed Image Retrieval: Support users in inputting both an image and a text instruction, comprehensively understanding both types of information to retrieve more precise target images
- Multimodal Embedding: Map images and text into a unified vector space, allowing data from different modalities to be compared and retrieved using vector similarity.
- Instruction Fine-Tuning: Fine-tune the model based on synthesized multimodal instruction data, enabling it to better understand and execute complex multimodal tasks, improving the model's generalization ability and task adaptability.
Technical Principles of BGE-VL
- Data Synthesis Method (MegaPairs)：
- Data Mining: Mine diverse image pairs from massive image-text corpora, and use multiple similarity models (such as CLIP, etc.) to find candidate images related to the query image.
- Instruction Generation: Generate open-domain retrieval instructions based on multimodal large language models (MLLM) and large language models (LLM), summarize the relationships between image pairs, and write high-quality retrieval instructions.
- Triplet Construction: Generate multimodal triplet data containing "query image, query statement, target image" for model training. The data requires no manual annotation and is efficient and scalable.
- Multimodal Model Architecture：
- CLIP-Based Architecture: BGE-VL-Base and BGE-VL-Large use a CLIP-like architecture, in which an image encoder and a text encoder map images and text into the same vector space, and contrastive learning optimizes model performance.
- Multimodal Large Model Architecture: BGE-VL-MLLM is based on a more complex multimodal large model architecture, handling complex multimodal interaction and instruction understanding tasks.
- Instruction Fine-Tuning: Fine-tune the model based on synthesized multimodal instruction data to enhance the model's ability to understand and execute multimodal tasks.
- Contrastive Learning and Optimization: During training, the model uses contrastive learning to optimize the vector representations of multimodal embeddings, bringing related images and text closer in the vector space while pushing unrelated data farther apart. Through training on large-scale synthetic data, the model learns more generalized multimodal feature representations and performs excellently across various multimodal tasks.
BGE-VL Project Address
- HuggingFace Model Repository：https://huggingface.co/collections/BAAI/megapairs
Application Scenarios of BGE-VL
- Intelligent Search: Users upload images or enter text to quickly find related content and improve search accuracy.
- Content recommendation: Based on the content or interests uploaded by users, recommend similar image and text materials to enhance the personalized experience.
- Image editing assistance: Help designers quickly find reference images with similar styles and improve creative efficiency.
- Intelligent customer service: Combine image and text to understand user issues, provide more intuitive solutions, and improve service efficiency.
- Cultural heritage research: Based on image and text retrieval, quickly find related artifacts or research materials to support archaeology and conservation work.