What is Infinity-MM
Infinity-MM is a tens-of-millions-scale multimodal instruction dataset launched by the BAAI Research Institute, containing 43 million samples with a data volume of 10TB. The dataset has undergone quality filtering and deduplication to ensure high quality and diversity, and can improve the performance of open-source vision-language models (VLMs). BAAI has introduced a synthetic data generation method based on open-source VLMs to further expand the dataset's scale and diversity. Based on Infinity-MM, BAAI successfully trained the 2-billion-parameter multimodal model Aquila-VL-2B, achieving state-of-the-art performance among models of the same scale.
Main Features of Infinity-MM
- Improving Open-Source Model Performance: Infinity-MM improves the performance of open-source vision-language models (VLMs) by providing large-scale and high-quality instruction data, bringing them close to or reaching the level of closed-source models.
- Dataset Construction: It contains 43 million multimodal samples that have undergone strict screening and deduplication, covering multiple types such as visual question answering, text recognition, document analysis, and mathematical reasoning.
- Synthetic Data Generation: Based on open-source VLMs and detailed image annotations, it generates diverse instructions closely related to image content, expanding the dataset scale and improving data diversity.
- Model Training and Evaluation: The Infinity-MM dataset is used to train a 2-billion-parameter VLM, Aquila-VL-2B, and the model shows excellent performance on multiple benchmarks.
- Advancing Multimodal Research: By providing a large-scale, high-quality dataset, it promotes research and application development in the field of multimodal AI.
Technical Principles of Infinity-MM
- Data Collection and Preprocessing: Infinity-MM's data comes from multiple public datasets and undergoes deduplication and quality filtering to ensure the dataset's high quality and diversity.
- Synthetic Data Generation Method：
- Image and Instruction Labeling System: Open-source recognition models (such as RAM++) are used to automatically label images and extract key information, forming the semantic basis of the images.
- Instruction Label System: A three-level instruction label system is designed, covering instructions at different levels and of different types.
- Establishment of Correspondence Between Images and Instruction Labels: The correspondence between image labels and instruction labels is counted to quickly retrieve matching instruction task labels.
- Question Generation and Filtering: The model is instructed to generate specific questions based on images and instruction types, and reasonableness judgment is performed.
- Answer Generation and Filtering: After generating questions, corresponding instruction answers are further generated and strictly filtered to ensure matching with the image content or task.
- Staged Training Strategy: The Aquila-VL-2B model is based on a staged training method, gradually improving the model's ability to understand and process visual information.
- Multimodal Architecture: The Aquila-VL-2B model is based on the LLaVA-OneVision architecture, combining a text tower (Qwen2.5-1.5B-instruct) and a vision tower (Siglip400m).
- Improved Training Efficiency: BAAI's self-developed FlagScale framework adapts model training and improves training efficiency, reaching 1.7 times that of the original DeepSpeed-based training code.
Infinity-MM Project Address
- HuggingFace Model Repository：https://huggingface.co/datasets/BAAI/Infinity-MM
- arXiv Technical Paper：https://arxiv.org/pdf/2410.18558
Application Scenarios of Infinity-MM
- Visual Question Answering (VQA): Based on paired data of images and related questions, train models to understand and answer questions about image content.
- Image Captioning: Generate descriptive text for images, with wide applications in fields such as social media, content management, and image retrieval.
- Document Understanding and Analysis: Extract and understand visual and textual information in documents, suitable for automated office work, intelligent document processing, and information extraction.
- Mathematical and Logical Reasoning: Train models to solve mathematical problems and logical reasoning tasks, very useful for educational technology, automated testing, and intelligent tutoring systems.
- Multimodal Interaction Systems: Combine visual and linguistic information to enhance the naturalness and efficiency of human-computer interaction, suitable for intelligent assistants and customer service robots.