What is Mini-InternVL
Mini-InternVL is a "mini version" of the InternVL large model, a series of lightweight multimodal large language models jointly launched by Shanghai AI Laboratory with Tsinghua University, Nanjing University, and other institutions. It includes three parameter versions: 1B, 2B, and 4B, achieving relatively high performance with smaller parameter counts. Among them, Mini-InternVL-4B achieves about 90% of the performance of InternVL2-76B with only 5% of the parameter count. Mini-InternVL uses InternViT-300M as the visual encoder and combines it with different pre-trained language models. Based on a dynamic resolution input strategy and pixel shuffle operation, it reduces the number of visual tokens and improves processing efficiency. Mini-InternVL performs excellently on multiple general multimodal benchmarks and can adapt to downstream tasks in specific domains using a simple transfer learning framework.
Main Features of Mini-InternVL
- Multimodal understanding and reasoning: Given image and text inputs, understand and reason about the semantic relationships within them.
- Cross-domain adaptability: Based on knowledge distillation and transfer learning techniques, adapt to different domains and tasks.
- Lightweight and efficient: Mini-InternVL achieves performance comparable to large models while maintaining a small model parameter count (100 million to 4 billion). This enables efficient operation in resource-constrained environments (such as consumer-grade GPUs or edge devices), reducing deployment costs and computational resource requirements.
- Visual instruction tuning: It has the ability to tune according to visual instructions, better understanding and executing user instructions based on images.
- Dynamic resolution input: Supports a dynamic resolution input strategy, dividing the image into tiles of different sizes according to its aspect ratio and processing them accordingly.
Technical Principles of Mini-InternVL
- Visual encoder (InternViT-300M): As the core part of the model, the visual encoder is responsible for converting input images into feature representations that the model can understand. InternViT-300M is a lightweight visual model that inherits rich visual knowledge from the more powerful InternViT-6B model based on knowledge distillation. The distillation process enables InternViT-300M to perform excellently in multiple visual domains while maintaining a small model parameter count.
- Knowledge distillation: Transfers knowledge from a large teacher model to a small student model, allowing the student model to inherit the performance of the teacher model. In Mini-InternVL, InternViT-6B serves as the teacher model and, based on computing a negative cosine similarity loss, transfers the knowledge of hidden states to InternViT-300M.
- MLP projector: The MLP (multilayer perceptron) projector is used to connect the visual encoder and the language model. It projects the feature vectors output by the visual encoder into a space suitable for processing by the language model, enabling visual information and text information to be effectively fused and interacted with.
- Pre-trained language models (LLMs): Mini-InternVL combines different pre-trained language models, such as Qwen2-0.5B, InternLM2-1.8B, and Phi-3mini.
- Dynamic resolution input strategy: Based on a dynamic resolution input strategy. This strategy divides the image into 448×448 tiles according to its aspect ratio, combines the tiles into a fixed sequence, and ultimately generates an image representation with a resolution of 2688×896. The model also adds a thumbnail for each tile to provide global context information.
- Pixel shuffle operation: Based on the pixel shuffle operation, the model reduces the image resolution to one quarter of the original, decreasing the number of visual tokens.
Mini-InternVL's project address
- GitHub repository：https://github.com/OpenGVLab/InternVL
- HuggingFace model library：https://huggingface.co/collections/OpenGVLab/internvl-adaptation
- arXiv technical paper：https://arxiv.org/pdf/2410.16261
Mini-InternVL's application scenarios
- Autonomous driving: Used for environmental perception, behavior prediction, and path planning, processing multi-view images, recognizing and predicting the behavior of traffic participants, and generating safe and efficient driving paths.
- Medical image processing: Assisting with disease diagnosis, image annotation, and treatment plan recommendations, analyzing medical images, and providing diagnostic support and treatment recommendations.
- Remote sensing: Conducting land use classification, disaster monitoring, and environmental monitoring, identifying different types of land use situations, assessing the impact of natural disasters, and monitoring environmental changes.
- Document and chart understanding: Extracting document content, parsing tables and charts, generating document summaries and chart explanations, and supporting data visualization and analysis.
- Video understanding: Extracting key frames and content from videos, recognizing the behavior and events of people in videos, generating video summaries, and answering video-related questions.