What is BlueLM-V-3B
BlueLM-V-3B is an algorithm and system co-design method jointly launched by vivo AI Lab and MMLab of The Chinese University of Hong Kong, supporting efficient deployment of multimodal large language models (MLLM) to mobile devices. The model is characterized by small size (2.7B language parameters and 400M vision parameters), fast speed (24.4 token/s generation speed), and strong performance (score of 66.1 on the OpenCompass benchmark). Based on an optimized dynamic resolution scheme and hardware-aware deployment, it improves the model's inference efficiency and performance on mobile phones.
Main features of BlueLM-V-3B
- Multimodal understanding: Processes and integrates multiple data forms such as text and images, providing richer interaction and deeper contextual understanding.
- Real-time processing: Enables real-time responses on mobile devices, suitable for scenarios requiring rapid feedback, such as augmented reality or real-time translation.
- Privacy protection: Supports local processing on the device, reduces data transmission, and enhances user privacy protection.
- Efficient deployment: Optimizes the model to adapt to the computing and memory constraints of mobile devices, ensuring efficient operation on resource-constrained hardware.
- High performance: Achieves performance comparable to models with larger parameter counts despite a relatively small number of parameters.
- Cross-language capability: Supports multilingual understanding, enhancing the model's applicability in different language environments.
Technical principles of BlueLM-V-3B
- Algorithm design：
- Dynamic resolution processing: Redesigns the dynamic image resolution strategy to adapt to high-resolution image understanding, reduces the number of image tokens, and lowers deployment complexity.
- Relaxed aspect ratio matching method: Introduces parameter α to select a more suitable aspect ratio, reduces image upscaling, and optimizes training and deployment efficiency.
- System design：
- Batch image encoding: Uses the parallel processing capability of the NPU to process image patches in batches, accelerating the image encoding process.
- Pipeline parallel processing: In the image encoding process, designs pipeline parallel processing to hide the execution latency of Conv2D operations.
- Token downsampler: Based on merging and fusing information, reduces the number of image tokens to adapt to the processing capability of the NPU.
- Block-wise computation: To process long input tokens, adopts a block strategy to balance parallel processing and NPU performance.
- Model quantization: Uses INT8 and INT4 precision to quantize model weights, maintaining INT16 precision for LLM activations and FP16 precision for ViT activations, balancing computational efficiency and model accuracy.
- Overall framework: Loads both the ViT and LLM models simultaneously during model initialization. After the user uploads an image, processing starts immediately while instructions are input, improving response speed and memory usage efficiency.
Project address of BlueLM-V-3B
- arXiv technical paper：https://arxiv.org/pdf/2411.10640
Application scenarios of BlueLM-V-3B
- Augmented reality (AR): Provides real-time augmented reality experiences on mobile devices, such as a phone camera recognizing objects in the real world and providing related information.
- Real-time translation: In cross-language communication, translate text in speech or images in real time, helping users overcome language barriers.
- Educational assistance: As a learning tool, it helps students understand complex concepts and provides an interactive learning experience with images and text.
- Visual question answering (VQA): Users ask questions related to image content based on taking or uploading pictures, and the model provides accurate answers.
- Image and document understanding: In office automation, understand and process the content in images and documents, such as automatically recognizing document information such as invoices and contracts.