What is Qianfan-VL
Qianfan-VL is a visual understanding large model launched by Baidu AI Cloud Qianfan for enterprise-level multimodal application scenarios. The model includes three size versions: 3B, 8B, and 70B, with excellent general capabilities and specialized enhanced capabilities for vertical scenarios such as OCR and education. The model is developed based on open-source models and completes the full-process computing tasks on Baidu's self-developed Kunlun Core P800, demonstrating outstanding performance and efficiency. Qianfan-VL supports multimodal tasks such as complex chart understanding, visual reasoning, and mathematical problem solving, providing high-precision visual understanding solutions for enterprise-level applications.
Main Features of Qianfan-VL
- Multi-Size Models: Provides three specifications of models: 3B, 8B, and 70B, meeting the diverse needs of enterprises and developers of different scales, applicable from real-time on-device scenarios to complex reasoning and computing scenarios.
- Enhanced OCR and Document Understanding: Features full-scenario OCR recognition capabilities, able to accurately recognize handwriting, mathematical formulas, natural scene text, etc., and supports structured extraction of card, certificate, and receipt information; at the same time, its complex layout document understanding capability is outstanding, able to automatically analyze layout elements, accurately parse tables and charts, and achieve intelligent document question answering and structured parsing.
- Thinking and Reasoning Capability: The 8B and 70B models support activating chain-of-thought capability through special tokens, covering multiple complex scenarios such as complex chart understanding, visual reasoning, and mathematical problem solving, and can combine visual information with external knowledge for compositional reasoning, providing clear problem-solving ideas and step displays.
- General Capabilities: Performs excellently in general multimodal tasks such as object recognition, image description, and visual question answering, supports mixed Chinese-English understanding, and has good cross-modal alignment capability, providing strong support for intelligent applications in different scenarios.
Technical Principles of Qianfan-VL
-
Multimodal Architecture: The 3B model is based on the Qwen2.5 architecture, and the 8B and 70B models are based on theLlama 3.1architecture, with vocabulary expansion and localization enhancement through 3T Chinese-English corpus, supporting mixed Chinese-English understanding. Based onInternViT initialization, it supports dynamic chunking to process images of different resolutions, with support for up to 4K resolution input.Through an MLP adapter, it achieves seamless bridging between visual and language modalities, ensuring the accuracy and efficiency of information transfer.
-
Capability Enhancement Training Pipeline：
-
Four-Stage Training Strategy: Through four stages—cross-modal alignment, general knowledge injection, domain-enhanced knowledge injection, and post-training—it gradually improves the model's general capabilities and domain capabilities.
-
High-Precision Data Synthesis Technology: Builds a large-scale data synthesis pipeline for multimodal tasks, covering core tasks such as document recognition, mathematical problem solving, chart understanding, table recognition, formula recognition, and natural scene OCR. Through refined pipeline design and intermediate process data construction, it achieves large-scale production of high-quality training data.
-
-
Large-Scale Parallel Training: Based ona three-dimensional parallel combination of data parallelism (DP), tensor parallelism (TP), and pipeline parallelism (PP), it significantly improves training efficiency through dynamic load balancing, gradient synchronization optimization, ZeRO-3 state sharding technology, and other means.Based on Baidu's self-developed Kunlun Core P800 chip, through a hardware separation design of communication operators and matrix multiplication operators, it achieves parallel communication and computation, significantly improving hardware utilization.
-
Inference Optimization: The model performs highly efficient inference on Kunlun chips, GPUs, and other chips, supports parallel computing at a scale of 5,000 cards for a single task, and ensures the model's efficient processing capability in practical applications.
Qianfan-VL project address
- Project official website：https://baidubce.github.io/Qianfan-VL/
- GitHub repository：https://github.com/baidubce/Qianfan-VL
- HuggingFace model library：https://huggingface.co/collections/baidu/qianfan-vl-68d0b9b0be8575c17267c85c
- arXiv technical paper：https://github.com/baidubce/Qianfan-VL/blob/main/docs/qianfan_vl_report_comp.pdf
Qianfan-VL application scenarios
- OCR recognition scenario: The model can accurately recognize text information in various documents, bills, handwritten notes, etc., supports multiple fonts and complex backgrounds, and provides efficient solutions for enterprise document processing, data entry, and more.
- Math problem-solving scenario: Through visual recognition of math problems and reasoning-based calculation, it supports various problem types such as geometry and algebra, provides intelligent tutoring tools for the education field, and helps students understand and solve math problems.
- Document understanding scenario: Automatically parses document structure, extracts key information, supports understanding and analysis of complex tables and charts, and improves the efficiency of enterprise document management, information retrieval, and knowledge management.
- Chart analysis scenario: Extracts data from bar charts, line charts, pie charts, and other charts for analysis, supports trend prediction, association reasoning, and more, providing strong support for data analysis and decision-making.