Qianfan-OCR - Baidu Qianfan's end-to-end document intelligence model

What is Qianfan-OCR

Qianfan-OCR is an end-to-end document intelligence model launched by Baidu Qianfan, based on a 4B-parameter vision-language architecture that integrates document parsing, layout analysis, text recognition, and semantic understanding into one. In the OmniDocBench v1.5 evaluation, the model ranked first among end-to-end models with a score of 93.12. Through the Layout-as-Thought mechanism, it achieves explicit modeling of layout structure and supports understanding of complex tables and charts. The model is open source and can be efficiently deployed on a single A100 GPU.

Qianfan-OCR's main features

• Document image parsing: The model supports directly extracting structured text content from scanned documents or images without preprocessing.
• Layout analysis and understanding: Supports automatically identifying elements such as headings, paragraphs, tables, and charts in documents as well as their spatial relationships.
• Text recognition and conversion: Accurately converts printed or handwritten text in images into editable text.
• Key information extraction: Supports locating and extracting specific fields from complex documents, such as dates, amounts, and names.
• Chart reasoning and analysis: The model can understand the numerical meaning and trends of visual content such as bar charts and line charts.
• Multi-format output: Supports generating structured data formats such as Markdown, JSON, and HTML.

Qianfan-OCR's technical principles

• End-to-end unified architecture: Qianfan-OCR adopts a unified vision-language architecture, replacing the traditional multi-stage "detection-recognition-understanding" pipeline with an end-to-end approach. The model maps document images directly to structured output, avoiding error accumulation and loss of visual information caused by staged processing.
• Layout-as-Thought mechanism: To address the issue that end-to-end models lack explicit layout modeling, the team introduced the Layout-as-Thought mechanism. Before outputting the final result, the model triggers a structural thinking stage through a <think> token, first generating layout information such as element positions, types, and reading order, and then completing content parsing based on this prior knowledge, thereby possessing both structural awareness and semantic understanding within a unified framework.

Qianfan-OCR's key information and usage requirements

• Model scale: 4B-parameter vision-language architecture
• Evaluation results: First among end-to-end models on OmniDocBench v1.5 (93.12 points), surpassing Gemini 3-Pro on the KIE leaderboard
• Core innovation: Adopts the Layout-as-Thought mechanism, supporting explicit modeling of layout structure
• Deployment performance: Throughput of 1.024 pages/second on a single A100 GPU (W8A8 quantization)
• Open-source status: Model weights have been released on HuggingFace, with a supporting Skills toolchain
• Hardware environment: It is recommended to equip NVIDIA A100 or an equivalent GPU for inference deployment
• Software dependencies: Requires installation of the vLLM inference framework, supporting W8A8 quantization to reduce VRAM usage
• Access method: Call online through the Baidu Qianfan platform, or deploy privately based on open-source weights
• Input format: Supports common document image formats (PDF, PNG, JPG, etc.)
• Output format: Configures structured output such as Markdown, JSON, and HTML according to requirements

Core Advantages of Qianfan-OCR

• Leading Architecture: The model adopts an end-to-end unified vision-language architecture, replacing the traditional multi-stage Pipeline, eliminating error accumulation between modules and greatly simplifying system deployment and operation complexity.
• Layout Understanding: The original Layout-as-Thought mechanism, through <think> tokens, explicitly models the position, type and reading order of document elements, significantly improving parsing accuracy in complex layout scenarios.
• Top Performance: In the OmniDocBench v1.5 evaluation, it ranks first among end-to-end models with 93.12 points, and won 5 best results out of 6 tasks including chart understanding.
• Outstanding Efficiency: A single A100 GPU with W8A8 quantization can achieve a throughput of 1.024 pages/second, eliminating the cost of CPU detection and multi-model heterogeneous orchestration compared to traditional solutions.
• Out-of-the-Box: Supports online calls on the Baidu Qianfan platform, private deployment with HuggingFace open-source weights, and provides a complete Skills toolchain and multi-format output capabilities.

How to Use Qianfan-OCR

• Online Calls: Visit the Baidu Qianfan platform console, select the built-in Qianfan-OCR model in the model center, create an application to obtain an API Key, and then upload document images through the standard HTTP interface to obtain structured parsing results in real time.
• Private Deployment Download the open-source model weights from HuggingFace, install the vLLM inference framework and configure the W8A8 quantization parameters, start the model service on a server equipped with an A100 GPU, and implement offline calls through the local API.
• Toolchain Integration Clone the official Skills repository on GitHub, carry out secondary development based on the provided document intelligence toolkit, embed OCR capabilities into existing business systems, and support custom output formats and batch document processing workflows.

Qianfan-OCR Project Address

• GitHub Repository：https://github.com/baidubce/Qianfan-VL
• HuggingFace Model Library：https://huggingface.co/baidu/Qianfan-OCR
• arXiv Technical Paper：https://arxiv.org/pdf/2603.13398

Comparison of Qianfan-OCR with Similar Competing Products

Application Scenarios of Qianfan-OCR

• Enterprise Document Digitization: Supports batch processing of scanned contracts, invoices, reports, and more, automatically extracting key fields and generating a structured database.
• Financial document review: The model can identify amounts, dates, and account information in bank statements, insurance policies, and account statements, assisting risk control and compliance review.
• Medical record management: Parses symptoms, diagnoses, and medication records in handwritten or printed medical records to enable rapid archiving and retrieval of electronic medical records.
• Academic paper processing: Converts PDF documents into Markdown format, preserving formulas, charts, and citation structures, facilitating knowledge base construction.
• Archival restoration of historical documents: The model can recognize text in low-quality images such as ancient books and old newspapers, assisting the digital preservation of cultural heritage.