What is dots.ocr
dots.ocr is a multilingual document layout parsing model open-sourced by Xiaohongshu hi lab. The model is based on a 1.7-billion-parameter vision-language model (VLM), capable of unified layout detection and content recognition while maintaining good reading order. Although the model is small in scale, its performance reaches the industry-leading level (SOTA), performing excellently on benchmarks such as OmniDocBench, and its formula recognition performance can rival that ofDoubao-1.5and gemini2.5-pro and other larger-scale models, with significant advantages in low-resource language parsing. dots.ocr provides a concise and efficient architecture; switching tasks requires only changing the input prompt, with fast inference speed, suitable for various document parsing scenarios.
Main Features of dots.ocr
- Multilingual Document Parsing: Supports document parsing in multiple languages, covering elements such as text, tables, formulas, and images.
- Layout Detection and Content Recognition: Unifies layout detection and content recognition in a single vision-language model, maintaining good reading order.
- Efficient Inference: Based on a 1.7-billion-parameter vision-language model, with fast inference speed, suitable for large-scale document processing.
- Flexible Task Switching: By changing the input prompt, different tasks such as layout detection and content recognition can be easily switched.
- Diverse Output Formats: Supports multiple output formats such as JSON and Markdown, and provides layout visualization images.
Technical Principles of dots.ocr
- Vision-Language Model (VLM): dots.ocr is based on a 1.7-billion-parameter vision-language model that combines the strengths of a vision encoder and a language model. The vision encoder is responsible for extracting visual features from document images, while the language model is used to understand and generate text content.
- Three-Stage Training Process：
- Vision Encoder Pretraining: Trains a 1.2-billion-parameter vision encoder from scratch using a large-scale image-text pair dataset.
- Continued Vision Encoder Pretraining: Adds high-resolution input support and aligns with the language model, further enhancing visual feature extraction capability.
- VLM Training: Trains with a pure OCR dataset to optimize the model's performance on document parsing tasks.
- Supervised Fine-Tuning (SFT): Performs supervised fine-tuning with diverse datasets, including manually annotated data, synthetic data, and open-source datasets. Based on an iterative data flywheel mechanism, it continuously optimizes model performance and improves data quality and diversity. Based on the method of "large model ranking + rule-based post-validation," it corrects the reading order to ensure that the order of layout elements conforms to human reading habits.
- Task Switching Mechanism: Uses input prompts to specify the model's task, such as layout detection, content recognition, formula parsing, etc. The prompt guides the model to generate the corresponding output, enabling the model to flexibly handle different document parsing needs.
Project Address of dots.ocr
- GitHub Repository：https://github.com/rednote-hilab/dots.ocr
- HuggingFace Model Library：https://huggingface.co/rednote-hilab/dots.ocr
- Online Experience Demo：https://dotsocr.xiaohongshu.com/
Application Scenarios of dots.ocr
- Document Digitization and Content Extraction: Efficiently converts paper documents or PDF files into editable digital formats, accurately extracting structured content such as text, tables, and formulas, facilitating electronic document management.
- Academic Research and Publishing: Quickly parse formulas, charts, and text in academic papers, helping researchers efficiently obtain key information and accelerating academic research and knowledge dissemination.
- Financial and accounting document processing: Automatically extract data and tables from financial reports, support financial data analysis and compliance checks, and improve the efficiency of financial business processing.
- Education sector: Parse educational materials such as textbooks and exam papers, extract questions and answers, assist in the digitization and online delivery of teaching content, and support the development of educational informatization.
- Enterprise internal document management: Support processing documents such as internal meeting minutes and project reports, extract key information, and optimize enterprise operational efficiency..