SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

FireRed-OCR - Xiaohongshu's Open-Source Document Structure Parsing Vision-Language Model

Author: AI小集 · Original source: AI工具集

What is FireRed-OCR

FireRed-OCR is a lightweight document structure parsing vision-language model open-sourced by the Xiaohongshu team. With only 2B parameters, it achieved a comprehensive score of 92.94% in the authoritative OmniDocBench v1.5 evaluation, surpassing super-large models such as GPT-5.2, Gemini-3.0 Pro, and Qwen3-VL-235B, realizing the breakthrough of "a small model defeating large models." The model is based on the Qwen3-VL-2B-Instruct architecture and adopts a three-stage progressive training strategy. FireRed-OCR is specifically designed to solve the "structural hallucination" problem in document parsing, and can accurately extract complex tables, mathematical formulas, hierarchical headings, and other content, converting them into standard Markdown format.

FireRed-OCR

FireRed-OCR's Main Features

  • Complex Table Extraction: Accurately identifies and extracts table structures from cluttered PDFs and scanned documents, maintaining row-column correspondence and avoiding the table misalignment problems common in traditional OCR.
  • Mathematical Formula Parsing: Accurately recognizes mathematical formulas in documents and converts them into standard LaTeX or Markdown format, ensuring formula syntax validity and readability.
  • Hierarchical Structure Restoration: Intelligently identifies heading levels (H1-H6), paragraph indentation, list markers, and more in documents, generating Markdown hierarchical structures that conform to specifications.
  • Multi-format Document Conversion: Supports one-click conversion of PDFs, scanned images, academic papers, financial reports, and other document formats into structured Markdown text.
  • Anti-Structural Hallucination: Through GRPO reinforcement learning optimization, significantly reduces common document parsing errors such as content fabrication, row order confusion, and hierarchy disorder.
  • Multi-scenario Adaptation: Suitable for professional scenarios such as financial report digitization, academic paper parsing, contract document structuring, and book content extraction.
  • Lightweight Deployment: With a 2B parameter scale, it supports local deployment and API calls, reducing computing costs, and is suitable for small and medium-sized enterprises and individual developers.

FireRed-OCR's Technical Principles

  • Base Architecture: Built on the Qwen3-VL-2B-Instruct multimodal large model, inheriting its powerful visual understanding and text generation capabilities.
  • Three-stage Progressive Training Strategy：
    • Stage 1 (Multi-task Pre-alignment): Simultaneously trains three tasks—region detection, region recognition, and layout-to-Markdown—establishing the model's perception of document spatial layout.
    • Stage 2 (Specialized SFT): Conducts supervised fine-tuning on high-quality, standardized Markdown datasets to ensure output logical consistency and hierarchical expression accuracy.
    • Stage 3 (Format-constrained GRPO): Applies the Group Relative Policy Optimization reinforcement learning algorithm to optimize output quality through a format reward mechanism.
  • Four Major Reward Mechanisms：
    • Formula syntax validity reward: Ensures mathematical formulas conform to LaTeX syntax specifications.
    • Table completeness reward: Ensures complete and corresponding table row-column structures.
    • Hierarchy closure reward: Verifies that Markdown heading level tags are correctly closed.
    • Text accuracy reward: Improves text recognition precision and content fidelity.
  • Structural Hallucination Suppression: Addressing common issues in document parsing such as scrambled table rows, fabricated formulas, and disordered hierarchy, it jointly optimizes through format constraints and reinforcement learning, significantly reducing the hallucination rate.
  • End-to-end optimization: Directly generates structured Markdown from visual input, without the traditional OCR multi-stage pipeline (detection → recognition → layout analysis → formatting), reducing error accumulation.

FireRed-OCR project address

  • Github repository：https://github.com/FireRedTeam/FireRed-OCR

FireRed-OCR application scenarios

  • Financial report digitization: Accurately extracts complex tables and financial data from listed company financial reports and audit reports, converting them into structured Markdown for easy financial analysis and data warehousing.
  • Academic paper parsing: Recognizes mathematical formulas, figure and table captions, and reference hierarchy in research papers, generating standard academic format text to assist literature management and knowledge extraction.
  • Contract document structuring: Converts scanned contracts and legal documents into editable structured text, preserving clause hierarchy and key information, improving the efficiency of legal document processing.
  • Book and magazine digitization: Processes scanned books, journals, and magazines, restoring table of contents hierarchy and main text layout, quickly building a searchable digital library.
  • Educational material organization: Parses formulas and table content in textbooks, exam papers, and lecture notes, converting them into structured formats suitable for online learning, supporting content development for educational platforms.
  • Archive digitization: Helps enterprises and institutions convert historical paper archives and handwritten notes into structured electronic documents, achieving permanent preservation and intelligent retrieval of archives.