GLM-OCR - Zhipu's Open-Source Lightweight Multimodal OCR Model

• General Text Recognition: Supports photos, screenshots, scanned documents, and PDFs, recognizing printed text, handwriting, seals, code, and other special text.
• Complex Table Parsing: Accurately understands structures such as merged cells and multi-level headers, and directly outputs HTML code without the need for secondary table creation.
• Structured Information Extraction: Intelligently extracts key fields from cards, receipts, and tables, and outputs standard JSON format for integration with business systems.
• Formula and Code Recognition: Supports accurate recognition of mathematical formulas, program code, and other professional technical content.
• Multilingual and Mixed Layout Support: Supports processing complex layouts such as vertical text and multilingual mixed text.
• Batch Document Processing: Supports large-scale document recognition and outputs well-structured formats, providing a high-quality data foundation for RAG.

GLM-OCR's Technical Principles

• Overall Architecture: GLM-OCR adopts the classic "encoder-decoder" architecture design and is overall inherited from the GLM-V series. The architecture consists of three core modules: the CogViT visual encoder on the vision side (400M parameter scale), a lightweight connector layer responsible for cross-modal information fusion, and the backend GLM-0.5B language decoder.
• Visual Encoding: The visual encoder adopts Zhipu's self-developed CogViT architecture and introduces the CLIP contrastive learning strategy for large-scale pretraining on billions of image-text pairs. This gives the model powerful text detection and layout semantic understanding capabilities, enabling it to effectively handle challenges in complex documents such as multi-column layouts, mixed text and images, and rotated text.
• Cross-Modal Fusion: To achieve efficient fusion of visual and linguistic information, GLM-OCR designs a lightweight and efficient connector layer structure. It incorporates the SwiGLU activation mechanism and introduces a 4x downsampling strategy, enabling precise filtering and retention of key visual tokens and efficient compression and transmission of high-density visual semantic information to the backend language decoder, supporting high-precision OCR recognition output.
• Training Optimization: In its training strategy, GLM-OCR is the first to introduce multi-token prediction loss (MTP) into OCR model training. By simultaneously predicting multiple future tokens, it enhances the density of the loss signal and significantly improves model learning efficiency. Through continuous and stable full-task reinforcement learning training, it further optimizes the model's overall recognition accuracy and cross-domain generalization ability in complex document scenarios.
• Inference Process: At the system level, GLM-OCR adopts a two-stage technical paradigm of "layout analysis → parallel recognition". It performs document layout analysis based on PP-DocLayout-V3 to precisely locate regions such as text, tables, and images; OCR recognition is executed in parallel, ultimately achieving stable, high-quality, and highly efficient parsing results in document scenarios with diverse layouts and complex structures.

GLM-OCR's project address