What is GOT-OCR2.0
GOT-OCR 2.0 is an advanced Optical Character Recognition (OCR) model that propels OCR technology into the 2.0 era. The GOT-OCR 2.0 end-to-end model consists of a high-compression encoder and a long-context decoder, and can process a variety of optical characters including text, mathematical formulas, molecular formulas, charts, sheet music, and geometric figures. GOT-OCR 2.0 supports multiple languages, especially Chinese and English, and can output various formatted results such as Markdown and LaTeX. The model features interactive OCR capabilities, including region-level recognition and a dynamic resolution strategy, as well as multi-page OCR technology, making it suitable for high-resolution images and batch document processing. GOT-OCR 2.0 has 580M parameters and a model size of 1.43GB, providing an accurate and efficient OCR solution.
Main features of GOT-OCR2.0
- Multilingual and multimodal recognition: Supports text recognition in multiple languages, including Chinese and English, as well as handwritten and printed text.
- Diverse input and output: Can process multiple input formats such as photos, documents, and slices, and supports output formats such as plain text, Markdown, TikZ, SMILES, and Kern.
- Long text processing: The decoder supports tokens up to 8K, making it suitable for processing long text materials such as academic papers and legal documents.
- Interactive OCR capabilities: Region-level recognition guided by coordinates or colors, providing a more flexible user experience.
- Dynamic resolution strategy: Adapts to ultra-high-resolution images, such as large posters or stitched PDF pages, while maintaining recognition accuracy.
- Multi-page OCR technology: Batch processes multi-page documents, improving the processing efficiency of long PDF files or multi-image documents.
Technical principles of GOT-OCR2.0
- Encoder-decoder architecture：
- Encoder: Responsible for compressing the input image into a series of image tokens, where the tokens capture the visual information in the image.
- Decoder: Receives the image tokens output by the encoder and converts them into text output. The decoder supports long context and can process long text.
- High-compression encoder: The encoder compresses a 1024×1024 pixel image into image tokens of size 256×1024, which helps process high-resolution images.
- Long-context decoder: The decoder supports token sequences up to 8K long and can process long documents containing large amounts of text.
- Multi-stage training strategy：
- Pre-training stage: The encoder is pre-trained on large amounts of text data to learn visual representations of text.
- Joint training stage: The encoder is trained together with the new decoder to adapt to a broader range of OCR tasks.
- Post-training stage: The decoder is further trained to support advanced features such as fine-grained OCR, dynamic resolution, and multi-page OCR.
Project address of GOT-OCR2.0
- GitHub repository：https://github.com/Ucas-HaoranWei/GOT-OCR2.0
- HuggingFace model library：https://huggingface.co/stepfun-ai/GOT-OCR2_0
- arXiv technical paper：https://arxiv.org/pdf/2409.01704
Application scenarios of GOT-OCR2.0
- Document digitization: Converts paper documents (such as books, manuscripts, legal documents, academic papers, etc.) into electronic formats for easy storage, retrieval, and editing.
- Scene text recognition: Recognize and extract text in natural scenes, such as street signs, billboards, menus, etc.
- Receipt Processing: Automatically recognize and extract text information on invoices, receipts, and bills, simplifying financial and accounting processes.
- Identity Verification and Security: In scenarios that require verifying personal identity, such as banking, airport security checks, etc., recognize information on passports, ID cards, or driver's licenses.
- Logistics and Transportation: Automatically recognize barcode and address information on packages, improving the efficiency of logistics sorting and delivery.
- Medical Record Management: Recognize and digitize doctors' handwritten prescriptions, medical records, and other medical documents.