What is mPLUG-DocOwl2
mPLUG-DocOwl 2 is a multimodal large language model for multi-page document understanding launched by the mPLUG team at Alibaba Tongyi Lab. Without relying on optical character recognition (OCR) technology, it achieves efficient understanding and processing of document images through high-resolution document image compression technology. mPLUG-DocOwl 2 has reached a new state of the art (SOTA) on multi-page document understanding benchmarks, consuming 324 tokens per document image page, reducing GPU memory usage and time to first packet, and improving processing speed. The model's training is divided into three stages: single-page pretraining, multi-page pretraining, and multi-task instruction fine-tuning. mPLUG-DocOwl 2 supports understanding single-page documents and can also handle complex problems in multi-page documents, such as cross-page content association and structure parsing.
Main features of mPLUG-DocOwl2
- Multi-page document understanding: Extracts and understands information directly from multi-page document images without relying on OCR technology.
- High-resolution image processing: Through the high-resolution document image compression module, each document image page is compressed into 324 visual tokens, reducing GPU memory usage and time to first packet.
- Multi-page question answering capability: Answers questions about the content of multi-page documents, providing detailed explanations and relevant page numbers.
- Document structure parsing: Parses and represents the hierarchical structure of multi-page documents, outputting it in JSON format for further data processing and analysis.
- Cross-page content association: Understands and associates cross-page content in multi-page documents, providing cross-page structure understanding.
- Efficient processing: On a single A100-80G GPU, processes up to 60 pages of high-definition document images simultaneously, improving processing efficiency.
Technical principles of mPLUG-DocOwl2
- High-resolution document image compression (High-resolution DocCompressor): Based on low-resolution global visual features as guidance, compresses high-resolution document images into fewer visual tokens through a cross-attention mechanism.
- Shape-adaptive Cropping: The adaptive cropping module cuts according to the shape and size of the document to adapt to different page layouts.
- Visual feature extraction: Uses a visual encoder (such as ViT) to extract the visual features of each slice, and performs feature merging and dimension alignment through the H-Reducer module.
- Cross-attention mechanism: During compression, global image features are used as queries and slice features as key-value pairs, achieving feature compression through cross-attention layers.
- Combination of global and local visual features: Combines global visual features (capturing layout information) and local visual features (preserving text and image details) to achieve more accurate document understanding.
Project address of mPLUG-DocOwl2
- GitHub repository：https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl2
- arXiv technical paper：https://arxiv.org/pdf/2409.03420v2
Application scenarios of mPLUG-DocOwl2
- Legal document analysis: Automatically parses legal documents and cases, extracts key information, and supports legal research and case preparation.
- Medical record management: Extracts important data from medical records and reports, supporting patient care, research, and administrative management.
- Academic research: Helps researchers quickly understand and summarize large amounts of literature, accelerating scientific discovery and knowledge innovation.
- Financial report analysis: Automated processing of annual reports, financial statements, and other financial documents, extracting key financial indicators and trends.
- Government document processing: Automated processing of government-issued announcements, regulations, and policy documents to improve government service efficiency.