What is Mini-Monkey
Mini-Monkey is a lightweight multimodal AI model jointly launched by Huazhong University of Science and Technology and South China University of Technology. It adopts the Multi-Scale Adaptive Cropping strategy (MSAC) and the Scale Compression Mechanism (SCM), effectively solving the jagged edge effect caused by traditional image cropping and improving performance in high-resolution image and document understanding tasks. Mini-Monkey has achieved excellent results in multiple benchmark tests and has strong potential in the fields of multimodal understanding and document intelligence.
Features of Mini-Monkey
- Multi-Scale Adaptive Cropping strategy (MSAC): By generating representations at different scales, it allows the model to select objects that have not been segmented, thereby improving the ability to recognize small objects or irregularly shaped objects.
- Scale Compression Mechanism (SCM): A training-free and parameter-free mechanism used to reduce computational overhead by focusing on visual tokens at the detail layer to extract key visual features.
- High-resolution image processing: Optimizes the processing capability for high-resolution images and reduces semantic incoherence caused by image cropping.
- Document understanding tasks: Performs excellently in document understanding and effectively avoids the problems caused by traditional cropping strategies when processing small text and irregular shapes in documents.
Technical Principles of Mini-Monkey
- Multi-Scale Adaptive Cropping strategy (MSAC)：The strategy divides the image into grids of different scales and layers them according to the aspect ratio of the grids, providing features at different scales.The detail layer provides high-resolution image details, while the adaptive layer avoids repeatedly cutting the same object based on the aspect ratio generated by the detail layer, ensuring semantic consistency between different layers.
- Scale Compression Mechanism (SCM)：SCM is used to reduce the computational overhead brought by MSAC by generating attention maps to identify and extract key visual features.It uses the first few layers of a trained large language model (LLM) to select visual Tokens, without additional training or parameters.
- Attention weight allocation：In SCM, visual Tokens of different importance are distinguished by assigning high attention weights and low attention weights.Tokens with lower attention weights will be filtered out, while important Tokens are retained for further processing.
- Shared LLM layers：SCM uses block layers of the LLM to share the computation of processing different visual Tokens, which helps improve the efficiency and performance of the model.
Project Address of Mini-Monkey
-
arXivTechnical Paper：https://arxiv.org/pdf/2408.02034v2
Application Scenarios of Mini-Monkey
- Document understanding: When processing document images, it can accurately recognize and understand the textual content in documents, including complex documents such as ancient books and handwritten notes.
- Image recognition: In scenarios that require recognizing small objects or irregularly shaped objects in images, such as medical image analysis and satellite image interpretation, Mini-Monkey can provide more accurate recognition results.
- Multimodal information processing: Scenarios that combine image and text information for comprehensive analysis, such as social media content analysis and evaluation of the matching degree between advertising images and copywriting.
- Content generation: In applications that need to generate descriptions or stories based on image content, Mini-Monkey can provide richer and more accurate text content.
- Decision support system: In decision support systems that require rapid processing and analysis of large amounts of visual data, such as security surveillance, traffic management, etc.