What is ParGo
ParGo is an innovative multimodal large language model connector proposed through collaboration between the ByteDance team and Sun Yat-sen University, enhancing the alignment between visual and language modalities in multimodal large language models (MLLMs). By combining local tokens and global tokens, it uses carefully designed attention masks to separately extract local and global information, enhancing the modeling of relationships between local regions while controlling the number of tokens, considering both image details and the global perspective, and overcoming the problem of traditional methods ignoring details.
Main features of ParGo
- Efficient connection between visual and language modalities: ParGo adopts a joint global + local perspective approach, using two key modules, the Partial-Global Perception Block (PGP) and the Cascaded Partial Perception Block (CPP), to map visual features into Partial tokens and Global tokens, separately extracting local and global information from images. This achieves an efficient connection between visual features and large language models (LLMs), overcoming the problem of traditional methods overly focusing on salient regions.
- Improving the performance of multimodal large language models: It performs excellently on multiple MLLM benchmarks, for example, on the MME benchmark it improves by 259.96 compared with the traditional Q-Former projector. Especially in tasks that emphasize detail perception ability, ParGo significantly outperforms other projectors, enabling more accurate text recognition, better description of image details, and more effective identification of local elements.
- Self-supervised learning enhances contextual understanding: A self-supervised learning strategy is introduced during the training phase, enhancing the model's ability to understand context by predicting the content of occluded parts, improving the model's generalization performance and reducing dependence on large-scale annotated data.
Technical principles of ParGo
- Joint global + local perspective: ParGo adopts two types of learnable tokens and, based on the attention mechanism, simultaneously maps visual features into large language models (LLMs) from both local and global perspectives.
- Partial-Global Perception Block (PGP): In ParGo, the features of the visual encoder are mapped into two different types of tokens: Partial tokens and Global tokens, which can separately extract local and global information from images.
- Partial tokens: Each token interacts with only part of the visual features, focusing on the local information of the image.
- Global tokens: Global tokens interact with all visual features, capturing the global information of the image.
- Cascaded Partial Perception Block (CPP): ParGo introduces the Cascaded Partial Perception (CPP) module before the Partial-Global Perception module. The core of the CPP module is a self-attention mechanism with specially designed masks. As the number of layers increases, each Partial token can access more neighboring tokens, gradually expanding its perception range.
- Self-supervised learning strategy: During the training phase, ParGo introduces a self-supervised learning strategy, that is, enhancing the model's ability to understand context by predicting the content of occluded parts.
Project address of ParGo
- Github repository：https://github.com/bytedance/ParGo
- arXiv technical paper：https://arxiv.org/pdf/2408.12928
Application scenarios of ParGo
- Visual question answering system: ParGo can understand visual cues in images and parse semantic information in text, performing excellently in visual question answering tasks.
- Image caption generation: ParGo performs particularly well on tasks such as COCO Caption and can generate high-quality image captions. It can describe the global information of an image and also capture local details in the image, generating more accurate and richer captions.
- Cross-modal retrieval: ParGo can be used for cross-modal retrieval tasks, helping users find relevant images through text queries, or find relevant text through image queries.
- Sentiment analysis: ParGo can understand sentiment information in images and text, providing more accurate results in sentiment analysis tasks. Image content understanding: ParGo can be used for image content understanding tasks, helping systems better understand the details and global information in images.