CogVLM2 - Zhipu AI's New Generation Multimodal Large Model

What is CogVLM2

CogVLM2 is a new generation multimodal large model launched by Zhipu AI, achieving significant performance improvements in visual and language understanding. It supports text lengths of up to 8K and image inputs with a resolution of 1344*1344, and possesses strong document image understanding capabilities. The model adopts a 5-billion-parameter visual encoder and a 7-billion-parameter visual expert module. Through a deep fusion strategy, it optimizes the interaction between visual and language modalities, ensuring that language processing capabilities are maintained while enhancing visual understanding. The open-source version of CogVLM2 supports both Chinese and English, with a model size of 1.9 billion parameters, but the number of parameters activated during actual inference is approximately 12 billion, demonstrating efficient performance in multimodal tasks.

Improvements of CogVLM2

The improvements of the CogVLM2 model compared to the previous generation mainly include the following aspects:

• Performance Improvement: On multiple key benchmarks such as OCRbench and TextVQA, CogVLM2's performance has significantly improved. For example, performance on OCRbench improved by 32%, and performance on TextVQA improved by 21.9%.
• Document Image Understanding: CogVLM2 enhances its understanding and question-answering capabilities for document images, especially performing well on the DocVQA benchmark.
• Support for High-Resolution Images: The model supports image resolutions of up to 1344*1344 pixels, enabling it to process higher-definition images.
• Support for Long Text: CogVLM2 supports text inputs of up to 8K, enabling the model to process longer documents and more complex language tasks.
• Bilingual Support: CogVLM2 provides an open-source model version supporting both Chinese and English, enhancing the model's multilingual capabilities.

Model Information of CogVLM2

CogVLM2 has open-sourced two CogVLM2 models based on Meta-Llama-3-8B-Instruct as the language foundation model, namely cogvlm2-llama3-chat-19B and cogvlm2-llama3-chinese-chat-19B. Interested users can go to GitHub, Hugging Face, or ModelScope to download or experience them online.

Model Name	cogvlm2-llama3-chat-19B	cogvlm2-llama3-chinese-chat-19B
Foundation Model	Meta-Llama-3-8B-Instruct	Meta-Llama-3-8B-Instruct
Language	English	Chinese, English
Model Size	19B	19B
Task	Image understanding, dialogue model	Image understanding, dialogue model
Model Link	🤗 Huggingface 🤖 ModelScope 💫 Shizhi AI 🐙GitHub	🤗 Huggingface 🤖 ModelScope 💫 Shizhi AI
Experience Link	📙 Official Page	📙 Official Page 🤖 ModelScope
Int4 Model	Not yet available	Not yet available
Text Length	8K	8K
Image Resolution	1344 * 1344	1344 * 1344

Model Architecture of CogVLM2

The model architecture of CogVLM2 has been optimized and innovated on the basis of inheriting the previous generation model. The specific features are as follows:

1. Visual Encoder: CogVLM2 adopts a visual encoder with 5 billion parameters, responsible for feature extraction and encoding of input images.
2. Visual Expert Module: A 7-billion-parameter visual expert module is integrated into the large language model. Through unique parameter settings, this module finely models the interaction between visual and language sequences.
3. Deep Fusion Strategy: CogVLM2 adopts a deep fusion strategy, enabling the visual modality and the language modality to be more tightly integrated, thereby enhancing the model's visual understanding capability while maintaining its advantages in language processing.
4. MLP Adapter: An MLP (Multilayer Perceptron) Adapter is used in the model to adjust and adapt features between different modalities.
5. Downsampling Module: To better process and understand high-resolution documents or web page images, CogVLM2 introduces a dedicated downsampling module after the visual encoder, effectively extracting key information and reducing the sequence length input to the language model.
6. Word Embedding: The model includes a Word Embedding layer, which is used to convert text into numerical vectors that the model can understand.
7. Multi-Expert Module Structure: CogVLM2 is designed with a multi-expert module structure, so that during inference, the number of actually activated parameters is only about 12 billion. This design both ensures the model's performance and improves inference efficiency.
8. Language Base Model: CogVLM2 uses Meta-Llama-3-8B-Instruct as the language base model, providing the model with powerful language understanding and generation capabilities.

CogVLM2 Model Performance

The CogVLM2 team conducted quantitative evaluations on a series of multimodal benchmarks, including TextVQA, DocVQA, ChartQA, OCRbench, MMMU, MMVet, MMBench, etc. As can be seen from the table below, the two models of CogVLM2, despite their smaller model size, achieve SOTA performance on multiple benchmarks; and on other performance metrics, they can also reach levels close to closed-source models (such as GPT-4V, Gemini Pro, etc.).

Model	Open Source	Model Size	TextVQA	DocVQA	ChartQA	OCRbench	MMMU	MMVet	MMBench
LLaVA-1.5	✅	13B	61.3	–	–	337	37.0	35.4	67.7
Mini-Gemini	✅	34B	74.1	–	–	–	48.0	59.3	80.6
LLaVA-NeXT-LLaMA3	✅	8B	–	78.2	69.5	–	41.7	–	72.1
LLaVA-NeXT-110B	✅	110B	–	85.7	79.7	–	49.1	–	80.5
InternVL-1.5	✅	20B	80.6	90.9	83.8	720	46.8	55.4	82.3
QwenVL-Plus	❌	–	78.9	91.4	78.1	726	51.4	55.7	67.0
Claude3-Opus	❌	–	–	89.3	80.8	694	59.4	51.7	63.3
Gemini Pro 1.5	❌	–	73.5	86.5	81.3	–	58.5	–	–
GPT-4V	❌	–	78.0	88.4	78.5	656	56.8	67.7	75.0
CogVLM1.1 (Ours)	✅	7B	69.7	–	68.3	590	37.3	52.0	65.8
CogVLM2-LLaMA3 (Ours)	✅	8B	84.2	92.3	81.0	756	44.3	60.4	80.5
CogVLM2-LLaMA3-Chinese (Ours)	✅	8B	85.0	88.4	74.7	780	42.8	60.5	78.9