What is CogAgent-9B
CogAgent-9B is a specialized Agent task model trained based on GLM-4V-9B. It relies solely on screen screenshots as input and does not require textual representations such as HTML. CogAgent-9B supports high-resolution image processing, has bilingual (Chinese and English) interaction capabilities, can predict and execute GUI operations, and enables automated tasks. It has achieved leading results on multiple GUI operation datasets. The model has been open-sourced, promoting the development of the large-model Agent ecosystem. CogAgent-9B can be widely used in GUI interaction scenarios such as personal computers, mobile phones, and in-vehicle systems.
Main features of CogAgent-9B
- GUI understanding and operation: CogAgent-9B can understand and operate graphical user interfaces (GUIs), performing tasks such as clicking buttons and entering text.
- Screenshot input: The model relies solely on screen screenshots as input and does not require additional textual representations such as HTML, making it flexible to apply on various devices.
- High-resolution processing: Supports high-resolution image input up to 1120×1120 pixels and can process more complex visual information.
- Bilingual interaction: Supports screen screenshots and language interaction in Chinese and English, enhancing the model's international application capabilities.
- Predicting GUI operations: Based on the user-specified task and historical operations, the model can predict the next GUI operation.
- Automated task execution: CogAgent-9B can simulate user operations and automatically execute a series of GUI tasks.
- Cross-platform applications: Suitable for various GUI interaction-based scenarios such as personal computers, mobile phones, and in-vehicle systems.
- Leading performance: Achieved leading results on multiple GUI operation datasets, demonstrating its outstanding performance.
Technical principles of CogAgent-9B
- Vision-language model (VLM): CogAgent-9B is built on the powerful vision-language model GLM-4V-9B and can process visual data (such as screen screenshots) and textual information to achieve understanding and operation of GUI elements.
- Dual-stream attention mechanism: CogAgent-9B adopts a dual-stream attention mechanism that maps visual elements (such as buttons and icons) to their textual labels or descriptions, enhancing the model's ability to predict user intent and execute related operations.
- GUI Grounding pre-training: During the pre-training stage, CogAgent-9B introduces the GUI Grounding pre-training method, constructing correspondences between interface sub-regions and layout representations through screen screenshots and layout pairs, improving the model's basic understanding of visual input and GUI interfaces.
- Rich and improved datasets: The CogAgent-9B team extensively collected and integrated various datasets, including unsupervised data and GUI instruction fine-tuning datasets. The richness and diversity of these datasets provide a broad training and testing foundation for the model.
- Optimization of pre-training and post-training strategies: In the pre-training stage, CogAgent-9B introduces GUI Referring Expression Generation (REG) and GUI Referring Expression Comprehension (REC) tasks to construct correspondences between interface sub-regions and layout representations. In the post-training stage, a more scientific GUI agent post-training strategy is adopted, giving the model stronger analysis, reasoning, and prediction capabilities.
- Model inference and chain-of-thought optimization: CogAgent-9B decomposes the chain of thought into Status (current screen state), Plan (global plan), Action (natural language description of the next step), and Operation (formal language description of the next step), and trains on data mixed from multiple modes via random sampling, flexibly adjusting and controlling the actual output during the reasoning process.
- Action space refinement: CogAgent-9B clarifies the basic action space and adds advanced actions such as LLM, QUOTE_TEXT, and LAUNCH, enhancing the model's ability to use tools and interact.
CogAgent-9B project address
- Github repository：https://github.com/THUDM/CogAgent
- HuggingFace model library：https://huggingface.co/THUDM/cogagent-9b-20241220
Application scenarios of CogAgent-9B
- Automated testing: In software development, CogAgent-9B can be used for automated testing, simulating user operations to test an application's GUI, improving testing efficiency and coverage.
- Intelligent assistant: As an intelligent personal assistant, CogAgent-9B can help users automatically complete daily tasks, such as schedule management and email processing.
- Customer service: In the customer service field, CogAgent-9B can assist customer service personnel through automated operations, quickly responding to customer needs and executing related operations.
- Smart home control: CogAgent-9B can be integrated into smart home systems, controlling various smart devices in the home through the GUI.
- Smart cockpit: In the automotive field, CogAgent-9B can be used in smart cockpit systems, interacting with in-vehicle infotainment systems through the GUI to provide a safer and more convenient driving experience.