CogAgent

CogAgent is an open-source, end-to-end GUI Agent (graphical interface agent) based on a vision-language model (VLM). The official repository is zai-org/CogAgent, which provides model descriptions, inference and fine-tuning methods, App examples, and other materials.

Main versions and progress

• December 2024: Open-sourced the latest version, the CogAgent-9B-20241220 model. Compared with previous generations, it has significantly improved GUI perception, reasoning accuracy, action space completeness, task generality, and generalization ability, and supports Chinese-English bilingual interaction via screenshots and natural language.
• June 2024: CogAgent was accepted by CVPR 2024 and recognized as a conference Highlight (top 3%).
• December 2023: Open-sourced the first GUI Agent: CogAgent (see the introduction for the original repository), and released the paper CogAgent Paper.
• November 2023: Released the first generation of CogAgent; the related code and model weights are available in the CogVLM & CogAgent official repository.

Model introduction CogAgent-9B-20241220 is based on GLM-4V-9B, a bilingual open-source VLM base model. Through data collection and optimization, multi-stage training, and strategy improvements, this version has made notable progress in GUI perception, reasoning prediction accuracy, action space completeness, and cross-task generalization, and supports Chinese-English bilingual interaction with screenshot and language input. This model has been applied to Zhipu AI's GLM-PC product.

Capability demonstration The model achieves leading results on GUI Agent tasks and GUI Grounding benchmarks across multiple platforms and categories. In the technical blog, it is compared with API-based commercial models (GPT-4o-20240806, Claude-3.5-Sonnet), commercial APIs plus GUI Grounding models (GPT-4o + UGround, GPT-4o + OS-ATLAS), and open-source GUI Agent models (Qwen2-VL, ShowUI, SeeClick). The results show that CogAgent leads on GUI localization (Screenspot), single-step operations (OmniAct), the Chinese step-by-step self-developed benchmark (CogAgentBench-basic-cn), and multi-step operations (OSWorld);on OSWorld, it is only slightly inferior to Claude-3.5-Sonnet, which focuses on Computer Use, and to the solution combining GPT-4o with an external GUI Grounding model.

Inference and fine-tuning costs

• Inference requires at least 29GB of VRAM (BF16 precision). INT4 inference is not recommended because the performance loss is obvious; INT4 inference uses about 8GB of VRAM, and INT8 about 15GB. The relevant lines have been commented out in inference/cli_demo.py and can be uncommented for use, but only NVIDIA devices are supported.
• The above GPUs refer to A100 or H100; other devices need to calculate the required GPU/CPU memory accordingly.
• During supervised fine-tuning (SFT), the codebase freezes the vision encoder, uses a batch size of 1, and trains with 8 A100s; the total input tokens (including images, with images taking 1600 tokens) total 2048 tokens. This codebase cannot perform SFT fine-tuning without freezing the vision encoder. LoRA fine-tuning does not freeze the vision encoder, uses a batch size of 1, and uses 1 A100; the total input tokens (including 1600 image tokens) are also 2048. Under the above settings, SFT fine-tuning requires at least 60GB of VRAM per GPU (8 cards in total), and LoRA fine-tuning requires at least 70GB of VRAM on a single card (it cannot be split).
• SFT fine-tuning has not been tested on Ascend devices; it has only been tested on an Atlas800 training server cluster. The inference code needs to be modified according to the loading mechanism in the Ascend device download link.
• The online demo link does not support controlling the computer; it only allows viewing the model's inference results. Local deployment is recommended.

Model input and output cogagent-9b-20241220 is an agentic execution model, not a dialogue model. It does not support continuous dialogue, but it does support continuous execution history. A new session must be started each time, and the past history must be provided to the model. To obtain the best GUI Agent performance, a strict input-output format is used. The user input concatenation order is: query = f'{task}{history}{platform}{format}'. If the model needs to continue answering, '[Continue]\n' can be appended after {task}, concatenated as: query = f'{task}[Continue]\n{history}{platform}{format}'.

• task field: user task description, in text format, similar to a prompt, and needs to be concise and clear.
• platform field: supports three platforms - Windows 10 and 11 use WIN; macOS 14 and 15 use Mac; Android 13, 14, 15 (and other GUI operation variants that are similar) use Mobile. If it is not these systems, the effect may be suboptimal.
• format field: return format options include: Answer in Action-Operation-Sensitive format (the default demonstration return type in this repository, returning actions, corresponding operations, and sensitivity levels); Answer in Status-Plan-Action-Operation format (returning status, plan, and corresponding operations); Answer in Status-Action-Operation-Sensitive format (returning status, actions, corresponding operations, and sensitivity); Answer in Status-Action-Operation format (returning status and actions); Answer in Action-Operation format (returning actions and corresponding operations).
• history field: concatenate in the above order.
• The output content may include: sensitive operations (<<sensitive operation>> and <<general operation>>, returned only when the Sensitive format is requested); Plan, Status, and Action fields (returned only when the corresponding fields are requested); General answer section (a summary before the formatted answer); Grounded Operation field (describing the specific operation, including location, type, and action details; the box attribute indicates the execution coordinate area, element_type indicates the element type, and element_info describes the element, all wrapped in the "operation instruction"). Note: Even if there are no historical actions, 'History steps:' still needs to be appended to the prompt. The model output is a strictly formatted string and does not support JSON output, so it must be parsed strictly according to the official requirements. The model must input images; pure text dialogue cannot complete GUI Agent tasks.

Running the model Python 3.10.16 or above needs to be installed, and dependencies need to be installed. Local inference based on transformers canrun the corresponding command; the command-line interaction code needs to provide the image path. If the model returns a result containing a bounding box, it will output an image with the bounding box annotated, indicating the area where the operation should be executed, and save it to output_image_path with the file name {your_input_image_name}_{round}.png. format_key indicates the expected response format. The platform field specifies the platform (such as Mac), so when platform is set to Mac, all uploaded screenshots must come from macOS. If you need to run the online Web demo, it supports continuously uploading images for interactive inference. You can run the corresponding command, and the experience is the same as the HuggingFace Space online demo. The model returns the corresponding bounding box and execution category.

App and fine-tuning The official basic demo App is provided, explaining how to deploy the model on a server with a GPU and run cogagent-9b-20241220 locally to perform automated GUI operations. The official statement says it cannot guarantee the safety of AI behavior, so caution is required when using it. The examples are for academic reference only, and it does not assume legal responsibility for problems arising from them. If you are interested in the App, you can view the documentation; if you want to fine-tune the model, you can refer to the relevant instructions.