What is AgentCPM-GUI
AgentCPM-GUI is an open-source on-device GUI Agent jointly launched by Tsinghua University and the ModelBest team, optimized for Chinese applications. The model is built on MiniCPM-V (8 billion parameters) and can accept smartphone screenshots as input to autonomously execute user-specified tasks. The model is pretrained on large-scale Chinese Android app interface data, significantly improving its understanding and grounding capabilities for GUI elements. AgentCPM-GUI achieves SOTA performance on both the Chinese Grounding Benchmark and the Agent Benchmark, making it the first open-source GUI Agent finely optimized for Chinese applications.
Main Features of AgentCPM-GUI
- Chinese Application Operation: Understands and operates a variety of Chinese applications, such as Amap, Dianping, Bilibili, and Xiaohongshu.
- Task Automation Execution: After receiving user instructions, automatically breaks down task steps and accurately executes them in the corresponding applications, such as placing orders or playing videos.
- High-Quality GUI Grounding: Accurately locates GUI elements on the screen, such as buttons, input boxes, and labels.
- OCR Grounding and Recognition: Recognizes text content on the screen and executes corresponding operations based on text descriptions.
Technical Principles of AgentCPM-GUI
- Pretraining: Pretrained on large-scale Chinese Android app interface data, covering common general GUI controls such as buttons, input boxes, labels, and icons. Based on high-quality GUI Grounding pretraining, it improves the model's understanding and grounding capabilities for visual interface elements. It has OCR Grounding capabilities, accurately locating and recognizing text content on the screen.
- Reinforcement Fine-Tuning (RFT): Designs reward functions across three dimensions—action format reward, action type reward, and action parameter reward—to guide the model in autonomously generating high-quality chain-of-thought processes. Under the guidance of the reward functions, the model continuously optimizes its own strategy to obtain higher rewards and improve task execution success rates.
- Compact Action Space Design: Based on a compact JSON format, the average action length is compressed to 9.7 tokens, reducing computation and memory requirements during inference. The compact action space design makes the model more suitable for deployment on mobile devices, speeds up on-device inference, and alleviates privacy and security issues.
AgentCPM-GUI Project Address
- GitHub Repository：https://github.com/OpenBMB/AgentCPM-GUI
- HuggingFace Model Library：https://huggingface.co/openbmb/AgentCPM-GUI
Application Scenarios of AgentCPM-GUI
- Intelligent Assistant: Helps users complete various application operations based on voice or text instructions, such as ordering takeout or playing music.
- Automated Testing: Used to automate testing of the functions and interfaces of Chinese applications, improving testing efficiency.
- Elderly Care: Simplifies smartphone operations for the elderly, such as video calls and checking the weather.
- Assistance for Visually Impaired People: Combines voice instructions and screen readers to help visually impaired people use applications more conveniently.
- Enterprise Application Automation: Automates repetitive tasks in enterprises, such as data entry and report generation, improving efficiency.