AppAgent

AppAgent is a multimodal agent framework based on a large language model (LLM), designed to enable agents to operate smartphone applications like human users. It interacts with mobile applications through a simplified action space (such as taps and swipes), without requiring access to the system backend, making it applicable to various applications. Agents can learn to use new applications through autonomous exploration or by observing human demonstrations, and generate a knowledge base to perform complex tasks. The project was developed by teams including Tencent QQGYLab, with the paper published at CHI 2025, and the code has been open-sourced.

Purpose

AppAgent's main purpose is to enable agents to automatically operate Android applications and complete user-specified tasks. It simulates the way humans interact with mobile applications, such as tapping and swiping, and therefore does not rely on system backend interfaces, making it broadly applicable to various applications.

Main Capabilities

• Multimodal understanding: The agent is driven by a multimodal model and can simultaneously receive text and visual input, thereby understanding screen content and making decisions.
• Simplified action space: The agent operates applications through basic actions such as tapping and swiping, without requiring system permissions.
• Learning mechanism: The agent can learn how to use applications through autonomous exploration or by observing human demonstrations, and generate a knowledge base (documentation) for reference in subsequent tasks.
• Knowledge base reuse: During the deployment phase, the agent can refer to the previously generated knowledge base to perform complex tasks, improving the success rate.
• Grid overlay: An optional feature that displays a grid overlay on the screen to tap or swipe UI elements that are not labeled with numeric tags.
• CAPTCHA handling: Experiments show that AppAgent has the ability to pass CAPTCHA.

How to Use

Using AppAgent requires the following steps:

1. Preparation: Install Android Debug Bridge (adb) on a PC, prepare an Android device and enable USB debugging, and connect the device via USB. If there is no Android device, an Android Studio emulator can be used. Clone the repository and install dependencies (Python 3).
2. Configure the agent: Modify the config.yaml file in the root directory to configure the multimodal model. By default, gpt-4-vision-preview is used, requiring an OpenAI API key and a request interval. qwen-vl-max (Tongyi Qianwen-VL) can also be used, requiring the creation of an Alibaba Cloud account and obtaining a Dashscope API key, filling in DASHSCOPE_API_KEY and setting Change the MODEL field to Qwen. Note that GPT-4V is not free, costing about $0.03 per request/response pair.
3. Exploration phase: Run learn.py, choose autonomous exploration or human demonstration mode, and provide the app name and task description. In autonomous exploration mode, the agent attempts the task on its own and generates an element document; in human demonstration mode, the user must follow the prompts to operate, and the agent records the interactive elements and generates a document.
4. Deployment phase: Run run.py, enter the app name, select the document library to use, and provide the task description. The agent will automatically execute the task. If there is no document, you can also choose to run without a document (success rate is not guaranteed).

Applicable Scenarios

AppAgent is suitable for scenarios that require automated operation of Android apps, such as automatically completing specific tasks, testing app features, and assisting users with operations. It is especially suitable for apps whose system backend cannot be accessed or is inconvenient to access.

Limitations

• Depends on multimodal models: GPT-4V or models such as qwen-vl-max need to be configured. GPT-4V is paid and relatively costly.
• Performance differences: qwen-vl-max is currently free but performs worse than GPT-4V.
• Document quality affects the success rate: the coverage and accuracy of the knowledge base directly affect the task completion rate. It is recommended to manually check and correct the documents.
• Platform limitations: currently mainly targeted at the Android platform, requiring connection to a device via adb or use of an emulator.
• Project updates: AppAgentX has been released as the next-generation GUI agent, featuring an evolutionary mechanism.