Open-AutoGLM - Zhipu's open-source mobile AI Agent framework

What is Open-AutoGLM

Open-AutoGLM is Zhipu's open-source mobile intelligent assistant framework, built based on AutoGLM , and can automate mobile operations through natural language instructions. Through the Phone Use capability framework, Open-AutoGLM converts natural language instructions into actual operations such as tapping, swiping, and typing, and can automatically complete tasks in scenarios such as food delivery, social networking, and customer service. The framework uses cloud phone technology to ensure privacy and security. The purpose of open-sourcing Open-AutoGLM is to promote the industry's common progress, protect user privacy, and accelerate the explosion of Agent technology. It currently supports more than 50 mainstream Chinese applications, and provides a complete toolchain and detailed documentation to help developers get started quickly and carry out secondary development.

Main features of Open-AutoGLM

• Natural language understanding and task execution: Users can describe their needs in natural language, and the AI automatically parses the intent and executes the operations.
• Multimodal interface understanding: Understands mobile screen content through a vision-language model and identifies interface elements for operation.
• Automated operations: Supports multiple operations such as tapping, swiping, typing text, long-pressing, and double-tapping, simulating real user behavior.
• Sensitive operation confirmation and manual takeover: When privacy-related or sensitive operations are involved, the AI will request manual confirmation or takeover to ensure security.
• Remote debugging and control: Supports remote ADB debugging via WiFi or network, allowing operation without a physical device connection.
• Supports multiple applications: Covers more than 50 mainstream Chinese applications, including scenarios such as social messaging, e-commerce shopping, food delivery, and video entertainment.
• Cloud phone environment deployment: Runs in virtual devices in the cloud to ensure user privacy and data security.

How to use Open-AutoGLM

• Preparation
  • Install Python: Make sure Python is installed on your computer (version 3.10 or above is recommended).
  • Install ADB: Download and install ADB https://developer.android.com/studio/releases/platform-tools。安装完成后，将 Add the ADB path to the system environment variables.
  • Configure an Android device: Make sure the Android device has Developer Mode and USB debugging enabled, and is connected to the computer.
• Download and installation
  • Download Open-AutoGLM：
    • Visit the Open-AutoGLM GitHub repository https://github.com/zai-org/Open-AutoGLM。
    • Click "Code" in the upper right corner of the page” button, select "Download ZIP”, and download the project code.
    • Extract the downloaded file to a convenient local directory (such as C:\Open-AutoGLM or ~/Open-AutoGLM）。
  • Install dependencies：
    • Open a command-line tool (Command Prompt or PowerShell on Windows, Terminal on Mac/Linux).
    • Switch to the extracted project directory:
    • cd path/to/Open-AutoGLM
    • Enter the following command to install dependencies:
    • pip install -r requirements.txt
  • Start the model service：
    • Enter the following command to start the model service (this is a required step, but you do not need to understand the code):
    • python3 -m vllm.entrypoints.openai.api_server --model zai-org/AutoGLM-Phone-9B --port 8000
    • If you see a prompt that the service has started successfully, you can proceed to the next step.
• Run the task: Open a new command line window and enter the following command to execute the task (you only need to replace the task description):python main.py --base-url http://localhost:8000/v1 --model "autoglm-phone-9b" "任务描述"

Open-AutoGLM project address

• GitHub repository：https://github.com/zai-org/Open-AutoGLM
• HuggingFace model library：https://huggingface.co/zai-org/AutoGLM-Phone-9B

Application scenarios of Open-AutoGLM

• Food delivery ordering: Users only need to describe their needs in natural language, and Open-AutoGLM can automatically open the Meituan app, search for KFC, select the family bucket combo, and complete the order placement.
• Social media interaction: In social apps such as WeChat and Weibo, users can instruct the AI to "like a friend's latest update" or "comment on a Douyin video," and the AI will automatically identify the relevant content and perform interactive actions such as liking and commenting.
• Office automation: In office software (such as WPS, Microsoft Office), users can issue voice or text commands such as "create a document named 'Project Plan' and write the meeting content into it," and the AI can complete document creation and content editing.
• Smart home control: Through smart home apps (such as Xiaomi Smart Home), Open-AutoGLM can accurately identify and control the corresponding devices, enabling automated switching of home scenarios.
• Transportation and travel: In map or ride-hailing apps (such as Amap and Didi Chuxing), the AI will plan routes in real time and complete the ride-hailing operation, making it convenient and efficient.