What is Droidrun
Droidrun is an AI phone operation tool that supports AI Agents operating Android phones like humans. Droidrun consists of an LLM agent running on a computer or cloud and the DroidRun Portal App installed on the phone, combining visual parsing, UI extraction, and LLM reasoning to achieve complex task automation. Droidrun supports users in completing tasks such as app launching and UI operations based on simple natural language commands. Droidrun supports multiple LLM providers such as OpenAI, Anthropic, and Gemini, and provides a Python API for custom automation. Droidrun provides new possibilities for building powerful AI assistants and achieving complex task automation.
Main features of Droidrun
- Visual + UI parsing: Combines visual models (recognizing screen content) and UI structure extraction (parsing the control tree) to achieve precise interaction.
- Intelligent self-healing mechanism: Intelligently detects errors (such as network interruptions and interface changes), automatically adjusts the operation path, and ensures the robustness of the automation process.
- Natural language control: Users directly control Android devices based on natural language instructions, such as opening apps and sending messages.
- Multi-LLM support: Supports multiple large language model (LLM) providers, such as OpenAI, Anthropic, Gemini, etc., and users can choose different models as needed.
- Minimalist command-line operation: Provides an easy-to-use command-line interface, making it convenient for users to quickly deploy and execute automation tasks.
- Rich extensibility: Provides an extensible Python API, allowing users to customize automation scripts to meet personalized needs.
- Automated testing support: Executes fixed UI processes and verifies success, suitable for test engineers' automated testing scenarios.
Technical principles of Droidrun
- Natural language processing (NLP): Parses users' natural language instructions based on large language models (LLMs). The model understands the user's intent and generates the corresponding operation sequence.
- Visual parsing: Obtains images of the phone screen based on screenshot functionality, and uses visual models (such as OCR technology) to parse text and controls on the screen. This allows the LLM to "see" the phone interface, more accurately understand the current context, and generate correct operation instructions.
- UI structure extraction: Extracts UI structure information from the screen based on Android's Accessibility Service API. By analyzing AccessibilityNode information, DroidRun can identify controls such as buttons and input boxes on the screen, achieving precise operations.
- Operation instruction generation: The LLM generates a specific operation instruction sequence based on the parsed natural language instructions and screen context information
- DroidRun Portal App: The DroidRun Portal App installed on the Android phone is responsible for receiving instructions from the computer or cloud, and converting the instructions into actual phone operations based on ADB or Wi-Fi.
Droidrun official website address
- Official website address：droidrun.ai
- GitHub repository：https://github.com/droidrun/droidrun
Application scenarios of Droidrun
- AI phone assistant: Enables AI to operate phones like a human and complete various operations.
- Cross-app data interaction: Achieves data transfer and task flow between different apps.
- Automated task execution: Automatically completes complex tasks such as booking a ride and recording videos.
- Data Extraction and Entry: Efficiently extract and enter data in the application.
- Multi-Device Collaborative Control: Supports batch operations and task assignment across multiple devices