What is AutoDroid-V2
AutoDroid-V2 is a mobile GUI automation script agent based on a small language model (SLM) launched by the Institute for AI Industry Research of Tsinghua University. It can improve the performance of GUI agents on devices based on code generation technology. AutoDroid-V2 analyzes the exploration history of applications and automatically generates detailed application documentation to guide the SLM in generating multi-step scripts to complete complex user tasks. Compared with traditional step-by-step GUI agents, AutoDroid-V2 significantly improves task completion rates and execution efficiency, reduces dependence on large cloud models, and lowers user privacy risks and centralized service costs. AutoDroid-V2 performs excellently in multiple benchmark tests, demonstrating the potential for fully deploying GUI agents on devices.
Main features of AutoDroid-V2
- Automated UI task execution: Based on generating and executing multi-step scripts, it automatically completes various UI tasks for users on mobile devices, such as opening applications, entering text, clicking buttons, etc.
- Code generation and execution: It transforms the UI task automation problem into a code generation problem, uses a small language model (SLM) on the device to generate executable code scripts, and uses a code interpreter to execute the scripts efficiently.
- Application documentation generation: It automatically analyzes the exploration history of applications and generates detailed application documentation, recording the application's GUI states, important elements, and their interaction relationships, providing guidance for script generation.
Technical principles of AutoDroid-V2
- arXiv technical paper：https://arxiv.org/pdf/2412.18116
Application scenarios of AutoDroid-V2
- Daily life assistance: When users are busy, it automatically replies based on message content, such as automatically replying to work emails, greetings from friends, etc., avoiding missing important information.
- Daily work: During meetings, it automatically records meeting content, extracts key information, and generates meeting minutes, making it convenient to review after the meeting and execute related tasks, improving meeting efficiency.
- Application testing: After an application update, it automatically executes regression test cases, checks whether the functions of the new version of the application are normal, promptly discovers and fixes potential bugs, and ensures the stability and reliability of the application.
- Smart home: According to the user's daily routine and current ambient light, it automatically adjusts the brightness and color temperature of lights at home, such as automatically dimming the lights at night to create a comfortable sleeping environment and improve quality of life.
- Special industry applications: After patients are discharged, it automatically sends follow-up reminders and health questionnaires, collects patients' recovery conditions, helps doctors promptly understand changes in patients' conditions, and provides better follow-up treatment and rehabilitation guidance.