What is AndroidLab
AndroidLab is a framework for training and systematically evaluating Android autonomous agents, integrating text and image modality operation environments, a unified action space, and reproducible benchmark testing. AndroidLab supports large language models and multimodal models, including 138 tasks covering nine applications. Based on AndroidLab, an Android instruction dataset was developed to improve the success rate of open-source models. The framework narrows the performance gap between open-source and closed-source models, promotes the development of open-source solutions, and is open-sourced on GitHub.
Main Features of AndroidLab
- Multimodal Operation Environment: Provides a standardized multimodal operation environment that supports large language models (LLMs) and multimodal models (LMMs) to interact within the same operation space.
- Benchmark Testing: Designs a benchmark containing 138 tasks, covering nine common applications. Task types include operation tasks and query tasks, involving complex real-world interaction scenarios.
- Evaluation Metrics: Introduces evaluation metrics such as task success rate (SR), sub-goal success rate (Sub-SR), reverse redundancy rate (RRR), and reasonable operation ratio (ROR) to precisely evaluate the agent's task performance.
- Dataset Construction: Creates the Android Instruct dataset, combining automated exploration and manual annotation to generate high-quality operation data, improving the performance of open-source models in task completion rate and operation efficiency.
- Model Training and Optimization: Supports training and optimization of open-source and closed-source models, significantly narrowing the performance gap between open-source and closed-source models based on instruction tuning.
Technical Principles of AndroidLab
- Operation Modes：
- XML Mode: Designed for LLMs with text input, transmitting interface states based on compressed XML information. The model directly selects elements to perform operations.
- SoM Mode: Designed for LMMs, using the Set-of-Mark method to combine screenshots with mark information. The model selects numbered marked elements for operations.
- Reasoning and Operation Framework：
- ReAct Framework: Based on step-by-step reasoning and operation output, the model outputs operations while displaying the intermediate reasoning process.
- SeeAct Framework: Separates reasoning and operation, conducted in two rounds of interaction. In the first round, the model generates detailed reasoning steps, and in the second round, it executes specific operations.
- Task Design and Reproducibility: Each task is designed with multiple sub-goals, and XML tree structure matching is used to verify the completion of each sub-goal, ensuring result accuracy and operation efficiency.
- Evaluation System: A task completion-based evaluation system that judges directly from device and screen states, providing comprehensive and precise agent performance evaluation.
- Dataset and Model Training: Builds datasets based on automated exploration and manual annotation, trains and fine-tunes models, improving model performance in multimodal and text input tasks.
Project Address of AndroidLab
- GitHub Repository：https://github.com/THUDM/Android-Lab
- arXiv Technical Paper：https://arxiv.org/pdf/2410.24024
Application Scenarios of AndroidLab
- Automated Testing: Used for automated testing of mobile applications, simulating user operations to detect application functionality and performance.
- Intelligent Assistant Development: Develop intelligent assistants that understand and execute complex user instructions, such as voice assistants and chatbots.
- Human-Computer Interaction Research: Research and improve human-computer interaction methods, especially on mobile devices, and how to enable machines to better understand and respond to human instructions.
- Artificial Intelligence Research: Provide an experimental platform to research and develop the performance of large language models (LLMs) and large multimodal models (LMMs) in practical applications.
- Education and Training: As an educational tool, help students and researchers understand the working principles of the Android operating system and application development.