What is AndroidGen
AndroidGen is a framework launched by the Zhipu technical team to enhance the capabilities of Agents based on large language models (LLMs), especially in situations where data is scarce. The framework collects human task trajectories and trains language models based on these trajectories, developing Agents that do not require manually annotated trajectories and significantly improving the ability of LLMs to perform complex tasks.
Main features of AndroidGen
- Data collection and training without manual annotation: AndroidGen can, without manually annotated trajectories, collect human task trajectories and train language models based on these trajectories to develop efficient Agents.
- Enhance the task execution capability of Agents: Through four core modules (ExpSearch, ReflectPlan, AutoCheck, and StepCritic), AndroidGen significantly improves the ability of LLMs to perform complex tasks.
- xpSearch (Experience Search): By retrieving completed similar trajectories, LLMs perform in-context learning, thereby enhancing the Agent's capabilities and helping it generalize from simple tasks to complex tasks.
- ReflectPlan (Reflective Planning): Conducts self-reflection on the current environment and updates the plan state, enhancing the Agent's long-term reasoning ability.
- AutoCheck (Automatic Check): Proactively verifies the effectiveness of each Agent operation, reducing the risk of task failure caused by operational errors.
- StepCritic (Step Evaluation): Decomposes tasks into multiple sub-goals and provides step-by-step trajectory evaluation, offering fine-grained labels for model optimization.
- Efficient data collection pipeline: AndroidGen builds an efficient data collection pipeline that can generate a large number of high-quality Android browsing trajectories.
Technical principles of AndroidGen
- Model training: Uses LoRA technology to fine-tune GLM-4-9B and Llama-3-70B on automatically constructed datasets to obtain Android Agent models. Without manual trajectory annotation, it fully utilizes the information in the dataset by training each step in the trajectory as an independent sample.
- Mixed planning and execution steps: Mixes planning and execution steps for fine-tuning, enabling LLMs to possess both planning and execution capabilities.
- Data collection process：
- Task formulation: Based on GPT-4o, generates about 300 task instructions according to the instructions in AndroidWorld.
- Agent sampling: Samples the trajectory of each task based on AndroidWorld and GPT-4o.
- Trajectory recording: Records the environment and operation information of each step, constructing reproducible Android navigation trajectories.
- Trajectory evaluation: Uses StepCritic to evaluate the recorded trajectories, ensuring that each sub-goal has been completed.
- Trajectory enhancement: Expands the high-quality dataset, ultimately constructing a dataset containing more than 1,000 trajectories.
Performance results of AndroidGen
- AndroidWorld benchmark test：
- AndroidGen significantly improves the capabilities of Agents based on the same foundation model, with more significant performance improvements compared to M3A and SeeAct.
- The average score of AndroidGen + GPT-4o reached 46.8, far exceeding other combinations.
- The average score of GLM-4-9B + AndroidGen, which has smaller model parameters and is open source, exceeds that of GPT-4o + M3A, which has larger model parameters and is closed source.
- AitW（Android in the Wild） In evaluations with eight globally popular mobile applications (such as Google Maps, YouTube, etc.), AndroidGen also performs excellently, demonstrating its ability to understand and interact with natural language instructions in real device environments.
Application scenarios of AndroidGen
- Automated task processing: Through natural language instructions, the Agent can automatically complete tasks such as sending emails, setting reminders, and querying information.
- Cross-application operations: The Agent can interact between different applications and complete operations such as copying data from one application to another.
- Intelligent navigation: On an Android device, the Agent can navigate according to user instructions, for example, opening a specific application or finding files.
- Intelligent interaction: Through natural language understanding, the Agent can interact with users and provide a more intelligent user experience.