What is AGUVIS
AGUVIS is a unified purely visual framework jointly launched by the University of Hong Kong and Salesforce, designed specifically for autonomous GUI agents, capable of operating on various platforms (such as web, desktop, and mobile devices). AGUVIS is based on the association of image observations and natural language instructions with visual elements, and adopts a consistent action space to achieve cross-platform generalization. AGUVIS combines explicit planning and reasoning to enhance the agent's autonomous navigation and interaction in complex digital environments. Through a large-scale dataset and a two-stage training process, the framework achieves performance surpassing existing methods in both offline and online scenarios, becoming the first purely visual GUI agent that can independently complete tasks without relying on external closed-source models.
Main features of AGUVIS
- Cross-platform autonomous GUI interaction: Autonomously executes GUI tasks on different platforms (such as websites, desktops, and mobile devices).
- Association of image observations with natural language instructions: Maps natural language instructions to visual interface elements to enable image-based interaction.
- Explicit planning and reasoning: Integrates planning and reasoning capabilities, enabling the agent to analyze the environment and generate effective operation steps.
- Large-scale dataset construction: Creates a large-scale GUI agent trajectory dataset containing multimodal reasoning and grounding.
Technical principles of AGUVIS
- Purely visual framework: Adopts a purely visual approach, unifying interface observations as images and grounding instructions to image coordinates, improving generalization ability across environments.
- Unified action space: Uses a standardized action space and plugin system to perform consistent learning and interaction across different platforms.
- Vision-language model (VLM): VLM serves as the foundation, such as Qwen2-VL, processing high-resolution images of arbitrary resolution and dynamically converting them into visual tokens.
- Two-stage training paradigm：
- First stage: foundational training: Focuses on enabling the model to understand and interact with objects in a single GUI screenshot.
- Second stage: planning and reasoning training: Builds on foundational training, introducing more complex decision-making and reasoning processes, and training the model based on diverse agent trajectory data.
- Inner Monologue: Generates detailed inner monologue during training, including observation descriptions, thoughts, and low-level action instructions, simulating the agent's thinking process and improving its planning ability.
- Plugin system: Provides flexibility for actions that cannot be directly mapped to the existing action space, enabling the model to adapt to new environments and tasks.
AGUVIS project address
- Project official website：aguvis-project
- GitHub repository：https://github.com/xlang-ai/aguvis
- arXiv technical paper：https://arxiv.org/pdf/2412.04454
Application scenarios of AGUVIS
- Automated testing: Automates testing of various graphical user interfaces in software development, including websites, desktop applications, and mobile applications, ensuring software stability and reliability.
- Virtual assistant: Acts as a virtual assistant, helping users automate daily computer tasks such as schedule management, email processing, and data entry.
- Business process automation: Automates the execution of specific business processes in enterprises, such as financial report generation and customer data management, improving work efficiency.
- Education and Training: Create interactive educational software that simulates real-world GUI operations for teaching and training purposes.
- Customer Service Automation: Automatically handle customer requests in the customer service field, providing rapid responses and solutions based on the GUI interface.