What is Aria-UI
Aria-UI is a large multimodal model jointly launched by the University of Hong Kong and Rhymes AI, designed for graphical user interface (GUI) grounding tasks. Based on a purely visual approach, it does not rely on auxiliary inputs such as HTML or AXTree. Using a large-scale, diverse data synthesis pipeline, it generates high-quality element descriptions and instruction samples from Common Crawl and public data, adapting to diverse instructions in different environments. Aria-UI innovatively integrates dynamic action history information, using text or text-image interleaved formats to enhance the model's grounding capability in multi-step task scenarios. In extensive benchmark tests including offline and online agent tasks, Aria-UI achieves excellent performance, demonstrating strong zero-shot generalization ability and cross-platform applicability, making it a powerful solution for general GUI grounding tasks.
Main Features of Aria-UI
- GUI Element Grounding: Accurately maps language instructions to target elements in the GUI, achieving precise element grounding and providing a foundation for subsequent interaction operations.
- Multimodal Input Processing: Processes input information from multiple modalities including GUI images, text instructions, action history, etc., fully understanding and utilizing the rich information in multimodal data.
- Diverse Instruction Adaptation: Based on diverse instruction samples generated by a large-scale, diverse data synthesis pipeline, it effectively adapts to various instruction expressions in different environments.
- Dynamic Context Understanding: Integrates dynamic action history information, using text or text-image interleaved formats to understand the current dynamic context in multi-step task scenarios, providing important reference for accurate element grounding.
- High-Resolution Image Processing: Supports image resolutions up to 3920×2940, splitting images into small blocks for processing, significantly expanding the range of processable image sizes.
Technical Principles of Aria-UI
- Purely Visual Approach: Adopts a purely visual approach, directly extracting visual features from GUI images, using visual information to understand and ground target elements.
- Multimodal MoE Model: Built on the Aria multimodal MoE (Mixture of Experts) model, with 3.9B activated parameters, skilled at processing multimodal data.
- Data Synthesis and Training: Based on a two-stage data synthesis pipeline, it generates high-quality element descriptions and diverse instruction samples from Common Crawl and public data, covering the three major GUI environments of Web, desktop, and mobile, providing large-scale, diverse training data for model training and enhancing the model's ability to recognize different instructions and elements.
- Context-Aware Data Expansion: Uses publicly available agent trajectory data to simulate grounding tasks with context, constructing two context settings: text action history and text-image interleaved history. Based on the data synthesis pipeline, it generates detailed step-by-step instructions for all grounding steps in the trajectory data, enabling the model to better understand and execute tasks in dynamic environments.
- Super-Resolution Support: Splits images into small blocks and processes them, supporting image resolutions up to 3920×2940, maintaining image details and accuracy.
Project Address of Aria-UI
- Project Website：https://ariaui.github.io/
- GitHub Repository：https://github.com/AriaUI/Aria-UI
- HuggingFace Model Library：https://huggingface.co/Aria-UI
- arXiv Technical Paper：https://arxiv.org/pdf/2412.16256
Application Scenarios of Aria-UI
- Automated Testing: Web application testing automatically clicks webpage buttons and enters information to verify whether functions work properly.
- User interaction assistance: Voice commands control home devices, such as “turn on the light” automatically clicking the switch button.
- Intelligent customer service: E-commerce platform customer service quickly locates product details and answers users' inquiry questions.
- Education industry: Automatically operates a code editor to demonstrate the programming process and results.
- Office automation: Automatically operates financial software, generates reports, and improves work efficiency.