What is ShowUI
ShowUI is a vision-language-action model jointly launched by the Show Lab of the National University of Singapore and Microsoft, capable of improving the work efficiency of graphical user interface (GUI) assistants. The model is based on UI-guided visual token selection to reduce computational cost, uses interleaved vision-language-action streams to unify diverse requirements in GUI tasks, and manages vision-action history to enhance training efficiency. ShowUI uses a small-scale but high-quality instruction-following dataset, achieving 75.1% zero-shot screenshot grounding accuracy with 256K data, and improving training speed by 1.4 times, demonstrating potential in the field of GUI visual agents.
ShowUI's main features
- UI-guided visual token selection: Constructs screenshots as UI connection graphs, adaptively identifies redundant relationships, and uses them as criteria for selecting tokens in the self-attention module, reducing computational cost.
- Interleaved vision-language-action streams: Flexibly unifies diverse requirements in GUI tasks, effectively manages vision-action history, and improves training efficiency.
- Small-scale high-quality GUI instruction-following dataset: Based on carefully curated data and adopting a resampling strategy to solve the problem of data type imbalance, improving model accuracy and efficiency.
- Zero-shot screenshot grounding: The ability to directly understand and operate on screenshots without additional training.
- GUI automation: Automates the execution of GUI tasks, such as clicking, typing, etc., improving human-computer interaction efficiency.
ShowUI's technical principles
- UI-guided visual token selection：
- Divides screenshots into regular patches, with each patch as a node.
- Identifies adjacent patches with the same RGB values, constructs a UI connection graph, and groups visually redundant regions together.
- In the self-attention module, selectively processes visual tokens based on the UI connection graph, reducing computational load.
- Interleaved vision-language-action streams：
- Structures GUI actions, represented in JSON format, unifying actions across different devices.
- Based on alternating processing of vision, language, and action data, manages complex interaction history.
- During training, uses multi-turn dialogue to improve data utilization efficiency.
- Data curation and resampling strategy：
- Carefully curates and selects high-quality training data, rather than simply aggregating all available data sources.
- Based on a resampling strategy, solves the data imbalance problem between different devices and task types.
- Efficiently processing high-resolution UI screenshots: For high-resolution UI screenshots, optimizes the model to effectively process long token sequences and reduce computational cost.
- Model architecture：
- Based on the Qwen2-VL-2B model, integrates a visual encoder and a language model to process visual and textual data.
- Based on a specific data recipe and training strategy, improves the model's performance in GUI tasks.
ShowUI's project address
- GitHub repository：https://github.com/showlab/ShowUI
- HuggingFace model library：https://huggingface.co/datasets/showlab/ShowUI-desktop-8K
- arXiv technical paper：https://arxiv.org/pdf/2411.17465
- Online experience Demo：https://huggingface.co/spaces/showlab/ShowUI
ShowUI's application scenarios
- Web automation: Automatically executes operations on web pages such as clicking, typing, scrolling, etc., used in automated testing, data scraping, or simulating user behavior.
- Mobile Application Testing: Automatically perform various user interactions in mobile applications, such as swiping, tapping, filling out forms, etc., to conduct application functional testing.
- Desktop Software Automation: Automate repetitive tasks in desktop software, such as file management, data entry, settings adjustments, etc.
- Virtual Assistant: As part of a virtual assistant, execute specific GUI operations according to the user's natural language instructions.
- Game Automation: In games that support automation scripts, automatically perform operations such as character movement, item pickup, combat, etc.