Qwen-UI-Agent - Alibaba Tongyi's GUI agent foundation model

What is Qwen-UI-Agent

Qwen-UI-Agent is a GUI agent foundation model launched by the Alibaba Tongyi team, capable of operating interfaces on phones, computers, and web pages like a real person. The model is trained on 100+ real devices and 150+ apps, supports mixed GUI and command-line execution, and can complete long-horizon cross-app tasks. It has built-in safety mechanisms: dangerous requests are directly refused, and sensitive operations such as payments and deletions are proactively handed back to the user for confirmation. On benchmarks such as MobileWorld and WebArena, it comprehensively surpasses GPT-5.6, Claude Opus, and other flagship models, making it one of the GUI Agents with the strongest real-world usability today.

Qwen-UI-Agent's main features

• Cross-device GUI operation: automatically clicks, inputs, and swipes on phones, computers, and browsers to complete interface interactions.
• Real-device multi-app workflows: executes complex long-chain tasks across 150+ real apps, such as checking a map → finding a restaurant → booking a table → sending a calendar entry.
• Mixed GUI + CLI execution: can both click through interfaces and run command lines, supporting multiple batched actions in one output to speed things up.
• Safety boundary control: illegal requests are directly refused, and sensitive operations such as payments, file deletion, and privacy authorization proactively trigger a pop-up handed back to the user for confirmation.
• Long-horizon task handling: stably executes ultra-long tasks of 100+ steps, such as research, data analysis, and report generation.
• Precise interface localization: refreshes SOTA on grounding benchmarks such as ScreenSpot-Pro, accurately identifying the positions of on-screen elements.

Qwen-UI-Agent's technical principles

• Real-device training environment: builds a real-device cluster of 100+ real phones and 150+ apps, directly collecting real trajectories for training and evaluation, bridging the performance gap between simulated environments and real devices.
• Hybrid action space: the model simultaneously masters GUI operations (clicking, swiping, input) and CLI commands, and introduces a Batched Actions mechanism that supports outputting multiple actions in parallel from a single decision, making CLI account for nearly half of computer tasks and significantly shortening the overall number of execution steps.
• Long-horizon online reinforcement learning: supports end-to-end online RL training on ultra-long trajectories of more than 100 steps, relying on about 10,000 concurrent environments for parallel rollout, combined with adaptive curriculum learning, to continuously tackle highly difficult long-horizon tasks.
• Safety alignment mechanism: embeds safety judgment throughout the entire task execution chain, identifying illegal or high-risk requests through upfront classification and directly terminating them; for sensitive scenarios such as payments, data deletion, and privacy authorization, it triggers at key nodes ask_user takeover, ensuring dangerous requests are not executed and sensitive operations are not decided arbitrarily.
• Multimodal Grounding and retention of general capabilities: while training specifically for GUI, it retains the foundation model's general reasoning and agentic capabilities, enabling it to both precisely localize on-screen elements (refreshing multiple grounding SOTA) and handle long-tail knowledge and tool-calling needs in the real world.

Follow WeChat and reply "open source" to join theAI open-source project exchange group

How to use Qwen-UI-Agent

• Read the technical report: visit the arXiv paper to learn about the model architecture, training methods, and detailed evaluation results on 8 benchmarks.
• Visit the project homepage: Open tongyi-mai.github.io/Qwen-UI-Agent to watch real-device demos and interaction cases across 5 real-world scenarios.
• Follow the open-source repository: Follow GitHub Tongyi-MAI/MAI-UI to get updates on potentially forthcoming model weights, code, and deployment documentation.
• Run benchmark evaluations: Use the Docker environment of the companion repository Tongyi-MAI/MobileWorld to conduct reproducible mobile GUI capability tests on the model.

Core advantages of Qwen-UI-Agent

• Real-device closed loop: Trained and evaluated on 100+ real phones and 150+ apps, with the self-built MobileWorld-Real real-device benchmark, directly bridging the "simulation-to-real" gap.
• Comprehensive SOTA: Comprehensively surpasses or matches flagship models such as GPT-5.6, Claude Opus 4.8, and Gemini 3.1 Pro on 8 core benchmarks including MobileWorld, OSWorld, WebArena, and ScreenSpot-Pro.
• Hybrid execution acceleration: Unifies GUI clicks and CLI commands, supports outputting multiple actions in batch from a single decision, with about 40% batched actions in computer tasks, saving nearly 58% of execution steps overall.
• Long-horizon online reinforcement learning: Supports end-to-end online RL for ultra-long trajectories of 100+ steps, relying on about 10,000 concurrent environments for parallel rollout, continuously tackling complex long-horizon tasks.
• Built-in safety boundaries: Illegal/dangerous requests are directly refused and terminated; sensitive operations such as payments, data deletion, and privacy authorization actively stop at key points, returning a pop-up to the user for confirmation before continuing.

Qwen-UI-Agent project address

• Project official website：https://tongyi-mai.github.io/Qwen-UI-Agent/
• GitHub repository：https://github.com/Tongyi-MAI/MAI-UI
• Technical paper：https://tongyi-mai.github.io/Qwen-UI-Agent/Qwen-UI-Agent-Technical-Report.pdf

Comparison of Qwen-UI-Agent with similar competitors

Application scenarios of Qwen-UI-Agent

• Local life venue exploration: Automatically works across Amap, Dianping, and Xiaohongshu to complete address search, popularity comparison, and reputation summarization, handling date venue exploration in one stop.
• Trip and meeting coordination: Check 12306 high-speed rail schedules, calculate subway commute time, and automatically create a meeting schedule with reminders in DingTalk.
• In-depth research report: Collect financial report data across websites, run analysis scripts, and automatically generate Excel, PPT, and Word deliverables.
• Cross-device invoice organization: Extract invoice photos from a phone album, transfer them to a computer for categorized archiving, and generate a structured Excel reimbursement form.
• Proactive flight cancellation response: After detecting a flight cancellation notice, automatically search for alternative flights and high-speed rail options, recommend the best option that can make the meeting, and return the decision to the user.