What is Doubao 1.5·UI-TARS
Doubao 1.5·UI-TARS is a GUI-oriented Agent model launched by ByteDance Doubao. Based on human-like capabilities such as perception, reasoning, and action execution, the model interacts continuously and smoothly with graphical interfaces. It integrates visual understanding, logical reasoning, interface element localization, and operation into a single model, requiring no predefined workflows or manual rules and enabling end-to-end task automation. Doubao 1.5·UI-TARS is now available on the Volcano Ark platform.
Main features of Doubao 1.5·UI-TARS
- Graphical interface interaction capability: Based on perception, reasoning, and action execution, it interacts continuously and smoothly with graphical user interfaces to complete complex tasks.
- Visual understanding and localization: Understands visual information on the screen, supports box localization and point localization for multiple targets and small targets, and performs localization counting, describing localized content, and more.
- Logical reasoning and decision-making: Combines visual information and task instructions to perform logical reasoning and generate reasonable operation steps.
- High execution efficiency: Based on the Ark Doubao large model inference service, with the highest throughput in the industry, an initial 5 million TPM, and extremely low inference latency, with TPOT 30 ms.
- Native GUI Agent: Requires no predefined processes or manual rules, enabling end-to-end automated GUI interaction task execution.
Technical principles of Doubao 1.5·UI-TARS
- Vision large model (VLM): The model is based on a powerful vision large model to understand and process visual information in graphical interfaces, including images, text, icons, and more.
- Multimodal fusion: Integrates visual perception, logical reasoning, and action execution capabilities into a single model, enabling fusion processing of multimodal information.
- End-to-end learning: Based on large amounts of annotated data and reinforcement learning, the model learns an end-to-end mapping from task input to operation output, without the need for manually defined rules.
Doubao 1.5·UI-TARS project official website
- Project official website：https://www.volcengine.com/docs
Application scenarios of Doubao 1.5·UI-TARS
- Automated office work: Automatically handles tasks such as documents, spreadsheets, and emails to improve efficiency.
- Software testing: Simulates user operations, detects software issues, and improves quality.
- Intelligent customer service: Answers user questions in real time and provides operation guidance.
- Robot interaction: Guides robots to complete complex operations, applied in industry and logistics.