What is UFO²
UFO² is a multi-Agent operating system (AgentOS) for the Windows desktop launched by Microsoft, enabling automation of complex desktop tasks through deep system integration and natural language interaction. UFO² is based on a central HostAgent that decomposes tasks and coordinates multiple application-specific AppAgents to execute operations, combining GUI interaction with native API calls to improve the efficiency and robustness of task execution. UFO² introduces hybrid control detection, continuous knowledge integration, and a non-intrusive user experience design, and supports running in an isolated virtual desktop to avoid disturbing user operations. UFO² performs excellently across multiple real Windows applications, significantly improving the success rate and execution efficiency of automated tasks.
Main features of UFO²
- Deep operating system integration: Supports deep integration into the Windows system, enabling fine-grained control over desktop applications.
- Non-intrusive user experience: UFO² supports running in an isolated virtual desktop, allowing users and agents to operate simultaneously without interfering with each other.
- Multi-turn interaction support: Supports multi-turn task execution, letting users progressively refine instructions or intervene in the agent's operations during a session.
- Security assurance mechanism: Detects potentially dangerous operations and prompts the user for confirmation before execution, ensuring user data and system security.
Technical principles of UFO²
- Multi-agent architecture：
- HostAgent: Acts as the central control plane, responsible for parsing user instructions, decomposing tasks, dispatching AppAgents, and coordinating execution across applications.
- AppAgent: Application-specific execution modules equipped with application-specific APIs, knowledge bases, and hybrid GUI/API action interfaces to execute tasks efficiently.
- Hybrid control detection: Combines the structured data provided by Windows UI Automation (UIA) APIs with vision-based detection models to achieve reliable recognition of standard and custom UI elements.
- Unified GUI/API action layer: Based on the Puppeteer module, UFO² dynamically chooses GUI operations or native application API calls, optimizing task execution paths and reducing the fragility of GUI operations.
- Continuous knowledge integration: Based on retrieval-augmented memory (RAG) technology, it integrates external documents and historical execution records into the agent's knowledge base, supporting the agent to dynamically learn and improve at runtime.
- Speculative multi-action execution: Predicts multiple actions based on a single inference, verifies the feasibility of these actions at runtime, reduces the frequency of inference calls, and improves execution efficiency.
Project addresses for UFO²
- Project website：https://microsoft.github.io/UFO/
- GitHub repository：https://github.com/microsoft/UFO
- arXiv technical paper：https://arxiv.org/pdf/2504.14603
Application scenarios of UFO²
- Office automation: Automatically process Excel data, edit Word documents, create PowerPoint presentations, etc.
- Cross-application workflows: Coordinate multiple applications to complete complex tasks, such as importing data from Excel into Outlook.
- Enterprise task automation: Reduce manual intervention and efficiently complete repetitive work such as data entry and file processing.
- Intelligent customer service: Quickly respond to user requests and solve problems through natural language interaction.
- Education and training: Assist teaching, automatically demonstrate operations, or generate learning reports.