EvoCUA - Meituan's Open-Source General Multimodal Computer Operation Model

What is EvoCUA

EvoCUA (Evolving Computer Use Agent) is an open-source multimodal model from Meituan, focused on computer use automation tasks. EvoCUA enables end-to-end multi-turn operations of software such as Chrome, Excel, and PowerPoint through natural language instructions and screen screenshots, significantly improving computer use capabilities. On the OSWorld benchmark, EvoCUA ranks first among open-source models with a 56.7% task completion rate, surpassing multiple well-known models. EvoCUA's innovative data synthesis and training methods can significantly improve the computer use capabilities of open-source vision-language models without reducing general performance.

EvoCUA's Main Features

• Multi-turn Interaction: Supports multi-turn interaction with the desktop environment, gradually completing operations according to task requirements.
• Task Automation: Supports completing complex computer tasks according to natural language instructions, such as generating reports in Excel, searching for information in a browser, etc.
• Multimodal Input: Combines visual (screen screenshots) and linguistic (natural language instructions) input to understand task requirements more accurately.
• High-efficiency Execution: Through optimized algorithms and model structure, completes tasks in fewer steps, improving efficiency.
• Open Source and Extensible: As an open-source model, EvoCUA supports users in customizing and extending it according to their needs.

EvoCUA's Technical Principles

• Data Synthesis and Training Methods
  EvoCUA adopts innovative data synthesis methods to generate large amounts of high-quality training data, simulating real-world computer operation tasks. Its training method can significantly improve computer use capabilities without reducing the model's general performance. By optimizing the training strategy, the model performs excellently in multimodal tasks.
• Reinforcement Learning and Optimization: EvoCUA utilizes reinforcement learning techniques during training, optimizing the model's behavioral strategy through a reward mechanism. The model continuously learns through trial and error in multi-turn interactions, gradually finding the optimal operation path and achieving efficient task completion in complex tasks.
• Model Architecture: EvoCUA is built on large language models (LLM) and vision-language models (VLM), possessing strong language understanding and generation capabilities while also being able to process visual information. Through specific prompts and parsing mechanisms, the model converts natural language instructions into specific operation instructions, enabling automated operation of the desktop environment.