What is PC Agent-E
PC Agent-E is an efficient agent training framework jointly launched by Shanghai Jiao Tong University and SII. The framework uses 312 human-annotated computer usage trajectories and synthesizes diverse action decisions based on the Claude 3.7 Sonnet model, significantly improving data quality. The framework includes four key parts: trajectory collection, thought completion, trajectory enhancement, and agent training. PC Agent-E achieved a 241% performance improvement on the WindowsAgentArena-V2 benchmark, surpassingClaude 3.7 Sonnet's extended thinking mode, becoming the new-generation SOTA for open-source computer agents on the Windows system.
Main Features of PC Agent-E
- Efficient Training: Requires only 312 human-annotated trajectories, and significantly improves model performance based on data augmentation.
- Cross-platform Generalization: In the OSWorld benchmark, it demonstrates strong cross-platform capabilities and is applicable to different operating systems.
- Task Execution: Supports completing a variety of complex tasks, such as file operations, software usage, web browsing, etc.
- Data Augmentation: Based on synthesizing diverse action decisions, it enriches trajectory data and improves model generalization ability.
Technical Principles of PC Agent-E
- Trajectory Collection: Based on the PC Tracker tool, it records human operation trajectories, including task descriptions, screenshots, and keyboard/mouse operations. With a simple annotation process, it collects a small number of high-quality human operation trajectories.
- Thought Completion: Based on the Claude 3.7 Sonnet model, it adds the thinking logic behind each action step. Based on the provided task description, historical actions, and current state, it generates a reasoning process consistent with human thinking.
- Trajectory Boost: Synthesizes more action decisions for each step of the trajectory to capture the diversity of tasks. Using the Claude 3.7 Sonnet model, it generates multiple reasonable action decisions for each step, enriching the trajectory data.
- Agent Training: Training is conducted on the basis of the open-source model Qwen2.5-VL-72B. With a simple end-to-end training framework, it ensures that the model can efficiently learn and execute tasks.
- Evaluation and Validation: Model performance is validated on the WindowsAgentArena-V2 and OSWorld benchmarks. By adjusting the number of synthesized actions, it verifies the key role of the trajectory enhancement method in performance improvement.
Project Address of PC Agent-E
- Project Official Website：https://gair-nlp.github.io/PC-Agent-E/
- GitHub Repository：https://github.com/GAIR-NLP/PC-Agent-E
- HuggingFace Model Library：https://huggingface.co/henryhe0123/PC-Agent-E
- arXiv Technical Paper：https://arxiv.org/pdf/2505.13909
Application Scenarios of PC Agent-E
- Automated Office Work: Automatically completes tasks such as document editing and data analysis, improving office efficiency.
- Software Testing: Simulates user operations, discovers errors and problems in software, and improves software quality.
- Educational Assistance: As a virtual teaching assistant, it helps students complete computer operation tasks and provides instant guidance.
- Assisting People with Disabilities: Provides auxiliary operation functions to facilitate computer use for people with disabilities.
- Cross-platform Compatibility: Migrates and executes tasks between different operating systems, achieving seamless switching.