What is PC Agent
PC Agent is an advanced AI system jointly launched by Shanghai Jiao Tong University and the Generative AI Research Lab (GAIR). The system is based on simulating human cognitive processes to perform complex digital work such as organizing research materials, drafting reports, and creating presentations. PC Agent integrates PC Tracker for efficiently collecting human-computer interaction data, and uses a two-stage cognitive completion process to transform raw data into cognitive trajectories. PC Agent adopts a multi-agent system architecture, combining a planning agent and a grounding agent to achieve precise visual grounding and decision-making. Trained on a small amount of high-quality cognitive data, the system can handle complex workflows of up to 50 steps, demonstrating outstanding data efficiency and practical application potential.
PC Agent's Main Features
- Task Automation: Automates complex digital tasks such as organizing research materials, drafting reports, and creating presentations.
- Human-Computer Interaction Data Collection: Based on PC Tracker, collects detailed trajectories and cognitive context of user-computer interactions.
- Cognitive Trajectory Transformation: Uses a two-stage cognitive completion process to convert raw interaction data into trajectories rich in cognitive information.
- Complex Work Processing: Handles complex workflows involving multiple applications, such as switching between PowerPoint and a browser to collect materials.
- Multi-Agent Collaboration: Combines a planning agent and a grounding agent to achieve decision-making and precise visual grounding.
- Training with Small Amounts of Data: Executes complex workflows when trained on only a small amount of cognitive trajectory data.
PC Agent's Technical Principles
- PC Tracker：
- Data Collection: Runs in the background, records the user's keyboard and mouse activity, captures screen screenshots, and collects human-computer interaction data.
- Event Tracking: An event-based tracking strategy records key user operation events rather than continuous video streams, reducing storage requirements.
- Unified Action Space: Encapsulates keyboard and mouse operations into a unified action space, simplifying AI's understanding of human behavior.
- Cognitive Completion Process：
- Data Refinement: Optimizes the quality of raw interaction data based on trajectory filtering, action filtering, and standardization.
- Action Semantic Completion: Supplements semantic information for click-related actions and generates high-quality descriptions of click targets.
- Thought Process Reconstruction: Based on action semantic information, reconstructs the implicit reasoning process behind each action.
- Multi-Agent System：
- Planning Agent: Responsible for action decision-making and acquires effective planning capabilities based on learning human cognitive trajectories.
- Grounding Agent: Responsible for executing click-related actions and has a self-verification mechanism to achieve near-human precision.
- Error Correction Mechanism: When the grounding agent finds that the target the planning agent attempts to click does not exist on the screen, the planning agent is prompted to reformulate the action plan.
PC Agent's Project Address
- Project Website：gair-nlp.github.io/PC-Agent
- GitHub Repository：https://github.com/GAIR-NLP/PC-Agen
- arXiv Technical Paper：https://arxiv.org/pdf/2412.17589
PC Agent's Application Scenarios
- Office Automation: Create and edit documents, spreadsheets, and presentations, automate daily office tasks, and improve work efficiency.
- Research and Academia: Organize and analyze research data, automatically generate literature reviews, and assist in writing academic papers.
- Content Creation: Automate layout and formatting, improving the efficiency and quality of content creation.
- Project Management: Automatically update project reports, coordinate team tasks, and optimize project management processes.
- Customer Service: Manage customer data and provide fast, personalized customer support.