SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

ComAct - Shanghai AI Lab and others introduce a research paradigm for software automation operations

Author: AI小集 · Original source: AI工具集

What is ComAct

ComAct (COM-as-Action) is a research paradigm that redefines professional software automation operations, proposed by Jiaxin Ai and others. The paradigm identifies Windows Component Object Model (COM) as a unified executable abstraction, transforming professional software interaction from fragile sequential visual control into deterministic program synthesis, solving the error accumulation problem of GUI-based agents in long-horizon tasks, as well as the dilemma of API-based methods being limited by heterogeneous protocols and closed-source interfaces.

ComAct

Main features of ComAct

  • COM-as-Action paradigm shift: Transforms professional software interaction from fragile sequential visual control (GUI) into deterministic program synthesis, using Windows COM interfaces as a unified executable abstraction to directly call the software's internal structured programming interfaces.
  • ComCADBench benchmark: Builds the first benchmark for evaluating agents operating real industrial CAD software, containing 1,000 tasks, covering the three major platforms SolidWorks, Inventor, and AutoCAD as well as seven engineering activities including 3D modeling, 2D sketching, assembly, and engineering drawings.
  • ComActor self-correcting agent: Adopts a three-stage progressive training framework—text-to-code supervised fine-tuning (SFT), agentic SFT based on multi-turn interaction trajectories, and GRPO reinforcement learning based on continuous geometric rewards—enabling the agent to generate COM scripts, read terminal feedback, diagnose and rewrite failed steps.
  • ComForge scalable training platform: Supports concurrent hosting of 1,000+ Dockerized Windows environments, pre-installed with real CAD software, providing code_execute(COM scripts),gui_action(mouse/keyboard) and reset three action types, enabling large-scale parallel training and evaluation.
  • Cross-application workflow integration: As a unified semantic programming interface, COM naturally supports collaborative operations and complex workflow orchestration across multiple professional software (Microsoft Office, Adobe suite, industrial CAD, etc.).
  • Robust execution of long-horizon tasks: By generating a single deterministic script to replace hundreds of error-prone sequential GUI operations, it eliminates long-horizon error accumulation and maintains over 80% end-to-end success rate in long-horizon multi-task pipelines, significantly outperforming traditional GUI-based agents.
Dig into GitHub

Follow WeChat and reply "open source" to join theAI open source project exchange group

Technical principles of ComAct

  • COM-as-Action paradigm shift: Identifies Windows Component Object Model (COM) as a unified executable abstraction, redefining professional software interaction from fragile sequential visual control (GUI) as deterministic program synthesis, directly calling the software's internal functions through structured programming interfaces.
  • Validated instruction-code pair data construction: Synthesizes a validated instruction-COM code corpus through a dedicated data construction pipeline, ensuring the correctness of training data at the syntactic and semantic levels, providing a high-quality foundation for supervised fine-tuning.
  • Three-stage progressive training framework：
    • Stage one (text-to-code SFT): Performs supervised fine-tuning on the validated corpus, enabling the model to learn to generate syntactically valid COM scripts;
    • Stage two (Agentic SFT): Conduct iterative debugging training based on multi-turn interaction trajectories and screen screenshots/terminal feedback, enabling the agent to self-recover from execution failures;
    • Stage Three (GRPO Reinforcement Learning): Adopts Group Relative Policy Optimization (Group Relative Policy Optimization), combined with a continuous geometric reward based on Chamfer Distance, to directly optimize task-level geometric fidelity, bridging the gap of "syntactically correct but geometrically inconsistent".
  • Self-Correcting Execution Loop: After the agent generates a COM script, it executes it, reads the traceback or execution result returned by the terminal, autonomously diagnoses the cause of the error, and rewrites the failed steps, forming a "generate-execute-feedback-correct" closed loop.
  • ComForge Scalable Infrastructure: Builds a training platform based on Dockerized Windows containers, concurrently hosting 1,000+ environments pre-installed with real CAD software, supporting code_execute(COM scripts),gui_action(mouse/keyboard), and reset three action types, enabling large-scale parallel training and evaluation.
  • Cross-Application Unified Action Space: Leveraging the characteristic of COM operating at the semantic object level, it unifies the heterogeneous interfaces of different professional software (SolidWorks, AutoCAD, Inventor, Microsoft Office, etc.) into a standardized executable action space, naturally supporting cross-application workflow orchestration.

How to Use ComAct

  • Environment Preparation (ComForge): Deploy a Dockerized Windows container environment pre-installed with the target professional software (such as SolidWorks, AutoCAD, Inventor); ComForge supports concurrently hosting 1,000+ parallel instances for training or evaluation.
  • Task Instruction Input: Input a natural language instruction to the ComActor agent (such as "create a cube with rounded corners and generate an engineering drawing"); the agent parses it into the corresponding COM operation objectives.
  • COM Script Generation: Based on its three-stage training capabilities, ComActor automatically generates syntactically valid COM scripts (such as Python calling COM interfaces), directly manipulating the software's internal object model rather than simulating mouse clicks.
  • Execution and Feedback Loop: The script is executed in the ComForge environment; the agent reads the execution result or error traceback (traceback) returned by the terminal, autonomously diagnoses the problem, and rewrites the failed steps, forming a self-correcting closed loop.
  • Result Verification and Output: After the task is completed, the agent outputs the final generated files (such as 3D models, engineering drawings), and verifies geometric fidelity through the continuous geometric reward mechanism (Chamfer Distance).
  • Cross-Application Workflow Orchestration: Leveraging the unified COM interface, it chains multiple professional software applications to complete complex pipeline tasks (such as modeling in CAD → exporting → generating a BOM table in Excel).

ComAct Official Website Address

  • Project Official Website：https://knowledgexlab.github.io/ComAct/
  • GitHub Repository：https://github.com/KnowledgeXLab/ComAct
  • arViv Technical Paper：https://arxiv.org/pdf/2606.13239

Core Advantages of ComAct

  • Deterministic Program Synthesis Replaces Fragile Visual Control: Transforms professional software interaction from sequential GUI operations relying on pixel localization into generating a single deterministic COM script, fundamentally eliminating long-horizon error accumulation and avoiding the cascading failures of traditional GUI-based agents where "one misstep leads to all subsequent steps going wrong".
  • Unified Executable Abstraction Spanning Heterogeneous Interfaces: Leveraging Windows COM as a unified semantic programming interface, it integrates heterogeneous protocols from various professional software such as SolidWorks, AutoCAD, Inventor, and Microsoft Office, natively supporting complex cross-application workflow orchestration.
  • Overcoming the scarcity of tools for closed-source software: The COM interface exposes an application's internal functionality at the semantic object level. Even for commercial software whose source code is not public, it can provide high-fidelity local API-level access, breaking through the bottleneck of API-based methods being limited by inaccessible commercial interfaces.
  • Significantly leading in long-horizon task robustness: In long-horizon multi-task pipelines, it maintains an end-to-end success rate of over 80%, while traditional GUI-based baseline models achieve a task success rate close to 0%on ComCADBench, demonstrating extremely strong long-horizon planning and execution resilience.
  • Self-correction and iterative debugging capability: Through Agentic SFT training, the agent possesses a closed-loop capability of "generate-execute-read traceback-diagnose-rewrite," enabling it to autonomously recover from COM script execution failures without human intervention.
  • Directly optimizing geometric fidelity: The third-stage GRPO reinforcement learning introduces a continuous geometric reward based on Chamfer Distance, directly optimizing task-level geometric precision and bridging the execution gap of "syntactically correct but geometrically inconsistent."

Application Scenarios of ComAct

  • Industrial CAD automated design: In real industrial CAD software such as SolidWorks, AutoCAD, and Inventor, automatically converts natural language requirements into COM script execution, covering the entire engineering workflow including 3D modeling, 2D sketching, assembly design, drawing generation, mass property analysis, and interference detection.
  • Long-horizon multi-task pipeline orchestration: Chains together complex design pipeline tasks, such as from conceptual modeling → detailed part design → assembly verification → drawing output → BOM table generation, replacing hundreds of error-prone sequential GUI operations with a single deterministic script while maintaining a high end-to-end success rate.
  • Cross-application collaborative workflows: Uses the unified COM interface to integrate Microsoft Office (Excel, Word), the Adobe suite, and industrial CAD software, enabling cross-application automation such as "automatically updating an Excel bill of materials and generating a Word technical report after a CAD model change."
  • Design verification and quality inspection: Automatically performs analysis tasks such as interference detection, geometric dimension verification, and mass property calculation. Through the continuous geometric reward mechanism, it ensures the geometric fidelity of output results and assists engineers in quickly troubleshooting design conflicts.
  • Parametric batch modeling: Batch-generates or modifies standardized parts/assemblies based on input parameters, suitable for scenarios requiring large amounts of repetitive modeling tasks (such as standard parts library generation and series product variant design), significantly improving design efficiency.