SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Step-GUI - AI Agent series models launched by StepFun

Author: AI小集 · Original source: AI工具集

What is Step-GUI

Step-GUI is a series of AI Agent models launched by StepFun, including the cloud-side model Step-GUI and the on-device model Step-GUI Edge (formerly known as Gelab-Zero). The cloud-side model excels at handling complex tasks, while the on-device model is lightweight and can run on local devices such as mobile phones, protecting user privacy. Through a self-evolving training process and the Calibrated Step Reward System (CSRS), Step-GUI converts interaction trajectories generated by the model into high-quality training data, significantly reducing annotation costs and improving performance. Step-GUI performs excellently in multiple benchmarks, such as AndroidWorld and ScreenShot-Pro. Step-GUI has introduced the GUI-MCP protocol to achieve standardized interaction between models and devices, protect user privacy, and evaluate model performance in real-world mobile applications through the AndroidDaily benchmark, promoting the practical and standardized development of GUI automation tools.

Step-GUI

Main features of Step-GUI

  • Automated task execution: Operates various applications like a human user, such as clicking buttons, swiping the screen, entering text, etc., to complete tasks ranging from simple to complex.
  • Cross-platform adaptation: Through the GUI-MCP protocol, Step-GUI can run on multiple devices (such as mobile phones, computers, and car systems) and supports different operating systems (such as Android, iOS, Windows, and macOS).
  • Privacy protection: Adopts a device-cloud collaborative architecture, keeping sensitive data on the local device side and transmitting only necessary semantic information to the cloud, ensuring user privacy.
  • Efficient task processing: Achieves efficient processing of complex tasks through the division of labor and collaboration between cloud-side large models and on-device small models. The cloud-side model is responsible for high-level reasoning and task decomposition, while the on-device model is responsible for specific operation execution.
  • Real-world scenario simulation: Simulates real user behavior patterns to complete daily high-frequency tasks such as social chatting, shopping payments, and video watching, making it suitable for a variety of practical application scenarios.

Technical principles of Step-GUI

  • Multimodal large language model: Step-GUI is based on powerful multimodal large language models (such as Qwen3-VL), and can understand and generate natural language instructions, converting them into specific GUI operations. By combining visual input (such as screen screenshots) with language instructions, the model achieves precise localization and operation of interface elements.
  • Self-evolving training framework: Through the Calibrated Step Reward System (CSRS), the operation trajectories generated by the model are converted into high-quality training data. CSRS ensures the accuracy and reliability of the data through trajectory-level verification signals, while using detailed reasoning chains generated by LLMs to improve the model's generalization ability.
  • Device-cloud collaborative architecture: Step-GUI adopts a device-cloud collaborative working mode. The on-device model (such as Step-GUI Edge) is responsible for executing specific operation tasks on the local device, ensuring privacy protection and low-latency response; the cloud-side model is responsible for handling complex logical reasoning and task decomposition, providing stronger semantic understanding and generalization ability.
  • GUI-MCP protocol: To achieve standardized cross-platform interaction, Step-GUI proposes the GUI-MCP (Model Context Protocol) protocol. Through a layered architecture, the protocol abstracts device operations into atomic operations and composite tasks, supports low-level operations (such as clicking and swiping) and high-level task delegation (such as "buy coffee"), while ensuring that data is processed locally to protect user privacy.
  • Reinforcement learning and data iteration: Step-GUI is trained using reinforcement learning (such as Group Relative Policy Optimization, GRPO), continuously optimizing the model's decision-making capabilities through interaction with the environment. At the same time, through a self-evolving training process, the model can continuously learn from data it generates itself, improving performance.

Step-GUI project address

  • GitHub repository：https://github.com/stepfun-ai/gelab-zero
  • Technical paper：https://github.com/stepfun-ai/gelab-zero/blob/d1cd0c7be83e234b66dbec4c5554f5fde44dce08/report/Step-GUI_Technical_Report.pdf

Step-GUI application scenarios

  • Personal daily task automation: Step-GUI can automatically reply to social messages, manage group chats, place orders and make payments in life service apps, helping users handle daily affairs efficiently.
  • Enterprise-level automation tasks: The tool can automatically complete data entry, document editing, and email sending in office software, improving enterprise office efficiency and optimizing workflows.
  • Smart home and Internet of Things: Step-GUI can control smart home devices, set scene modes, and realize automated control of devices such as lights and air conditioners, improving the convenience of home life.
  • Gaming and entertainment: Step-GUI can automatically complete repetitive tasks in games, or automatically play, favorite, and comment on videos on video platforms, enhancing the entertainment experience.
  • Education and learning: Step-GUI can automatically play courses and submit assignments on online learning platforms, and create tasks and record progress in learning management tools, assisting learning management.