SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Gemini 2.5 Computer Use - Google's Computer Use Model

Author: AI小集 · Original source: AI工具集

What is Gemini 2.5 Computer Use

Gemini 2.5 Computer Use is a computer use model based on Gemini 2.5, launched by Google DeepMind. The model enables AI to directly control the browser, performing operations such as clicking, scrolling, and typing. Through visual understanding and reasoning capabilities, the model helps users complete various tasks, such as retrieving information from web pages or organizing notes. The model performs excellently in benchmark tests and is also very fast. Developers can use it through Google AI Studio and Vertex AI, and users can try it out in Browserbase's hosted demo environment.

Gemini 2.5 Computer Use

Main Features of Gemini 2.5 Computer Use

  • Browser operations: Directly perform basic operations in the browser such as clicking, scrolling, and typing to help users complete web tasks.
  • Task automation: The model can handle multi-step complex tasks, such as retrieving information from one website and entering it into another system, or scheduling follow-up appointments.
  • Visual understanding and reasoning: Parse web page content visually, identify page elements, and reason out the next operation based on user requests.
  • Safety mechanisms: Before executing each action, an independent safety service evaluates the risk; for high-risk operations, it requests user confirmation to ensure operational safety.

Technical Principles of Gemini 2.5 Computer Use

  • Core tool: Implemented through the newly added computer_use tool in the Gemini API, supporting the model to interact directly with the user interface.
  • Input and output：
    • Input: User request, screenshot of the current environment, and history of recently executed actions. You can specify whether to exclude certain UI actions or add custom functions.
    • Output: The model generates a response, usually a function call representing a UI action (such as click, type, or scroll). For certain high-risk operations, the model requests user confirmation.
  • Loop process: The model runs in a loop process. After each action is executed, the system returns the latest screenshot and current URL to the model, restarting the loop. The loop continues until the task is completed, an error occurs, or it is terminated due to safety mechanisms or a user decision.
  • Safety mechanisms: During the reasoning phase, an independent safety service evaluates each action the model intends to execute to ensure operational safety. Developers can set that before specific high-risk operations, the agent must refuse or request user confirmation. For example, preventing the model from bypassing CAPTCHAs or controlling medical devices.

Project Address of Gemini 2.5 Computer Use

  • Project official website：https://blog.google/technology/google-deepmind/gemini-computer-use-model/
  • Technical paper：https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-2-5-Computer-Use-Model-Card.pdf

Application Scenarios of Gemini 2.5 Computer Use

  • UI testing: Help developers quickly test user interfaces, automatically execute various interactive operations, and significantly improve software development efficiency.
  • Personal assistant: Provide users with personalized task automation services, such as automatically filling out forms, scheduling appointments, or organizing information.
  • Workflow automation: Simplify repetitive tasks such as data entry, information collection, and cross-platform operations to improve work efficiency.
  • Customer service: Automatically handle customer requests, such as filling out tickets or querying information in customer support systems, improving response speed.
  • Education and Training: Assisting online learning platforms, helping students complete exercises or simulated operations, and enhancing the learning experience.