What is Computer Use Preview
Computer Use Preview is Google's open-source AI browser automation tool. The tool is based on the visual recognition capabilities of the Gemini model and can simulate human operation of web pages through "screenshot - analysis - action" without relying on fixed element locators. The tool can handle complex tasks, such as cross-website data transfer and visual interaction, and performs excellently in task completion tests, with a score of 69%, outperforming similar tools. Computer Use Preview integrates the Gemini API and Vertex AI, has powerful task understanding capabilities, is suitable for web testing, data collection, and daily operations, and is an ideal choice for zero-code beginners.
Main features of Computer Use Preview
- Natural language driven: Users can describe tasks in simple natural language, and the AI automatically plans and executes operations without writing complex scripts, greatly lowering the barrier to use.
- Intelligent interaction: Supports handling complex browser operations, such as navigation, clicking, form filling, scrolling, etc. The Gemini model can understand dynamic changes on the page and adapt to different scenarios.
- Dual environment support
- Playwright (local run): Controls a local Chrome browser instance, suitable for scenarios that require local data processing.
- Browserbase (cloud environment): Connects to a Browserbase instance, suitable for scenarios that require cloud resource support or distributed operations.
- Debugging friendly: Supports screenshots and mouse highlighting, allowing users to monitor the execution process in real time, making debugging and optimizing operations convenient.
- AI large model integration: Integrates the Gemini API and Vertex AI, providing powerful task understanding capabilities and the ability to handle complex instructions and dynamic web page content.
How to use Computer Use Preview
- Preparation: Download the project files from the Computer Use Preview GitHub page and extract them locally.
- Set up the runtime environment: Install Python (version 3.8 or above is recommended), create and activate a virtual environment to isolate project dependencies.
- Install dependencies: In the project folder, run commands to install project dependencies and the Playwright browser.
- Configure API key: Obtain a Gemini API key from Google Cloud and add it to the environment variables.
- Run the tool: Run the tool with natural language instructions, for example: "Go to Google and type 'Hello World'".
- Optional configuration: Specify the runtime environment (such as Playwright or Browserbase) and configure related parameters as needed.
Computer Use Preview project address
- GitHub repository：https://github.com/google-gemini/computer-use-preview
- Online experience address：Browserbase
Application scenarios of Computer Use Preview
-
Web automated testing: Used to quickly test web page functionality and verify whether operations such as button clicks and form submissions work properly.
-
Data collection: The tool can automatically scrape data from web pages, such as news and product information, without manual operation.
-
Cross-website data transfer: Automatically transfers data from one website to another system, simplifying the data migration process.
-
Daily task automation: The tool supports completing daily repetitive tasks, such as automatic login and scheduled queries, saving time and effort.
-
Visual interaction tasks: The tool can handle complex visual interactions, such as recognizing web page elements and performing classification or drag-and-drop operations.