glass

Purpose

glass is an MCP server written in Rust that provides AI coding agents with closed-loop operation capabilities for external native GUI applications: build → view → interact → debug. It enables agents to launch GUI applications, capture screen content, inject mouse and keyboard input, read application logs, and detect visual changes, so they can independently build and debug UI applications instead of repeatedly asking the user "does this look right?".

Main capabilities

• Launch and drive: Agents can launch GUI applications and drive them externally as a black box, suitable for native GUI applications written with any toolkit or language.
• View and verify: Supports screenshots (glass_screenshot), accessibility tree snapshots (glass_a11y_snapshot), and frame difference detection (glass_diff, returning changed_pct and bbox text, so routine checks do not need to consume vision tokens).
• Interaction: Inject mouse and keyboard input, supporting clicks, scrolling, and text input. glass_capabilities with scope: "session" can report whether these capabilities will affect your desktop.
• Element finding: glass_find_elements is used to inspect candidate elements when the target is not unique or not clear enough.
• Log reading: Can read application logs to assist debugging.
• Session evidence recording: Optionally enabled via --trace-dir, retaining tool inputs and request results in limited storage, after which glass-mcp trace inspect and trace export can be used to verify and package evidence as a ZIP. Disabled by default; retained application content may contain sensitive data.

Platform support

glass provides the following backends:

• Linux: Two backends, X11 and Wayland.
• Windows: Native backend.
• Android: AVD emulator, driven from any host via adb.
• iOS: Native applications in the simulator, via xcrun simctl, with input and accessibility tree via idb_companion, including two-finger pinch.
• macOS: Native backend.

By default, the Linux display and mobile simulators separate input from your desktop; native Windows and macOS currently share input. See docs/reference/platforms.md for the full matrix and system requirements. Transport is MCP over stdio (default) or network HTTP.

Usage

1. Download the glass build for your platform from the Releases page.
2. Configure the host according to the platform setup guide: Linux (docs/how-to/setup-linux.md, X11 or Wayland; Xvfb / sway + bubblewrap), Windows (docs/how-to/setup-windows.md, prebuilt .exe + Sandboxie), macOS (docs/how-to/setup-macos.md, install the notarized .dmg, no build required), Android (docs/how-to/setup-android.md, AVD emulator, any host), iOS (docs/how-to/setup-ios.md, simulator, macOS host only).
3. Connect glass to your agent, and run glass-mcp doctor to check the environment.
4. Optional: use glass-mcp --tool-profile lean to reduce the agent's tool definitions and perform operations with glass_do; use glass-mcp tools --json to inspect the tool list.
5. It is recommended to install the open-source glass-drive Agent Skill to help the agent drive glass more reliably and avoid rediscovering the "verify cheaply first, then view" loop in the initial stage.

Example: Have the agent use glass to run examples/tasks-demo/tasks_demo.py (with accessibility enabled), reproduce the bug where "clicking Add does not add a task", locate and fix the bug in the code by driving the UI and inspecting the accessibility tree (rather than only screenshots), and then verify that the task actually appears.

Applicable scenarios

• AI coding agents building and debugging native GUI applications.
• Scenarios that require automated verification of UI interaction results.
• Cross-platform (Linux, Windows, macOS, Android, iOS) GUI testing and debugging.
• Scenarios where a bug needs to be reproduced from the accessibility tree without screenshots.
• Scenarios where runtime evidence needs to be recorded and reviewed (optional).

Limitations

• For canvas or custom-rendered applications without an accessibility tree, pixel-driven approaches are required (screenshots, click coordinates, diff detection).
• Native Windows and macOS input is shared with the desktop.
• The iOS backend only supports macOS hosts.
• Session evidence recording is off by default, and retained application content may contain sensitive data.
• For the degree of support for specific platform capabilities (full/partial/unsupported/planned), refer to docs/reference/platforms.md.
• glass is open core and licensed under Apache-2.0.