Touchpoint is a cross-platform Python library open-sourced by Touchpoint-Labs on GitHub, positioned as "giving AI agents eyes and hands on any desktop." Its core idea is not to use pixel scraping or vision models, but instead to read the system's native accessibility (a11y) API to obtain structured information about every element on the screen, including name, role, state, and position, thereby achieving speed, reliability, and no need for vision models.
In terms of use, Touchpoint is used to let AI agents read and interact with desktop UIs, covering Linux, macOS, and Windows, and has built-in support for Chromium and Electron applications (such as Slack, Discord, VS Code, etc.), obtaining complete web content through CDP (Chrome DevTools Protocol). It is suitable for scenarios that require automating desktop applications, reading interface content, or letting LLM agents control desktop programs. The official example shows an AI agent researching data in Chrome and then creating a formatted Excel spreadsheet, with the entire task completed in about 12 minutes.
In terms of main capabilities, Touchpoint provides a unified API, where each element has a unique ID, such as atspi:1234:1:2.0 or cdp:9222:TID:4; action functions accept either an Element object or a bare ID string, making it convenient to save references across steps. It comes with an MCP (Model Context Protocol) server, which can connect to MCP-compatible clients such as Claude, Cursor, Windsurf, and VS Code, and can also be used with local models, allowing LLM agents to control desktop applications out of the box.
The MCP server supports two modes: in vision mode (the default), the agent uses screenshot() to view the screen and interacts through element IDs or coordinates, suitable for frontier models with strong visual capabilities; in no-vision mode, the agent uses snapshot() to obtain a compact structured text tree of the active window, and then directly operates on element IDs, suitable for any model, including local models without visual capabilities. Most action tools append automatic verification markers, such as (new window: ...), (focus moved), (no change detected), allowing the agent to detect state changes without screenshots. The MCP server has built-in instructions that teach the agent the correct workflow for each mode, including the "locate → act → verify" loop, when to use read_text versus find, and how to recover from errors.
In terms of usage, the installation command is pip install touchpoint-py, and it requires Python 3.10+. Everything is included: platform-native backends, CDP support for browsers and Electron applications, the MCP server, and screenshot capabilities; platform-related dependencies are automatically installed through pip environment markers. For MCP client configuration, the official documentation provides configuration methods for Claude Desktop (macOS configuration path ~/Library/Application Support/Claude/claude_desktop_config.json; Windows configuration path %APPDATA%\Claude\claude_desktop_config.json; if using a virtual environment, use the full path "/path/to/venv/bin/touchpoint-mcp"), VS Code (workspace .vscode/mcp.json), Cursor (~/.cursor/mcp.json), Windsurf (~/.codeium/windsurf/mcp_config.json), and OpenClaw (mcpServers in ~/.openclaw/openclaw.json). Browsers and Electron applications need to be started with a debugging port, and Touchpoint enables automatic discovery by default.
In terms of API reference, it provides categories such as Discovery, Search & Wait, Actions, Input, Screenshot & Config. elements(), find(), and get_element() support format="flat", format="json", or format="tree" (only elements) to return preformatted strings. The source parameter can control the source of results; source="ax" is still accepted as a compatibility alias for source="cdp_ax", and new code shouldIt is recommended to use cdp_ax to avoid confusion with the native macOS AX backend. CDP results are merged with native backend results, returning both toolbar and window controls from AT-SPI2/UIA/AX and the full web page content from CDP in the same elements() call.
Architecturally, a two-layer design is adopted: the Backend reads the accessibility tree and performs structured actions (click, set value, focus), being element-aware and reliable; the InputProvider simulates raw keyboard and mouse input, being coordinate-based and not element-aware, and is used as an automatic fallback when native accessibility actions are unavailable. CDP and platform backends run in parallel, and results are merged. Window management is implemented on the Linux AT-SPI2, Windows UIA, and macOS AX backends. tp.diagnostics() returns a JSON-friendly health report, including the active backend, input provider, CDP targets, optional platform tools, configured timeouts, and macOS apps recently skipped due to AX message timeouts.
Applicable scenarios include: enabling AI agents to retrieve data and operate office software in browsers such as Chrome, automating cross-platform desktop application interaction, reading interface structure in environments without vision models, and connecting desktop control capabilities to MCP-compatible LLM clients.
Regarding limitations, it is officially explicitly labeled as Alpha stage; although feature-complete and tested on the three major platforms, the API may change before 1.0 based on user feedback. Known limitations include: Wayland input — the Linux InputProvider uses xdotool, which requires X11; under pure Wayland (without XWayland), keyboard/mouse simulation is unavailable, but the accessibility tree and native actions still work; synchronous CDP — CDP calls block waiting for WebSocket responses, JavaScript dialogs (alert, confirm, prompt) are automatically dismissed to prevent deadlocks, and an asynchronous rewrite is planned; there is no built-in browser navigation API, so agents need to interact directly with UI elements to navigate (find the address bar, enter the URL, press Enter); CDP windows are page targets rather than operating system windows, but window management still works, and tp.activate_window() will bring the target to the foreground via CDP; performing minimize/fullscreen/close/move/resize on an already manifested cdp: windowwill be routed to the underlying native OS window (resolved by the owning PID) and handled by the platform backend, and ActionFailedError is thrown only when no native OS window corresponding to that target can be found (for example, it has been closed); backend role/state consistency is still uneven, and both macOS AX and Windows UIA have been significantly improved in 0.3.0, but Windows still relies on more heuristics and has more unmapped long-tail roles. Follow-up plans include high priority: asynchronous CDP architecture (non-blocking WebSocket, proper dialog queuing, concurrent multi-tab queries); medium priority: backend role/state consistency (closing the remaining role mapping gaps, especially the UIA long-tail roles on Windows), Wayland input backend (using libei / xdg-desktop-portal RemoteDesktop when X11 is unavailable), tooltip and notification visibility, element caching.
