DesktopCtl is an open-source "computer use" tool for AI agents, adopting a local-first runtime architecture and not relying on the cloud. Its positioning is to enable AI agents to observe and control the user's computer, with operation methods covering the screen, mouse, and keyboard.
In terms of main capabilities, DesktopCtl allows users to bring any AI model, and it can work even if the model does not have visual capabilities. It provides a GPU-accelerated text recognition and computer vision pipeline, supports a selector-first automation approach (through the --text and --token parameters) for element location, and falls back to the coordinate method when necessary. The tool has built-in explicit waiting mechanisms and post-action verification for agents, and provides a stable JSON contract externally, making agent integration easier. In addition, it also supports enumerating and invoking native menus through macOS Accessibility.
In terms of usage, DesktopCtl is divided into two binary components: DesktopCtl.app (desktopctld) is the daemon, responsible for perception, state, execution, and verification; desktopctl is a stateless command-line interface that executes actions and queries through local IPC. For installation, the Rust toolchain and the just command runner are required, and Accessibility and Screen Recording permissions must be granted to DesktopCtl.app. Currently, only macOS on Apple Silicon is supported. The installer places the CLI at ~/.local/bin/desktopctl and the app at ~/Applications/DesktopCtl.app, and does not modify shell startup files; if needed, users must add ~/.local/bin to PATH themselves. Release versions carry a local ad-hoc signature, but are not signed with a Developer ID or notarized; the installer only removes the com.apple.quarantine attribute from the installed DesktopCtl app and CLI, and does not change global Gatekeeper or other system security settings.
The file system layout adopts a Linux-style unified application directory, with path priority being DESKTOPCTL_HOME, XDG_DATA_HOME/desktopctl, and finally $HOME/.local/share/desktopctl. Configuration is stored in config.toml, persistent runtime metadata is in state/, launcher sessions are in workspaces/, logs and journal output are in logs/, disposable screenshots/OCR/debug output are in cache/, and Unix runtime sockets are kept in a private temporary runtime directory. On first launch, valid JSON configuration and launcher sessions are securely migrated from legacy configuration locations, without overwriting existing targets or touching conflicting legacy sources; migration errors are reported but legacy files are not deleted.
For demonstration, you can list the native menus, then invoke a given item via the returned #menu_* ID; the menu list by default omits the Apple/system menu, separators, and long-list overflow content. Use --system to include the Apple menu, and use --all to show all items.
The applicable scenario is for AI agents performing GUI automation tasks in a local macOS environment, such as observing the screen, recognizing text, locating controls by text or token, executing click/type/wait workflows, and verifying results.
In terms of limitations: currently only macOS is supported, and only Apple Silicon; it is in active development; the app is not signed or notarized with a Developer ID; specific pricing standards and price information are not provided in the documentation, and are not currently available. Future plans include adding doctor, richer window/app introspection and --explain failure output commands, improving local computer vision and semantic UI tokenization, and multi-platform support.
