SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillslocalmac-ai-ocr

localmac-ai-ocr

Office & productivity

View repository

Updated 10-15 22:41

About this skill

Localmac AI OCR

Overview

This skill is used to coordinate two ready-made command-line tools:

  • scripts/gui Responsible for window activation, screenshots, coordinate clicking, and keyboard input
  • scripts/ocr Responsible for calling AI Studio OCR to output text, scores, and coordinate boxes, and supports finding and clicking by text

Prefer using the cloud aistudio-ocr. Unless the user explicitly requests it, do not fall back to local Paddle, to avoid it being slow and consuming memory.

Do not put the real API key、token、api url into the skill, script invocation examples, or repository files. Uniformly require reading it from environment variables or command parameters.

This skill comes with executable scripts and does not depend on the business repository itself after installation. The current instructions assume the commands can be executed directly from the skill root directory.

The executable entry point is at:

scripts/ocr
scripts/gui

For first-time installation, it is recommended to run:

scripts/setup.sh

When to use

  • The user asks you to operate the local GUI, an RDP window, or a desktop app
  • You need to extract Chinese text and coordinate boxes from screenshots
  • You need to "first take a screenshot, then recognize, then click by text"
  • You need to verify OCR speed, complete output, hit boxes, and annotated images
  • You need to precisely find text in market/quote software, table interfaces, and button-dense interfaces

Do not prioritize web scraping or DOM automation in these scenarios; first determine whether it is a desktop/RDP interface.

Workflow

1. First confirm the command entry point

First confirm that the script entry point is executable:

test -x scripts/ocr
test -x scripts/gui

If the current directory is not the skill root directory, first switch to the directory containing scripts/ before continuing. Do not expose the user's local directory name in the skill, submitted content, or external descriptions.

2. First install the minimal dependencies

It is recommended to run the bundled installation script directly:

scripts/setup.sh

If manual installation is needed, then run the following commands:

cd scripts
uv sync

3. First configure the required items

The cloud OCR of this skill has two required configurations:

  • AISTUDIO_OCR_API_URL
  • AISTUDIO_OCR_TOKEN

Recommended method:

cp scripts/.env.example scripts/.env

Then edit scripts/.envand fill in the real values. Do not commit .env back to the repository.

You can also pass them in temporarily via environment variables, but still do not write the real values into scripts, the skill, or commit records.

4. First do an environment health check

First confirm the OCR token and GUI dependencies, then take screenshots or click:

AISTUDIO_OCR_API_URL="$AISTUDIO_OCR_API_URL" AISTUDIO_OCR_TOKEN="$AISTUDIO_OCR_TOKEN" scripts/ocr doctor
scripts/gui doctor --json

doctor Only check whether it is configured, and do not print the real token or the API address. When running scripts, uniformly use uv run, and do not manually manage .venv or build the Python path yourself.

5. In GUI scenarios, first locate the window and then take a screenshot

Standard order:

scripts/gui list-windows
scripts/gui activate --title 打新股
scripts/gui capture /tmp/rdp.png --rect 0,30,2243,1266

Rules:

  • First list-windows, then activate
  • Screenshots should preferably fall into /tmp
  • If the region is known, pass --rect; if uncertain, first take a full-screen screenshot and then crop

6. For OCR, prioritize using ocr，gui ocr only for lightweight detection

Full coordinates, searching, clicking, and speed testing all go through ocr：

AISTUDIO_OCR_API_URL="$AISTUDIO_OCR_API_URL" AISTUDIO_OCR_TOKEN="$AISTUDIO_OCR_TOKEN" scripts/ocr ocr /tmp/rdp.png --format json
AISTUDIO_OCR_API_URL="$AISTUDIO_OCR_API_URL" AISTUDIO_OCR_TOKEN="$AISTUDIO_OCR_TOKEN" scripts/ocr ocr /tmp/rdp.png --format json --annotate-output /tmp/rdp-annotated.png
AISTUDIO_OCR_API_URL="$AISTUDIO_OCR_API_URL" AISTUDIO_OCR_TOKEN="$AISTUDIO_OCR_TOKEN" scripts/ocr find /tmp/rdp.png --query 腾讯控股 --mode contains --center --format json
AISTUDIO_OCR_API_URL="$AISTUDIO_OCR_API_URL" AISTUDIO_OCR_TOKEN="$AISTUDIO_OCR_TOKEN" scripts/ocr click-text /tmp/rdp.png --query 腾讯控股 --mode contains

When you only need to quickly view text, you can use:

AISTUDIO_OCR_API_URL="$AISTUDIO_OCR_API_URL" AISTUDIO_OCR_TOKEN="$AISTUDIO_OCR_TOKEN" scripts/gui ocr /tmp/rdp.png --format text

7. Look at the hit box before clicking

Do not click blindly. By default, follow this order first:

  1. ocr find --format json --center
  2. Check the number of hits and whether the text is ambiguous
  3. If necessary, add --index N、--mode exact or narrow the screenshot area
  4. then execute click-text or gui click

If there are many texts with the same name in the interface, crop the image first before recognition, and do not directly click the first hit item on the full screen.

8. Verified stable click workflow

This is the order that has been verified in a real desktop environment and is most suitable for agent reuse:

  1. First take a screenshot to /tmp
  2. Use ocr find --center --format json --annotate-output ... to find the target text
  3. Check the number of hits
  4. If there is only 1 hit, directly click-text --index 1
  5. If there are multiple hits, prioritize:
    • Change --mode exact
    • or explicitly pass --index N
    • or narrow the screenshot area before recognition
  6. After clicking, immediately take another screenshot to verify whether the interface has changed

Verified command template:

scripts/gui capture /tmp/current.png
AISTUDIO_OCR_API_URL="$AISTUDIO_OCR_API_URL" AISTUDIO_OCR_TOKEN="$AISTUDIO_OCR_TOKEN" scripts/ocr find /tmp/current.png --query 自动化 --mode contains --center --format json --annotate-output /tmp/current-annotated.png
AISTUDIO_OCR_API_URL="$AISTUDIO_OCR_API_URL" AISTUDIO_OCR_TOKEN="$AISTUDIO_OCR_TOKEN" scripts/ocr click-text /tmp/current.png --query 自动化 --mode contains --index 3
scripts/gui capture /tmp/after-click.png

This workflow is suitable for:

  • Left-side menu
  • Top tabs
  • Text in the menu bar at the top right, such as "WeChat Input Method"
  • Text-dense interfaces such as chat windows, log windows, and configuration pages

Note:

  • click-text now by default converts screenshot pixel coordinates into actual screen points before clicking, adapting to Retina displays
  • If the screenshot comes from a local area rather than the full screen, you need to add --screen-rect "x,y,width,height"
  • If you only want the agent to review without clicking immediately, first keep the find --annotate-output result image

Quick commands

  • Full OCR JSON:
AISTUDIO_OCR_API_URL="$AISTUDIO_OCR_API_URL" AISTUDIO_OCR_TOKEN="$AISTUDIO_OCR_TOKEN" scripts/ocr ocr /tmp/rdp.png --format json
  • Full OCR JSON + output an annotated image with boxes:
AISTUDIO_OCR_API_URL="$AISTUDIO_OCR_API_URL" AISTUDIO_OCR_TOKEN="$AISTUDIO_OCR_TOKEN" scripts/ocr ocr /tmp/rdp.png --format json --annotate-output /tmp/rdp-annotated.png
  • Plain text only:
AISTUDIO_OCR_API_URL="$AISTUDIO_OCR_API_URL" AISTUDIO_OCR_TOKEN="$AISTUDIO_OCR_TOKEN" scripts/ocr ocr /tmp/rdp.png --format text
  • Find text and output the center point:
AISTUDIO_OCR_API_URL="$AISTUDIO_OCR_API_URL" AISTUDIO_OCR_TOKEN="$AISTUDIO_OCR_TOKEN" scripts/ocr find /tmp/rdp.png --query 00700 --mode exact --center --format json
  • When there are multiple hits, take the 2nd result:
AISTUDIO_OCR_API_URL="$AISTUDIO_OCR_API_URL" AISTUDIO_OCR_TOKEN="$AISTUDIO_OCR_TOKEN" scripts/ocr find /tmp/rdp.png --query 自动化 --mode contains --index 2 --center --format json
  • Draw boxes on the recognition results:
AISTUDIO_OCR_API_URL="$AISTUDIO_OCR_API_URL" AISTUDIO_OCR_TOKEN="$AISTUDIO_OCR_TOKEN" scripts/ocr annotate /tmp/rdp.png /tmp/rdp-annotated.png --query 腾讯控股
  • Test speed 10 times:
AISTUDIO_OCR_API_URL="$AISTUDIO_OCR_API_URL" AISTUDIO_OCR_TOKEN="$AISTUDIO_OCR_TOKEN" scripts/ocr benchmark /tmp/rdp.png --backend aistudio-ocr --repeat 10

Common mistakes

  • Taking a screenshot without activating the window, resulting in capturing an old frame
  • Not configuring both AISTUDIO_OCR_API_URL and AISTUDIO_OCR_TOKEN
  • Writing real configuration into the repository instead of putting it into scripts/.env
  • Too many text hits but directly click-text
  • Running OCR on the entire desktop, resulting in slowness and many ambiguities; you should crop to the target area first
  • Continuing to click coordinates when an existing popup is blocking, causing the operation to go off track

Reference

  • For common command recipes, see references/command-recipes.md
  • Executable scripts are located at scripts/
  • If the task is "screenshot -> find text -> click", prioritize executing according to the RDP recipe in the reference
  • If you need to customize the cloud API address, only reference the variable name, do not write the real value

Agent API quick reference

When targeting agents, prioritize remembering the semantics of the following parameters:

  • scripts/ocr ocr IMAGE --format json Purpose: return all recognition items; fields usually include text、score、bbox、quad
  • --center Purpose: additionally add center=[x,y]to the JSON result, making it easier for subsequent clicks or logging
  • --relative Purpose: additionally add relative coordinates to the JSON result, suitable for cross-resolution comparison
  • --rows Purpose: aggregate adjacent text into lines, suitable for menus, table rows, and log lists
  • --annotate-output PATH Purpose: synchronously draw boxes for the recognition results corresponding to the current command onto the image Default label content: recognized text, center coordinates, confidence
  • --annotate-no-text Purpose: do not display recognized text in the annotated image, keeping only coordinates and confidence
  • --annotate-no-center Purpose: do not display center coordinates in the annotated image
  • --annotate-no-score Purpose: do not display confidence in the annotated image
  • --annotate-color '#RRGGBB' Purpose: Adjust the box line and label colors, making it easier to review against dark or light backgrounds
  • scripts/ocr find IMAGE --query 文本 --mode contains --format json Purpose: Filter out matching items. Look here before clicking, do not click blindly
  • --index N Purpose: Take only the Nth matching item, starting from 1 in reading order
  • scripts/ocr click-text IMAGE --query 文本 Purpose: Click by the center of the recognition box Default behavior: If multiple matches are found, an error will be reported to avoid misclicking; when you really need to specify a particular one, explicitly pass --index N
  • --screen-rect "x,y,width,height" Purpose: Tell the tool "which area on the screen this image corresponds to", so that coordinates can be correctly converted when clicking on a region screenshot