SuperAPI AI Directory

pyautogui-helper

General & other

View repository

Updated 10-15 21:02

About this skill

PyAutoGUI & OpenCV Helper

PyAutoGUI is an intuitive library that can control the operating system's mouse and keyboard and analyze screen images to automate various GUI environments [1]. However, simple API calls alone will face screen loading delays, slight image resolution errors, multilingual input errors, and computational bottlenecks during real-time screen recognition [3] [8].

This skill PyAutoGUI's powerful control features과 OpenCV's ultra-fast and high-precision image analysis technologyby fusing them, provides optimal development guidelines and proven utilities for building a high-quality automation pipeline that operates stably without malfunctions [1] [2].

1. Core Use Cases

This skill is essential for projects that require advanced GUI automation such as the following:

  • High-speed screen analysis and real-time response: When you need to capture game or real-time dashboard screens with fast frame changes and detect targets without delay [8].
  • High-precision image search (Template Matching): When detecting targets in image environments with slight pixel errors or compression loss by gradually relaxing the accuracy threshold (threshold) [2].
  • Simultaneous detection of multiple objects: When multiple identical icons or checkboxes exist on the screen, identifying all of them and elaborately filtering duplicate matching coordinates [1].
  • Multithreading-based control: When designing a parallel processing architecture so that screen recognition operations and mouse/keyboard control actions do not interfere with each other [8].
  • Safety devices and Korean input workaround: When designing a Fail-Safe safety device to prevent loss of control and when Korean typing using the clipboard is needed [1] [3].

2. Recommended Development Workflow

When writing high-performance automation scripts combining PyAutoGUI and OpenCV, follow the following 4-step standard process.

+------------------------+      +------------------------+
| 1. 환경 구성 및 패키지  | ---> | 2. 안전 설정 및 초기화  |
+------------------------+      +------------------------+
                                             |
                                             v
+------------------------+      +------------------------+
| 4. 멀티스레딩 및 복구  | <--- | 3. 고속 캡처 및 매칭   |
+------------------------+      +------------------------+

Step 1: Environment setup and package installation

  • Required dependencies: For high-performance template matching and clipboard control, install the following packages in advance.
    sudo pip3 install pyautogui opencv-python pyperclip pillow numpy
    
  • OS-specific permissions: In the macOS environment, you must System Settings > Screen Recording grant permission and restart the running app for capture to work properly [3].

Step 2: Safety settings and initialization

To prevent infinite loop accidents that completely lose system control when a script malfunctions, explicitly set global safety devices and delay times at the beginning [1].

import pyautogui
pyautogui.FAILSAFE = True  # 마우스를 모서리로 가져가면 강제 종료
pyautogui.PAUSE = 0.1       # 모든 제어 명령 사이에 0.1초 미세 지연 부여

Step 3: High-speed screen capture and progressive matching (LOCATE-INTERACT)

Instead of pure PyAutoGUI image matching, Pillow ImageGrab memory capture and OpenCV matchTemplateCombine to maximize performance [2].

from scripts.pyautogui_utils import wait_and_click_opencv

# 'button.png' 이미지가 화면에 로드될 때까지 최대 10초 대기 후 정밀 클릭
# 매칭 감도(threshold)를 자동으로 완화하며 유연하게 검색합니다.
success = wait_and_click_opencv('button.png', timeout=10, initial_threshold=0.95)

Step 4: Implement multithreading-based parallel control

To maintain real-time detection performance, separate the thread that continuously captures and analyzes the screen from the thread that controls the mouse/keyboard, and when transferring data between threads, use a lock (threading.Lock) to synchronize [8].


3. Resource and Reference Guide

For effective development, actively utilize the detailed references and utility code built into this skill.

  • PyAutoGUI Core API Reference: api_reference.md
    • Basic control specifications such as mouse coordinate system, click control, keyboard shortcut map, and message boxes.
  • OpenCV Screen Recognition API Reference: opencv_reference.md
    • Detailed summary of core vision APIs such as grayscale conversion, template matching method (TM_CCOEFF_NORMED), image blurring and binarization, and ORB feature point detection.
  • Practical Problem Solving and Advanced Design Patterns: best_practices.md
    • Design guide for high-speed capture techniques, progressive matching sensitivity control code, multi-object deduplication algorithms, and multithreading parallel detector design.
  • Verified Utility Code: pyautogui_utils.py
    • Complete utility scripts with immediately implemented screen information acquisition, OpenCV high-speed capture, multi-object coordinate detection, and safe Korean input functionality.

References