SuperAPI AI Directory

autojs-task-runner-yashu

General & other

View repository

Updated 10-16 08:48

About this skill

Phone Task Executor (AI → AutoJS)

Purpose

Treat the AI on the PC as the "brain" and AutoJS on the phone as the "hands": the user describes in one sentence what they want done on the phone, the AI breaks it down into a chain of steps and sends them step by step to the phone for execution,only reporting back to the user once when everything is done or has failed。

User-side usage tutorial:<skill_dir>/使用示例.md— a scenario-organized collection of examples of "what the user says to the AI" (controlling the phone / viewing the screen / transferring files / long tasks / accumulating templates and projects / safety boundaries). When a new user uses it for the first time, or when the user asks "what can it do / how do I use it", proactively recommend this document.

Typical user group: users who can speak and type but have limited mobility—speech-to-text → AI → phone, indirectly controlling the phone.

Token-saving design (runs through this skill, must be followed)

This skill is aimed at long-term high-frequency users, and AI computing power costs money,every round of interaction must save tokens:

  1. Pass only the path, not the code: keep a copy of the template script on the PC (scripts/tasks/), and when sending, pass only path + args(a few dozen bytes); the phone downloads and executes it by name from the relay;
  2. Minimal receipt: by default templates return only {ok:1} / {ok:0, err:"原因"}. The AI must not require the script to send back the entire UI tree or large blocks of text;
  3. Fully automatic continuous run: intermediate steps produce no user conversation, executing silently step by step, with a single report at the final state;
  4. Screenshots instead of screen reading: when you need to "see" the screen, use GET /screenshot(binary directly transferred to the PC and written to disk, not passing through the AI context), and the AI then uses Read to read the image. The phone is forbidden to send backthe entire UI control tree JSON (i.e., the raw large JSON dumping all on-screen controls); but it is allowed to use inspect_control_* and other templatesto structurally retrieve onlythe needed text/id/desc, see the third level of the fallback chain in "Step 3". (Note:Read reading images depends on whether the caller's model is multimodal—non-multimodal models will have image reading filtered out, which is normal; handle it according to the rules in "Step 3: Viewing the Screen": first present_files present the image to the user, and if the AI itself cannot read it, use ocr/inspect_control_*/ask the user instead; do not arbitrarily judge the model's capabilities.)
  5. Two-level template reading: when planning, run only node scripts/scan-tasks.js to get the name+description of all templates (do not read the scripts, do not read the body); only after selecting a template, Read its tasks/<name>/TASK.md full text. The description is the basis for selection, and the body is the decision contract.
  6. 2K text budget: before task receipts enter the context, run-task.js strictly enforces this—anything over 2000 characters is automatically truncated to "a 400-character preview + the full-text disk path" (scripts/task-results/<taskId>.txt,resultTruncated:true); for large on-screen text from OCR, prefer using {"query":"关键词"} to filter and reduce at the source (see the ocr template).

Prerequisites (four permissions on the phone side, none may be missing)

The AutoJS6 app must be installed on the phone. AutoJS6 official website:https://docs.autojs6.com/#/

  1. Accessibility service: Settings → Apps → AutoJs6 → Accessibility → enable;
  2. Screenshot permission: Automatically requested when the phone-side script starts; in the popup, tap "Start Now";
  3. Floating window permission: Settings → Apps → AutoJs6 → Permission management → Floating window → Allow;
  4. Background running: Settings → Apps → AutoJs6 → Set battery saver policy to "Unrestricted", and allow auto-start in auto-start management (otherwise, after locking the screen or switching to the background, the resident client is killed by the system and the task is interrupted).

Convention

  • <skill_dir> = this skill's root directory (the directory containing SKILL.md).
  • One-off scripts written on the spot by the AI are written to the <skill_dir>/temp/ directory.
  • On the AI side, to write AutoJS multi-file projects (projects that need to be deployed to the phone), write them uniformly to the <skill_dir>/scripts/autojs-project/<工程名>/ directory (one subdirectory per project; for the internal structure, refer to the autojs-min-project-template template in the same directory). This directory already has the project1/ slot preset and ready to use, scripts/autojs代码参考例子/autojs-projects/ColorWheelthis is an autojs color wheel interface project for reference. One-off single-file temporary scripts are still placed in temp/(see "On-the-spot script specification"). For hard constraints on project coding (ES5 / var only, etc.), see "Deploying real projects → Project code writing specification".
  • Default port 9421. Changes must be synchronized in three places: ① curl localhost:9421/health;② autojs-task-phone-client.js 's SERVER_PORT;③ run-task.js 's DEFAULT_URL。
  • Shell path convention (hard rule): Commands issued locally go through Git Bash(POSIX shell), and the path separator is alwaysforward slash /(e.g. /c/Users/...)。It is strictly forbidden to write Windows backslashes in commands \——POSIX shell treats \ as an escape character and will silently swallow it, causing skills\autojs to become skillsautojs, path concatenation errors, andcd/command cannot find the file.It is recommended to first cd into the skill directory (in POSIX forward-slash form; verified usable in this environment), then use the relative path scripts/run-task.js to call the script + write only the template name, so that not even the tasks/... long path needs to be assembled.
  • MSYS argument rewriting pitfall (measured in this environment): Git Bash willarguments starting with / Parameters at the beginning(e.g. --target-dir /sdcard/Download) automatically rewrite to Windows paths (measured: /sdcard/Download was changed to D:/software/Git/Git/sdcard/Download; the phone side assembled an invalid path and the delivery failed). For --target-dir、delete-folder --path suchphone path argumentspassed in /sdcard form, the command must be prefixed with MSYS_NO_PATHCONV=1(e.g. MSYS_NO_PATHCONV=1 node scripts/pc-to-phone.js a.txt --target-dir /sdcard/Download)。
  • node argument path pitfall (measured in this environment): whenthe script's absolute path is passed directly as a node argument,/c/Users/... this kind of POSIX form will be treated by the Windows version of node as a "relative path under the current drive root" (e.g. resolved to d:\c\Users\...) and the file cannot be found. The correct approach is as above: first cd into the skill directory, then node scripts/run-task.js ...; if you must pass an absolute script path to node, use the C:/Users/.../run-task.jswith a drive letter prefix, do not write /c/Users/.../run-task.js。

Phone-side directory architecture v3 (scripts-from-computer)

Phone-side AutoJS script root directory (files.getSdcardPath() + "脚本" dynamically concatenated,hardcoding physical paths is prohibited across the whole chain) under:

  • user-written scripts: permanently stay in the root directory; this skill never touches them (the skill itself also leaves no files in the root directory);
  • scripts-from-computer/: quarantine zone for all artifacts this skill delivers from the PC—
    • client/ resident client itself (self-updates land directly, can be started manually; when the client starts it "self-returns" here);
    • single/ template single script + temporary probe script (deployed by run-task);
    • project/<工程名>/ AI project works (deploy-project deployment, run-project execution, delete-project cleanup);
    • data/ companion data (task-args/<taskId>.json per-request parameters,relay-config.json relay address,beep.wav and other resources);
    • files/ default destination for general files sent from PC to phone (when pc-to-phone / send-file-to-phone does not explicitly specify --target-dir ).
  • Naming convention: folders/files created by this skill are always lowercase kebab-case, to avoid file-not-found issues caused by case ambiguity.

Script list

  • scripts/autojs-relay-server.js: PC-side relay service (HTTP + WebSocket, Node.js ES6), AI-started. For architecture details see scripts/relay/ARCHITECTURE.md。
  • scripts/autojs-task-phone-client.js: phone-side resident client (strictly ES5, all variables declared with var), which the user copies to the phone to run in AutoJs6. Keep-alive, reconnect on disconnect, download and execute scripts by name, return results.
  • scripts/run-task.js: AI-side bridge script—POSTs template name/script file + parameters to the relay /run(asynchronous task ticket); by default it polls the task ticket for up to 30 seconds waiting for a receipt and prints it (outer shape compatible with older versions); for long tasks add --wait 0 to return the taskId immediately. For the four invocation methods and hard path rules see "Step 2"; task ticket control:--status <taskId>(query status/progress/result, queryable across sessions),--stop <taskId>(force kill),--list [数量](list recent task tickets). Running without parameters prints the full usage.
  • scripts/self-test.js:One-click regression self-test—runs health check, parameter/clipboard round-trip, screenshot compression, delete-path guardrails, full project lifecycle, and task ticket listing all at once, outputting a ✅/❌ summary and exit code. After changing the relay/client/template, just run it to prove you haven't broken anything (node scripts/self-test.js)。
  • scripts/get-lan-ip.js: get the computer's LAN IP (node <skill_dir>/scripts/get-lan-ip.js --first), used when filling in the IP on the phone. See references/获取电脑局域网IP.md。
  • scripts/pc-to-phone.js:Transfer any file from the computer to the phone(one-click entry for the PC→phone channel, a high-frequency must-have). node scripts/pc-to-phone.js <本地文件> [--target-dir <手机目录>] [--target-name <手机文件名>] [--keep] Usage notes: paths are always written relative to the skill directoryrelative paths;<本地文件> isdata parameter(the file to be transferred), don't confuse it with the node script path;--target-dir defaults to being dynamically derived via /health as scripts-from-computer/files/(if explicitly specified, it goes exactly where specified),--keep keeps a transit copy on the computer (by default cleaned up after the receipt). For the reverse direction (phone→computer) use the template download-file。
  • scripts/deploy-project.js:Deploy a multi-file AutoJS project (including images/audio and other resources) from the computer to the phone and run it—for complex projects (multi-module / with resources, which a single-file template cannot accommodate) use this instead of a single-file template. Usage node scripts/deploy-project.js <工程目录> [--name <工程名>] [--main <入口.js>] [--run|--no-run] [--args '<json>'] [--keep], for parameter defaults and mechanism see the section "Deploying a real project (multi-file + resources)".
  • scripts/run-project.js:Run a project already deployed to the phone (without re-delivering files). Use this when repeatedly running an already deployed project with different parameters. Usage node scripts/run-project.js <工程名> [--main <入口.js>] [--args '<json>'] [--wait 秒](Same task-order model as run-task.js: by default it waits 30 seconds for a receipt,--wait 0 and returns taskId immediately).
  • scripts/update-phone-client.js:One-click update on PC and restart of the phone-side resident client (no manual work needed). After changing phone-side code, run it to complete "push the new version directly to scripts-from-computer/client/(overwrite) + trigger self-restart". Usage node scripts/update-phone-client.js(by default uses the client bundled with the skill), or --path <其他位置的客户端.js>(data parameter, not a script path). For the mechanism, see the "Auto-update phone client" section.
  • scripts/delete-project.js:Delete a project already deployed on the phone (for cleanup)。node scripts/delete-project.js demo(deletes scripts-from-computer/project/demo) or --path <任意 sdcard 存储内路径>. The phone side has safety guardrails: the passed-in path is first normalized (resolving symlinks and ..), and only paths within sdcard storage are allowed.
  • scripts/tasks/<name>/:Task template library, one folder per task, containing <name>.js(script) and TASK.md(instructions). When AI plans, it first runs scripts/scan-tasks.js to scan the name+description of all templates and pick a template (the historical measured duration of successful tasks is automatically appended to the end of the description as [平均 X 秒]—trimmed mean: remove the largest 1/3 and smallest 1/3, then take the average, excluding cold-start/popup interference outliers; rolling retention of the most recent 10 runs). Only after selecting does it Read the full text of the corresponding tasks/<name>/TASK.md . For developers creating/extending templates, see references/AI_新建模板手册.md(AI template-building master manual: decision process + category skeleton + self-check checklist; copy it to extend).
  • scripts/scan-tasks.js: scan the task template library, extracting only the name+description from the preamble of each TASK.md .node scripts/scan-tasks.js outputs JSON(for AI planning, most token-efficient);node scripts/scan-tasks.js --human outputs "total N + readable list"(for user reading aloud). Relay GET /templates also reuses the same scanning logic.

Reference documents (references)

<skill_dir>/references/ contains the authoritative references for the phone-side AutoJs6 API and template building; consult as needed (before writing/modifying scripts, check the corresponding document to avoid pitfalls):

  • references/autojs6_device_docs.md — Device module:device.width/device.height for resolution,getBattery() battery,getMusicVolume() volume, brightness,vibrate() vibration, etc. If you need screen width/height or device info, check it.
  • references/AutoJs6_Engines_引擎文档.md — UiSelector / control-locating API:text()/id()/bounds()/desc() selectors and engines engine management. Essential when writing control-based scripts such as inspect_control_*/tap-text .
  • references/autojs6_images_doc.md — images image module:captureScreen() screenshot,images.clip() crop,images.toBytes() convert to bytes. Essential when writing screenshot/crop-screenshot .
  • references/AutoJS6-OCR文档.md — OCR text recognition: parameters and usage for screen text extraction. Essential when writing ocr templates.
  • references/autojs6_http_doc.md — http network request module: usage for sending HTTP requests (used under the hood when the phone side sends data back).
  • references/AutoJS6_Files_Doc.md — files file read/write module:files.read()/files.write() , etc.; templates reading the __TASK_ARGS_PATH parameter file injected by the task order depend on it.
  • references/现场脚本规范.md — On-site script conventions(one of the three main sections of the original text, read as needed): temp/ one-off script conventions, standard receipt skeleton, high-frequency pitfalls of creating files vs creating directories.
  • references/部署真实工程.md — Deploy a real project: multi-file project structure, the full deploy/run/delete-project workflow and hard constraints.
  • references/自动更新手机客户端.md — Auto-update phone client: update-phone-client bootstrap mechanism and unresponsive troubleshooting.
  • references/截图权限与弹框处理.md — Screenshot permission and the "Start Now" popup: any captureScreen()/ocr() scripts must have the built-in prerequisite code of "tap Start Now in the background + requestScreenCapture + sleep"; add it according to this document.
  • references/获取电脑局域网IP.md — Get the PC's LAN IP: how to obtain the IP to enter when the phone connects to the PC relay (the script already automatically excludes virtual network adapters / Clash fake-ip).
  • references/手机端常驻客户端说明.md — Phone-side persistent client and connection-status floating ball: client responsibilities and command list, floating ball's three colors (red = not connected / green = connected / blue = task executing), breathing/dragging/snapping behavior, adjustable constants, and the troubleshooting note that "the floating ball always being blue during screenshotting is normal". Consult when modifying client code or adjusting the floating ball.
  • references/手机连接排障.md — Five-step troubleshooting for phone not connected: get IP, change SERVER_IP, run the client on the phone, permission/WiFi check, connection confirmation (full version of "handling when the phone is not connected").
  • references/AI_新建模板手册.md — AI new/extend template master manual: decision flow + category skeleton + core specifications + self-check list; copy it to extend new templates (see the reference in the "On-site Script Specifications" section for details).
  • references/AI_AutoJS编码强制规范.md — Must-read before writing any phone-side JS script: strict ES5 (var only), no sleep/time-consuming operations on the UI thread, time-consuming APIs must threads.start multithreading, low-level hard constraints such as color literal int overflow, plus a pre-coding self-check list. The most frequent "silent crashes/freezes/timeouts" in this skill all stem from violating it,read before writing/modifying JS。
  • references/引擎_self_识别与isSelf判定.md — Determine whether "an engine is itself" (isSelf / self-protection): why you cannot use eng === myEngine, the correctsingle-id elementdetermination method (the old id+filename dual-element method is deprecated), defensive writing style and temporary-name pitfalls, with the id/source/cwd three-field real-device measured evidence table. Any time you write list-running-scripts / stop-script-by-id / any template that needs to be marked self or self-protected must check this.
  • scripts/autojs代码参考例子/ — AutoJS JS code example library: a batch of .js examples you can directly refer to. In the future, when writing/modifying phone-side JS scripts and encountering API usage or writing-style questions, go to scripts/autojs代码参考例子/ and browse by category

Execution SOP (AI must follow this order)

Installing dependencies is a prerequisite

cd scripts && npm install

Step 0: health check

curl -s http://localhost:9421/health 2>/dev/null | grep -q '"status":"ok"'
  • Command fails/no response → relay not started:node <skill_dir>/scripts/autojs-relay-server.js start it, wait about 1 second, then run health again to confirm (if already running, never restart—restarting triggers port cleanup and kicks off already-connected phones; the script has built-in fingerprint self-protection as a fallback).
  • {"phone":"connected"} → proceed to step 1.
  • {"phone":"disconnected"} → follow "handling when the phone is not connected".

Step 1: break down the task (two-level reading to save tokens)

Break the user's one sentence into a chain of steps.First run node <skill_dir>/scripts/scan-tasks.js to get the name+description of all templates and pick a template (see "Token-saving design" #5 for the two-level reading principle);only after selecting a template, Read its tasks/<name>/TASK.md full text, understand the scenario/pitfalls/fallbacks, and if necessary modify the script according to "temporary copy first" before deploying. Only for steps that templates cannot cover, write a script on the spot (write to <skill_dir>/temp/, follow "On-site Script Specifications").

Step 2: deploy step by step (templates first)

run-task.js is already cwd-independent,Recommended to call directly by template name; no need to piece together tasks/... long paths(the template name is automatically resolved to tasks/<name>/<name>.js and delivered to the phone to download by name):

# 推荐:先 cd 到技能目录(<skill_dir> = 本技能根目录,即 autojs-task-runner-yashu/;用相对路径,别写死你电脑的绝对路径),再用相对路径调脚本 + 模板名
cd <skill_dir> && \
  node scripts/run-task.js open-app --args '{"name":"MT管理器"}'
# 等价写法:node scripts/run-task.js --path tasks/open-app/open-app.js --args '{"name":"MT管理器"}'
# 等价写法:node scripts/run-task.js --name open-app --args '{"name":"MT管理器"}'

**Path hard rules (must be memorized; the easiest pitfall in this skill)**: the separator is always a forward slash / backslashes are forbidden \; the node parameter forbids /c/... -style absolute paths; for task parameters, write only the template name, do not manually concatenate tasks/...— the sole authoritative explanation of these three is in the "Conventions" section above.

  • Response {"success":true,"taskId":"...","status":"success","result":"{\"ok\":1}"} → this step succeeded; continue to the next step (result is a JSON string and needs to be parsed once more; taskId is the task ticket number, referenced when reporting errors and troubleshooting).Note status field is the task ticket lifecycle status(submitted/running/success/failed/stopped):success only means "the task ticket ran to completion normally and a receipt was obtained";business success or failure depends on result in ok— for example, when startup fails (script 404), status is still success while result.ok=0、result.phase="client"; if the relay side's submission timeout circuit breaker triggers, then result.phase="relay";
  • {"ok":0,"err":"..."} → immediately switch strategy and retry once (e.g., if tap-text cannot be found, take a screenshot to check the screen); if it still fails, terminate and report to the user which step it got stuck on;
  • Long/short task determination (rule of thumb): ① A single atomic operation (click/swipe/keypress/screenshot/open app, etc., constant duration and <30 seconds) → short, use the default synchronous wait; ② Duration varies with scale (batch N files/apps) or is unpredictable (recording a lesson/waiting for UI conditions/large file transfer) → long; ③ If unsure → use the default (timeout is not failure; taskId takes over); ④ At the end of the template description, [平均 X 秒] is the historical measured duration (trimmed mean, automatically recorded for successful tasks); judge directly based on it;
  • Long task submission (after determining it is long): add --wait 0 to return taskId immediately without blocking the conversation:
    • node scripts/run-task.js --status <taskId> check status/progress/heartbeat (in the template, use the injected __reportProgress("3/10 下载中") to report progress; for the long task template skeleton, see tasks/long-task-demo/);
    • if stuck, you can node scripts/run-task.js --stop <taskId> force-kill it;--list list recent task tickets — results are persisted on the relay computer,and can still be retrieved later by ticket number after the session ends;
    • During the waiting period, the phone can concurrently perform other operations as usual (screenshot/tap/transfer files).
  • After a UI navigation-type step, insert a step wait(500~1500ms) before operating the next step.
  • Short tasks wait 30 seconds by default, then output {"success":false,"timeout":true,...} → note that this is not a failure: the task is still running on the phone, and you can use --status to continue querying (if it is still not accepted by the phone 60 seconds after submission, the relay will trip to failed,err contains "submission timeout circuit breaker",result.phase="relay", and the task ticket will not hang forever). If the task ends as failed and err is "engine has exited but no receipt was received" → the script crashed silently; the first suspect isreceipt did not go through autojs_result broadcast(plain console.log will not be sent back) — for the mechanism, standard receipt skeleton, and the "broadcast immediately after creation" rule, see references/现场脚本规范.md, this is the most common mistaken-assumption error; during troubleshootingrule it out first。
  • Oversized receipt handling (resultTruncated:true): at this point the receipt has only a 400-character preview + the full-text persistence path (resultFile),It is strictly forbidden to read the entire persisted full text into the context——Use grep / line offsets to fetch only the needed fragments. Intelligent routing: ① Intermediate steps with clear intent (e.g. "find the coordinates of a certain button") → directly grep the persisted file to extract fragments and continue the task; in the final report state "results exceeded the limit and were extracted as needed; the full text is at <path>"; ② Unclear intent, or the full text itself is the deliverable → Ask the reverse questionUser chooses: just see the summary / hand the full-text file to the user / change keywords and filter to re-fetch / switch to local means (crop region screenshot, region OCR) and redo.

Step 3: Look at the screen (when needed)

curl -s http://localhost:9421/screenshot

Returns {"success":true,"path":"<PC本地路径>"}, use Read to read the PNG to judge the interface state, then decide the next step. Screenshots do not pass through the AI context; only reading an image consumes tokens, so use it as needed.

  • Look only at a local region: if the bounding box of a control is already known (e.g. via inspect_control_by_* detected bounds), use the crop-screenshot template to crop that region by the left/top/right/bottom quadruple and send back the computer PNG (looking at "what a certain control looks like" is the most economical, better than taking the whole screen and then visually finding the block). For invocation see its TASK.md。
  • The colored dot in the corner of a screenshot is not a malfunction: the round glowing dot on the right edge of the phone screen is the "connection status floating ball" (red = disconnected, green = connected, blue = task executing). Taking a screenshot is itself a task (the client turns blue first and then captures the screen), soin every screenshot taken through the relay it is necessarily blue, which is normal; do not misjudge the connection or task state based on it. For details see references/手机端常驻客户端说明.md。

Whether the screenshot result must be shown to the user —— two cases (key distinction):

  • Case A · The user explicitly requests a screenshot / screenshot, or the screenshot itself is the deliverable:/screenshot and crop-screenshot return the absolute path of the local PC PNG(in the form {"success":true,"path":"..."}). This is ahard rule, cannot be omitted——AI cannot just throw the receipt to the user and end, must: ① use present_files to present the PNG as a result file (generate a preview/card) so the user can directly see that image; ② even if the preview cannot be opened, the absolute path of the image must still be included in the reply body to ensure the user can open and view it themselves.
  • Case B · The screenshot is only for AI internal verification(e.g. "after opening Douyin, take a screenshot to confirm whether it really opened"): at this time the screenshotdoes not need to be pushed to the user. After the AI itself Read / ocr judges the interface state, it only needs to give theverification conclusionin the final report (e.g. "I have opened Douyin for you" / "it did not open, stuck at XX"). In this case not showing the imagedoes not count as violating the hard rule——the user wants the result, not that intermediate screenshot. Judgment principle: is the screenshot "for the user to see" or "for yourself to verify"?The former must be shown; the latter only reports the conclusion. Showing it does not conflict with the AI itself Reading to judge the next step; both can be done at the same time.

Regarding "whether the AI can read images"——test according to the actual situation, do not make arbitrary judgments: the capabilities of the AI models used by the caller vary greatly——some are multimodal and can directly read images/videos, while some only support text and image reading will be filtered (manifested as Content filtered or inability to read the content). Accordingly:

  • Case A "Show the image to the user" is a hard rule, unrelated to AI capability: the user can always see the image, so regardless of whether the AI itself can read images, it must present_files + attach the absolute path;
  • "AI reads the image to determine the next step" is best-effort(only affects the validation method of Case B): if the AI is unsure whether it can read images, first Read test it once—if it can recognize, make decisions based on that; if it gets filtered, switch to using ocr to get text / inspect_control_* to get controls / or just ask the user,do not assert "the model supports/does not support" based on a single error or subjective assumption;
  • base it on actual testing: this skill does not presuppose that the AI must be able to read images, nor does it assume it cannot.

Hard rule for landing files (text / data-type deliverables, same level as screenshots, cannot be omitted): whenever task execution on the PCgenerates a file(in the script receipt's file field, or an export file the AI itself Wrote), in the final report the AImustwrite out the file'sabsolute path in full(with drive letter, in the form D:/xxx/yyy.txt, forward slashes or backslashes are both acceptable, but it must be complete and directly locatable),must not write only the file name (such as user_apps_xxx.txt) or only give a relative path. At the same time still use present_files to present the file as a result (open preview/card). Reason: skill outputs often land in hidden directories such as .workbuddy/ , and only with the complete absolute path can the user copy it into the Explorer address bar to find the file; if only given the file name, they have no way to locate it. Applicable: all templates that "write results to disk" (such as get-all-apps landing the user's App list), screenshot files, export files the AI itself lands on disk, etc. If the receipt includes a file field, just copy that value as-is to the user.

The three priorities for recognizing screen content (fallback chain)

The essence of "looking at the screen" isreading out the text/state on the screen and handing it to the AI to determine the next step. The priorities are ordered bytoken savings + reliability—use the cheapest, most stable method first, and only fall back to a more expensive solution when the previous layer is insufficient, do not skip levels, do not get stuck at a certain level:

  1. Level 1 · Control tree text extraction (highest priority): use the inspect_control_by_* template to traverse the UI control tree, retrieving all controls' text/id/desc。
    • Applicable: when you need to determine "whether the interface really changed and which control changed", or when you need structured control semantics;
    • Features: more accurate than OCR (controls carry their own semantics), and does not require a large model to read images,most token-efficient; if the returned data is larger than needed, filter as required;
    • Fallback condition: if the control tree isincomplete / cannot retrieve the target control / cannot satisfy the judgment need, or the APP has blocked accessibility and control information cannot be obtained, then fall back to Level 2.
  2. Level 2 · OCR text extraction: call the ocr template to recognize screen text (a plain text array, usually also very token-efficient).
    • Applicable: when the model does not need to read images, but only needs "what text is on the screen" to make a judgment;
    • Screens with lots of text (settings pages/list pages) should preferentially include {"query":"关键词"} filtering, and only take the matching items;detail:true use it only when per-item coordinates/confidence are truly needed—both are to prevent a large response from blowing up the context;
    • Fallback condition: if the OCR resultis insufficient to determine whether the interface has actually changed(e.g., the interface is purely graphical, with no obvious text), oris insufficient to meet the user's needs(the user wants structure/controls rather than text), → fall back to the third level.
  3. Third level · screenshot + model direct reading (lowest priority): use GET /screenshot or crop-screenshot to screenshot back to the PC, letting the model directly read the image to recognize the content.
    • Applicable: only when the first two levels are both unsatisfied and the caller's modelsupports multimodality(can read images/videos), use this layer;
    • Characteristics: most complete information, most worry-free, butmost expensive, most token-consuming, so it is placed last and not used as the default solution;
    • Fallback condition: if the modeldoes not support multimodality(image reading is filtered / Content filtered / content cannot be read), this level is unavailable, and when the first two levels also both fail, ask the user directly; do not guess blindly.

Fallback operation principles:

  • Test level by level from the first level downward:Can the control tree satisfy it? Can OCR satisfy it? Only when the first two levels are both not viable, then use screenshot + model direct reading;
  • Used forinternal verificationscreenshots / OCR / control results (case B), only the conclusion needs to be given in the final-state report (e.g., "Douyin has been opened"); it is not mandatory to push the image to the user;Only when the user explicitly asks to see it, or the screenshot is the deliverable (case A), must the hard rule of "show to the user" be followed (present_files + attach the absolute path). The same applies to OCR and control results: for internal verification, only report the conclusion; only show it if the user wants to see it.
  • When even the third level cannot obtain usable information, ask the user directly; do not guess blindly.

Step 4: one-time final-state report

All steps completed → report the result in one sentence (e.g., "I have helped you open Settings and connect to WiFi"); failure → report which step it got stuck at, the reason, and suggest how the user can manually remedy it.Do not disturb the user during the intermediate process.

User Q&A: query template list (do not execute tasks)

When the user asks "how many phone task templates are there now"/"what templates are there"/"list all phone tasks", etc.pure queryquestions,do not go through the execution SOP, directly run the scanner and read out the result (no phone online required, does not consume phone task tokens):

node <skill_dir>/scripts/scan-tasks.js --human

Output in the form:

共 8 个手机任务模板:
· tap-point — 按屏幕坐标 (x,y) 点按
· tap-text — 点按含指定文字的控件
...

Just read this list to the user directly. If an external program/relay needs to fetch the data, you can also curl -s http://localhost:9421/templates get JSON {count, tasks}, but in the user Q&A scenario, preferentially use the --human command line; the result is friendlier.

Note: this is a "read-only query"—do not start a task for it, do not issue any scripts, and do not require the phone to be online.

On-site script specification (one-off scripts · when templates cannot cover it)

One-off scripts written on the spot by AI are written to <skill_dir>/temp/(at delivery time the PC reads the content and "inlines it directly", location-independent; after landing on the phone it uniformly goes into scripts-from-computer/single/), use node scripts/run-task.js <文件> --args '{...}' to deliver.Before writing, you must first read references/现场脚本规范.md(strict ES5/var only, no time-consuming operations on the UI thread, time-consuming APIs go into threads.start, standard receipt skeleton, the high-frequency pitfalls of creating a file vs creating a directory—the most frequent "silent crash/no receipt received" all stem from violating it). To create a new reusable template, read references/AI_新建模板手册.md。

Deploy a real project (multiple files + resources)

For complex tasks (multi-module / with image and audio resources), use deploy-project to deploy the entire project directory to the phone scripts-from-computer/project/<名>/ and automatically run the entry; PC projects are uniformly placed in <skill_dir>/scripts/autojs-project/<工程名>/. Hard constraints: strict ES5/var only (if require encounters ES6 the entire entry silently crashes), the entry must end with a standard receipt, modules use relative require.For the complete structure and usage, see references/部署真实工程.md。

Automatically update the phone client (one click from PC, no manual work)

When phone-side code (autojs-task-phone-client.js) is changed, on the PC run node scripts/update-phone-client.js to complete in one click "deliver the new version directly to scripts-from-computer/client/ + trigger self-restart", with no manual work needed.For the mechanism and freeze troubleshooting, see references/自动更新手机客户端.md。

Handling when the phone is not connected (AI must take the initiative)

Trigger condition: in step 0, health returns {"phone":"disconnected"} or the relay is not up.Never just throw out a single line "please connect the phone"—follow the five-step process in references/手机连接排障.md to proactively guide the user through connecting: ① the AI runs get-lan-ip.js itself to get the LAN IP → ② guide the user to change the client var SERVER_IP → ③ run on the phone in AutoJs6 autojs-task-phone-client.js → ④ same WiFi + four permissions (accessibility/floating window/background/screenshot) → ⑤ after the user reports "connected", recheck health. For the complete steps and the wording to give the user, see that document; note: relay restarts/network blips will self-heal and reconnect, and only the death of the client engine requires this process.

Security boundaries (red lines)

  • Payment, transfer, deletion, authorization-type irreversible operations: even in fully automatic mode, before executing that step you must first confirm with the user once;
  • one UI task at a time: although run-type task lists support concurrency (receipts are routed back by taskId without crossing lines), the phone screen can only do one thing at a time, so UI automation tasks must still be delivered serially step by step and must not be concurrent; synchronous short operations such as screenshots/updating the client/deleting projects are still mutually exclusive on the relay side (concurrency will return 429).

Notes

  • Recognizing screen-off/lock-screen symptoms (high-frequency misjudgment point): screenshot returnall-black frame + clipboard read/writereturns empty + abnormal OCR content, while open-app and other "fake successes" (am start does not require the screen to be on)—there is a high chance the phone is screen-off/auto-locked; ask the user to wake and unlock it and retry, and do not investigate in the direction of a script bug.
  • Phone-side scripts are strictly ES5 / var only: the AutoJs6 Rhino engine does not fully support ES6, and this is a hard constraint.
  • On-site scripts should not use setInterval to stay alive: scripts delivered via /run are managed by the relay, and a normal exit is enough; only resident clients need setInterval to stay alive.
  • Performance: steps involving UI tree searches (tap-text, etc.) are a bit slower when the interface hierarchy is deep; this is normal, just wait for the receipt.
  • Screenshot scripts must have permission prelude code built in: for any script that calls captureScreen() / ocr(...) on mobile, the very beginning must contain "background thread uses textMatch regex multi-candidate to automatically dismiss the authorization dialog (button text varies by ROM: Xiaomi "Start Now", some models "Start Screenshot", etc.) + requestScreenCapture() + sleep(500)" this code block, otherwise the screenshot fails or the script hangs (manifesting as "empty receipt"). The template screenshot / crop-screenshot / ocr / tap-text / unfollow-official-accounts already has it built in; when creating a new screenshot template, add it as in references/截图权限与弹框处理.md .