Purpose
Treat the AI on the PC as the "brain" and AutoJS on the phone as the "hands": the user describes in one sentence what they want done on the phone, the AI breaks it down into a chain of steps and sends them step by step to the phone for execution,only reporting back to the user once when everything is done or has failed。
User-side usage tutorial:
<skill_dir>/使用示例.md— a scenario-organized collection of examples of "what the user says to the AI" (controlling the phone / viewing the screen / transferring files / long tasks / accumulating templates and projects / safety boundaries). When a new user uses it for the first time, or when the user asks "what can it do / how do I use it", proactively recommend this document.
Typical user group: users who can speak and type but have limited mobility—speech-to-text → AI → phone, indirectly controlling the phone.
Token-saving design (runs through this skill, must be followed)
This skill is aimed at long-term high-frequency users, and AI computing power costs money,every round of interaction must save tokens:
- Pass only the path, not the code: keep a copy of the template script on the PC (
scripts/tasks/), and when sending, pass onlypath+args(a few dozen bytes); the phone downloads and executes it by name from the relay; - Minimal receipt: by default templates return only
{ok:1}/{ok:0, err:"原因"}. The AI must not require the script to send back the entire UI tree or large blocks of text; - Fully automatic continuous run: intermediate steps produce no user conversation, executing silently step by step, with a single report at the final state;
- Screenshots instead of screen reading: when you need to "see" the screen, use
GET /screenshot(binary directly transferred to the PC and written to disk, not passing through the AI context), and the AI then uses Read to read the image. The phone is forbidden to send backthe entire UI control tree JSON (i.e., the raw large JSON dumping all on-screen controls); but it is allowed to useinspect_control_*and other templatesto structurally retrieve onlythe neededtext/id/desc, see the third level of the fallback chain in "Step 3". (Note:Readreading images depends on whether the caller's model is multimodal—non-multimodal models will have image reading filtered out, which is normal; handle it according to the rules in "Step 3: Viewing the Screen": firstpresent_filespresent the image to the user, and if the AI itself cannot read it, useocr/inspect_control_*/ask the user instead; do not arbitrarily judge the model's capabilities.) - Two-level template reading: when planning, run only
node scripts/scan-tasks.jsto get the name+description of all templates (do not read the scripts, do not read the body); only after selecting a template, Read itstasks/<name>/TASK.mdfull text. The description is the basis for selection, and the body is the decision contract. - 2K text budget: before task receipts enter the context,
run-task.jsstrictly enforces this—anything over 2000 characters is automatically truncated to "a 400-character preview + the full-text disk path" (scripts/task-results/<taskId>.txt,resultTruncated:true); for large on-screen text from OCR, prefer using{"query":"关键词"}to filter and reduce at the source (see the ocr template).
Prerequisites (four permissions on the phone side, none may be missing)
The AutoJS6 app must be installed on the phone. AutoJS6 official website:https://docs.autojs6.com/#/
- Accessibility service: Settings → Apps → AutoJs6 → Accessibility → enable;
- Screenshot permission: Automatically requested when the phone-side script starts; in the popup, tap "Start Now";
- Floating window permission: Settings → Apps → AutoJs6 → Permission management → Floating window → Allow;
- Background running: Settings → Apps → AutoJs6 → Set battery saver policy to "Unrestricted", and allow auto-start in auto-start management (otherwise, after locking the screen or switching to the background, the resident client is killed by the system and the task is interrupted).
Convention
<skill_dir>= this skill's root directory (the directory containing SKILL.md).- One-off scripts written on the spot by the AI are written to the
<skill_dir>/temp/directory. - On the AI side, to write AutoJS multi-file projects (projects that need to be deployed to the phone), write them uniformly to the
<skill_dir>/scripts/autojs-project/<工程名>/directory (one subdirectory per project; for the internal structure, refer to theautojs-min-project-templatetemplate in the same directory). This directory already has theproject1/slot preset and ready to use,scripts/autojs代码参考例子/autojs-projects/ColorWheelthis is an autojs color wheel interface project for reference. One-off single-file temporary scripts are still placed intemp/(see "On-the-spot script specification"). For hard constraints on project coding (ES5 / var only, etc.), see "Deploying real projects → Project code writing specification". - Default port 9421. Changes must be synchronized in three places: ①
curl localhost:9421/health;②autojs-task-phone-client.js'sSERVER_PORT;③run-task.js'sDEFAULT_URL。 - Shell path convention (hard rule): Commands issued locally go through Git Bash(POSIX shell), and the path separator is alwaysforward slash
/(e.g./c/Users/...)。It is strictly forbidden to write Windows backslashes in commands\——POSIX shell treats\as an escape character and will silently swallow it, causingskills\autojsto becomeskillsautojs, path concatenation errors, andcd/command cannot find the file.It is recommended to firstcdinto the skill directory (in POSIX forward-slash form; verified usable in this environment), then use the relative pathscripts/run-task.jsto call the script + write only the template name, so that not even thetasks/...long path needs to be assembled. - MSYS argument rewriting pitfall (measured in this environment): Git Bash willarguments starting with
/Parameters at the beginning(e.g.--target-dir /sdcard/Download) automatically rewrite to Windows paths (measured:/sdcard/Downloadwas changed toD:/software/Git/Git/sdcard/Download; the phone side assembled an invalid path and the delivery failed). For--target-dir、delete-folder --pathsuchphone path argumentspassed in/sdcardform, the command must be prefixed withMSYS_NO_PATHCONV=1(e.g.MSYS_NO_PATHCONV=1 node scripts/pc-to-phone.js a.txt --target-dir /sdcard/Download)。 - node argument path pitfall (measured in this environment): whenthe script's absolute path is passed directly as a node argument,
/c/Users/...this kind of POSIX form will be treated by the Windows version of node as a "relative path under the current drive root" (e.g. resolved tod:\c\Users\...) and the file cannot be found. The correct approach is as above: firstcdinto the skill directory, thennode scripts/run-task.js ...; if you must pass an absolute script path to node, use theC:/Users/.../run-task.jswith a drive letter prefix, do not write/c/Users/.../run-task.js。
Phone-side directory architecture v3 (scripts-from-computer)
Phone-side AutoJS script root directory (files.getSdcardPath() + "脚本" dynamically concatenated,hardcoding physical paths is prohibited across the whole chain) under:
- user-written scripts: permanently stay in the root directory; this skill never touches them (the skill itself also leaves no files in the root directory);
scripts-from-computer/: quarantine zone for all artifacts this skill delivers from the PC—client/resident client itself (self-updates land directly, can be started manually; when the client starts it "self-returns" here);single/template single script + temporary probe script (deployed by run-task);project/<工程名>/AI project works (deploy-project deployment, run-project execution, delete-project cleanup);data/companion data (task-args/<taskId>.jsonper-request parameters,relay-config.jsonrelay address,beep.wavand other resources);files/default destination for general files sent from PC to phone (when pc-to-phone / send-file-to-phone does not explicitly specify--target-dir).
- Naming convention: folders/files created by this skill are always lowercase kebab-case, to avoid file-not-found issues caused by case ambiguity.
Script list
scripts/autojs-relay-server.js: PC-side relay service (HTTP + WebSocket, Node.js ES6), AI-started. For architecture details seescripts/relay/ARCHITECTURE.md。scripts/autojs-task-phone-client.js: phone-side resident client (strictly ES5, all variables declared with var), which the user copies to the phone to run in AutoJs6. Keep-alive, reconnect on disconnect, download and execute scripts by name, return results.scripts/run-task.js: AI-side bridge script—POSTs template name/script file + parameters to the relay/run(asynchronous task ticket); by default it polls the task ticket for up to 30 seconds waiting for a receipt and prints it (outer shape compatible with older versions); for long tasks add--wait 0to return the taskId immediately. For the four invocation methods and hard path rules see "Step 2"; task ticket control:--status <taskId>(query status/progress/result, queryable across sessions),--stop <taskId>(force kill),--list [数量](list recent task tickets). Running without parameters prints the full usage.scripts/self-test.js:One-click regression self-test—runs health check, parameter/clipboard round-trip, screenshot compression, delete-path guardrails, full project lifecycle, and task ticket listing all at once, outputting a ✅/❌ summary and exit code. After changing the relay/client/template, just run it to prove you haven't broken anything (node scripts/self-test.js)。scripts/get-lan-ip.js: get the computer's LAN IP (node <skill_dir>/scripts/get-lan-ip.js --first), used when filling in the IP on the phone. Seereferences/获取电脑局域网IP.md。scripts/pc-to-phone.js:Transfer any file from the computer to the phone(one-click entry for the PC→phone channel, a high-frequency must-have).node scripts/pc-to-phone.js <本地文件> [--target-dir <手机目录>] [--target-name <手机文件名>] [--keep]Usage notes: paths are always written relative to the skill directoryrelative paths;<本地文件>isdata parameter(the file to be transferred), don't confuse it with the node script path;--target-dirdefaults to being dynamically derived via /health asscripts-from-computer/files/(if explicitly specified, it goes exactly where specified),--keepkeeps a transit copy on the computer (by default cleaned up after the receipt). For the reverse direction (phone→computer) use the templatedownload-file。scripts/deploy-project.js:Deploy a multi-file AutoJS project (including images/audio and other resources) from the computer to the phone and run it—for complex projects (multi-module / with resources, which a single-file template cannot accommodate) use this instead of a single-file template. Usagenode scripts/deploy-project.js <工程目录> [--name <工程名>] [--main <入口.js>] [--run|--no-run] [--args '<json>'] [--keep], for parameter defaults and mechanism see the section "Deploying a real project (multi-file + resources)".scripts/run-project.js:Run a project already deployed to the phone (without re-delivering files). Use this when repeatedly running an already deployed project with different parameters. Usagenode scripts/run-project.js <工程名> [--main <入口.js>] [--args '<json>'] [--wait 秒](Same task-order model as run-task.js: by default it waits 30 seconds for a receipt,--wait 0and returns taskId immediately).scripts/update-phone-client.js:One-click update on PC and restart of the phone-side resident client (no manual work needed). After changing phone-side code, run it to complete "push the new version directly toscripts-from-computer/client/(overwrite) + trigger self-restart". Usagenode scripts/update-phone-client.js(by default uses the client bundled with the skill), or--path <其他位置的客户端.js>(data parameter, not a script path). For the mechanism, see the "Auto-update phone client" section.scripts/delete-project.js:Delete a project already deployed on the phone (for cleanup)。node scripts/delete-project.js demo(deletesscripts-from-computer/project/demo) or--path <任意 sdcard 存储内路径>. The phone side has safety guardrails: the passed-in path is first normalized (resolving symlinks and..), and only paths within sdcard storage are allowed.scripts/tasks/<name>/:Task template library, one folder per task, containing<name>.js(script) andTASK.md(instructions). When AI plans, it first runsscripts/scan-tasks.jsto scan the name+description of all templates and pick a template (the historical measured duration of successful tasks is automatically appended to the end of the description as[平均 X 秒]—trimmed mean: remove the largest 1/3 and smallest 1/3, then take the average, excluding cold-start/popup interference outliers; rolling retention of the most recent 10 runs). Only after selecting does it Read the full text of the correspondingtasks/<name>/TASK.md. For developers creating/extending templates, seereferences/AI_新建模板手册.md(AI template-building master manual: decision process + category skeleton + self-check checklist; copy it to extend).scripts/scan-tasks.js: scan the task template library, extracting only the name+description from the preamble of eachTASK.md.node scripts/scan-tasks.jsoutputs JSON(for AI planning, most token-efficient);node scripts/scan-tasks.js --humanoutputs "total N + readable list"(for user reading aloud). RelayGET /templatesalso reuses the same scanning logic.
Reference documents (references)
<skill_dir>/references/ contains the authoritative references for the phone-side AutoJs6 API and template building; consult as needed (before writing/modifying scripts, check the corresponding document to avoid pitfalls):
references/autojs6_device_docs.md— Device module:device.width/device.heightfor resolution,getBattery()battery,getMusicVolume()volume, brightness,vibrate()vibration, etc. If you need screen width/height or device info, check it.references/AutoJs6_Engines_引擎文档.md— UiSelector / control-locating API:text()/id()/bounds()/desc()selectors andenginesengine management. Essential when writing control-based scripts such asinspect_control_*/tap-text.references/autojs6_images_doc.md— images image module:captureScreen()screenshot,images.clip()crop,images.toBytes()convert to bytes. Essential when writingscreenshot/crop-screenshot.references/AutoJS6-OCR文档.md— OCR text recognition: parameters and usage for screen text extraction. Essential when writingocrtemplates.references/autojs6_http_doc.md— http network request module: usage for sending HTTP requests (used under the hood when the phone side sends data back).references/AutoJS6_Files_Doc.md— files file read/write module:files.read()/files.write(), etc.; templates reading the__TASK_ARGS_PATHparameter file injected by the task order depend on it.references/现场脚本规范.md— On-site script conventions(one of the three main sections of the original text, read as needed): temp/ one-off script conventions, standard receipt skeleton, high-frequency pitfalls of creating files vs creating directories.references/部署真实工程.md— Deploy a real project: multi-file project structure, the full deploy/run/delete-project workflow and hard constraints.references/自动更新手机客户端.md— Auto-update phone client: update-phone-client bootstrap mechanism and unresponsive troubleshooting.references/截图权限与弹框处理.md— Screenshot permission and the "Start Now" popup: anycaptureScreen()/ocr()scripts must have the built-in prerequisite code of "tap Start Now in the background + requestScreenCapture + sleep"; add it according to this document.references/获取电脑局域网IP.md— Get the PC's LAN IP: how to obtain the IP to enter when the phone connects to the PC relay (the script already automatically excludes virtual network adapters / Clash fake-ip).references/手机端常驻客户端说明.md— Phone-side persistent client and connection-status floating ball: client responsibilities and command list, floating ball's three colors (red = not connected / green = connected / blue = task executing), breathing/dragging/snapping behavior, adjustable constants, and the troubleshooting note that "the floating ball always being blue during screenshotting is normal". Consult when modifying client code or adjusting the floating ball.references/手机连接排障.md— Five-step troubleshooting for phone not connected: get IP, change SERVER_IP, run the client on the phone, permission/WiFi check, connection confirmation (full version of "handling when the phone is not connected").references/AI_新建模板手册.md— AI new/extend template master manual: decision flow + category skeleton + core specifications + self-check list; copy it to extend new templates (see the reference in the "On-site Script Specifications" section for details).references/AI_AutoJS编码强制规范.md— Must-read before writing any phone-side JS script: strict ES5 (var only), no sleep/time-consuming operations on the UI thread, time-consuming APIs mustthreads.startmultithreading, low-level hard constraints such as color literal int overflow, plus a pre-coding self-check list. The most frequent "silent crashes/freezes/timeouts" in this skill all stem from violating it,read before writing/modifying JS。references/引擎_self_识别与isSelf判定.md— Determine whether "an engine is itself" (isSelf / self-protection): why you cannot useeng === myEngine, the correctsingle-id elementdetermination method (the old id+filename dual-element method is deprecated), defensive writing style and temporary-name pitfalls, with theid/source/cwdthree-field real-device measured evidence table. Any time you writelist-running-scripts/stop-script-by-id/ any template that needs to be marked self or self-protected must check this.scripts/autojs代码参考例子/— AutoJS JS code example library: a batch of .js examples you can directly refer to. In the future, when writing/modifying phone-side JS scripts and encountering API usage or writing-style questions, go to scripts/autojs代码参考例子/ and browse by category
Execution SOP (AI must follow this order)
Installing dependencies is a prerequisite
cd scripts && npm install
Step 0: health check
curl -s http://localhost:9421/health 2>/dev/null | grep -q '"status":"ok"'
- Command fails/no response → relay not started:
node <skill_dir>/scripts/autojs-relay-server.jsstart it, wait about 1 second, then run health again to confirm (if already running, never restart—restarting triggers port cleanup and kicks off already-connected phones; the script has built-in fingerprint self-protection as a fallback). {"phone":"connected"}→ proceed to step 1.{"phone":"disconnected"}→ follow "handling when the phone is not connected".
Step 1: break down the task (two-level reading to save tokens)
Break the user's one sentence into a chain of steps.First run node <skill_dir>/scripts/scan-tasks.js to get the name+description of all templates and pick a template (see "Token-saving design" #5 for the two-level reading principle);only after selecting a template, Read its tasks/<name>/TASK.md full text, understand the scenario/pitfalls/fallbacks, and if necessary modify the script according to "temporary copy first" before deploying. Only for steps that templates cannot cover, write a script on the spot (write to <skill_dir>/temp/, follow "On-site Script Specifications").
Step 2: deploy step by step (templates first)
run-task.js is already cwd-independent,Recommended to call directly by template name; no need to piece together tasks/... long paths(the template name is automatically resolved to tasks/<name>/<name>.js and delivered to the phone to download by name):
# 推荐:先 cd 到技能目录(<skill_dir> = 本技能根目录,即 autojs-task-runner-yashu/;用相对路径,别写死你电脑的绝对路径),再用相对路径调脚本 + 模板名
cd <skill_dir> && \
node scripts/run-task.js open-app --args '{"name":"MT管理器"}'
# 等价写法:node scripts/run-task.js --path tasks/open-app/open-app.js --args '{"name":"MT管理器"}'
# 等价写法:node scripts/run-task.js --name open-app --args '{"name":"MT管理器"}'
**Path hard rules (must be memorized; the easiest pitfall in this skill)**: the separator is always a forward slash
/backslashes are forbidden\; the node parameter forbids/c/...-style absolute paths; for task parameters, write only the template name, do not manually concatenatetasks/...— the sole authoritative explanation of these three is in the "Conventions" section above.
- Response
{"success":true,"taskId":"...","status":"success","result":"{\"ok\":1}"}→ this step succeeded; continue to the next step (result is a JSON string and needs to be parsed once more; taskId is the task ticket number, referenced when reporting errors and troubleshooting).Notestatusfield is the task ticket lifecycle status(submitted/running/success/failed/stopped):successonly means "the task ticket ran to completion normally and a receipt was obtained";business success or failure depends onresultinok— for example, when startup fails (script 404),statusis stillsuccesswhileresult.ok=0、result.phase="client"; if the relay side's submission timeout circuit breaker triggers, thenresult.phase="relay"; {"ok":0,"err":"..."}→ immediately switch strategy and retry once (e.g., if tap-text cannot be found, take a screenshot to check the screen); if it still fails, terminate and report to the user which step it got stuck on;- Long/short task determination (rule of thumb): ① A single atomic operation (click/swipe/keypress/screenshot/open app, etc., constant duration and <30 seconds) → short, use the default synchronous wait; ② Duration varies with scale (batch N files/apps) or is unpredictable (recording a lesson/waiting for UI conditions/large file transfer) → long; ③ If unsure → use the default (timeout is not failure; taskId takes over); ④ At the end of the template description,
[平均 X 秒]is the historical measured duration (trimmed mean, automatically recorded for successful tasks); judge directly based on it; - Long task submission (after determining it is long): add
--wait 0to return taskId immediately without blocking the conversation:node scripts/run-task.js --status <taskId>check status/progress/heartbeat (in the template, use the injected__reportProgress("3/10 下载中")to report progress; for the long task template skeleton, seetasks/long-task-demo/);- if stuck, you can
node scripts/run-task.js --stop <taskId>force-kill it;--listlist recent task tickets — results are persisted on the relay computer,and can still be retrieved later by ticket number after the session ends; - During the waiting period, the phone can concurrently perform other operations as usual (screenshot/tap/transfer files).
- After a UI navigation-type step, insert a step
wait(500~1500ms) before operating the next step. - Short tasks wait 30 seconds by default, then output
{"success":false,"timeout":true,...}→ note that this is not a failure: the task is still running on the phone, and you can use--statusto continue querying (if it is still not accepted by the phone 60 seconds after submission, the relay will trip to failed,errcontains "submission timeout circuit breaker",result.phase="relay", and the task ticket will not hang forever). If the task ends as failed and err is "engine has exited but no receipt was received" → the script crashed silently; the first suspect isreceipt did not go throughautojs_resultbroadcast(plainconsole.logwill not be sent back) — for the mechanism, standard receipt skeleton, and the "broadcast immediately after creation" rule, seereferences/现场脚本规范.md, this is the most common mistaken-assumption error; during troubleshootingrule it out first。 - Oversized receipt handling (
resultTruncated:true): at this point the receipt has only a 400-character preview + the full-text persistence path (resultFile),It is strictly forbidden to read the entire persisted full text into the context——Use grep / line offsets to fetch only the needed fragments. Intelligent routing: ① Intermediate steps with clear intent (e.g. "find the coordinates of a certain button") → directly grep the persisted file to extract fragments and continue the task; in the final report state "results exceeded the limit and were extracted as needed; the full text is at <path>"; ② Unclear intent, or the full text itself is the deliverable → Ask the reverse questionUser chooses: just see the summary / hand the full-text file to the user / change keywords and filter to re-fetch / switch to local means (crop region screenshot, region OCR) and redo.
Step 3: Look at the screen (when needed)
curl -s http://localhost:9421/screenshot
Returns {"success":true,"path":"<PC本地路径>"}, use Read to read the PNG to judge the interface state, then decide the next step. Screenshots do not pass through the AI context; only reading an image consumes tokens, so use it as needed.
- Look only at a local region: if the bounding box of a control is already known (e.g. via
inspect_control_by_*detected bounds), use thecrop-screenshottemplate to crop that region by theleft/top/right/bottomquadruple and send back the computer PNG (looking at "what a certain control looks like" is the most economical, better than taking the whole screen and then visually finding the block). For invocation see itsTASK.md。 - The colored dot in the corner of a screenshot is not a malfunction: the round glowing dot on the right edge of the phone screen is the "connection status floating ball" (red = disconnected, green = connected, blue = task executing). Taking a screenshot is itself a task (the client turns blue first and then captures the screen), soin every screenshot taken through the relay it is necessarily blue, which is normal; do not misjudge the connection or task state based on it. For details see
references/手机端常驻客户端说明.md。
Whether the screenshot result must be shown to the user —— two cases (key distinction):
- Case A · The user explicitly requests a screenshot / screenshot, or the screenshot itself is the deliverable:
/screenshotandcrop-screenshotreturn the absolute path of the local PC PNG(in the form{"success":true,"path":"..."}). This is ahard rule, cannot be omitted——AI cannot just throw the receipt to the user and end, must: ① usepresent_filesto present the PNG as a result file (generate a preview/card) so the user can directly see that image; ② even if the preview cannot be opened, the absolute path of the image must still be included in the reply body to ensure the user can open and view it themselves.- Case B · The screenshot is only for AI internal verification(e.g. "after opening Douyin, take a screenshot to confirm whether it really opened"): at this time the screenshotdoes not need to be pushed to the user. After the AI itself
Read/ocrjudges the interface state, it only needs to give theverification conclusionin the final report (e.g. "I have opened Douyin for you" / "it did not open, stuck at XX"). In this case not showing the imagedoes not count as violating the hard rule——the user wants the result, not that intermediate screenshot. Judgment principle: is the screenshot "for the user to see" or "for yourself to verify"?The former must be shown; the latter only reports the conclusion. Showing it does not conflict with the AI itself Reading to judge the next step; both can be done at the same time.Regarding "whether the AI can read images"——test according to the actual situation, do not make arbitrary judgments: the capabilities of the AI models used by the caller vary greatly——some are multimodal and can directly read images/videos, while some only support text and image reading will be filtered (manifested as
Content filteredor inability to read the content). Accordingly:
- Case A "Show the image to the user" is a hard rule, unrelated to AI capability: the user can always see the image, so regardless of whether the AI itself can read images, it must
present_files+ attach the absolute path;- "AI reads the image to determine the next step" is best-effort(only affects the validation method of Case B): if the AI is unsure whether it can read images, first
Readtest it once—if it can recognize, make decisions based on that; if it gets filtered, switch to usingocrto get text /inspect_control_*to get controls / or just ask the user,do not assert "the model supports/does not support" based on a single error or subjective assumption;- base it on actual testing: this skill does not presuppose that the AI must be able to read images, nor does it assume it cannot.
Hard rule for landing files (text / data-type deliverables, same level as screenshots, cannot be omitted): whenever task execution on the PCgenerates a file(in the script receipt's
filefield, or an export file the AI itself Wrote), in the final report the AImustwrite out the file'sabsolute path in full(with drive letter, in the formD:/xxx/yyy.txt, forward slashes or backslashes are both acceptable, but it must be complete and directly locatable),must not write only the file name (such asuser_apps_xxx.txt) or only give a relative path. At the same time still usepresent_filesto present the file as a result (open preview/card). Reason: skill outputs often land in hidden directories such as.workbuddy/, and only with the complete absolute path can the user copy it into the Explorer address bar to find the file; if only given the file name, they have no way to locate it. Applicable: all templates that "write results to disk" (such asget-all-appslanding the user's App list), screenshot files, export files the AI itself lands on disk, etc. If the receipt includes afilefield, just copy that value as-is to the user.
The three priorities for recognizing screen content (fallback chain)
The essence of "looking at the screen" isreading out the text/state on the screen and handing it to the AI to determine the next step. The priorities are ordered bytoken savings + reliability—use the cheapest, most stable method first, and only fall back to a more expensive solution when the previous layer is insufficient, do not skip levels, do not get stuck at a certain level:
- Level 1 · Control tree text extraction (highest priority): use the
inspect_control_by_*template to traverse the UI control tree, retrieving all controls'text/id/desc。- Applicable: when you need to determine "whether the interface really changed and which control changed", or when you need structured control semantics;
- Features: more accurate than OCR (controls carry their own semantics), and does not require a large model to read images,most token-efficient; if the returned data is larger than needed, filter as required;
- Fallback condition: if the control tree isincomplete / cannot retrieve the target control / cannot satisfy the judgment need, or the APP has blocked accessibility and control information cannot be obtained, then fall back to Level 2.
- Level 2 · OCR text extraction: call the
ocrtemplate to recognize screen text (a plain text array, usually also very token-efficient).- Applicable: when the model does not need to read images, but only needs "what text is on the screen" to make a judgment;
- Screens with lots of text (settings pages/list pages) should preferentially include
{"query":"关键词"}filtering, and only take the matching items;detail:trueuse it only when per-item coordinates/confidence are truly needed—both are to prevent a large response from blowing up the context; - Fallback condition: if the OCR resultis insufficient to determine whether the interface has actually changed(e.g., the interface is purely graphical, with no obvious text), oris insufficient to meet the user's needs(the user wants structure/controls rather than text), → fall back to the third level.
- Third level · screenshot + model direct reading (lowest priority): use
GET /screenshotorcrop-screenshotto screenshot back to the PC, letting the model directly read the image to recognize the content.- Applicable: only when the first two levels are both unsatisfied and the caller's modelsupports multimodality(can read images/videos), use this layer;
- Characteristics: most complete information, most worry-free, butmost expensive, most token-consuming, so it is placed last and not used as the default solution;
- Fallback condition: if the modeldoes not support multimodality(image reading is filtered /
Content filtered/ content cannot be read), this level is unavailable, and when the first two levels also both fail, ask the user directly; do not guess blindly.
Fallback operation principles:
- Test level by level from the first level downward:Can the control tree satisfy it? Can OCR satisfy it? Only when the first two levels are both not viable, then use screenshot + model direct reading;
- Used forinternal verificationscreenshots / OCR / control results (case B), only the conclusion needs to be given in the final-state report (e.g., "Douyin has been opened"); it is not mandatory to push the image to the user;Only when the user explicitly asks to see it, or the screenshot is the deliverable (case A), must the hard rule of "show to the user" be followed (
present_files+ attach the absolute path). The same applies to OCR and control results: for internal verification, only report the conclusion; only show it if the user wants to see it. - When even the third level cannot obtain usable information, ask the user directly; do not guess blindly.
Step 4: one-time final-state report
All steps completed → report the result in one sentence (e.g., "I have helped you open Settings and connect to WiFi"); failure → report which step it got stuck at, the reason, and suggest how the user can manually remedy it.Do not disturb the user during the intermediate process.
User Q&A: query template list (do not execute tasks)
When the user asks "how many phone task templates are there now"/"what templates are there"/"list all phone tasks", etc.pure queryquestions,do not go through the execution SOP, directly run the scanner and read out the result (no phone online required, does not consume phone task tokens):
node <skill_dir>/scripts/scan-tasks.js --human
Output in the form:
共 8 个手机任务模板:
· tap-point — 按屏幕坐标 (x,y) 点按
· tap-text — 点按含指定文字的控件
...
Just read this list to the user directly. If an external program/relay needs to fetch the data, you can also curl -s http://localhost:9421/templates get JSON {count, tasks}, but in the user Q&A scenario, preferentially use the --human command line; the result is friendlier.
Note: this is a "read-only query"—do not start a task for it, do not issue any scripts, and do not require the phone to be online.
On-site script specification (one-off scripts · when templates cannot cover it)
One-off scripts written on the spot by AI are written to <skill_dir>/temp/(at delivery time the PC reads the content and "inlines it directly", location-independent; after landing on the phone it uniformly goes into scripts-from-computer/single/), use node scripts/run-task.js <文件> --args '{...}' to deliver.Before writing, you must first read references/现场脚本规范.md(strict ES5/var only, no time-consuming operations on the UI thread, time-consuming APIs go into threads.start, standard receipt skeleton, the high-frequency pitfalls of creating a file vs creating a directory—the most frequent "silent crash/no receipt received" all stem from violating it). To create a new reusable template, read references/AI_新建模板手册.md。
Deploy a real project (multiple files + resources)
For complex tasks (multi-module / with image and audio resources), use deploy-project to deploy the entire project directory to the phone scripts-from-computer/project/<名>/ and automatically run the entry; PC projects are uniformly placed in <skill_dir>/scripts/autojs-project/<工程名>/. Hard constraints: strict ES5/var only (if require encounters ES6 the entire entry silently crashes), the entry must end with a standard receipt, modules use relative require.For the complete structure and usage, see references/部署真实工程.md。
Automatically update the phone client (one click from PC, no manual work)
When phone-side code (autojs-task-phone-client.js) is changed, on the PC run node scripts/update-phone-client.js to complete in one click "deliver the new version directly to scripts-from-computer/client/ + trigger self-restart", with no manual work needed.For the mechanism and freeze troubleshooting, see references/自动更新手机客户端.md。
Handling when the phone is not connected (AI must take the initiative)
Trigger condition: in step 0, health returns {"phone":"disconnected"} or the relay is not up.Never just throw out a single line "please connect the phone"—follow the five-step process in references/手机连接排障.md to proactively guide the user through connecting: ① the AI runs get-lan-ip.js itself to get the LAN IP → ② guide the user to change the client var SERVER_IP → ③ run on the phone in AutoJs6 autojs-task-phone-client.js → ④ same WiFi + four permissions (accessibility/floating window/background/screenshot) → ⑤ after the user reports "connected", recheck health. For the complete steps and the wording to give the user, see that document; note: relay restarts/network blips will self-heal and reconnect, and only the death of the client engine requires this process.
Security boundaries (red lines)
- Payment, transfer, deletion, authorization-type irreversible operations: even in fully automatic mode, before executing that step you must first confirm with the user once;
- one UI task at a time: although run-type task lists support concurrency (receipts are routed back by taskId without crossing lines), the phone screen can only do one thing at a time, so UI automation tasks must still be delivered serially step by step and must not be concurrent; synchronous short operations such as screenshots/updating the client/deleting projects are still mutually exclusive on the relay side (concurrency will return 429).
Notes
- Recognizing screen-off/lock-screen symptoms (high-frequency misjudgment point): screenshot returnall-black frame + clipboard read/writereturns empty + abnormal OCR content, while
open-appand other "fake successes" (am startdoes not require the screen to be on)—there is a high chance the phone is screen-off/auto-locked; ask the user to wake and unlock it and retry, and do not investigate in the direction of a script bug. - Phone-side scripts are strictly ES5 / var only: the AutoJs6 Rhino engine does not fully support ES6, and this is a hard constraint.
- On-site scripts should not use setInterval to stay alive: scripts delivered via /run are managed by the relay, and a normal exit is enough; only resident clients need setInterval to stay alive.
- Performance: steps involving UI tree searches (tap-text, etc.) are a bit slower when the interface hierarchy is deep; this is normal, just wait for the receipt.
- Screenshot scripts must have permission prelude code built in: for any script that calls
captureScreen()/ocr(...)on mobile, the very beginning must contain "background thread usestextMatchregex multi-candidate to automatically dismiss the authorization dialog (button text varies by ROM: Xiaomi "Start Now", some models "Start Screenshot", etc.) +requestScreenCapture()+sleep(500)" this code block, otherwise the screenshot fails or the script hangs (manifesting as "empty receipt"). The templatescreenshot/crop-screenshot/ocr/tap-text/unfollow-official-accountsalready has it built in; when creating a new screenshot template, add it as inreferences/截图权限与弹框处理.md.