webfetch-plus

WebFetch Plus

Execution principles

• Set the current skill path to WFP_PATH
• For any command execution, you must first cd "$WFP_PATH"
• Ordinary scraping can use a pipe, skipping the safety prompt
• Pipe input splits arguments by whitespace; URLs and argument values must not contain spaces
• The script follows runtime/node/package.json to automatically install CloakBrowser and Playwright; the first installation uses a process lock, and other calls wait for it to finish
• Agent / non-TTY environments must not run the manual verification flow; the script will automatically select from /tmp/wfp-states/*.json the latest session matching the target domain, and only reports that interactive verification is needed when none is found
• The sole authority for the default state path is the runtime's getDefaultStatePath()：/tmp/wfp-states/<hostname-clean>.json（hostname-clean replace non-alphanumeric characters and hyphens in the URL hostname with - result). Do not guess or separately implement this naming on the caller side.

External side effects

• Access the passed-in HTTP(S) URL and launch a local browser;--stealth Chromium may be downloaded on first use
• Body text is by default written to /tmp/wfp-tasks/
• Failure evidence is always written to /tmp/wfp-evidence/, unaffected by --output-dir impact
• --save-state writes Playwright storageState JSON (cookies / localStorage); the file may contain login state, do not commit it to git
• --out and --output-dir are written to the path specified by the caller; do not let multiple calls share the same --out path

Scrape web pages

Pass the URL and arguments through a pipe:

echo "https://example.com" | bash "bin/wfp.sh"

You can also pass arguments directly:

bash bin/wfp.sh "https://example.com" --wait 3000

The script outputs the body text file path on stdout.

Dynamic page timeout

When a dynamic page keeps the connection open for a long time,networkidle may repeatedly time out during navigation. In this case, switch to --wait-until domcontentloadedand increase --timeoutaccording to the page's actual load time; for example:

bash bin/wfp.sh 'https://example.com' --wait-until domcontentloaded --timeout 90000

CAPTCHAs, login, and session reuse

When encountering human verification or login, run the launcher in an interactive TTY, complete the verification, then press Enter. The launcher will save the target's default session in headed + stealth mode; when --wait-until is not explicitly passed, domcontentloadedis used, and explicit values take precedence:

bash bin/wfp-human.sh 'https://example.com/protected-page'

Subsequent ordinary scraping will reuse the session:--state takes precedence; when not provided, the script verifies from /tmp/wfp-states/*.json that the cookie domain or localStorage origin matches the target domain, and adopts the latest file. When the session expires and is blocked again, just rerun the launcher to overwrite the default state.

When blocked by risk controls and there is no reusable session, stderr will output a copyable launcher command with the actual target URL. Automated tasks and Agents do not run the launcher and only use existing or automatically discovered sessions. Non-TTY macOS calls will automatically open Terminal.

Arguments

Argument	Description
--out <path>	Write directly to the specified file; the same path must not be used concurrently
--output-dir <path>	Write the unique body text file directly into this directory
--visible	Explicitly open a browser window (headed); by default it is implicitly headless
--state <path>	Explicitly load a Playwright storageState JSON (takes precedence over automatic discovery; the file must already exist)
--save-state <path>	Save storageState JSON after successful scraping
--wait <ms>	Additional wait time after page load
--timeout <ms>	Page navigation timeout, default 45000
`--wait-until <load	domcontentloaded
--retries <1-3>	Maximum number of attempts, default 3; upon first WAF detection, subsequent attempts automatically --stealth
--selector <css>	Extract only the content of the specified CSS selector
`--format <markdown	text
--stealth	CloakBrowser custom Chromium enhanced anti-WAF

Manual verification is not part of the public wfp.sh parameter. Only via bash bin/wfp-human.sh '<url>' [fetch options] launch; the launcher enforces headed and stealth, and saves the default state. It rejects --state、--save-state、--out、--visible and other managed parameters.

Execution flow

传入 URL →（优先 --state，否则自动匹配本地会话）→ 抓取
  → 若 WAF：自动 stealth 重试 → 仍失败则提示 wfp-human.sh 启动器
  → 启动器内部启用人工验证 → 等人回车 → reload 目标 URL → 保存 state
  → 成功则可选 --save-state → stdout 输出正文路径

Success determination: stdout outputs the unique body path

• Empty title + extremely short body + large HTML / visible challenge text → failure
• When a real title already exists and the body is long enough, do not misjudge as failure due to residual challenge-related CSS or text in the HTML

Failure handling contract：

1. Non-zero exit; stderr outputs the failure summary path, evidence's suggestion retains the same suggestion
2. First WAF → remaining retries automatically --stealth
3. stealth still fails and there is no matching /tmp/wfp-states/*.json session → final stderr outputs only one fully copyable launcher command with the actual target URL:bash bin/wfp-human.sh '<url>'
4. Manual challenge on the same page allows multiple Enter presses; if the challenge does not disappear, it prompts to try again and does not immediately close the page and reopen for three rounds

General rules：

• Default headless; the launcher's internal manual challenge forces headed, and does not hide the launch window
• The launcher does not automatically solve sliders
• Unattended tasks only use --state, do not run the launcher