Unified entry (business first)
core/system-Core/lib/crawl/index.js — plugins, stream, HTTP businessimport only from here。
import {
fetchWithPolicy,
runWebFetch,
buildWebFetchRuntime,
assertUrlSafeForFetch,
PlaywrightAgentSession,
createLocalFontScreenshotHelper,
DEFAULT_DEVICE_SCALE_FACTOR,
DOM_TWEAK_LABEL_COLON_HALF,
} from '../lib/crawl/index.js'
Capability layering (when to use which)
| Scenario | Capability | Implementation file (all in lib/crawl/) |
|---|---|---|
| Simple API / no JS rendering | fetchWithPolicy |
via ../net/fetcher.js export |
| Content extraction, Readability, Firecrawl | runWebFetch |
web-fetch-executor.js |
MCP web_fetch |
stream/web.js |
internally uses crawl |
| JS rendering, interaction, screenshots | PlaywrightAgentSession |
playwright-session.js |
| MCP controlled browser | stream/browser.js |
browser_* tools |
| Screenshot fonts/styles consistent with production | createLocalFontScreenshotHelper |
page-screenshot-enhance.js |
Selection: HTTP can get body → runWebFetch; need rendering or PNG → Playwright.
SSRF
ssrf-guard.js:assertUrlSafeForFetch、SsrFBlockedErrorPlaywrightAgentSession.gotoandrunWebFetchshared by- Alias:
assertUrlSafeForBrowserNavigation
PlaywrightAgentSession common API
| Method | Description |
|---|---|
PlaywrightAgentSession.launch(options) |
Default deviceScaleFactor=2 |
PlaywrightAgentSession.using(options, fn) |
auto close |
attachScreenshotHelper(helper) |
mount enhanced screenshot assistant |
goto(url, { waitUntil, timeoutMs }) |
navigation (including SSRF) |
gotoAndCapture(url, { selector, settleMs, ... }) |
prepare→goto→wait→screenshot |
captureRegion(selector) |
region screenshot |
regionText(selector) |
region innerText |
Directory structure
lib/crawl/
index.js # 对外唯一入口
ssrf-guard.js
web-fetch-executor.js
web-fetch-utils.js
web-fetch-visibility.js
web-shared.js
external-content-wrap.js
playwright-session.js
page-screenshot-enhance.js
lib/net/fetcher.js # 通用 HTTP(超时/重试/代理),由 crawl 再导出
Removed redundant directories:lib/openclaw-web/、lib/agent-browser/、nav-ssrf.js。
and workflow
stream/web.js→web_fetchstream/browser.js→browser_*(browser_screenshotsupportsselector)
Common pitfalls
- Do not use
importthe deletedopenclaw-web/、agent-browser/path. - Do not create wrapper files that only re-export; write extensions in
lib/crawl/and inindex.jsexport. - Prefer
using+gotoAndCapture, do not hand-write the four-step screenshot flow.
Reference
docs/system-core.md(web / browser chapters)core/system-Core/plugin/lkwg.js