SuperAPI AI Directory

xrk-crawl

Coding & development

View repository

Updated 11-05 18:20

About this skill

Unified entry (business first)

core/system-Core/lib/crawl/index.js — plugins, stream, HTTP businessimport only from here。

import {
  fetchWithPolicy,
  runWebFetch,
  buildWebFetchRuntime,
  assertUrlSafeForFetch,
  PlaywrightAgentSession,
  createLocalFontScreenshotHelper,
  DEFAULT_DEVICE_SCALE_FACTOR,
  DOM_TWEAK_LABEL_COLON_HALF,
} from '../lib/crawl/index.js'

Capability layering (when to use which)

Scenario Capability Implementation file (all in lib/crawl/)
Simple API / no JS rendering fetchWithPolicy via ../net/fetcher.js export
Content extraction, Readability, Firecrawl runWebFetch web-fetch-executor.js
MCP web_fetch stream/web.js internally uses crawl
JS rendering, interaction, screenshots PlaywrightAgentSession playwright-session.js
MCP controlled browser stream/browser.js browser_* tools
Screenshot fonts/styles consistent with production createLocalFontScreenshotHelper page-screenshot-enhance.js

Selection: HTTP can get body → runWebFetch; need rendering or PNG → Playwright.

SSRF

  • ssrf-guard.js:assertUrlSafeForFetch、SsrFBlockedError
  • PlaywrightAgentSession.goto and runWebFetch shared by
  • Alias:assertUrlSafeForBrowserNavigation

PlaywrightAgentSession common API

Method Description
PlaywrightAgentSession.launch(options) Default deviceScaleFactor=2
PlaywrightAgentSession.using(options, fn) auto close
attachScreenshotHelper(helper) mount enhanced screenshot assistant
goto(url, { waitUntil, timeoutMs }) navigation (including SSRF)
gotoAndCapture(url, { selector, settleMs, ... }) prepare→goto→wait→screenshot
captureRegion(selector) region screenshot
regionText(selector) region innerText

Directory structure

lib/crawl/
  index.js                  # 对外唯一入口
  ssrf-guard.js
  web-fetch-executor.js
  web-fetch-utils.js
  web-fetch-visibility.js
  web-shared.js
  external-content-wrap.js
  playwright-session.js
  page-screenshot-enhance.js
lib/net/fetcher.js          # 通用 HTTP(超时/重试/代理),由 crawl 再导出

Removed redundant directories:lib/openclaw-web/、lib/agent-browser/、nav-ssrf.js。

and workflow

  • stream/web.js → web_fetch
  • stream/browser.js → browser_*(browser_screenshot supports selector)

Common pitfalls

  • Do not use import the deleted openclaw-web/、agent-browser/ path.
  • Do not create wrapper files that only re-export; write extensions in lib/crawl/ and in index.js export.
  • Prefer using + gotoAndCapture, do not hand-write the four-step screenshot flow.

Reference

  • docs/system-core.md(web / browser chapters)
  • core/system-Core/plugin/lkwg.js