website-mirror

Website Mirror

One question decides everything:

Will the bytes I save still contain the content with no network?

Mirroring fails silently when content is fetched at view-time. Answer that first, then pick a tool.

Answer it: where does the content live?

curl -sL <url> (or view-source) and grep for text you can see on the page:

• In the HTML → any tool works (static site).
• Rendered by JS from data already inline (var data=[…], __NEXT_DATA__) → works offline if the saved file keeps its scripts (monolith and wget do).
• Fetched from an API at runtime (fetch('/api/…'), infinite scroll) → the saved page renders blank offline. Resolve the API and bake the data in, or capture already-rendered HTML with firecrawl.

Pick the tool

Local set on this machine (command -v to confirm elsewhere): wget, monolith, firecrawl, yt-dlp.

Goal	Tool	Command
Whole static site, browseable offline	wget	wget --mirror --convert-links --adjust-extension --page-requisites --no-parent --wait=1 --random-wait <url>
One page → single self-contained file	monolith	monolith <url> -o page.html
JS/API content, or rendered HTML → markdown	firecrawl	firecrawl scrape <url> / firecrawl crawl <url> (also firecrawl MCP)
Video / audio from a page	yt-dlp	yt-dlp <url>

• monolith crawls nothing - one page only. It DOES inline statically-referenced assets, cross-domain included (a CDN stylesheet or image ends up embedded). What dies offline is anything that injects or fetches at runtime - analytics, Turnstile, social buttons - because the script runs (or fails) when the page is opened, not when it is captured. Not a mirror bug.
• wget's --mirror = -r -N -l inf; the -k -E -p flags are what make the copy actually browseable offline (converted links, .html extensions, page requisites). Bare wget -r gives broken links and missing assets.

robots and blocks - respect them

• wget honours robots.txt by default and will silently mirror nothing on a Disallow. -e robots=off overrides it - a deliberate choice, not a default. A blanket User-agent: * / Disallow: / (e.g. IMDb) means the site is off-limits to generic clients; use its licensed or official data route, don't evade access controls.
• Soft-block tell: HTTP 200 or 202 with a 0-byte body is bot mitigation, not success. A naive http_code < 400 check treats it as a win and ships an empty dataset - check size_download / actual bytes.

Be gentle

--wait=1 --random-wait, serial not aggressively parallel. The delay is the courtesy, not an afterthought.

Verify offline (don't re-fetch)

Inspect the saved files only: grep for real content, and confirm converted links point at local files that exist. Re-fetching during verification hides exactly the gap you are checking for.