crawl4ai-seo

crawl4ai-seo

SEO crawler for website audits and competitive analysis. Answers the question: "what actually lies on the pages and how the site is structured from the inside".

Does not replace SERP tools (positions, visibility) — complements them with data on page content and structure.

What it can do

Task	Description
Site inventory	Full page inventory: URL, status, title, H1, meta, canonical, word count
On-page audit	Finding issues: empty title/H1, duplicate headings, broken canonical, thin content
Internal linking audit	Internal link graph, orphan pages (0 inbound), weakly linked pages
Navigation audit	Breadcrumbs, nav blocks, menu consistency, hub pages without outbound links
Landing comparison	Comparison of shortlist URLs by on-page signals (title, H1, content, links)
Competitor research	Running competitor pages through the same pipeline, comparing templates

Config

Global defaults: config/defaults.example.json, local override — config/defaults.json. Details: config/README.md.

Rule: in defaults do not store a specific site or client parameters. Everything site-specific comes through launch params and is recorded in cache/jobs/<job_id>/resolved_config.json.

Workflow

Before any crawl

Check the environment:
```
python3 scripts/doctor.py
```
If crawl4ai/playwright are not ready — stop at seed/discovery, do not imitate a crawl.
Determine the research goal:
- Full site audit → site inventory + navigation audit
- Landing page audit → landing comparison
- Competitor analysis → competitor research
- Internal linking audit → internal linking audit
Collect launch params:
- target.domain — research domain
- job.project — project/client
- job.label — launch label

Main pipeline

# 1. Создать job
python3 scripts/init_job.py --domain https://example.com --project client-a --label full-audit

# 2. Собрать seed URL из sitemap/robots
python3 scripts/seed_urls.py --job-id <job_id>

# 3. Запустить crawl
uv run --script scripts/crawl_batch.py --job-id <job_id>

# 4. Построить навигационный отчёт
python3 scripts/build_navigation_report.py \
  --seed-job-id <seed_job_id> \
  --crawl-job-id <crawl_job_id> \
  --report-dir reports/<project>/<label>

# 5. Сравнить страницы
python3 scripts/compare_pages.py --job-id <job_id>

Quick option (without a separate init)

python3 scripts/seed_urls.py --domain https://example.com --project client-a --label quick-check
uv run --script scripts/crawl_batch.py --job-id <job_id>

Scripts

Script	Purpose
`doctor.py`	Environment check: uv, Python, crawl4ai, playwright, config
`init_job.py`	Creating a job: `job_id`, `launch_params.json`, `resolved_config.json`
`seed_urls.py`	Discovery URL from sitemap/robots/file → `seed.json` (works without crawl4ai)
`crawl_batch.py`	Batch crawl via crawl4ai → `pages.ndjson`, `links.ndjson`, `summary.json`
`compare_pages.py`	Tabular comparison of crawled pages by on-page signals
`build_navigation_report.py`	Consolidated navigation audit: orphans, weak hubs, breadcrumbs, menu drift
`common.py`	Shared helpers (do not run directly)

Extracted SEO Signals

For each page (pages.ndjson) the following are extracted:

On-page: title, description, keywords, canonical, robots, og:title, og:description, og:image, H1, headings hierarchy, word count.

Navigation: path depth, breadcrumbs (texts + URLs), nav blocks count, nav links count, nav URLs sample.

Links: internal/external links count. В links.ndjson — full graph: source → target, anchor text, nofollow, same_domain.

Navigation Audit Findings

build_navigation_report.py automatically finds:

Problem	What it means for SEO
Orphan pages (0 inbound links)	The search engine will not find the page or will not pass weight to it
Weakly linked (1 inbound)	The page receives minimum link equity
Missing breadcrumbs	No navigation chain for the bot and the user
Breadcrumb inconsistency	Different trail signatures in the same section — confusion in the hierarchy
Weak nav template	Hub page with a suspiciously small number of nav links
Menu inconsistency	Pages lose part of common nav targets (the template "drifts")
Weak hubs	A category/section gives fewer than 5 internal links
Duplicate titles	Several pages with the same title — cannibalization
Canonical mismatches	canonical points not to itself
Technical junk	login URLs, php endpoints, asset links in the navigation layer
Linked-not-seeded	The site links to URLs that are not in the sitemap

Integration with other skills

Scenario	Partner skill	Order
SERP → on-page audit	`yandex-search-api`	Get a shortlist from SERP → run it through crawl4ai
Organic landings → audit	`yandex-metrika`	Take top organic pages → check on-page quality
Indexing + audit	`yandex-webmaster`	Compare indexed pages with crawled inventory
Anti-block fallback	`scrapedo-web-scraper`	If crawl4ai is blocked — fallback via Scrape.do

Cache Layout

cache/jobs/<job_id>/
  launch_params.json      # raw params этого запуска
  resolved_config.json    # effective config после merge defaults + params
  manifest.json           # operational state и artifact paths
  seed.json               # seed URLs и метаданные discovery
  pages.ndjson            # одна запись на страницу
  links.ndjson            # одна запись на link edge
  summary.json            # компактная сводка
  markdown/               # сохранённый markdown контент

Each job is isolated. Parallel runs on different sites are safe.

Output Hygiene

stdout: only preview and file paths.
Full data: only in job files.
For analysis use rg, head, wc -l on ndjson files, do not launch the browser again.

About this skill