Web Crawling 网页爬虫 | Web Scraping 网页爬取 | LLM 优化输出
Intelligent web crawler and scraping tool, supports search, full-site crawling, dynamic page scraping. Free web crawler and scraper with LLM-optimized Markdown output.
核心功能 | Core Features
- 🔍 Web Search 网页搜索 - DuckDuckGo search, no API key required
- 🕷️ Web Crawling 网页爬虫 - Site crawler, spider, sitemap recognition
- 📝 Web Scraping 网页抓取 - Smart scraper, data extraction
- 📄 LLM-Optimized Output - Fit Markdown, saves 80% tokens
- ⚡ Dynamic Page Scraping - JavaScript rendered page scraping
Quick Start | Quick Start
Installation | Installation
pip install crawl4ai-skill
Web Search | Web Search
# Search the web with DuckDuckGo
crawl4ai-skill search "python web scraping"
Web Scraping | Single-page scraping
# Scrape a single web page
crawl4ai-skill crawl https://example.com
Web Crawling | Full-site crawler
# Crawl entire website / spider
crawl4ai-skill crawl-site https://docs.python.org --max-pages 50
Use Cases | Use Cases
Use Case 1: Web Crawler for Documentation | Documentation site crawler
# Crawl documentation site with spider
crawl4ai-skill crawl-site https://docs.fastapi.com --max-pages 100
Crawler Output | Crawler Output:
- ❌ Removed: navigation bar, sidebar, ads
- ✅ Retained: title, body text, code blocks
- 📊 Token：50,000 → 10,000（-80%）
Use Case 2: Search + Scrape | Search + scrape
# Search and scrape top results
crawl4ai-skill search-and-crawl "Vue 3 best practices" --crawl-top 3
Use Case 3: Dynamic Page Scraping | Dynamic page scraping
Scraping JavaScript-rendered pages (Xueqiu, Zhihu, etc.):
# Scrape JavaScript-heavy pages
crawl4ai-skill crawl https://xueqiu.com/S/BIDU --wait-until networkidle --delay 2
Commands | Commands
| Command | Description |
|---|---|
search <query> |
Web search 网页搜索 |
crawl <url> |
Web scraping 单页爬取 |
crawl-site <url> |
Web crawling 全站爬虫 |
search-and-crawl <query> |
Search + scrape 搜索并爬取 |
Common Options | Common Options
# Web Search 搜索
--num-results 10 # Number of results
# Web Scraping 爬取
--format fit_markdown # Output format
--output result.md # Output file
--wait-until networkidle # Wait strategy for dynamic pages
--delay 2 # Additional wait time (seconds)
--wait-for ".selector" # Wait for specific element
# Web Crawling 爬虫
--max-pages 100 # Max pages to crawl
--max-depth 3 # Max crawl depth
Output Formats | Output Formats
fit_markdown (Recommended)
Smart extraction, saves 80% tokens. Smart extraction, save 80% tokens.
crawl4ai-skill crawl https://example.com --format fit_markdown
raw_markdown
Preserve full structure. Preserve full structure.
crawl4ai-skill crawl https://example.com --format raw_markdown
Why This Crawler? | Why This Crawler?
✅ Free Crawler - No API key required, works out of the box
✅ Smart Scraper - Automatic denoising, extracts core content
✅ Site Crawler - Supports sitemap, recursive crawling
✅ Dynamic Scraping - Supports JavaScript-rendered pages
✅ Search Integration - DuckDuckGo search built in