web-scraper

Web Scraper (Crawl4AI)

Based on Crawl4AI high-performance web crawler. Crawl4AI is the #1 GitHub Trending project of 2025 (59k+ Stars), designed specifically for AI Agents and LLMs.

Core Advantages

Feature	Description
🏆 Industry Benchmark	59k+ GitHub Stars, Apache-2.0 open source
💰 Completely Free	No API Key required
🛡️ Strong Anti-Crawling	Playwright browser engine + stealth mode + random UA
🚀 Ultra Fast	Async concurrency + memory-adaptive scheduling, 6x faster than alternatives
🎯 LLM Optimized	Automatically outputs clean Markdown, suitable for RAG and LLM consumption
📄 Dynamic Pages	Supports JavaScript rendering, infinite scrolling, SPA
🔧 Intelligent Extraction	PruningContentFilter automatically removes noisy content

Use Cases

• User says "help me read this article" "scrape this webpage"
• User says "collect recent news" "organize industry information"
• Batch scrape full content of webpages from search results
• Can serve as the content acquisition layer for in-depth research workflows

Tech Stack

Crawl4AI v0.8.x
├── Playwright (Chromium 浏览器引擎)
├── AsyncWebCrawler (异步并发)
├── PruningContentFilter (智能内容过滤)
├── DefaultMarkdownGenerator (Markdown 生成)
└── MemoryAdaptiveDispatcher (内存自适应调度)

Quick Start

Single URL

from crawl4ai import AsyncWebCrawler

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun("https://example.com")
    print(result.markdown)  # 干净的 Markdown

Batch Concurrency

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode

urls = [
    "https://36kr.com/article1",
    "https://techcrunch.com/article2",
    "https://theverge.com/article3",
]

config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)

async with AsyncWebCrawler() as crawler:
    results = await crawler.arun_many(urls, config=config)
    for result in results:
        if result.success:
            print(f"✅ {result.url}: {len(result.markdown.raw_markdown)} chars")

Intelligent Content Filtering

from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
from crawl4ai.content_filter_strategy import PruningContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

config = CrawlerRunConfig(
    cache_mode=CacheMode.BYPASS,
    markdown_generator=DefaultMarkdownGenerator(
        content_filter=PruningContentFilter(threshold=0.4, threshold_type="fixed")
    ),
)

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun("https://news.ycombinator.com", config=config)
    print(result.markdown.fit_markdown)  # 过滤后的高质量内容

Anti-Crawling Capabilities

Crawl4AI has built-in multi-layer anti-crawling mechanisms:

Layer	Mechanism	Description
Browser	Playwright Chromium	Real browser rendering, not HTTP requests
User-Agent	user_agent_mode="random"	Random UA each time
Stealth	Built-in stealth mode	Hides automation fingerprints
Proxy	proxy parameter	Supports proxy pools
JavaScript	Full JS rendering	Supports dynamically loaded pages
Cookies	Session management	Supports login state persistence

Configuring Anti-Crawling

from crawl4ai import BrowserConfig

browser_config = BrowserConfig(
    headless=True,
    user_agent_mode="random",      # 随机 UA
    ignore_https_errors=True,
)

Concurrency Performance

Crawl4AI uses MemoryAdaptiveDispatcher to automatically adjust concurrency based on system resources:

# 批量抓取 - 自动调整并发度
results = await crawler.arun_many(urls, config=config)

# 流式处理 - 边爬边处理
config = CrawlerRunConfig(stream=True, cache_mode=CacheMode.BYPASS)
async for result in await crawler.arun_many(urls, config=config):
    if result.success:
        process(result)

Output Format

Markdown (default)

# Article Title

Published on 2024-01-15 by John Doe

Main content paragraph 1...

## Section Heading

Content with **bold** and [links](https://example.com).

| Column 1 | Column 2 |
|---|---|
| Data | Data |

Output Fields

result.markdown.raw_markdown    # 原始 Markdown
result.markdown.fit_markdown    # 过滤后的 Markdown (需配置 content_filter)
result.extracted_content        # 结构化提取结果 (需配置 extraction_strategy)
result.success                  # 是否成功
result.url                      # 原始 URL
result.error_message            # 错误信息

Typical Scenarios

Scenario 1: Collect a week of news

# Step 1: 通过 api_calling 调用 Jina Search 获取 URL（通过搜索类 Skill 获取 URL）
results = await api_calling(url="https://s.jina.ai/AI行业新闻 最近一周", method="GET", headers={"Accept": "application/json"})
urls = [r["url"] for r in results]

# Step 2: Crawl4AI 并发抓取 (10个URL约5-10秒)
config = CrawlerRunConfig(
    cache_mode=CacheMode.BYPASS,
    markdown_generator=DefaultMarkdownGenerator(
        content_filter=PruningContentFilter(threshold=0.4)
    ),
)
async with AsyncWebCrawler() as crawler:
    results = await crawler.arun_many(urls, config=config)

# Step 3: 提取成功的文章
articles = [r for r in results if r.success]
for article in articles:
    content = article.markdown.fit_markdown  # 过滤后的干净内容

Scenario 2: Handle dynamic pages (SPA)

config = CrawlerRunConfig(
    cache_mode=CacheMode.BYPASS,
    wait_until="networkidle",         # 等待网络空闲
    delay_before_return_html=1.0,     # 额外等待 JS 渲染
    scan_full_page=True,              # 滚动到底部
)

CLI Usage

Crawl4AI also provides a CLI tool crwl:

# 基本爬取
crwl https://example.com -o markdown

# 带内容过滤
crwl https://example.com -o markdown-fit

# JSON 输出 + 详细日志
crwl https://example.com -o json -v --bypass-cache

Installation

pip install crawl4ai
crawl4ai-setup  # 安装 Playwright 浏览器依赖

Configured as auto_install: true, the Python package is automatically installed on first use.crawl4ai-setup Requires manually running a one-time browser installation.

Comparison with Other Solutions

Solution	Stars	Anti-Crawling	Dynamic Pages	Speed	Cost
Crawl4AI	59k	Playwright+Stealth	✅	6x fastest	Free
Firecrawl	77k	Cloud browser	✅	Fast	Paid ($83/month)
httpx+BS4	-	❌ None	❌	Medium	Free
Jina Reader	-	Cloud	✅	Fast	Free tier
Scrapy	54k	Requires plugin	❌	Fast	Free

Best Practices

✅ Recommended

• News websites, blogs, technical articles, documentation
• Dynamically loaded pages (React/Vue/SPA)
• Websites with strict anti-scraping measures
• Batch scraping of search results
• Pages that require full JavaScript rendering

Configuration Recommendations

# 新闻/博客 (轻量)
config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)

# 重度反爬网站
browser_config = BrowserConfig(
    headless=True,
    user_agent_mode="random",
)

# SPA / 动态页面
config = CrawlerRunConfig(
    wait_until="networkidle",
    scan_full_page=True,
    delay_before_return_html=1.0,
)

Reference Resources

• GitHub: https://github.com/unclecode/crawl4ai (59k+ Stars)
• Documentation: https://docs.crawl4ai.com/
• License: Apache-2.0 (completely free for commercial use)