ScrapeGraphAI is an AI-based Python scraping library that creates scraping pipelines for websites and local documents through LLMs and direct graph logic. Users only need to describe the information they want to extract in natural language, and the library can automatically complete the scraping.
Main capabilities
- Supports extracting information from websites or local files (XML, HTML, JSON, Markdown, etc.).
- Provides a variety of standard scraping pipelines, the most commonly used being SmartScraperGraph, which can extract information from a single page based on a user prompt and source URL.
- Supports multi-page information extraction, generating Python scripts, and even generating audio files.
- Each graph has corresponding multi-version variants that support parallel LLM calls.
- Can flexibly switch between different LLMs, including using OpenAI, Groq, Azure, Gemini, MiniMax, etc. via API, or using local models via Ollama.
Usage
- Installation: Can be installed via PyPI; installing in a virtual environment is recommended to avoid library conflicts. The official materials do not provide a specific installation command.
- To use local models, you must first install Ollama and download models with the ollama pull command.
- Basic usage: Specify the user prompt and source URL, then call the corresponding pipeline; the output is in dictionary format.
- Supports integration with mainstream frameworks and platforms: LLM frameworks such as Langchain, Llama Index, Crew.ai, Agno, CamelAI; low-code frameworks such as Pipedream, Bubble, Zapier, n8n, Dify, Toolhouse; an MCP server is also provided.
Use cases
- Data exploration and research purposes.
- Need to extract structured information from web pages or local documents according to natural language instructions.
- Scenarios where you want to use local LLMs for self-hosting, cost control, or meeting data residency requirements.
Limitations
- Officially stated to be for data exploration and research purposes only, with no responsibility for misuse.
- The open-source library requires users to manage browsers, proxies, and extensions themselves.
- Specific fees, pricing, performance results, etc. are not provided in the materials.
Open source and hosted API ScrapeGraphAI offers two forms: a self-runnable open-source library, and a hosted cloud API used through Python and JS/TS SDKs. The open-source library is suitable for users who need full control, local/self-hosted data, local LLMs, or fine-grained cost adjustments and can manage their own infrastructure; the hosted API is suitable for users who want zero infrastructure, hosted JS rendering and anti-scraping, built-in crawling and scheduled monitoring tasks, and rapid launch, and is billed by credits.
Other information
- The project uses the MIT license.
- Provides a Python SDK (https://github.com/ScrapeGraphAI/scrapegraph-py）和 JS/TS SDK（https://github.com/ScrapeGraphAI/scrapegraph-js）。
- API documentation:https://docs.scrapegraphai.com/introduction
- Collects anonymous usage metrics, which can be opted out of by setting the environment variable SCRAPEGRAPHAI_TELEMETRY_ENABLED=false.
- Provides proxy discount codes: SCRAPEGRAPH35 (35% off mobile and residential proxies), SCRAPEGRAPH40 (40% off ISP/static proxies).
