SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI agentsolostep-mcp-server

olostep-mcp-server

Web scraping, crawling and search API: extract content as Markdown or JSON, batch-process URLs and get answers with citations.

Open website

About this tool

Olostep MCP Server is the official Model Context Protocol (MCP) server implementation provided by Olostep, used to connect Olostep's web scraping, content extraction, and search infrastructure to any MCP-compatible AI agent, enabling it to scrape, crawl, batch extract, and search web pages in real time.

Purpose

This server is intended for scenarios that require AI agents to access real-time web data, providing scraping, crawling, batch extraction, and search capabilities. Before use, you need an Olostep API key, which can be obtained by registering on the Olostep website.

Main Capabilities

  • Scrape website content in HTML, Markdown, JSON, or plain text format (with optional parser support).
  • Parser-based web search that returns structured results.
  • AI answers with cited sources, supporting optional JSON structured output.
  • Batch scrape up to 10,000 URLs.
  • Autonomously crawl an entire site starting from a starting URL.
  • Website URL discovery and mapping (with include/exclude filtering).
  • Route requests by country to obtain geolocated content.
  • Configurable wait time for JavaScript-heavy websites.
  • Robust error handling and reporting.
  • Simple API key configuration.

Usage

You can connect to the Olostep MCP Server in several ways:

  1. Remote endpoint (recommended): no local installation required; connect directly to the hosted MCP server. Authentication is done using the Olostep API key as a Bearer token in the Authorization header.
  2. Docker: pull and run the official Docker image; you can also build locally from source.
  3. npx: run without installation, supporting Windows PowerShell, Windows CMD, or global installation.
  4. Smithery CLI: install in your device terminal.

Client configuration examples:

  • Cursor: create or edit .cursor/mcp.json in the project root directory; or add it via Cursor Settings > Features > MCP Servers with the command env OLOSTEP_API_KEY=your-api-key npx -y olostep-mcp.
  • Claude Desktop: add to claude_desktop_config.json, or use Docker.
  • Claude Code: add the remote endpoint, or use the local method.
  • Windsurf: add to ./codeium/windsurf/model_config.json.
  • VS Code: add to .vscode/mcp.json.
  • Metorial: supports one-click installation (search for Olostep in the Metorial dashboard and enter the API key) or manual configuration.

Environment variables:

  • OLOSTEP_API_KEY: Olostep API key (required).
  • ORBIT_KEY: optional, used to route requests through Orbit.

Available Tools

  1. Scrape Website (scrape_website): extract content from a single URL, supporting multiple formats and JavaScript rendering. Parameters include url_to_scrape (required), output_format (html, markdown, json, or text, default markdown), country (optional country code), wait_before_scraping (0-10000 milliseconds), parser (optional parser ID).
  2. Search the Web (search_web): search the web based on a query and get structured results (non-AI, parser-based). Parameters include query (required), country (optional, default US).
  3. Answers (AI) (answers): search the web and return AI-driven answers, supporting custom JSON structures, with sources and citations. Parameters include task (required), json (optional JSON schema/object or output shape description). Returns answer_id, object, task, result (if JSON is provided), sources, created.
  4. Batch Scrape URLs (batch_scrape_urls): scrape up to 10,000 URLs simultaneously. Returns batch_id, status, total_urls, created_at, formats, country, parser, urls.
  5. Create Crawl (create_crawl): Starts an asynchronous crawl, autonomously discovering and scraping the entire website, returning a crawl_id, running in the background; you must then call get_crawl_results to poll the status and retrieve pages. Returns crawl_id, object, status, start_url, max_pages, created, formats, country, parser. Note: crawls and batches are independent resources; do not pass crawl_id to get_batch_results.
  6. Create Map (create_map): Retrieves all URLs on a website, used for discovery and analysis. Returns map_id, object, url, total_urls, urls, search_query, top_n.
  7. Get Webpage Content (get_webpage_content): Retrieves webpage content in clean markdown format, supporting JavaScript rendering. Parameters include url_to_scrape (required), wait_before_scraping (default 0), country (optional).
  8. Get Website URLs (get_website_urls): Searches for and retrieves relevant URLs on a website, sorted by relevance to the query. Parameters include url (required), search_query (required).
  9. Get Batch Results (get_batch_results): Uses batch_id to retrieve the results of a previously submitted batch scraping task. Returns batch_id, status (processing or completed), total_urls, completed_urls, items (an array of scraping results for each URL, including url, custom_id, markdown_content, html_content, json_content, text_content, status, page_metadata).
  10. Get Crawl Results (get_crawl_results): Retrieves the status and scraped pages of the asynchronous crawl started by create_crawl; it is a required companion tool for create_crawl. Parameters include crawl_id (required), formats (markdown, html, json, text, default ["markdown"]), items_limit (1-100, default 20), cursor (default 0), search_query (optional filter). While in progress, returns crawl_id, status (in_progress), pages_completed, pages_total, and a prompt to call again in about 10 seconds; when completed, returns crawl_id, status (completed), pages_returned, next_cursor, has_more, and a pages array.

Error Handling

The server provides robust error handling: detailed error messages for API issues, network error reporting, authentication failure handling, and rate limit information.

Distribution and Platforms

The MCP server is provided as a Docker image: Docker Hub (olostep/mcp-server), the official Docker MCP Registry (mcp/olostep, coming soon, with signing and SBOM enhanced security), GitHub Container Registry (ghcr.io/olostep/olostep-mcp-server). The server is being added to the official Docker Desktop MCP Toolkit, and users will be able to discover it in the MCP Toolkit UI in Docker Desktop, install it with one click, configure it visually, and use it with any MCP-compatible client (Claude Desktop, Cursor, etc.). Status: being submitted to the Docker MCP Registry.

Applicable Scenarios and Limitations

Suitable for scenarios that require AI agents to access web data in real time, such as content extraction, large-scale data collection, website structure analysis, question answering with citations, etc. In terms of limitations: an Olostep API key is required; batch scraping is capped at 10,000 URLs; crawling is an asynchronous two-step operation; some features (such as the Docker MCP Registry submission and Docker Desktop MCP Toolkit integration) are in progress or coming soon.chronous crawl status and scraped pages; it is a required companion tool for create_crawl. Parameters include crawl_id (required), formats (markdown, html, json, text, default ["markdown"]), items_limit (1-100, default 20), cursor (default 0), search_query (optional filter). While in progress, returns crawl_id, status (in_progress), pages_completed, pages_total, and a prompt to call again in about 10 seconds; when completed, returns crawl_id, status (completed), pages_returned, next_cursor, has_more, and a pages array.

Error Handling

The server provides robust error handling: detailed error messages for API issues, network error reporting, authentication failure handling, and rate limit information.

Distribution and Platforms

The MCP server is provided as a Docker image: Docker Hub (olostep/mcp-server), the official Docker MCP Registry (mcp/olostep, coming soon, with signing and SBOM enhanced security), GitHub Container Registry (ghcr.io/olostep/olostep-mcp-server). The server is being added to the official Docker Desktop MCP Toolkit, and users will be able to discover it in the MCP Toolkit UI in Docker Desktop, install it with one click, configure it visually, and use it with any MCP-compatible client (Claude Desktop, Cursor, etc.). Status: being submitted to the Docker MCP Registry.

Applicable Scenarios and Limitations

Suitable for scenarios that require AI agents to access web data in real time, such as content extraction, large-scale data collection, website structure analysis, question answering with citations, etc. In terms of limitations: an Olostep API key is required; batch scraping is capped at 10,000 URLs; crawling is an asynchronous two-step operation; some features (such as the Docker MCP Registry submission and Docker Desktop MCP Toolkit integration) are in progress or coming soon.ation) are in progress or coming soon.

Related tools

Browse category