cn-scraper-mcp

Purpose

cn-scraper-mcp aims to let AI Agents search the Chinese internet directly. Every AI Agent (such as Codex, Claude Code, Cursor, Trae) can search web pages, but Chinese platforms usually require a login state, a browser environment, or platform-specific parameters. This project packages the relevant adaptations into an MCP Server, so an Agent can initiate a search in one sentence, for example taobao_search("儿童学习桌").

Supported Platforms and Capabilities

• Taobao: Requires a network fingerprint consistent with a browser and login Cookies. There is no hard rate limit, but the platform may tighten restrictions at any time, so high-frequency batch scraping is not recommended.
• JD: Depends on a logged-in local browser environment; the local Chrome generates the login state and dynamic signature, and the tool reads structured API responses, through guided_login("jd") can automatically initialize a persistent Profile.
• Xiaohongshu: Requires a residential IP, a local browser, and access parameters from search results; only residential IPs are allowed. Cloud browsers/data center IPs are blocked directly, and local Chrome must be used.
• Zhihu: Guest search has been closed, and all APIs require a login state.
• Pinduoduo: The platform has strict restrictions. Each browser session allows only the first search, after which it permanently shows “system busy.” A single search result has zero practical value. The engine code is retained but not recommended for use.
• Weibo: The search API requires a login state (SUB token), while hot search is accessible to guests.
• Douyin: Requires browser login and may require manual captcha handling; search requires a login state plus manually passing a slider captcha, supports a 120s wait for the user to manually verify, and automatically scrapes after passing,guided_login("douyin") can guide login.
• Bilibili: Search, trending, video details, and comments can directly use public APIs.
• Douban: Item search, item details, and short reviews/film reviews.
• Dianping: Merchant search, merchant details, and user reviews.
• Knowledge Planet: Paid communities, with content hidden behind a cookie-authenticated REST API.

Usage

Login and Cookies

Using CDP for automatic login and saving Cookies is recommended: after installing and connecting the MCP, directly have the Agent call the login tool. It will open local Chrome and go to the platform’s official login page; after the user scans the QR code or enters the password, the tool automatically reads the complete Cookies through CDP (including HttpOnly Cookies that JavaScript cannot read), then saves them locally to ~/.cn-scraper-cookies/. For JD, it is saved to a local persistent Chrome Profile. Available platform names include taobao、jd、xiaohongshu、zhihu、weibo、zsxq、douyin、pdd、douban and dianping. If there is already a logged-in Chrome started through a remote debugging port, the corresponding tool can also be called.

Installation and Connection

Chromium is preinstalled in the container, so no local browser is needed. For remote server deployment, you can switch to HTTP transport and connect via IP + port, or use Docker Compose. The remote MCP endpoint is configurable.

Note: Please replace /本机绝对路径/ with the real path; when the MCP client starts the process directly, it will not expand ~for you. The Docker image has built-in Chromium plus --no-sandbox; if JD headful mode requires Xvfb, set the environment variable XVFB_WRAPPER=1; Xiaohongshu still requires a residential IP—data center IPs will be blocked.

Client Configuration

• Codex:~/.codex/config.toml, after saving you can use codex mcp list to check the connection status.
• Claude Code / Cursor / Reasonix: These three clients all support the standard mcpServers JSON. For Claude Code, it is the project root directory .mcp.json; for Cursor, it is the global ~/.cursor/mcp.json, or the project directory .cursor/mcp.json; for Reasonix, it is the project root directory .mcp.json.
• Trae: The configuration file location may differ between versions. It is recommended to add a local stdio Server in the MCP management interface in settings: for the name, enter cn-scraper, for the command, enter cn-scraper-mcp, and leave the arguments blank.

If the client reports that the command cannot be found, first use where cn-scraper-mcp(Windows) or which cn-scraper-mcp(macOS/Linux) to find the full path, then replace command with that path.

Security and Privacy

cn-scraper-mcp runs locally and does not need to upload cookies, account passwords, or browser profiles to any relay server. Cookies are stored by default in ~/.cn-scraper-cookies/, and the JD login state is stored in the local Chrome profile. The login process takes place directly on the platform's official page, and the software does not read or save account passwords. Cookie values are not written to logs, nor are they returned to the Agent through MCP tool results; the tools only return non-sensitive information such as status, field names, and local paths. When initiating scraping or online login verification, credentials are only sent to the corresponding platform domain. The code is fully open source, and all credential handling processes can be reviewed. It is still recommended to protect local accounts just as you would protect a browser login state: do not share cookie files, do not commit credentials to Git, and restrict access permissions for local files.

Remote HTTP mode will cause MCP tools to access the local Cookie/Profile directory over the network. Do not expose public ports directly. It is recommended to place it behind an intranet, VPN, firewall whitelist, or reverse proxy with authentication. For platforms such as Xiaohongshu, JD, and Douyin that depend on a local browser, residential IP, or manual CAPTCHA, stability on a remote server depends on the server's network and graphical environment.

Manual Handling Mechanism

When a platform requires manual handling of login, CAPTCHA, or risk control pages, the tool returns a unified ACTION_REQUIRED structure and explains the platform, reason, handling action, handling link, and suggested retry tool in the action_required field. The current Douyin CAPTCHA has been integrated into this structure; in remote Docker scenarios, users still need to complete verification through a visible browser or a subsequent noVNC web page.

Applicable Scenarios and Limitations

Suitable for learning and research scenarios that require AI Agents to access Chinese platform data. In terms of limitations: Pinduoduo is not recommended; Xiaohongshu must use a residential IP and local Chrome; Douyin requires a login state and manual CAPTCHA completion; Taobao is not recommended for high-frequency batch scraping; the stability of some platforms deployed on remote servers depends on the server's network and graphical environment. This project uses the MIT license and is for learning and research purposes only. Batch scraping may violate platform terms of service, and you bear the risk yourself. Do not use it for spam, DDoS, or commercial-scale large-scale scraping. Project dependencies include curl_cffi (TLS fingerprint spoofing), FastMCP (MCP Server framework), and websockets (asynchronous WebSocket).