SearchClaw - An AI Deep Research Agent Launched by Renmin University of China

What Is SearchClaw

SearchClaw is a self-hosted AI deep research agent launched by RUC-NLPIR (Information Retrieval Lab, Renmin University of China), equipped with a Web interactive interface. After a user submits a question, the system automatically performs multiple rounds of web searches, page scraping, paper retrieval, and content synthesis, ultimately generating a detailed report with citation links. The project is built on FastAPI and ensures research quality through a "tools + hooks" architectural design. It supports multiple LLM providers and persistent memory, making it suitable for researchers and analysts conducting traceable automated investigations.

Main Features of SearchClaw

• Autonomous multi-round research loop: Automatically and iteratively performs searching, scraping, reading, and citing until the answer meets the standard.
• Multi-source information retrieval: Integrates web (Google/Serper), academic (Semantic Scholar/DBLP/arXiv), news (NewsAPI/Google News RSS), and WeChat official account article search.
• Deep browser integration: Renders JavaScript pages via Playwright/CDP and supports scraping with login sessions.
• Intelligent follow-up clarification: Can ask users follow-up questions during the research process to clarify requirements.
• Research plan decomposition: Automatically breaks down complex queries into trackable subtasks and executes them step by step.
• Quality gating system: Built-in hooks check the number of citations, source diversity, and answer completeness, and continue research if the standards are not met.
• Context compression management: A two-stage compression mechanism ensures long sessions do not exceed the context window limit.
• Cross-session persistent memory: Automatically saves and reuses high-quality sources, user preferences, and key facts from past research.

How to Use SearchClaw

• Clone the repository and install dependencies：git clone https://github.com/RUC-NLPIR/SearchClaw.git && pip install -e .
• Configure API keys (LLM and search sources): Set ANTHROPIC_API_KEY or OPENAI_API_KEY, and it is recommended to configure SERPER_API_KEY and JINA_API_KEY to improve search quality.
• Start the service：python -m src.main, and visit in the browser http://localhost:8000。
• Return results: Enter a research question, and the system streams back a comprehensive report with citations in real time over WebSocket.

Key Information and Usage Requirements of SearchClaw

• Environment requirements: Python 3.11+, supporting Linux/macOS/Windows.
• LLM support: Routed through litellm, compatible with mainstream models such as Anthropic, OpenAI, Google Gemini, xAI, Alibaba Tongyi, ByteDance Doubao, Zhipu GLM, Moonshot, as well as local vLLM/Ollama endpoints.
• Search dependencies: Automatically degrades to DuckDuckGo scraping when Serper is unavailable, and falls back to direct HTTP requests when Jina is unavailable.
• Optional browser: Execute pip install -e '.[browser]' && playwright install chromium to enable JS rendering support.
• Security configuration: For remote deployment, set an access password via SEARCH_CLAW_API_KEY to prevent unauthorized use.
• Persistent storage: Memory and session data are stored by default in JSON format on the local file system.

Core Advantages of SearchClaw

• Verifiable citation quality: Forcing multi-source citations and passing gate checks reduces hallucination risk, outperforming the fragmented information stitching of single-round RAG.
• Autonomous Planning and Error Correction: Automatically decomposes complex tasks and self-drives supplementary research when quality standards are not met, with no need for manual intervention in the iteration process.
• Multi-Source Heterogeneous Integration: The only open-source research tool that simultaneously covers English academic databases, Chinese WeChat public accounts, and real-time news, adapting to cross-border research scenarios.
• Fully Self-Hosted: Data never leaves the local environment, supports local LLM endpoints, and meets privacy compliance requirements for sensitive research data.

SearchClaw's project address

• GitHub repository：https://github.com/RUC-NLPIR/SearchClaw

Comparison of SearchClaw with similar competing products

SearchClaw's application scenarios

• Academic Literature Review: The tool can automatically retrieve papers from Semantic Scholar and arXiv, and generate field progress summaries with citations.
• Market Competitor Monitoring: Automatically tracks news sources and WeChat public accounts, analyzes competitor developments, and outputs timestamped reference evidence.
• Policy and Regulatory Research: The tool can scrape government websites and think tank reports, and integrate multilingual sources to form compliance analysis reports.
• Technology Selection Evaluation: It can search technical blogs, GitHub discussions, and academic papers to compare community reviews and performance data of different frameworks.