MindSearch - An AI Search Framework Launched by Shanghai AI Laboratory

What Is MindSearch

MindSearch is an open-source AI search framework launched by the joint research team of Shanghai AI Laboratory, combining large-scale information gathering and organization capabilities. Using the InternLM2.5 7B dialogue model, MindSearch can gather effective information from over 300 web pages within 3 minutes, completing tasks that typically take humans 3 hours. It uses a multi-agent framework to simulate human thinking, planning before searching, improving the accuracy and completeness of information. The project has been fully open-sourced, allowing users to experience and deploy it locally for free.

MindSearch's Main Features

• Complex Query Processing: Decomposes complex user queries into multiple smaller, manageable sub-questions to search for relevant information more precisely.
• Dynamic Graph Construction: By constructing a directed acyclic graph (DAG), it simulates the human thought process when solving problems, gradually refining the problem and exploring solutions.
• Parallel Information Retrieval: Based on a multi-agent architecture, it enables parallel search of multiple sub-questions, improving the speed and efficiency of information retrieval.
• Hierarchical Retrieval Strategy: WebSearcher adopts a coarse-to-fine retrieval strategy, first broadly gathering information, then selecting the most valuable pages for in-depth reading and information extraction.
• Context Management: Effectively manages context information in a multi-agent system, ensuring information consistency and completeness during information retrieval and integration.
• Response Generation: Synthesizes the retrieved information to generate accurate, comprehensive, and in-depth responses to answer the original complex query.
• Performance Improvement: In closed-set and open-set question-answering tasks, the above features significantly improve the quality of answers, including depth and breadth.
• Human Preference: The generated responses are more aligned with human preferences. Compared with other AI search engines, the answers provided by MindSearch are more favored by human evaluators.

MindSearch's Technical Principles

• WebPlanner: As an advanced planner, it is responsible for decomposing user queries into multiple sub-questions and, through a dynamic graph construction process (directed acyclic graph, DAG), simulating the mental model of multi-step information seeking.
• WebSearcher: Executes hierarchical information retrieval, retrieving and summarizing valuable information from the internet based on the sub-questions assigned by WebPlanner.
• Multi-Agent Collaboration: WebPlanner and WebSearcher act as independent agents, handling problem decomposition and information retrieval tasks respectively, enabling parallel task processing and effective information integration.
• Dynamic Graph Construction: Through code generation and execution, it dynamically constructs a logic graph for problem solving, enabling the LLM to gradually refine the problem and retrieve relevant information.
• Context Management: Effectively transfers contextual state between multiple agents, ensuring that key information is not lost during information retrieval and integration.

MindSearch's Project Address

MindSearch's Application Scenarios