What is LoHoSearch
LoHoSearch is a next-generation search agent evaluation benchmark launched by Meituan's LongCat team. It automatically generates questions based on a knowledge graph covering 7.62 million Wikipedia entities and 265 million relationship edges, replacing the traditional manual question-writing model. The benchmark includes 544 high-quality questions that have been manually verified, covering 11 domains such as music, film and television, and geography, creating challenges by controlling two dimensions: search space and structural complexity. The current strongest model GPT-5.5 has an accuracy of only 34.74%, far below its performance of over 90% on BrowseComp, providing a more discriminative evaluation standard for long-horizon search reasoning and context management.
Main Features of LoHoSearch
- Automated Knowledge Graph Construction: Uses the complete English Wikipedia as the data source to build an ultra-large-scale knowledge graph with 7.62 million entity nodes and 265 million directed edges. Each entity is annotated with its Wikidata type and in-degree popularity, providing a foundation for question selection from a global perspective.
- Dual-Dimension Difficulty Control: Systematically regulates question difficulty from two dimensions, search space and structural complexity, breaking through the cognitive ceiling of manual question writing.
- Tree-Structure and Graph-Structure Sampling: The tree structure creates difficulty by amplifying the search space, while the graph structure introduces circular dependencies and cross-constraints on top of a huge search space, further layering on structural complexity.
- Automated QA Generation and Verification: Extracts Wikipedia descriptions from subgraphs and rewrites them into natural language questions, followed by three layers of automatic screening—relation extraction, subgraph coverage checking, and answer satisfaction verification—supplemented by manual review to ensure quality.
- Multi-Model Performance Evaluation: Supports standardized evaluation of mainstream search agents, outputting core metrics such as accuracy, number of tool calls, and calibration error, intuitively reflecting the model's long-horizon reasoning ability.
Technical Principles of LoHoSearch
- Automated Knowledge Graph Construction: Uses the complete English Wikipedia as the data source, extracting 7.62 million entity pages as nodes and 265 million hyperlinks in the body text as directed edges, annotating each entity with its Wikidata P31 type and in-degree popularity to build a global knowledge network.
- Dual-Dimension Difficulty Definition: Quantifies question difficulty along two orthogonal dimensions, "search space" and "structural complexity," breaking through the cognitive limitations of manual question writing.
- Tree-Structure Subgraph Sampling: Creates difficulty by amplifying the search space, expanding multiple independent branches downward from a single answer node, with each branch corresponding to a constraint condition with a high candidate space, causing the cost of elimination to rise exponentially.
- Graph-Structure Subgraph Sampling: Introduces circular dependencies and cross-constraints on the basis of a huge search space. Multiple condition nodes are interconnected with each other, and solving requires simultaneously satisfying multiple sets of interlocking relationships, layering structural complexity to form a dual challenge.
- Relation Extraction and Natural Language Generation: Extracts the Wikipedia description of each entity from the sampled subgraph and uses a large model to rewrite it into a coherent natural language question, ensuring that each clue fully preserves the relational information in the original subgraph.
How to Use LoHoSearch
- Load the Data: Directly load via the
datasetslibrary to loadmeituan-longcat/LoHoSearch, obtaining 544 questions and standard answers. - View the Structure: The dataset contains question text, corresponding subgraph structures, domain labels, and unique answers verified by the knowledge graph.
- Evaluation Integration: Input the questions into the search agent under test, and record its reasoning trajectory and final output.
- Automatic Scoring: Compare against the standard answers provided by the dataset, verify whether the model output satisfies all constraints, and calculate the accuracy.
- Comparative Analysis: Use the subgraph complexity and domain labels included with the dataset to analyze the model's performance across different difficulty levels and topics by dimension.
Core Advantages of LoHoSearch
- Breaking Through the Ceiling of Manually Authored Questions: Traditional manual benchmarks are limited by the range of entities known to the annotators, whereas LoHoSearch relies on a global knowledge graph and can systematically identify "truly hard problems" with a high search space and high structural complexity.
- Significantly More Difficult Than Existing Benchmarks: The same model achieves 58.84% accuracy on BrowseComp but only 10.02% on LoHoSearch, while the average number of tool calls increases from 35 to 61, raising the challenge intensity by 74%.
- Stronger Discrimination: Existing context management strategies improve performance by 14.03% on BrowseComp but only 6.8% on LoHoSearch, which more faithfully reflects the marginal benefits of strategies in long-horizon complex scenarios.
- Independently Controllable Structural Complexity: Accuracy on graph-structured questions (8.01%) is significantly lower than on tree-structured questions (11.89%), proving that structural complexity is an additional source of difficulty independent of the search space and can be quantified separately.
LoHoSearch Project Address
- HuggingFace Model Repository：https://huggingface.co/datasets/meituan-longcat/LoHoSearch
- arXiv Technical Paper：https://arxiv.org/pdf/2606.12837
Comparison of LoHoSearch with Similar Competing Products
| Dimension | LoHoSearch | BrowseComp |
|---|---|---|
| Question Generation Method | Automated generation from knowledge graph + manual verification | Manual design |
| Number of Questions | 544 questions | Several hundred questions |
| Covered Domains | 11 domains (music, film and television, geography, etc.) | Domains not explicitly specified |
| Difficulty Control | Dual dimensions: search space + structural complexity | Manual subjective control |
| Strongest Model Accuracy | GPT-5.5：34.74% | GPT-5.5 Pro：90.1% |
| Median Number of Tool Calls | 59 calls | 26 calls |
| Context Strategy Improvement | +6.8% | +14.03% |
| Saturated state | Far from saturated, with high discrimination | Approaching saturation |
| Applicable Scenarios | Long-horizon search reasoning, context management evaluation | Basic search capability evaluation |
Application Scenarios of LoHoSearch
- Search Agent Capability Evaluation: Provide high-difficulty standardized tests for search agents with long-horizon reasoning capabilities, precisely identifying the model's weaknesses in complex information retrieval chains.
- Context Management Strategy Validation: Evaluate the true gains of context compression and verification strategies such as Summary, Discard-all, and Verify in ultra-long interaction scenarios, avoiding overestimation of strategy effectiveness on simple benchmarks.
- Model Iteration Benchmarking: Provide research teams with a non-saturated evaluation standard to support capability boundary probing and performance comparison during model version iteration.
- Search Algorithm Research: Provide a dataset and evaluation framework with controllable difficulty for research on complex search algorithms that require multi-step reasoning and cross-verification.
- Agent Product Selection: Helps enterprises distinguish the actual performance of different models in real complex query scenarios through LoHoSearch when searching for agent solutions for product selection.