What is WebSailor
WebSailor is an open-source web agent from Alibaba Tongyi Lab, focused on complex information retrieval and reasoning tasks. Through innovative data synthesis methods (such as SailorFog-QA) and training techniques (such as rejection sampling fine-tuning and the DUPO algorithm), it performs excellently on highly difficult tasks, surpassing multiple well-known models in evaluations such as BrowseComp and topping the open-source web agent leaderboard. WebSailor's reasoning reconstruction technology can efficiently handle complex tasks and generate concise and precise reasoning chains. It performs excellently in complex scenarios and demonstrates strong generalization ability in simple tasks.
WebSailor's Main Features
-
Complex Task Data Synthesis: WebSailor uses the SailorFog-QA method to generate highly uncertain complex task data, simulating complex information search scenarios in the real world.
-
Multi-Turn Tool Calling and Reasoning Reconstruction: With the help of open-source reasoning models, WebSailor can perform multi-turn tool calling, reconstruct the reasoning process, and efficiently handle complex problems.
-
Reinforcement Learning Algorithm: It adopts the DUPO algorithm to optimize training efficiency through a dynamic sampling strategy, significantly improving the model's decision-making ability.
-
Information Retrieval and Analysis: WebSailor can proactively search and access multiple web pages, analyze the relationships between information, and provide complete and accurate answers.
WebSailor's Technical Principles
- Data Synthesis (SailorFog-QA): WebSailor first generates highly uncertain complex task data through the SailorFog-QA method. It uses "knowledge graph random walk" technology to select obscure entities from knowledge bases such as Wikidata as starting points, randomly expand the knowledge graph, and construct complex nonlinear relationship networks. By obscuring problem descriptions (such as changing specific years to time periods, hiding some information, etc.), it further increases the initial uncertainty of the task.
- Model Training (RFT Cold Start): In the training phase, WebSailor uses rejection sampling fine-tuning (RFT) for cold start. RFT performs preliminary alignment of the model by selecting high-quality problem-solving trajectories, helping the model master basic reasoning and tool-calling habits.
- Reinforcement Learning (DUPO Algorithm): After completing the cold start, WebSailor enters the reinforcement learning phase, adopting the DUPO (Duplicated Sampling Policy Optimization) algorithm. The core of DUPO lies in its dynamic sampling strategy: filtering out overly simple samples before training, and repeatedly sampling challenging trajectories during training. This greatly improves training efficiency and enables the model to iterate quickly on complex tasks.
WebSailor's Project Address
- Github Repository：https://github.com/Alibaba-NLP/WebAgent
WebSailor's Application Scenarios
-
Complex Information Retrieval: WebSailor can handle vague and complex query problems, and through multi-step reasoning and cross-validation, quickly locate and verify answers in massive amounts of information.
-
Multi-Hop Question Answering Tasks: In multi-hop question answering scenarios, WebSailor can gradually break down complex problems and find answers through multi-turn tool calling and reasoning.
-
Scientific Research and Data Analysis: WebSailor can help researchers and analysts quickly sort out complex information networks, synthesize information from multiple sources, and provide complete and accurate conclusions.
-
Training and Optimization for Highly Difficult Tasks: WebSailor's training dataset SailorFog-QA simulates complex real-world information search scenarios. By constructing high uncertainty and complex relationship networks, it can handle complex tasks that have no preset solution path.