What is WebShaper
WebShaper is an innovative AI training data synthesis system launched by Alibaba Tongyi Lab. Through formal modeling and an agent expansion mechanism, it provides high-quality, scalable data for training AI agents. WebShaper introduces for the first time the set theory-based concept of "Knowledge Projection" (KP), constructing complex question structures through intersection, union, and recursive operations on KPs, precisely controlling reasoning paths and task complexity. WebShaper's Expander agent can start from simple "seed questions" and gradually expand them into complex reasoning tasks, letting AI "create questions" on its own. The training strategy combines supervised fine-tuning (SFT) and GRPO reinforcement learning, enabling the model to perform excellently in complex information retrieval tasks.
WebShaper's Main Features
- Formal Modeling: WebShaper proposes for the first time a set theory-based formal modeling method for IS (Information Seeking) tasks. Through "Knowledge Projection" (KP), complex information seeking tasks are decomposed into multiple set operations (such as intersection, union, and recursive operations). Each KP is a set containing specific entities, and through these operations, complex question structures can be constructed, precisely controlling reasoning paths and task complexity.
- Agent Expansion Mechanism: A major innovation of WebShaper is letting AI "create questions" on its own. Through the Expander agent, the system starts from a simple "seed question" and gradually expands it into complex reasoning tasks. The Expander agent calls tools such as search, summarization, and verification to gradually construct more complex, logically clear questions and verify the correctness of answers. This ensures a clear reasoning chain and controllable task structure.
- High-Quality Data Generation: Through formal modeling and the agent expansion mechanism, the training data generated by WebShaper is no longer "guesswork" but controllable, interpretable, and scalable high-quality tasks. It breaks through the boundaries of pre-retrieved data, achieving broader task types, capability stimulation, and knowledge coverage, and reducing errors and redundant information in data synthesis.
- Agent Training Strategy: WebShaper adopts a combined strategy of supervised fine-tuning (SFT) and GRPO reinforcement learning, enabling AI agents to gradually master reasoning and retrieval capabilities in ambiguous, multi-hop information. Training starts from high-quality training trajectories, and a reward mechanism guides the model through multi-step reasoning, avoiding "taking shortcuts" or "guessing answers".
WebShaper's Technical Principles
-
Formalization-Driven Framework: WebShaper systematically formalizes information retrieval tasks using set theory, with the core concept of "Knowledge Projections" (KP). KP is a set of entities based on specific relations.
-
Knowledge Projection Operations
-
R-Union: Used to handle uncertain conditions, for example, "players who participated between 2000-2010" can be represented through a union operation.
-
Intersection: Used to handle multi-condition constraints, for example, "players who participated in 2000 and were born in the 90s".
-
-
Task Expansion Mechanism: WebShaper starts with "seed tasks" and uses an Expander to gradually expand question complexity. Based on the formalization framework and combined with retrieval and verification tools, the Expander expands simple questions into complex questions, ensuring logical consistency and task difficulty.
-
Data Synthesis and Training: The generated complex questions are converted into training data, and the model is trained through supervised fine-tuning (SFT) and reinforcement learning (such as the GRPO algorithm) to enhance the model's reasoning ability in complex information retrieval tasks.
WebShaper's project address
- Github repository：https://github.com/Alibaba-NLP/WebAgent
- HuggingFace model library：https://huggingface.co/datasets/Alibaba-NLP/WebShaper
- arXiv technical paper：https://arxiv.org/pdf/2507.15061
WebShaper's application scenarios
- Literature organization and analysis: WebShaper can help researchers quickly collect and organize relevant literature and conduct interdisciplinary knowledge discovery.
- Market research: WebShaper can be used for market research, competitive analysis, and investment decision support. Corporate analysts can have AI systems automatically collect industry data, analyze market trends, and evaluate competitors' strategies.
- Intelligent learning assistant: WebShaper can become an intelligent learning assistant, helping students engage in deep learning and research-based learning.
-
Life decisions: WebShaper can be used immediately in scenarios such as travel planning, health inquiries, and life decisions, providing users with personalized information support.
-
Medical information inquiries: WebShaper can help users inquire about medical and health information and provide professional medical advice and health consultation.