What is WebRL
WebRL is a self-evolving online curriculum reinforcement learning framework jointly launched by Tsinghua University and Zhipu AI, which trains high-performance web agents using open large language models (LLMs). WebRL dynamically generates tasks, uses an outcome reward model (ORM) to evaluate whether tasks are successful, and employs adaptive reinforcement learning strategies to address challenges such as scarce training tasks, sparse feedback signals, and policy distribution drift in online learning. WebRL significantly improves the success rates of models such as Llama-3.1 and GLM-4 on the WebArena-Lite benchmark, surpassing proprietary LLM APIs and previously trained web agents, demonstrating its effectiveness in enhancing the web task capabilities of open-source LLMs.
Main Features of WebRL
- Self-Evolving Curriculum Learning: WebRL can generate new tasks from failed attempts, dynamically adjust the difficulty and complexity of tasks, and adapt to the agent's current skill level.
- Outcome Reward Model (ORM): WebRL trains an ORM to evaluate whether a task is successful, provides a binary reward signal (1 for success, 0 for failure), and guides the agent's learning process.
- Adaptive Reinforcement Learning Strategy: Based on a policy update algorithm constrained by KL divergence, WebRL limits distribution drift during policy updates, ensuring that the agent does not deviate too far from existing knowledge when learning new tasks.
- Experience Replay Buffer: WebRL uses an experience replay buffer to retain previous successful experiences, mitigate the risk of catastrophic forgetting, and reuse experience during training.
- Continuous Performance Improvement: Based on iterative self-evolution, WebRL enables the agent to continuously and consistently improve performance in online environments.
Technical Principles of WebRL
- Problem Formulation: WebRL models web tasks as a finite-horizon Markov decision process (MDP), defining states, actions, rewards, and transition probabilities.
- ORM Training: Based on training an LLM as the ORM, it automatically evaluates whether the agent's execution trajectory successfully completes the task and provides feedback signals.
- Reinforcement Learning: In online web environments, WebRL uses a self-evolving curriculum learning strategy to dynamically generate tasks and uses a KL-constrained policy update algorithm to prevent large drift in the policy distribution.
- Experience Replay: Based on an experience replay buffer to retain previous knowledge, mitigate the risk of catastrophic forgetting, and use stored successful trajectories to avoid the challenge of accurately estimating the intermediate states of erroneous trajectories.
- Self-Evolving Curriculum Learning Strategy: WebRL implements a two-step process of generation and filtering, generating tasks that are gradually more challenging while still suitable for the agent's current capabilities, and creates new instructions based on the In-breadth evolving technique.
- Policy Update: WebRL considers the KL divergence between the new and old policies during policy updates, ensuring smooth policy transitions and avoiding performance degradation caused by policy updates.
WebRL Project Address
- GitHub Repository：https://github.com/THUDM/WebRL
- arXiv Technical Paper：https://arxiv.org/pdf/2411.02337v1
Application Scenarios of WebRL
- Web Browsing Automation: WebRL trains agents to automatically complete web browsing tasks, such as information retrieval, filling out forms, online shopping, etc.
- Web Data Extraction: In scenarios that require extracting specific data from web pages (such as prices, reviews, news articles), WebRL helps automate the data extraction process.
- Customer Service Automation: In the field of customer service, as a chatbot, it uses web interaction to resolve user issues or complete transactions.
- Web Content Management: For website administrators who need to manage large amounts of web content, WebRL automates content updates, publishing, and maintenance tasks.
- E-commerce: On e-commerce platforms, it helps automate order processing, inventory management, and customer interaction.