What is RAGEN
RAGEN is an open-source reinforcement learning framework for training large language model (LLM) reasoning agents in interactive, stochastic environments. Based on the StarPO (State-Thinking-Action-Reward Policy Optimization) framework, it optimizes the entire trajectory through multi-turn interactions and supports multiple optimization strategies such as PPO and GRPO. RAGEN formalizes the interaction between the agent and the environment through an MDP, and introduces a progressive reward normalization strategy, effectively addressing instability in multi-turn reinforcement learning. The code structure of RAGEN has been optimized and is divided into three modules: the environment manager, the context manager, and the agent proxy, making extension and experimentation convenient. It supports multiple environments, such as Sokoban, FrozenLake, and others, demonstrating good generalization ability.
Main Features of RAGEN
- Multi-turn Interaction and Trajectory Optimization: RAGEN formalizes the interaction between the agent and the environment as a Markov Decision Process (MDP) through the StarPO (State-Thinking-Actions-Reward Policy Optimization) framework, optimizing the entire interaction trajectory, not just single-step actions. The full-trajectory optimization strategy helps the agent make more reasonable decisions in complex environments.
- Reinforcement Learning Algorithm Support: RAGEN supports multiple reinforcement learning algorithms, including PPO, GRPO, and BRPO, providing researchers with flexible algorithm choices.
- Easily Extensible Environment Support: RAGEN supports multiple environments, including Sokoban, FrozenLake, and others, and provides interfaces for adding custom environments, making it convenient for researchers to conduct experiments.
- Improved Stability and Efficiency: RAGEN effectively improves training stability and efficiency through techniques such as variance-based trajectory filtering, the introduction of a "critic," and decoupled clipping.
Technical Principles of RAGEN
- MDP Formalization: RAGEN formalizes the interaction between the agent and the environment as a Markov Decision Process (MDP), in which states and actions are token sequences. It supports LLMs in reasoning about environment dynamics.
- StarPO Framework: The framework trains through two alternating phases:
- Rollout Phase: Given an initial state, the LLM generates multiple reasoning-guided interaction trajectories, receiving the trajectory history at each step and generating an action.
- Update Phase: After generating trajectories, importance sampling is used to optimize the expected reward of the entire trajectory rather than single-step optimization, enabling long-term reasoning.
- Optimization Strategies: StarPO supports multiple reinforcement learning algorithms, such as PPO (Proximal Policy Optimization) and GRPO (Normalized Reward Policy Optimization), adapting to different training needs.
- Progressive Reward Normalization Strategy: To address instability in multi-turn training, RAGEN introduces strategies such as uncertainty-based filtering, removal of the KL penalty, and asymmetric PPO clipping.
- Modular Design: RAGEN adopts a modular architecture, including an environment state manager, a context manager, and an agent proxy, making extension and customization convenient.
RAGEN Project Address
- Project Website：https://ragen-ai.github.io/
- Github Repository：https://github.com/RAGEN-AI/RAGEN
- Technical Paper：https://ragen-ai.github.io/pdf/RAGEN.pdf
Application Scenarios of RAGEN
- Intelligent Dialogue Systems: RAGEN can be used to train dialogue systems, giving them better reasoning ability in interactions with users and providing more natural and accurate responses.
- Game AI: In complex and dynamic game environments, RAGEN can help agents carry out reasonable strategy planning and execution.
- Automated Reasoning: RAGEN can be applied to automated reasoning scenarios such as mathematical problem solving and programming tasks, improving the system's problem-solving capabilities.
- Enterprise Knowledge Management: RAGEN can be used for internal enterprise document assistants, locating information from company wikis and meeting minutes, and generating project reports or meeting summaries.
- Legal Consultation: In the legal field, RAGEN can match relevant legal provisions and precedents, and explain legal risks in plain language.
- Content Creation: RAGEN can be used for scenarios such as technical blog writing and news report generation. By retrieving GitHub code examples, technical documentation, and more, RAGEN can integrate information and output structured tutorials.