SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI developer platformsRAGEN

RAGEN

RAGEN is an open-source reinforcement learning framework for training reasoning LLM agents, based on the StarPO algorithm and supporting multi-turn trajectory-level reinforcement learning. It provides 10 built-in environments (such as Sokoban, FrozenLake, WebShop, DeepCoder, SearchQA, etc.) and a Gym-compatible interface, making it easy to add custom environments. The RAGEN-2 version further introduces SNR-Adaptive Filtering, a lightweight trajectory filtering method based on reward variance, as well as reasoning collapse diagnostics (detecting template collapse through conditional entropy and mutual information proxy metrics), helping to understand and fix hidden failure modes in agent reinforcement learning (such as Echo Trap and reasoning collapse). The framework supports two optimization methods, PPO and GRPO, and includes tools such as a diagnostics dashboard and training metrics guide.

Open website

About this tool

RAGEN (Reasoning AGENT) is a flexible reinforcement learning framework for training reasoning LLM agents through multi-turn reinforcement learning, built on the StarPO algorithm. The project was developed by MLL Lab (Manling Li's team at Northwestern University), aiming to diagnose the working mechanisms of agent reinforcement learning training and fix hidden failure modes such as Echo Trap (RAGEN) and reasoning collapse (RAGEN-2).

[Purpose] RAGEN targets research and engineering scenarios that require training multi-turn, trajectory-level LLM agents, providing a unified reinforcement learning optimization framework that supports flexible control over the reasoning process, reward assignment mechanisms, and prompt-rollout structure.

[Main Capabilities]

  1. StarPO framework: uniformly optimizes multi-turn agents, supporting trajectory-level and turn-by-turn training.
  2. 10 built-in environments: Sokoban, FrozenLake, WebShop, DeepCoder, SearchQA, Lean, Bandit, Countdown, MetaMathQA, Sudoku.
  3. Gym-compatible interface: makes it easy to add custom environments (see config/envs.yaml).
  4. Optimization methods: supports PPO (token-level advantage estimation via a value function) and GRPO (assigning normalized rewards to the entire trajectory).
  5. New in RAGEN-2: SNR-Adaptive Filtering (V2), lightweight rollout filtering based on reward variance to mitigate noisy gradient updates; reasoning collapse diagnostics (V2), providing mutual information proxy metrics to detect and monitor template collapse during training.
  6. Diagnostics and training quality: provides an extended benchmark suite, an extended MI diagnostics dashboard (with WandB visualizations of entropy, MI proxies, and gradient decomposition), and an RL training metrics guide.
  7. Four reasoning mechanisms: diagnose agent behavior along two axes (within-input diversity—conditional entropy H(Z|X); cross-input distinguishability—mutual information I(X;Z)); template collapse (high entropy, low MI) cannot be discovered by existing entropy-based metrics.

[Usage] Installation: run bash scripts/setup_ragen.sh --with-search to include the search environment; for the WebShop environment, see docs/experiment_webshop_release.md. Training: default training without filtering; canFiltered training using SNR-Adaptive Filtering (V2, Top-p). For more filtering strategies (Top-k, linear mode, etc.), see the Rollout Filtering Guide. Evaluation and configuration: see the Evaluation Guide (how to evaluate models and configure output formats), MI Metrics Reference, and the guide to adding custom environments. Experiment reproduction: main table, intervention sweep, FrozenLake, Sokoban gradients, Search, DeepCoder, and WebShop all have corresponding reproduction instructions.

[Applicable Scenarios] Suitable for LLM agent reinforcement learning research, especially scenarios such as multi-turn interaction, trajectory-level optimization, reasoning capability training, and training failure mode diagnosis. Related or inspired projects include ROLL, VAGEN, Search-R1, ZeroSearch, Agent-R1, OpenManus-RL, MetaSpatial, s3, etc.

[Limitations] The official documentation (released 2025.5.8) has been marked as outdated. RAGEN is developing the next-generation infrastructure and diagnostic tools, planned for release by the end of March 2026, including an asynchronous rollout engine, HTTP-based environment interfaces, hierarchical Env Wrapper, optional environment dependencies, an extended benchmark suite, an extended MI diagnostic dashboard, multimodal agent support (based on VAGEN), a public leaderboard, etc. Specific pricing, cost, platform support, and other information for the current version are not yet available.

Related tools

Browse category