RAGEN

RAGEN (Reasoning AGENt) is a flexible reinforcement learning framework for training reasoning LLM agents, built on the StarPO algorithm and supporting multi-turn reinforcement learning. The project also develops diagnostic tools to understand how agent reinforcement learning training works and to fix hidden failure modes such as the Echo Trap (RAGEN) and reasoning collapse (RAGEN-2).

Purpose: To train LLM agents with reasoning capabilities, optimize their reasoning and action strategies in environments through multi-turn reinforcement learning, and diagnose and improve the stability of the training process.

Main capabilities:

1. StarPO framework: unified optimization for multi-turn agents, supporting trajectory-level and turn-level training.
2. 10 built-in environments: Sokoban, FrozenLake, WebShop, DeepCoder, SearchQA, Lean, Bandit, Countdown, MetaMathQA, Sudoku.
3. Gym-compatible interface: makes it easy to add custom environments.
4. RAGEN-2 additionally introduces:
   • SNR adaptive filtering (V2): lightweight rollout filtering based on reward variance, used to mitigate noisy gradient updates.
   • Reasoning collapse diagnostics (V2): detects and monitors template collapse during training through a mutual information proxy metric.

StarPO principle: StarPO (State-Thinking-Action-Reward Policy Optimization) contains two interleaved phases: the rollout phase and the update phase. The LLM generates reasoning-guided actions, interacts with the environment, collects trajectory-level rewards, and jointly optimizes the reasoning and action policies.

• MDP modeling: The interaction between the agent and the environment is modeled as a Markov decision process, where both states and actions are token sequences, enabling the LLM to reason about environment dynamics, with the goal of maximizing the expected cumulative reward of multi-turn interactions.
• Rollout phase: Given an initial state, the LLM generates multiple trajectories. At each step, the model produces a reasoning-guided action: <think>...</think><ans> action </ans>, and the environment returns feedback (reward and next state).
• Update phase: StarPO uses importance sampling to optimize the entire trajectory, supporting PPO (token-level advantage estimation through a value function) and GRPO (assigning normalized rewards to the entire trajectory).

V2 template collapse diagnosis: Entropy alone cannot detect template collapse (reasoning appears diverse within a single input but becomes input-independent across inputs). RAGEN-2 decomposes reasoning quality into two dimensions:

• Within-input diversity: conditional entropy H(Z|X)
• Cross-input distinguishability: mutual information I(X;Z) SNR adaptive filtering uses reward variance as a lightweight proxy to select high-signal prompts at each iteration, directly targeting the root cause of template collapse.

Four reasoning modes: RAGEN-2 diagnoses agent behavior along two axes—within-input diversity (conditional entropy) and cross-input distinguishability (mutual information)—yielding four distinct reasoning modes. Input-driven reasoning adapts to the current state; templated reasoning produces nearly identical responses across different inputs. Template collapse (high entropy, low mutual information) is invisible to existing entropy-based metrics.

Usage: Running bash scripts/setup_ragen.sh --with-search can include search environments. For WebShop-related configuration, see docs/experiment_webshop_release.md. Training can be done without filtering (default), or with SNR adaptive filtering (V2, Top-p). For more filtering strategies (Top-k, linear mode, etc.), see Rollout Filtering Guide.

Applicable scenarios: Research or engineering scenarios that require training multi-turn reasoning LLM agents, such as agent reinforcement learning in environments like Sokoban, FrozenLake, WebShop, DeepCoder, SearchQA, etc., and that need to diagnose training quality, identify template collapse or reasoning collapse, and other issues.

Limitations: The official documentation states that some documents are outdated, and an updated version will be released later; currently no information on fees, pricing, etc. is provided.