Agent-R1

Agent-R1: Training powerful LLM agents with end-to-end reinforcement learning.

I. Purpose Agent-R1 is a unified, modular agent reinforcement learning (Agentic RL) framework for training multi-step LLM agents. It trains models through a step-native RL loop: the model observes the environment, generates actions, receives tool or environment feedback, and continues executing until the task is completed or terminated. Unlike single-turn RL pipelines that treat interaction as a single ever-growing prompt-response pair, Agent-R1 models each episode as a step-level MDP transition, making tool use, environment state, context management, reward assignment, and policy optimization explicit components of the same training foundation.

II. Background and Design Goals Modern LLM infrastructure already has powerful serving systems such as vLLM and SGLang, as well as powerful distributed training systems such as DeepSpeed, FSDP, and Megatron-LM. Agentic RL needs to reconnect the two sides into a rollout -> reward -> replay -> update loop, enabling the model to interact with tools and the environment over multiple turns.

Agent-R1 is built around three design goals:

1. Step-level trajectory representation: each transition stores the observation, action, environment feedback, reward, termination state, and next observation, while preserving action boundaries and avoiding fragile Token -> Text -> Token reconstruction.
2. Flexible context management: the environment determines what the model sees next, so history can be appended, truncated, summarized, rewritten, or augmented.
3. Algorithm-system decoupling: task workflows, environments, rollouts, rewards, advantage estimators, and policy objectives can evolve independently.

III. Core Idea: Step-level MDP In multi-turn agent training, the model does not merely continue a token sequence. Each model output may call a tool, change the environment state, receive external feedback, and shape the next observation. Therefore, Agent-R1 treats the agent step as the basic unit of interaction: a step records what the model saw, what action it produced, what feedback and reward the environment returned, and what observation should be exposed next. This step-level trajectory representation aligns rollout, replay, context construction, and credit assignment with real agent decisions, while still allowing token-level policy loss within each generated action. Agent-R1 uses hierarchical abstractions so that new tasks can reuse the same trainer without rewriting the entire RL stack.

1. Main Capabilities

• Step-level MDP training foundation and hierarchical abstraction.
• Online Policy Distillation (OPD) support (the opd branch provides general OPD training support).
• StepPO integration and training support.
• Recipe integration for HotpotQA, ALFWorld, WebShop, and academic paper search.
• Multimodal support for vision-language model agents.
• Process rewards for feedback on each tool call.
• Inference scripts and an interactive chat interface.
• A redesigned tool environment supporting more flexible agent-tool interaction patterns.
• Fixed GRPO and REINFORCE training crashes caused by NaN values.

1. Usage Agent-R1 uses the same environment setup as verl, and the current version requires verl==0.7.0. Simply clone this repository; there is no separate Agent-R1 installation step.

The main loop is:

1. Load samples containing prompt, agent_name, reward_model, and optional env_kwargs.
2. Create the configured AgentFlow and environment.
3. Generate actions from the current observation.
4. Parse the action, execute the tool or update the environment, and return feedback.
5. Record the step and continue until done=True or max_steps is reached.
6. Convert the structured trajectory into rewards, advantages, masks, and policy updates.

Recommended getting-started path:

• Read the Getting Started page for the minimal setup flow.
• Download the processed data release from ModelScope, then place or symlink each task's files to the path expected by the corresponding recipe.
• Use examples/gsm8k/run_steppo.sh as a sanity check that the environment is wired correctly.
• Go to the Agent Task Tutorial to see a minimal GSM8K + Tool example based on ToolEnv + BaseTool.
• Read Recipes and Algorithms to understand the current task integrations and launch script layout.

Processed data can be downloaded via the ModelScope CLI or git.

Phase 1: Basic training stack sanity check Using the processed data from the data release GSM8K file, or regenerate the minimal GSM8K dataset locally, then run the single-step script. This stage is only a setup check, used to confirm whether the environment, model path, dataset path, and training stack are wired correctly.

Stage 2: Try a minimal tool-calling example GSM8K + Tool is the simplest ToolEnv + BaseTool example. Use the processed GSM8K tool file from the data release, or regenerate the tool-augmented dataset locally, then launch the multi-step tool-calling script. This path uses the generic AgentEnvLoop with the built-in ToolEnv and the recipe-local calc_gsm8k_reward tool. The pure GSM8K script is still a single-turn environment integrity check.

VI. Building a new agent task For a new task, keep the trainer unchanged and implement the task-specific layer. A typical migration checklist:

• Data: output parquet rows containing prompt, reward_model, agent_name, and env_kwargs.
• Environment/tools: define how state updates, tool observations, rewards, and termination work.
• Agent flow: connect model actions to the environment loop and expose step records.
• Training script: set paths, rollout steps, batch size, estimator, and policy loss via Hydra overrides.

VII. Branch notes

• main contains the current v0.1.0 architecture, based on Step-level MDP and hierarchical abstraction.
• legacy keeps the previous implementation for reference. A newer verl source checkout containing the AgentFlow / asynchronous rollout stack required by this repository is needed.

VIII. Projects using Agent-R1

• TableMind: an autonomous programmatic agent for tool-augmented table reasoning.
• PaperScout: an autonomous academic paper search agent trained with Agent-R1 and Proximal Sequence Policy Optimization.
• Cast-R1: an agent framework that reformulates time series forecasting as sequential decision-making.
• StepPO: step-aligned policy optimization for agent reinforcement learning, a step-level Agentic RL method that treats agent steps as action units and aligns credit assignment with multi-turn agent decisions.

IX. Versions and release notes

• [2026.07.21] Support onlinepolicy distillation (OPD), general OPD training support is provided on the opd branch.
• [2026.05.30] Release a substantially revised Agent-R1 technical report, introducing step-level trajectory representation, flexible context management, and hierarchical abstraction for agent reinforcement learning.
• [2026.05.29] Integrate StepPO, expand recipe coverage, and release processed data; includes recipe integrations for HotpotQA, ALFWorld, WebShop, and academic paper search; processed datasets are available on ModelScope.
• [2026.03.23] Agent-R1 v0.1.0 is the first official release of the refactored architecture, introducing the Step-level MDP foundation and new hierarchical abstraction; the previous implementation is archived in the legacy branch.
• [2026.03.04] Release Claw-R1, extending Agentic RL to general agents such as OpenClaw through a middleware-style design.
• [2026.01.10] Release PaperScout.
• [2025.11.18] The Agent-R1 technical report is released on arXiv.
• [2025.05.06] Redesign the tool environment; fix crashes caused by NaN in GRPO and REINFORCE training.
• [2025.04.01] Add basic inference scripts and an interactive chat interface.
• [2025.03.18] Add multimodal support; move verl to a git submodule.
• [2025.03.16] Support process rewards for feedback on each tool call.

X. Acknowledgments This work was completed at the National Key Laboratory of Cognitive Intelligence, University of Science and Technology of China. Thanks to DeepSeek-R1, veRL, and RAGEN for their ideas and infrastructure, and to Professor Qi Liu and Professor Mingyue Cheng for their guidance and support.

Project homepage:https://agentr1.github.io/agent-r1 Documentation:https://agentr1.github.io/agent-r1/docs/ Repository:https://github.com/0russwest0/Agent-R1

Experiment snapshot: The Agent-R1 report evaluates Qwen3-4B in representative agent scenarios; the experimental setup, task coverage, optimizer comparison, and context management analysis are presented in the Experiments section.