rLLM is an open-source framework for training language agents via reinforcement learning (RL), with the goal of making reinforcement learning for large language models more accessible.
Main capabilities
- Arbitrary execution frameworks: supports more than 10 CLI execution frameworks (such as Claude Code, Codex, Terminus-2, mini-swe-agent, opencode, etc.) as well as Harbor-compatible task directories; custom agents such as LangGraph, OpenAI Agents SDK, and openai.OpenAI can also be wrapped via @rllm.rollout.
- Arbitrary sandbox environments: supports Docker, Daytona, Modal, or local environments, and provides snapshot and warm pool acceleration to keep rollout costs low at training scale.
- Multiple training backends with a unified API: supports verl (distributed multi-GPU), tinker (single machine), and fireworks (Fireworks platform), switchable with a single flag.
- More than 60 built-in evaluation benchmarks: covering mathematics, code, multiple choice, question answering, search, VLM, translation, agents, and more, such as Terminal-Bench 2.0, SWE-bench, SkillsBench, AIME, MATH-500, GPQA, etc. You can use rllm eval <name> to automatically pull and run them.
- Multiple training methods: GRPO, REINFORCE, RLOO, SFT, on-policy distillation, and more.
Usage rLLM requires Python >= 3.11. It can be installed directly via pip or built from source. The default installation runs the rLLM CLI to install dependencies for the tinker backend (single machine, Tinker API); other backends require referring to the relevant instructions. There are two usage modes:
- Option A: CLI (no code writing required).
- Option B: Python API. Define the rollout (agent) and evaluator (reward function), then hand them to the trainer. During training, config.base_url points to a gateway that transparently captures token IDs and logprobs, so agent code remains consistent between evaluation and training. For complete examples, refer to the cookbooks.
Workflow rLLM follows a simple pipeline: run the agent → collect trajectories → Compute rewards → Update the model. The agent runs as-is, and rLLM's model gateway captures LLM calls through URL-routed sessions (token IDs + logprobs) and structures them into an Episode (one task), which contains a Trajectory (one agent run) composed of Steps (one LLM call). The reward function scores the outcome, and the RL algorithm updates the model weights. The same agent code works for both evaluation and training. The underlying components include: a workflow engine that runs N parallel agent instances to collect rollouts; a model gateway that routes requests and captures token IDs + logprobs; a transformation pipeline that groups trajectories to compute advantages; and a training backend (verl, tinker, or fireworks) that handles policy updates.
Use cases and results This framework has been adopted by academic labs and industrial research teams and has achieved multiple state-of-the-art open-source results, such as DeepScaleR-1.5B, DeepCoder-14B, DeepSWE-32B, and FinQA-4B. Community projects span multiple directions, including Alibaba NLP's open-source AI researcher Tongyi DeepResearch, Terminal-Bench-RL for training long-horizon terminal agents with RL, PettingLLMs for multi-agent RL on-policy training, SETA for scaling terminal agent environments, LLM-in-Sandbox for running LLMs in a sandbox to build general agents, Vision-DeepResearch for the first long-horizon multimodal deep research MLLM, OpenSearch-VL for an open-source solution for frontier multimodal search agents, Cogito, Ergo Ludo for learning games through reasoning and planning, Cut the Bill, Keep the Turns for cost-effective multi-turn search RL, Experiential Reinforcement Learning for the experience-reflection-consolidation loop in sparse-reward RL, V1 for pairwise self-verification for parallel test-time scaling, TherapyGym for evaluating and aligning therapy chatbots' clinical fidelity and safety, SandMLE for a synthetic sandbox for training MLE agents, AxPO for agentic exploration policy optimization for multimodal agent reasoning, and Remember When It Matters, among others.
Additional information Related articles and blog posts include the rLLM UI real-time observability tool (March 2026), rLLM on-policy distillation (March 2026), an open-source solution for fully asynchronous training of deep research agents (February 2026), rLLM-FinQA: a 4B model surpasses 235B and rivals Gemini 2.5 Pro on financial analysis (February 2026), rLLM SDK: train arbitrary agent programs without modifying code (December 2025), rLLM v0.2: RL training for general agent programs (October 2025), DeepSWE: an open-source SWE agent through RL (July 2025), DeepCoder: a 14B coding model reaching O3-mini level (April 2025), and DeepScaleR: 1.5B surpasses O1-Preview (February 2025). This project was completed as part of the Berkeley Sky Computing Lab. The rLLM team receives funding from the Laude Institute, AWS, Hyperbolic, Fireworks AI, and Modal, and has research collaboration and compute support with Together AI. The related prior work DeepScaleR, DeepCoder, and DeepSWE can be cited.
