SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI developer platformsprime-rl

prime-rl

prime-rl is a training framework for large-scale reinforcement learning (RL), supporting asynchronous RL and agent training. It uses FSDP2 for training and vLLM for inference, scales to 1000+ GPUs, supports 1T+ MoE models, multi-node Slurm deployment, multimodal VLMs, end-to-end post-training with SFT/RL/evaluation, and natively integrates verifiers environments through the Environments Hub. The project is licensed under Apache 2.0, with code open-sourced on GitHub, and the community can contribute through issues and PRs.

Open website

About this tool

prime-rl is a framework for large-scale reinforcement learning, emphasizing ease of use and modifiability while scaling to 1000+ GPUs.

Purpose and Positioning prime-rl is aimed at large-scale RL training, with a particular focus on high-throughput agentic training. It supports end-to-end post-training, including SFT, RL training, and evaluations (evals).

Key Capabilities

  • Fully asynchronous RL for large-scale, high-throughput agent training.
  • Performance: built for training 1T+ MoE models on 1000+ GPUs, using FSDP2 for training and vLLM for inference, with support for FP8 inference, PD disaggregation, EP and CP parallelism, etc.
  • Native integration of verifiers environments through the Environments Hub, with built-in support for SWE and agentic environments.
  • End-to-end post-training: SFT, RL training, and evaluation.
  • Multi-node deployment with Slurm support.
  • Multimodal support, suitable for VLMs such as Qwen3.5.
  • Modifiable, modular, extensible design.
  • One-line SLURM deployment for frontier models, such as GLM-5 FP8 with P/D disaggregation, llm-d router, and Mooncake KV offload.

Model Support The trainer runs PrimeRL model implementations under src/prime_rl/trainer/models/: dense Llama, Qwen3, and Qwen3.5, as well as other families in the table, with expert parallelism (EP) support for MoE layers, context parallelism (CP) support for long sequences, and additional kernels such as quack-kernels. Other architectures will fail during the trainer setup stage. Development and testing hardware includes NVIDIA RTX 3090/4090/5090, A100, H100, H200, and B200.

Usage Using prime-rl currently requires at least one NVIDIA GPU. Installation can be done with a single command: clone the repository, initialize submodules, and install dependencies from the lock file. Note: environments are optional (opt-in) uv workspace members,uv sync --all-extras they are not installed; if you need to train on these environments, you can use uv sync --all-extras --all-packages to install all of them, or use uv sync --package prime-rl --package <env> Install the subset. Then you need to check that the environment uses Python 3.12, flash-attn is installed, the SFT trainer can be run (requires 1 GPU), the inference server can be run (requires 1 GPU), and the full RL stack (inference + orchestrator + trainer) can run end to end (requires 2 GPUs). Additional setup includes: logging runs to W&B requires login; restricted/private models or datasets on HuggingFace require login.

Training Examples Basic training (1 to 8 GPUs) includes: Reverse Text (train Qwen3-0.6B to reverse text, a few minutes on a single consumer GPU, suitable for getting started); Wordle (train Qwen3-1.7B to play Wordle, several hours on 2-4 H100s, exploring multi-turn training capabilities); Alphabet Sort (train Qwen3-4B-Instruct-2507 to sort names alphabetically, no SFT warmup required, LoRA-based multi-turn RL, about an hour plus on a single H100, exploring LoRA training); Wiki Search (train Qwen3-4B-Instruct-2507 to answer trivia questions by searching Wikipedia, demonstrating multi-turn scenarios with web search tool use); Hendrycks Sanity (run sanity-check experiments on Qwen3-1.7B with a filtered MATH subset, suitable for algorithm ablations). Advanced training (32 to 2048 GPUs) targets difficult reasoning and agent tasks, designed to run on a Slurm cluster, and includes: Qwen3-30B-A3B (training on math, SWE, and agent tool use); GLM-4.5-Air (training on search, SWE, and terminal tasks); GLM-5 (large-scale GLM-5 / GLM-5.3 training with P/D disaggregation, llm-d router, and FP8).

Documentation and Contributions The documentation directory contains in-depth guides: Overview (architecture, installation, copy-pasteable end-to-end RL runs), Configuration (TOML composition, CLI overrides, environment variables, validation), Training (RL, SFT, checkpoints, observability, rules of thumb), Eval (standalone evaluation against any inference server, online evaluation, resumption, platform upload), Scaling (single GPU to multi-node, FSDP/EP/CP, SLURM, benchmarking), Algorithms (async/off-policy training, AIPO loss, advantage and filtering plugins, trajectory merging), Advanced (custom modeling, multimodal training, LoRA), Development (test suite, pre-commit hooks, adding new models). The project welcomes community contributions, uses issues to track bugs, feature requests, and share the internal roadmap; when contributing via PR, you need to install pre-commit hooks, keep PRs in Draft until ready for review, link the issue in the description when resolving an issue, and run the test suite locally whenever possible. The project is licensed under Apache 2.0.

Limitations Currently using prime-rl requires at least one NVIDIA GPU. Architectures other than the listed model families will fail during the trainer setup stage. The environment installation must be selected explicitly, and the default uv sync --all-extras does not include environment packages.

Related tools

Browse category