Purpose
understand-r1-zero is an open-source project for large language model reinforcement learning (LLM RL) research, aiming to understand the mechanism of "R1-Zero-like training" from a critical perspective. The project analyzes the two core components, the base model and reinforcement learning, and provides a reproducible training method.
Main Findings and Capabilities
On the base model side: DeepSeek-V3-Base already exhibits the "Aha moment"; as a popular choice for R1-Zero-like training, the Qwen2.5 base model exhibits strong reasoning ability even without using a prompt template, with average benchmark scores improving by about 60% compared to traditional 4-shot prompting. On the reinforcement learning side: the project points out that GRPO leads to biased optimization, and proposes a simple correction, Dr. GRPO (GRPO Done Right), which improves token efficiency while maintaining reasoning performance. The project also analyzes the joint impact of templates and problem sets on RL dynamics: for Qwen2.5-Math-1.5B, mismatched templates (such as the R1 template) actually destroy reasoning ability before RL reconstruction, making the apparent improvement stand out; and as long as the template does not deviate too far from the pretraining distribution, even a small-scale and fully o.o.d. problem set (such as GSM8K) can similarly induce reasoning ability by reinforcing correct reasoning behavior rather than injecting new knowledge. In addition, Llama can also be RL-tuned from a base model, in which case domain-specific pretraining raises the RL upper bound; GRPO can even make Llama with mathematical knowledge "Aha" by increasing output length, but this may stem from its length bias, and that bias can be eliminated by Dr. GRPO.
Usage
The project recommends using a clean python==3.10 environment for development. R1-Zero training is implemented by extending the Learner and Actor components of the Oat framework; refer to the step-by-step guide in train_zero_math.py, and see the repository description for more example scripts. The minimalist R1-Zero training recipe provided by the project is: use the (unbiased) Dr. GRPO algorithm to perform RL tuning on Qwen2.5-Math-7B with the Qwen-Math template on MATH level 3-5 problems; according to the available information, on 8× A100 State-of-the-art performance can be achieved on GPUs with only 27 hours of compute. The project also provides scripts for serving DeepSeek-V3-Base and DeepSeek-R1-Zero on a k8s cluster. Example API calls require the k8s container to set the environment variables MASTER_ADDR and MASTER_PORT. Hardware requirements: FP8 requires 2 x 8 x H100/800/20, BF16 requires 4 x 8 x A100/A800; for more details, refer to the official sglang tutorial.
Use cases
Suitable for researchers and engineers studying LLM reinforcement learning, R1-Zero-style training mechanisms, GRPO optimization bias, and reasoning ability elicitation, as well as developers who need to reproduce or extend related experiments.
Updates and acknowledgements
Papers, models, and codebase released in March 2025; received an ICML 2025 AI for Math Workshop Best Paper Honorable Mention in July 2025. R1-Zero training is implemented based on the modular, research-friendly, and efficient Oat framework. Computing resources are supported by Sea AI Lab; the training code is built on top of Oat, using vLLM, DeepSpeed, and launchpad, DeepSeek models are served through SGLang, and the base models come from Qwen2.5-Math, Llama, and DeepSeek.
Limitations
The materials do not provide other features, pricing models, licensing agreements, or more detailed usage steps for the project beyond the above content; related details are not currently available.
