Open-Reasoner-Zero is an open-source reinforcement learning training project for large-scale reasoning tasks, aiming to advance scalable, simple, and easy-to-use reasoning model training. Its official repository provides the complete source code, training scripts, parameter configurations, training data, and model weights.
In terms of use, the project focuses on reinforcement learning training of reasoning capabilities on base models, using the same base model as DeepSeek-R1-Zero-Qwen-32B, and achieves better performance on the AIME2024, MATH500, and GPQA Diamond benchmarks while requiring only about one-tenth of the training steps. It is committed to enabling a broader range of researchers to participate in reasoning model training and accelerating research toward artificial general intelligence (AGI).
Main capabilities include: providing training scripts and weights for models of different scales (0.5B, 1.5B, 7B, 32B), among which ORZ-0.5B can run on a single A800/H800 GPU; training rewards and response length grow steadily with model scale, demonstrating good scalability. The project also released ORZ-R1-Distill-Qwen-14B, which, by applying the ORZ recipe to reasoning-enhanced models such as DeepSeek-R1-Distill-Qwen-14B, achieved results on reasoning benchmarks that surpass the larger DeepSeek-R1-Distill-Qwen-32B. In terms of datasets, it provides a total of 129k curated data, including the original 57k, the expanded 72k, and 13k hard-example data mined from the 129k, used for the annealing stage of ORZ-32B training, improving AIME2024 from about 41% to about 48%. In addition, a complete Critic model series (0.5B, 1.5B, 7B, 32B) has been released for in-depth research.
Usage: The project has released all curated high-quality training data in the data folder. Installation requires running the installation command, and the training scripts support multi-node (such as 16-node, 4-node, 2-node) and single-node/single-GPU operation. For example, on 16 nodes, first run a certain command on the master node, then run a certain command on all other nodes, and finally run a certain command on the master node; the training log will be displayed in the master node terminal. A single A800/H800 node can start ORZ-0.5B PPO training, and it can even run on a single GPU (without the need for logic such as ray start). The environment variable DEBUG_MODE is retained in the code for researchers to debug and iterate, and it is recommended to use python -m playground.orz_0p5b_ppo_1gpu for debugging. The policy model can be used like any chat model in transformers and vllm, because the chat template jinja is already included in the tokenizer; the Critic model can be loaded as in the training code.
Applicable scenarios and limitations: Suitable for research institutions, universities, and enterprise researchers conducting reasoning-oriented large-scale reinforcement learning training, model scalability research, data ablation experiments, etc. The project codebase adopts a single-controller trainer design, which is flexible and researcher-friendly; training and generation are colocated on the same GPU to maximize utilization. Its training framework is based on OpenRLHF, vllm, DeepSpeed, and ray, and the models are based on the Qwen2.5 series base models. Main limitations include: it requires relatively high computing resources (such as A800/H800 GPUs) for large-scale training; some details (such as specific fees, prices, platform support, etc.) are not yet provided.for trial iteration, and it is recommended to use python -m playground.orz_0p5b_ppo_1gpu for debugging. The policy model can be used like any chat model in transformers and vllm, because the chat template jinja is already included in the tokenizer; the Critic model can be loaded as in the training code.
Applicable scenarios and limitations: Suitable for research institutions, universities, and enterprise researchers conducting reasoning-oriented large-scale reinforcement learning training, model scalability research, data ablation experiments, etc. The project codebase adopts a single-controller trainer design, which is flexible and researcher-friendly; training and generation are colocated on the same GPU to maximize utilization. Its training framework is based on OpenRLHF, vllm, DeepSpeed, and ray, and the models are based on the Qwen2.5 series base models. Main limitations include: it requires relatively high computing resources (such as A800/H800 GPUs) for large-scale training; some details (such as specific fees, prices, platform support, etc.) are not yet provided.
