TinyZero is a minimalist reproduction project of DeepSeek R1-Zero, hosted on GitHub (https://github.com/Jiayi-Pan/TinyZero）。项目基于 built with veRL, with the goal of reproducing the reinforcement learning training results of DeepSeek R1 Zero on countdown and multiplication tasks.
In terms of main capabilities, the project demonstrates that through reinforcement learning, a 3B base language model can develop self-verification and search abilities on its own. The project claims that the "Aha moment" can be experienced at a cost of less than $30. The related experiment records are published on Weights & Biases (https://wandb.ai/jiayipan/TinyZero），并在 There is a corresponding discussion thread on Twitter (https://x.com/jiayi_pirate/status/1882839370505621655）。项目还发布了 Adaptive Parallel Reasoning, exploring a new dimension of scaling reasoning models.
In terms of usage, the project provides instructions related to Data Preparation. For out-of-VRAM issues that occur during code execution, it is recommended to add critic.model.enable_gradient_checkpointing=True in the script and refer to the related discussion. The project states that this solution is suitable for model sizes no larger than 1.5B; for the Qwen2.5-0.5B base model, it is known that it cannot learn reasoning. For models of 3B and above, the base model can develop complex reasoning skills. The project also conducted an Instruct ablation experiment using Qwen-2.5-3B Instruct, and explains that the data needs to be reprocessed to follow the chat template.
The applicable scenarios are mainly reinforcement learning experiments, reasoning ability research, and reproduction research of DeepSeek R1-Zero.
In terms of limitations, there is a clear Deprecation Notice at the top of the repository: the repository is no longer actively maintained, and when conducting RL experiments it is recommended to directly use the latest veRL library; the original documentation has been archived to OLD_README.md. In addition, the project has requirements for model size: the 0.5B base model cannot learn reasoning, and models of 1.5B and below require additional configuration to address VRAM issues.
The acknowledgements section mentions that the experiments were run based on veRL and used the Qwen2.5 series base models.
