oat-zero is a lightweight research project developed by the sail-sg team under the Agency for Science, Technology and Research (A*STAR) of Singapore, aimed at reproducing the reinforcement learning (RL) training process of DeepSeek-R1-Zero and deeply analyzing its self-reflection behavior. The core of the project is a paper titled "There May Not be Aha Moment in R1-Zero-like Training — A Pilot Study," which presents questioning findings about the "Aha moment" in R1-Zero-like training.
Purpose: This project is mainly used to explore whether purely reinforcement learning-trained language models produce emergent abilities such as the "Aha moment." The traditional view holds that during R1-Zero training, the model suddenly learns self-reflection, thereby solving complex reasoning problems, manifested as a sudden increase in response length. oat-zero conducts a detailed empirical study of this phenomenon by reproducing training on small-scale models (1B to 7B parameters).
Main findings:
- There may be no "Aha moment": The study found that self-reflection patterns (such as "Wait, let me think again") actually already appear in the base model before training (epoch 0), and do not emerge during training.
- Superficial Self-Reflection (SSR): The self-reflection statements produced by the base model do not necessarily lead to correct answers and are often merely superficial reflections.
- Reasons for the increase in response length: The growth in response length observed during RL training is not due to the emergence of self-reflection, but rather because reinforcement learning is optimizing preset rule-based reward functions, and the model tends to generate longer responses in order to obtain higher rewards.
Main capabilities:
- Reproduce R1-Zero-like training on the Countdown task (a mathematical reasoning task).
- Provide scripts for evaluating the self-reflection behavior of the base model.
- Reproduce the results of SimpleRL-Zero on the MATH dataset.
- Include detailed analysis of response length distribution and reward curves (as shown in the charts on the project page).
Usage: The project is developed based on the Oat framework, and the installation command is:
pip install oat-llm
Or use editable mode for local development:
git clone https://github.com/sail-sg/oat-zero.git
cd oat-zero
pip install -e .
To evaluate self-reflection behavior: run the script and pass the model name as an argument, for example:
python eval/self_reflection.py --model <model_name>
Reproduce the Countdown task results:
python train/countdown.py
To reproduce the MATH task results, refer to the instructions in the simpleRL/train directory.
Applicable scenarios:
- Scholars and engineers studying the reasoning capabilities of large language models.
- Researchers interested in the phenomenon of emergent abilities in reinforcement learning training.
- Developers who need to validate the R1-Zero training method on small-scale models.
Limitations:
- The project is research-oriented and is not a production-grade tool.
- It mainly reproduces specific tasks (Countdown, MATH), so its generality is limited.
- It depends on third-party libraries such as Oat and vLLM, and the corresponding environment must be configured.
- The project follows the MIT License.
Other information: The project code is hosted on GitHub (https://github.com/sail-sg/oat-zero），感谢 Technical contributions from Oat, SimpleRL, TinyZero, vLLM, as well as teams such as DeepSeek, Qwen, Rho, and Llama.
