Purpose
open-r1-reprod is a reproduction project of open-r1, with the goal of performing GRPO training on Qwen models of 0.5B, 1.5B, 3B, and 7B, and observing and analyzing phenomena such as the reasoning abilities of models of different sizes, various hacks of rule rewards, and multilingual mixed reasoning on the MATH-lighteval math dataset. The project is not exactly a success, but it also has some interesting findings, and discussion is welcome.
Main capabilities
The project uses the code of the open-r1 project, modifying the config_grpo.yaml configuration file for training. Training details can be found in config_grpo.yaml.
In the model evaluation stage, 2 modes of system are set to test the difference between different systems, and system2 is used by default. system_prompt is set so that the model can follow the format <think>\n...\n</think>\n<answer>\n...\n</answer>.
Rule-based reward functions include accuracy reward (accuracy_reward) and format reward (format_reward, tag_count_reward), and the total reward is the sum of the three. Among them, tag_count_reward is because the complete format is too difficult, and the model may find it hard to follow, so for answers with partial format, partial reward is given through tag_count_reward.
The learning rate uses cosine, with a maximum value of 3.0e-06.
Observed phenomena
Observing the reward curves, the 0.5B model shows almost no change in accuracy_reward and format_reward, and only has a certain improvement in tag_count_reward, reaching 50%. This indicates that a model that is too small cannot improve the accuracy of answers through GRPO, nor can it follow the format well.
The 1.5B model has a certain improvement in accuracy_reward, reaching 40%, and reaches close to 100% in format_reward and tag_count_reward. For the 3B and 7B models, there are similar curves, with the difference being the different degree of improvement in accuracy_reward, indicating that GRPO begins to take effect at the 1.5B model, and the larger the model, the better the effect.
Different parametersThe inference length of the model is generally stable, and there is no phenomenon of inference length increasing with more training.
Because the loss&kl&grad curves of the 7B model have very severe fluctuations, for example, in the loss curve it can reach 75.132, and the loss of the 0.5B, 1.5B, and 3B models cannot be observed, the loss&kl&grad curves of the 0.5B, 1.5B, and 3B models are plotted separately (right figure). The 7B model shows severe fluctuations around step 35, with the loss value reaching 75.132, and at step 95 the loss value reaches 4.1998. The 1.5B model also shows relatively large fluctuations, with the loss value reaching a maximum of 0.4545. kl&grad also shows the same trend as loss.
The severe fluctuations in the loss&kl&grad curves may affect the model's performance, but this is not reflected in the reward function curve. When testing the 1.5B model (including 1epoch, 2epoch, and 3epoch), the model often outputs a bunch of gibberish. The 7B model performs well at 1epoch (58step), but at 2epoch (116step) and 3epoch (174step), it also outputs a bunch of gibberish. These phenomena may be related to the severe fluctuations in the loss&kl&grad curves.
Model output issues
The following output issues were observed: there is no complete <think>\n...\n</think>\n<answer>\n...\n</answer> format; the answer does not start with <think>; </journalism> is fabricated out of thin air; multiple languages are mixed together in the answer, including think and answer.
Qwen2.5-7B-Instruct gives a wrong answer, while after 1epoch of training, the model successfully solves the problem. The 2epoch and 3epoch models output gibberish, which is not shown here, but for the training data, even if translated into Chinese, the 3epoch model can still answer correctly.
Modifying the system does not affect the inference answer, but affects the <think>\n...\n</think>\n<answer>\n...\n</answer> format.
Suggestions and findings
It is difficult for the 0.5B small model to learn reasoning ability through GRPO; using models above 1.5B forTraining is more appropriate.
For the format reward, the model can always bypass the correct way in unexpected ways to obtain the reward. The following approaches can be considered:
- Simplify <think>\n...\n</think>\n<answer>\n...\n</answer> to <think>\n...\n</think>\n, reducing the difficulty for the model to match the format.
- Concatenate the question with <think>\n to eliminate the possibility of the model placing the thinking process before <think>\n.
- Penalize cases where <think>\n appears multiple times.
- Cold start: first use a portion of correctly formatted data to let the model learn to match the correct format.
For the answer reward, a reasonable way to extract the answer needs to be designed, though of course this is difficult. I tried the OpenR1-Math-220k, s1K-1.1, Bespoke-Stratos-17k, and MATH-lighteval datasets, but either the dataset was too large or it was difficult to extract answers, so I ultimately chose the MATH-lighteval dataset. Different answer types encountered when selecting datasets:
- Numeric type: for example, the integer 2, 020, or the floating-point number 3.14.
- Boolean type: for example, True, true.
- Multiple-choice questions: for example, single-choice A, a, or multiple-choice BCD, bC.
- Restating the question: for example, 1+1=2, 1+2.14=3.14.
- The above answers with a specific derivation process: for example, ...... so the answer is XXX, The answer is XXX, Answer is XXX, x=XXX.
- The above answers with LaTeX formatting: for example, ...... so the answer is \boxed{XXX}, x=\boxed{XXX}, \fbox{XXX}.
- Proof problems: different proof approaches are all correct, and such problems have no solution.
For multilingual mixing and garbled text, consider adding a reward function that penalizes multilingual mixed and garbled responses.
Hyperparameters are very important, and the hyperparameters of models with different parameter counts may differ. Using the same hyperparameters to train 0.5B, 1.5B, 3B, and 7B qwen models, but the results varied greatly:
- The 0.5B model did not learn reasoning ability, and its format was also muddled through.
- The 1.5B model collapsed during training or was too severely overfitted, and its responses were always garbled.
- The 3B model showedthe phenomenon of multilingual mixing.
- The 7B model also collapsed during training or was too severely overfitted, and its responses were always garbled. It was fine at 1 epoch, but it became increasingly severe later.
When tested on aime-2024 and math-500, all models had scores that decreased as training increased, possibly due to overfitting or incorrect response formats.
Sudden changes in loss during training may have a large impact on the model. For example, both the 1.5B and 7B models showed sudden changes, and the final training results were also poor. However, what is puzzling is that when the sudden change occurred within 1 epoch, the 7B 1epoch model actually performed quite well, while the 2epoch and 3epoch models performed poorly.
Reference Projects and Data
This project mainly refers to the following projects and materials:
- https://github.com/huggingface/open-r1 (code process)
- https://huggingface.co/datasets/DigitalLearningGmbH/MATH-lighteval (data source)
