CP-Zero is a training project based on the R1-Zero method, mainly targeting the Code Contests dataset. The project uses rule-based rewards and the GRPO algorithm, with Qwen2.5-Coder-7B as the base model. The task setup is: given an algorithm problem description, the model generates code and tests the correctness of the code on test data. The reward consists of two parts: a format reward and a scoring reward based on the number of correct test data. The project provides the run script main.sh for reference.
The project shares some training findings: regarding the format score, because the base model is a Base model, the initial format is very difficult to answer correctly, but after about 100 steps there are basically no format errors. Regarding the proportion of compilation errors, there are few compilation errors in the initial steps, and after training begins it rapidly rises to a very high proportion, then quickly decreases, dropping to a relatively low proportion after 200-300 steps. The acceptance rate continuously improves with the number of training steps, and the reward score on the validation set also basically improves with the number of training steps. However, the project did not observe an Aha Moment, that is, no thinking such as reflection and summarization appeared in the think stage. After 2000 training steps, the model showed inconsistency between the content of think and the content of answer. For example, think wanted to implement a segment tree but the answer did not implement it, and the answer code seemed to become shorter and shorter. It is guessed that this may be because there are no strong examples in the dataset, so using a segment tree implementation is not cost-effective. The responses gradually became more anthropomorphic, for example saying I blabla, we blabla. The initial responses were standard listings of 1 2 3 4 5 similar to a solution explanation, but by around 3000 steps the grammar in think began to become incorrect.
The project made certain modifications on the verl framework, for example using multithreading to speed up the calculation of rule-base reward. The format score mainly references Logic-RL. The dataset currently uses deepmind/code_contests. 2025.04.01 update: added calling command execution code during generate. For details, refer to verl/workers/rollout/vllm_rollout/vllm_rollout.py. Currently, there is no significant improvement on the 7b model.
