GRPO-Zero is an open-source project that implements from scratch the GRPO (Group Relative Policy Optimization) algorithm adopted by DeepSeek R1. The project emphasizes minimal dependencies and low memory usage: almost all components are implemented in-house, relying only on tokenizers for tokenization and pytorch for training, without using transformers or depending on vLLM.
In terms of hardware requirements, the default configuration targets a single A40 GPU (48GB of VRAM), and training for several hours yields good results; it also supports training on a 24GB VRAM GPU (for example, an RTX 4090) by offloading the optimizer to the CPU. Since the policy network is updated only a few hundred times throughout the entire training process, the additional overhead caused by this offloading is small.
At the algorithmic level, the core idea of GRPO is: randomly sample multiple answers for each question, and define the advantage value of an answer as the normalized result of its reward, thereby eliminating the need for a value estimation network. The specific process implemented by the project is: at each training step, randomly sample N questions; sample M answers for each question; compute the reward for each answer; compute the mean and standard deviation of the rewards for each question; for each token in the answer, divide (reward minus mean) by the standard deviation to obtain the advantage; use the PPO surrogate objective to compute the policy gradient (for simplification, only one policy update is performed per iteration, in which case the gradient of the PPO objective is equivalent to a naive per-token policy gradient estimate); finally, update the policy network and return to the first step.
Building on the original GRPO, the project supports several improvements from the DAPO project:
- Token-level policy gradient loss: each token is equally weighted in the policy gradient loss.
- Removal of the KL divergence: the KL divergence is not used in the policy gradient loss, so a reference policy network is no longer needed, reducing GPU memory usage.
- Filtering of overly long episodes: skip unfinished episodes that exceed the context length limit to stabilize training. This feature is disabled by default so that the model's learning can be observed under a limited context length, and it can be enabled by setting skip_unfinished_episodes to true.
In terms of the training task, the project trains a Qwen2.5 model on the CountDown task. The task description is: given 3 or 4 numbers and a target number, the model needs to generate a mathematical expression that uses only the common four arithmetic operations (+, -, *, /) and whose result equals the target number.To accomplish this task, the project uses GRPO to train the model to first generate chain-of-thought reasoning before giving the final expression, and to output in the required format.
The reward function consists of the sum of two parts:
- Format reward: 0.1 reward is obtained when the model correctly follows the specified format containing the thinking and answer tags, otherwise 0.
- Answer reward: 1 reward is obtained when the final answer uses each given number exactly once and correctly evaluates to the target value, otherwise 0.
The project uses the Qwen2.5-3B-Instruct model for training. The materials provide the command entry point for running training, but do not show the specific contents of the command.
In terms of acknowledgments, the project is built on several excellent projects: DeepSeekMath proposed the GRPO algorithm; DAPO enhanced the original GRPO algorithm; TinyZero provided a GRPO implementation and created the CountDown-Tasks-3to4 dataset; nano-aha-moment provided a clear GRPO implementation and tutorial; Qwen2.5 provided the high-quality pretrained model used by the project.
