nanoGRPO is a lightweight implementation of Group Relative Policy Optimization (GRPO), mainly used for policy optimization training of language models.
Purpose
According to the project description, nanoGRPO is used for language model training, and emphasizes that users can "cooked you own tiny r1/o1", that is, use this project to train their own small r1/o1-like models. It provides a lightweight implementation of GRPO, suitable for developers who want to understand or practice GRPO under limited hardware conditions.
Main capabilities
The project gives the GRPO objective function:
$$ J_{GRPO}(\theta) = \frac{1}{G} \sum_{i=1}^{G} \left( \min \left( \frac{\pi_{\theta}(o_i|q)}{\pi_{\theta_{old}}(o_i|q)} A_i, \text{clip} \left( \frac{\pi_{\theta}(o_i|q)}{\pi_{\theta_{old}}(o_i|q)}, 1 - \epsilon, 1 + \epsilon \right) A_i \right) \right) $$
It also explains: GRPO is a form of PPO, whose advantage is estimated as the z score of grouped output, rather than using a value function.
Usage
The project gives the official address in the title:https://github.com/joey00072/nanoGRPO。文本中未提供具体安装命令、运行步骤或代码示例，相关细节暂未提供。
Applicable scenarios and limitations
The project mentions results obtained after training for 120 steps on an 8GB RTX 4060, indicating that it can be used under relatively small VRAM conditions. However, apart from this hardware and step count information, the text does not provide the dataset, model scale, hyperparameters, performance metrics, or comparison with other implementations, nor does it explain the license, dependencies, and scope of application; these contents are not yet provided.
