This is an open-source repository exploring the application of GRPO, with the goal of reproducing the results of DeepSeek R1 at a smaller scale, with a focus on implementing the core mechanism from scratch for understanding. The author trains the Qwen1.5B model on a single H100 GPU, using a grade school math dataset, and after about 400 training steps, signs of learning appear on the validation set.\n\nIn terms of implementation, the project draws heavily on Will Brown's work, but reorganizes the code into a form suitable for learning and experimentation. The key difference is directly computing the GRPO loss function rather than using an external reinforcement learning library, and organizing it into a multi-script repository structure.\n\nThe system includes the following modules: the core training loop implements GRPO (Generalized Reward-Powered Optimization), responsible for model training, evaluation, and metric tracking; the model loading and configuration module, currently supporting LLaMA and Qwen models through the Hugging Face transformers library, and designed to be easily extensible to other architectures; the dataset loading and preprocessing module, currently focused on GSM8K math problems, implementing custom training and evaluation data loaders; the evaluation metrics and reward function module, closely following the original DeepSeek implementation.\n\nTraining is conducted on a single H100 GPU, and after about 400 steps the validation set results show signs of learning.\n\nFuture directions include: adding self-play capabilities so that agents compete and learn from each other through relative rewards; implementing a soft reward structure, especially for complex reasoning tasks; extending to vision-language models (VLM) to improve world modeling capabilities, exploring the use of R1-style training to enhance VLM construction and maintenance of internal world models; and the need to improve speed and support multi-GPU training.\n\nThe environment variables required by the project and more details are not yet provided.
DeepSeekRL-Extended
This repository is an open-source project that implements the DeepSeek R1 training mechanism from scratch, mainly training the Qwen1.5B model on the elementary school math dataset GSM8K. By directly computing the GRPO loss function instead of relying on external reinforcement learning libraries, it aims to help learners understand the core principles and conduct small-scale experiments. The project includes modules for the training loop, model loading, data processing, and evaluation metrics, and plans to extend to directions such as self-play, soft rewards, and vision-language models.
