SimpleRL-reason is a simple reinforcement learning training codebase for improving model reasoning ability, developed by the HKUST-NLP team and open-sourced on GitHub. Its core feature is that it uses only rule-based rewards and the GSM8K/Math datasets, enabling reinforcement learning training without complex reward models.
Main capabilities: The project has successfully conducted zero RL training on 10 diverse base models, including Llama3 8B, Mistral 7B/24B, DeepSeekMath 7B, Qwen2.5 0.5B/1.5B/7B/14B/32B, and Qwen2.5-Math-7B. Using only 8K example data, the accuracy improvement ranges from 10 to more than 20 absolute percentage points. The project has open-sourced all 10 model checkpoints, training/evaluation code, and analysis tools, and provides papers, blogs, and a Hugging Face model collection.
Usage: The code is implemented based on Verl, supports custom environment settings and FSDP training. The training data includes three difficulty levels (Easy: GSM8K and MATH lv.1; Medium: MATH lv.1-4; Hard: MATH lv.3-5), and has been processed into two formats: simple prompts (abel) and complex prompts (qwen). The datasets can be downloaded directly. During training, a Ray cluster must first be started. The main training script is train_grpo_math_tune_ray.sh. Environment variables need to be specified in the script, and then the training job is submitted from the master node. The training process uses the GRPO algorithm and uses Ray and vLLM for acceleration.
Hardware requirements: Training Qwen-2.5-0.5B requires at minimum a single H/A100-80G GPU; training 7B and 14B models uses 2x8 H100-80G GPUs and completes in about 100 steps and 15 hours; training the 32B model uses 8x8 H100-80G GPUs and completes in 1.5 days.
Evaluation and visualization: Evaluation uses the Qwen Math codebase, but solving problems by calling code is prohibited. The eval_math_nodes.sh script provides a complete process for evaluation, result collection, and analysis. Evaluation results are saved in $RUN_NAME/eval_results, and metrics include clipping ratio, average response length, etc. The project also provides a Gradio-based visualization toolthat displays the model's reasoning process at different training steps and on benchmarks.
Applicable scenarios: Suitable for academic or engineering scenarios that study the use of reinforcement learning to improve language model reasoning ability, especially research on zero RL training and reasoning ability improvement under limited data. Limitations: GPU resources are required, training details depend on the Verl framework, and some models require a multi-GPU environment.
