Purpose
GSM8K-RLVR is a simplified implementation for experimenting with "Reinforcement Learning from Verifiable Rewards" (RLVR) on the GSM8K math problem-solving dataset. This repository references the RLVR concept proposed in the Tulu3 paper ("Pushing Frontiers in Open Language Model Post-Training") and is inspired by DeepSeek R1, with the goal of improving base language models in mathematical reasoning and structured output through RLVR. Its core objective is: using only GSM8K data and few-shot prompting, adapt and optimize a base model through RLVR so that it possesses better mathematical reasoning and structured output capabilities.
Main Capabilities and Design Points
- Base model focus: aimed at improving base model performance, without relying on a pretrained reward model.
- Few-shot prompting: utilize few-shot examples in each model input to establish the required data pattern and facilitate RL learning.
- Simplified prompt format: omit explicit <think> and <answer> tags, keeping completions straightforward.
- Dual reward system: includes two reward functions—
- Correctness: rewards accurate answers to GSM8K problems.
- Format Adherence: incentivizes outputting the final answer in the specific format "#### The final answer is {number}".
Usage and Experimental Setup
The repository uses RLVR and GRPO to perform RL fine-tuning on the Qwen/Qwen2.5-Math-1.5B model, and evaluates performance improvements on the GSM8K benchmark. The model was fine-tuned for two epochs using a reward function comprising correctness and formatting, where the correctness reward is: 1 for correct, 0 for incorrect; and the formatting reward is: 0.5 for correct format, otherwise 0. The evaluation method includes 8-shots evaluation. The authors also mention an interesting observation that arose due to a parameter-passing error: loading a LoRA adapter trained on Qwen/Qwen2.5-Math-1.5B also improved the performance of Qwen/Qwen2.5-1.5B on GSM8K, with the score increasing from 58.23% to 64.82%.
The authors alsoPoints out: although these results are impressive, a fairer comparison should be to perform SFT on GSM8K for both the base model and the RL fine-tuned version, and evaluate the models in a zero-shot manner. The authors also tried a Baseline+SFT that uses a language modeling loss on completions, but its performance was consistently worse than the few-shot setting of the Baseline; while Baseline+RLVR performed well even with only 1-shot examples. The only scenario where SFT outperforms RLVR is the zero-shot setting, where RLVR received 0.00% due to improper formatting.
Applicable Scenarios
This repository is suitable for researchers or developers who want to get started quickly and explore RLVR and reasoning ability improvement on GSM8K, and can serve as a starting point for related experiments.
Limitations and Notes
- Resource constraints: Due to the 24GB VRAM limit, the authors had to impose restrictions: the maximum sequence length generated during RL fine-tuning was limited to 300 tokens; the number of few-shot examples was limited to 2, and the number of tokens to 256. These restrictions may affect the model's ability to solve more complex problems that require longer explanations. The authors believe that removing these restrictions could bring further performance improvements.
- Dataset order bias: It was observed that the GSM8K dataset has a tendency for earlier samples to be more difficult. Therefore, if the dataset is not shuffled, the reported reward growth curve may appear overly optimistic because the model learns harder examples early in training; when the dataset is not shuffled, the results should be interpreted with caution.
- Future directions: The authors list follow-up plans, including evaluating RL fine-tuning on Qwen/Qwen2.5-0.5B to examine scalability, extending fine-tuning to models that generate code on GSM8K and execute it with a Python interpreter, and sharing the models on HuggingFace.
Information Not Yet Provided
In addition to the above, facts such as the repository's billing model, pricing, specific installation commands, license agreement, detailed environment dependencies, and complete usage steps are not yet provided.
