X-R1

Purpose

X-R1 is an easy-to-use, low-cost end-to-end reinforcement learning training framework, with the goal of accelerating the development of Scaling Post-Training. Inspired by DeepSeek-R1 and open-r1, it can train a 0.5B-scale R1-Zero model from a base model at extremely low cost and reproduce the "Aha Moment" phenomenon.

Main Capabilities

• Low-cost training: Training for 1 hour on 4 3090/4090 GPUs costs less than $7, and the "Aha Moment" can appear in the output after 37 steps in 10 minutes.
• Model scale: Supports reinforcement learning training for 0.5B-scale models, and can be extended to larger models such as 1.5B/7B/32B.
• Datasets: Provides 0.75k/1.5k/7.5k datasets for rapid training loops.
• Training techniques: Supports LoRA, LoRA+Zero3, GRPO, and QLoRA GRPO training, and supports training without KL divergence (setting the KL term beta to 0.0 and ignoring ref_model can improve performance by about 20%).
• Reward mechanisms: Supports medical-domain rewards, llm-as-a-reward, and more rule-based rewards.
• Evaluation and logging: Uses vLLM as the backend evaluation benchmark, outputs accuracy-metric and format-metric as well as JSON files; records GRPO online sampling data to log files, and provides wandb details.
• Chinese support: Supports Chinese mathematical reasoning and can easily produce a Chinese "Aha Moment". For example, training 3B-base with 4x3090 for about 16 hours using 7.5k Chinese math problems.
• Inference and models: Provides Colab inference, and has released models such as X-R1-0.5B, X-R1-1.5B, and X-R1-3B as well as configurations/wandb/models/logs.

Usage

1. Environment requirements: CUDA >= 12.4.
2. Training script examples:
   • 0.5B R1-Zero training: If using 4 GPUs, set --num_processes=3, where one GPU deploys vLLM as the online inference engine to accelerate GRPO sampling.
   • Training 3 epochs on 4x4090 takes about 1 hour and 20 minutes.
   • It is recommended to use --config recipes/X_R1_zero_3B_config.yaml to achieve better reasoning and format learning results.
   • Single-GPU 3090 training example for 7B LoRA; multi-GPU training example; test environment example; single-GPU LoRA option.
   • Log file location:./output/test.log.
3. Batch size and generation count settings: Taking 4 GPUs (1 vLLM + 3 training) as an example, set --num_processes=3, satisfying (per_device_train_batch_size * num_processes) % num_generations == 0; similarly for using 8 GPUs (1 vLLM + 7 training).
4. Evaluation: Use the vLLM backend to evaluate benchmarks, outputting accuracy-metric and format-metric as well as JSON files.
5. Chinese mathematical reasoning: X-R1 supports Chinese mathematical reasoning and can easily produce a Chinese "Aha Moment".

Use cases

• Researchers and developers who want to train small R1-Zero models at extremely low cost and observe the "Aha Moment" phenomenon.
• Scenarios that require rapid experimentation with reinforcement learning post-training methods (such as GRPO, LoRA, QLoRA).
• Chinese mathematical reasoning tasks.
• Reinforcement learning training experiments for models of different scales from 0.5B to 32B.

Limitations

• The project materials do not provide specific installation commands and complete code, only showing some run scripts and configuration instructions.
• It is not stated whether task types other than Chinese and mathematical reasoning are supported.
• Detailed information on fees, pricing, or commercial use licenses is not provided.
• Some features (such as QLoRA GRPO training, 7B configuration/results, more rule-based rewards, more base models, benchmark evaluation results) are marked as to be released or continuously being added, and may not yet be fully available.
• Training effectiveness and the time at which the "Aha Moment" appears may vary depending on hardware and configuration; the data given in the materials are based on a specific experimental environment.