The R1-Nature project aims to reproduce R1-like results on small models, and through comparative experiments demonstrate the differences in fine-tuning effects between different thinking reasoning CoT datasets and small models of different sizes (0.5B, 1B, 1.5B, 3B), so as to explain in a relatively simple way that the most important influencing factor in current O1-like and R1-like systems is the internal reasoning process of think thinking. The project also points out that there are still a large number of unclear detailed issues, such as the chain-of-thought reasoning explosion phenomenon and solutions, which require researchers' attention, rather than simple distillation plus RL learning.
The project environment requirements include PyTorch 2.3.0+, CUDA 12.1, transformers 4.48.2, LLAMACPP B4646 cuda12.4, one RTX 4090 24G GPU, 128GB memory, and LMStudio 0.3.9.
The project provides links to multiple datasets and fine-tuned models, including models obtained by distillation fine-tuning from qwen2.5 0.5b, llama3.2 1b, llama3.2 3b, qwen2.5 1.5b, and the official DeepSeek qwen2.5 1.5b, as well as the LoRA weights of each experimental model. It also provides GGUF models supporting device-side and edge computing devices, covering FP16 and Q4 K_M quantized versions.
In terms of datasets, ringo1-CoT_demo.json has about 0.1K entries, mainly English mixed with Chinese, sourced from Marco-o1; training a 1B model on a single 4090 for 1 epoch takes only a few minutes. openr1-SFT.json has about 160K entries, mixed Chinese and English, sourced from OpenO1; training a 1B model on a single 4090 for 1 epoch takes about 35 hours. magpie-r1.json has about 10K entries, entirely in English with relatively long chain-of-thought content, sourced from Magpie; training a 1B model on a single 4090 for 1 epoch takes about 30 minutes. The project uses the alpaca dataset, whose base corpus comes from alpaca_gpt4_data_zh.json, originally 50K entries. tiny_alpaca_XXX has only about 20 entries, used for preliminary observation and evaluation of the reasoning differences of the distilled target model. The main fine-tuning corpus has been tested in practice; for the same model, the dataset distilled from the locally released DeepSeek qwen 2.5 7B performs in terms of LOSSThe data obtained remotely, such as from GPT-4o, is inferior, with loss differences reaching up to 0.4 (range 0.8-1.2). alpaca_r1_data_zh-remote.json comes from the cloud, is purely Chinese, contains approximately the first 2.6K entries of alpaca, and takes about 8 minutes to train a 1B model on a single 4090 GPU for 1 epoch; alpaca_r1_data_zh-localpost.json comes from local sources, is purely Chinese, contains approximately the first 2K entries of alpaca, and takes about 8 minutes to train a 1B model on a single 4090 GPU for 1 epoch.
The project provides several scripts: jsonl2json.py, parquet2json.py, distillationgalpacalocal.py, deallocaldistil.py, distillationgalpacaremote.py, finetuneSFTR1.py, generatefinetune.py, and demo.py, which are used respectively for format conversion, obtaining distillation data, SFT LoRA fine-tuning, building a Gradio test, and testing and sharing the model.
Regarding the phenomenon of chain-of-thought reasoning explosion, the project observed that after a strong reasoning model acquires the ability, it leads to slower reasoning speed, and the reasoning content sometimes appears to enter an infinite loop. When max_length is set to 1024, the reasoning content can actually reach more than 8K characters, leading to failure. When choosing llama 3.2 1B, selecting the magie_10k dataset, and repeatedly performing SFT LoRA on all model layers for 3-5 epochs, this phenomenon is easily triggered. The project proposes several possible solutions: grop step func reward controls the direction and content of fine-tuning think through rewards; freezing some layers of the LLM during training, such as fine-tuning only the q\k modules and training 1-2 times, but when the amount of data is not large, the output format may be inconsistent; adopting the previous agent prompt technique to control the overall thinking process; referring to the ZeroCoT random interpolation CoT idea; distinguishing datasets at different stages to improve characteristics, scale, and quality. The project emphasizes that distilling high-quality data combined with multi-stage RL learning does not necessarily yield a nearly perfect product, and many detailed factors deserve attention. The project also found that 1B and 0.5B models exhibited surprising abilities such as reasoning and retrospective analysis, which still have some value even if they do not help solve the problem successfully. The subsequent plan is to produce reasoning traces on RWKV7 0.1B or even smaller parameter models.
