nanoAhaMoment is an open-source library for reinforcement learning training of large language models, developed by the McGill-NLP team. Its core positioning is "single-file, single-GPU, from-scratch implementation, efficient, full-parameter tuning," aiming to make the RL training process for LLMs transparent and easy to get started with.
In terms of purpose, the project implements a DeepSeek R1-zero style training pipeline and can be used for full-parameter reinforcement learning fine-tuning of 3B and 7B base models. It is suitable for researchers and developers exploring the training mechanisms of RL for LLM, especially those who want to understand the entire training process with minimal dependencies and simplified code.
Its main capabilities include: supporting training on a single 80G GPU, while also providing multi-GPU support to accelerate training and cover 7B models; using full-parameter tuning; being competitive with verl in efficiency, but with a simpler implementation. On 3B models, it can support context lengths up to 32K with multiple GPUs; on 7B models, it supports context lengths up to 16K with multiple GPUs. In June 2025, multi-GPU support and VinePPO episode generation (experimental) were added.
In terms of usage, the repository provides nano_r1.ipynb as an interactive single-file Jupyter notebook containing a tutorial; nano_r1_script.py is the Python script equivalent to the notebook, convenient for direct execution and supporting multiple GPUs. There is also notebooks/checkpoint_playground.ipynb for comparing different model checkpoints (including trained models) and conducting experiments. The official project also provides a trained 3B model on Hugging Face (McGill-NLP/nano-aha-moment-3b), with an accuracy of about 60% on the CountDown task.
Installation steps: after cloning the repository, make sure CUDA 12.4 is installed, then install PyTorch 2.5.1 (pip install torch==2.5.1 --index-url https://download.pytorch.org/whl/cu124），再安装其余依赖（pip install -r requirements.txt). You can also optionally use uv installation: uv sync or uv sync --extra compile to install flash-attention. To run training, open nano_r1.ipynb or nano_r1_script.py; if using uv, you can execute uv run nano_r1_script.py, or activate the environment and run python nano_r1_script.py. Multi-GPU training provides an example run command for 4 GPUs, and for 4xA100 80GB it provides batch size recommendations under different context lengths; other GPU types may require adjustment.
The applicable scenarios are mainly academic research, RL algorithm teaching, and experimental training of small and medium-sized models. The library's design philosophy is that every line of code is visible and understandable, inspired by TinyZero and Mini-R1, but more concise, clear, and fast. The project also includes detailed Karpathy-style explainer videos on YouTube (in two parts).
In terms of limitations, the materials do not provide information on pricing, commercial licensing, model performance benchmarks (except for about 60% on the CountDown task), other supported model sizes, or detailed performance comparison data; these contents are not yet provided.
