Hands-On Modern RL is an open-source, hands-on reinforcement learning course project whose goal is to bridge the gap from basic reinforcement learning concepts to LLM alignment, RLVR (Reinforcement Learning with Verifiable Rewards), and advanced agent systems. The project revolves around a core question: an agent takes actions, observes results, and improves its next action accordingly; when rewards are delayed, observations are incomplete, and every update changes the future distribution of experience, the traditional supervised learning paradigm is no longer sufficient, so a language for describing interaction, methods for estimating long-term returns, and algorithms capable of stably improving policies from limited data are needed.
[Course Positioning and Content Structure] The course starts with CartPole and multi-armed bandits, making states, actions, rewards, and policies visible; it then introduces Markov decision processes, value functions, and the Bellman equation as a shared language, and goes on to DQN, policy gradients, Actor-Critic, PPO, continuous control, and offline reinforcement learning. On this basis, RLHF, DPO, GRPO, and RLVR are treated as extensions of the same sequential decision-making framework, rather than as an isolated collection of acronyms. The book is divided into seven parts and twenty-six chapters:
- Prologue: From trial and error to modern agents.
- Part One: The language of sequential decision-making, establishing the mathematical objects needed to describe long-term decisions.
- Part Two: Learning values and policies with neural networks, introducing function approximation, and advancing along the two routes of value learning and policy learning toward PPO and continuous control.
- Part Three: Scaling data, tasks, and agents, covering scenarios with high interaction costs, expert demonstrations, or multi-agent multi-timescale tasks.
- Part Four: LLM alignment and post-training, expanding "action" into a segment of text, and moving into preference alignment, verifiable rewards, and inference-time computation.
- Part Five: Tool use and agent reinforcement learning, where the training unit becomes trajectories, and credit assignment, environment construction, and safety boundaries become central.
- Part Six: Reinforcement learning in a multimodal world, covering new state representations, reward sources, and evaluation criteria brought by vision, audio, robotic actions, and generative models.
- Part Seven: Safety, evaluation, and research frontiers, asking whether goals are reasonable, whether gains are real, and what new risks arise from capability scaling.
- Appendix: Implementation, mathematics, and engineering references.
[Main Capabilities and Learning Approach] Each chapter follows the rhythm of "problem-method-experiment-reflection": first a concrete task exposes the difficulty, then the needed concepts and formulas are introduced, then runnable code, training curves, and evaluation metrics are used to test the argument, and finally assumptions, failure modes, and the boundaries of the method are examined. The implementation code preserves the visible skeleton of each algorithm, allowing readers to trace trajectory collectionsets, advantage estimation, loss construction, and metric changes, and observe how reward hacking, KL drift, entropy collapse, distribution shift, or evaluation leakage can make seemingly successful training fail. The project provides eight sets of examples: four environment-based experiments, plus preference tuning, code reinforcement learning, research agents, and visual reasoning. The project collaborates with ModelScope to provide online training environments for classic reinforcement learning experiments. Each Studio comes with a companion Notebook under code/online-experiments, which can import the same training runtime, expose experiment parameters, print complete training logs, plot checkpoint evaluations, and display policy replays or result artifacts. Among them, experiments 03, 08, 10, 11, and 12 require reserving a ModelScope xGPU Notebook and checking CUDA before training.
[Usage and Repository Structure]
- Course site: walkinglabs.github.io/hands-on-modern-rl
- Source content: docs/
- Runnable examples: code/ (including a code index and dependency notes for each chapter, see code/README.md)
- Local verification: npm run verify Running the documentation site locally requires Node.js >= 18.0.0. After modifying the documentation structure, theme code, navigation, build scripts, or generated assets, before submitting a PR, you should run npm run verify, which checks formatting, lints the VitePress theme, builds the site, and validates the expected build artifacts. Most code examples use Python and are organized by chapter; small installations can use the chapter-specific requirements files; some chapters may require additional system libraries, GPU support, model downloads, or environment-specific configuration. For examples involving LLMs, VLMs, or heavy simulators, it is recommended to start from Chapter 01.
[Recommended Reading Path] For a first systematic read, it is recommended to proceed in chapter order: Chapters 1-4 establish the language of reinforcement learning and recursive thinking; Chapters 5-9 develop the core of deep reinforcement learning algorithms; Chapters 10-12 expand data and task settings. Readers focused on LLM post-training can proceed to Chapters 13-18 after completing Chapters 6-8; policy gradients, advantage estimation, PPO, and KL constraints directly explain the objectives used by RLHF, DPO, and GRPO. After that, you can choose topics related to agents or multimodal reinforcement learning from Chapters 19-24; Chapters 25-26 are recommended to be paired with any experimentRead together, because reward and evaluation errors will affect every method in the book. For each chapter it is recommended to: restate the problem it solves, derive the core formulas, run at least one experiment, and change one important assumption to explain the metric results.
[Target Audience and Learning Outcomes] This book is aimed at students, researchers, and engineers who have basic machine learning experience and want a systematic understanding of modern reinforcement learning. Readers should be familiar with Python and basic PyTorch, and have an introductory-level knowledge of linear algebra, probability, and calculus; the math appendix will rebuild these tools to the depth required by each chapter, so no prior advanced mathematics course is required. After completing the core chapters and experiments, you should be able to: formulate new decision-making problems using MDPs, value functions, Bellman equations, and credit assignment; implement, read, and diagnose DQN, REINFORCE, Actor-Critic, PPO, DPO, and GRPO; explain the relationships among SFT, reward modeling, preference optimization, RLHF, and RLVR in LLM post-training; design trajectories, rewards, training loops, and evaluation protocols for tool-use, code, and multimodal agents; identify failure modes behind training curves and test improvement proposals with controlled experiments.
[Planned Milestones and Limitations] The project's announced plans include: 2026-05-02 initial open-source browsable version; 2026-05-10 release of the first stable minor version, fixing early typos and stabilizing the content and code of Parts 1 and 2; late May 2026 improvement of reproducible LLM RL experiments and addition of a complete RLVR hands-on module with evaluation; early June 2026 gradual delivery of agent RL projects from single-tool use to complex Deep Research trajectory synthesis; late June 2026 addition of Unity-based embodied RL environments and trainable project examples; July 2026 and beyond expansion of multimodal frontier content, including complete VLM RL or Diffusion RL hands-on cases. Limitations that need to be stated: the course content is being expanded chapter by chapter, and chapters marked as "under construction" may still contain errors; the project explicitly states that this course was created with AI assistance and has not yet been fully reviewed, and may contain factual errors or code that does not run as expected, and Issues and Pull Requests are welcome. Due to limited computational resources, the project is seeking GPU support. The project uses the CC BY-NC-SA 4.0 license.
