What is rStar-Math
rStar-Math is an innovative research project launched by Microsoft Research Asia. Based on deep thinking driven by Monte Carlo Tree Search (MCTS), it enables small language models (SLMs) to reach or even surpass the level of OpenAI's large models in mathematical reasoning. rStar-Math does not rely on data distillation from more advanced models; instead, it uses self-evolving deep thinking to improve model performance. rStar-Math introduces three innovative methods: code-enhanced step-by-step verified reasoning trajectory synthesis, a Q-value-based process preference model (PPM) training method, and a four-round self-evolution training strategy. In the MATH benchmark, rStar-Math improves the accuracy of Qwen2.5-Math-7B from 58.8% to 90.0%, and in the AIME 2024 test it solves an average of 53.3% of the problems, surpassing OpenAI's o1-preview model. rStar-Math demonstrates the model's intrinsic self-reflection ability, identifying and correcting erroneous steps during the reasoning process.
Main features of rStar-Math
- Generate high-quality mathematical reasoning trajectories: Generate step-by-step verified reasoning trajectories based on Monte Carlo Tree Search (MCTS), ensuring the correctness and high quality of each step.
- Self-evolution: Use four rounds of self-evolution to gradually improve the performance of the policy model and the process preference model (PPM), handling more complex mathematical problems.
- Improve model accuracy: Significantly improve model accuracy in multiple mathematical benchmarks, for example, improving the accuracy of Qwen2.5-Math-7B from 58.8% to 90.0% in the MATH benchmark.
- Self-reflection ability: The model can identify and correct erroneous steps during the reasoning process, demonstrating self-reflection ability.
Technical principles of rStar-Math
- Code-enhanced step-by-step verified reasoning trajectory synthesis
- MCTS-driven deep thinking: Decompose complex mathematical problems into multiple single-step generation tasks, gradually build a search tree based on MCTS, and generate reasoning trajectories.
- Code execution verification: The policy model generates natural language (NL) reasoning steps and corresponding Python code.
- Q-value annotation: Based on two methods, terminal-guided annotation and PPM-enhanced annotation, automatically assign a Q-value to each step, guiding MCTS node selection and identifying high-quality steps.
- Process preference model (PPM) training method
- Avoid directly using Q-values: Traditional Q-values as reward labels suffer from noise and inaccuracy. rStar-Math trains the PPM by constructing step-level positive and negative preference pairs and using a pairwise ranking loss, improving the reliability of labels.
- Preference pair construction: For each step, select the two steps with the highest Q-values as positive examples and the two steps with the lowest Q-values as negative examples. The PPM is trained on these preference pairs to predict the reward label for each step.
- Four-round self-evolution
- Initial strong policy model: In the first round, DeepSeek-Coder-V2-Instruct is used as the initial policy model to perform MCTS rollout and generate training data.
- Reliable PPM training: In the second round, the updated policy model is used for more reliable Q-value annotation to train the first reliable PPM.
- PPM-enhanced MCTS: In the third round, the reliable PPM is used for MCTS to generate higher-quality reasoning trajectories and cover more mathematical and competition-level problems.
- Solve challenging problems: In the fourth round, increase the number of MCTS rollouts and use different random seeds to improve coverage of competition-level problems.
rStar-Math project address
- arXiv technical paper：https://arxiv.org/pdf/2501.04519
Application scenarios of rStar-Math
- Educational tutoring: Provide students with personalized mathematics learning tutoring, gradually solve complex mathematics problems, and improve problem-solving ability and comprehension.
- Scientific research support: Assist mathematicians and scientists in exploring complex mathematical problems, generate preliminary problem-solving ideas and verification steps, and accelerate the research process.
- Financial technology: In financial risk assessment and quantitative trading, predict market risks and optimize trading strategies based on precise mathematical models and reasoning.
- Engineering design: In engineering design and system optimization, use mathematical reasoning to optimize system parameters and improve system performance and reliability.
- Data analysis: In enterprise data analysis, based on mathematical models and reasoning, extract valuable information from large amounts of data to support market forecasting and business decision-making.