What Is OpenR
OpenR is an open-source framework jointly launched by University College London (UCL), Shanghai Jiao Tong University, the University of Liverpool, The Hong Kong University of Science and Technology (Guangzhou), and Westlake University. It combines search, reinforcement learning, and process supervision to enhance the reasoning capabilities of large language models (LLMs). The framework is inspired by OpenAI's o1 model, which significantly improves model reasoning capabilities by integrating reinforcement learning during the reasoning process. OpenR is the first framework to provide an open-source implementation of integrated techniques, supporting LLMs in achieving advanced reasoning based on effective data acquisition, training, and inference paths. OpenR supports online reinforcement learning training and multiple search strategies, follows the test-time scaling law, and supports models in providing refined outputs through generation or search at test time. OpenR provides an automated data pipeline that extracts reasoning steps from outcome labels, reducing manual annotation workload while ensuring the collection of valuable reasoning information.
OpenR's Main Features
- Integrated Training and Inference: Integrates data acquisition, reinforcement learning training (online and offline), and non-autoregressive decoding into a unified platform.
- Process Reward Model (PRM): Improves the LLM policy based on policy optimization techniques during training, and guides the LLM's search process during the decoding phase.
- Reinforcement Learning Environment: Models mathematical problems as Markov decision processes (MDPs) and optimizes the model policy based on reinforcement learning methods.
- Multi-Strategy Search and Decoding: Supports multiple search algorithms, such as Beam Search and Best-of-N, combining PRM-guided search and scoring.
- Data Augmentation and Automated Annotation: Generates synthetic samples based on automated methods, reducing reliance on manual annotation and improving data collection efficiency.
OpenR's Technical Principles
- Process Reward Model (PRM): PRM is used to evaluate the correctness of solution steps. It is trained based on supervised learning, using correct or incorrect judgments as classification labels to predict the subsequent tokens of each step.
- Policy Iteration: During training, PRM improves the LLM policy based on policy optimization techniques such as policy iteration. During the decoding phase, PRM guides the LLM's search process, directing reasoning toward more effective results.
- Markov Decision Process (MDP): Converts mathematical problems into MDPs, consisting of states, actions, and rewards. The model generates reasoning steps as actions, and the next state is determined based on the current state and action.
- Reinforcement Learning: Uses algorithms such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) for online reinforcement learning training to optimize the language outputs generated by the model.
- Search Algorithms: During the decoding phase, PRM is used to evaluate the accuracy of each solution step, combining with the language model for guided search and scoring or voting over multiple generations.
OpenR's Project Address
- Project official website：openreasoner.github.io
- GitHub repository：https://github.com/openreasoner/openr
- Technical paper：https://github.com/openreasoner/openr/blob/main/reports/OpenR-Wang.pdf
OpenR's Application Scenarios
- Mathematical Problem Solving: OpenR solves mathematical problems by generating and evaluating reasoning steps to find the correct solution path.
- Code Generation and Debugging: In software development, OpenR helps generate code snippets or find and fix errors in code during debugging.
- Natural Language Processing (NLP) Tasks: OpenR is used in NLP tasks that require deep understanding of text and logical reasoning, such as machine reading comprehension, question-answering systems, and text summarization.
- Educational assistance: In the field of education, OpenR serves as an auxiliary tool to help students understand complex concepts and problem-solving steps, providing personalized learning paths.
- Automated customer service: In the field of customer service, OpenR provides accurate answers and solutions by reasoning about users' questions and needs.