What is SCoRe
SCoRe (Self-Correction via Reinforcement Learning) is an innovative multi-turn reinforcement learning method introduced by Google DeepMind, designed to enhance the self-correction capabilities of large language models (LLMs). By training on data generated by the model itself, it enables the model to self-correct erroneous answers without external guidance. SCoRe's training consists of two stages: the first stage initializes the model through appropriate regularization constraints to avoid mode collapse during training; the second stage uses a reward mechanism to encourage the model to perform effective self-correction on its second attempt. Experimental results show that SCoRe improves self-correction ability on math problems and programming tasks by 15.6% and 9.1%, respectively, outperforming traditional supervised learning methods. SCoRe's success demonstrates the potential of reinforcement learning in enhancing large model performance, especially in application scenarios requiring high accuracy.
Main Features of SCoRe
- Self-Correction: SCoRe enables large language models to identify and correct their own errors without external feedback.
- Self-Generated Data Training: Trains based on data generated by the model itself, without relying on external annotations or teacher models.
- Performance Improvement: Improves the model's self-correction ability in math and programming tasks.
- Multi-Turn Learning: Gradually improves answers through multiple attempts to achieve the best response.
- Strong Adaptability: Can adapt to differences in data distribution between training and inference.
Technical Principles of SCoRe
- Multi-Turn Reinforcement Learning: SCoRe is based on a multi-turn RL framework, allowing the model to learn how to improve its behavior across multiple consecutive attempts.
- Regularization Constraints: Uses regularization techniques, such as KL divergence, in the model's first attempt to maintain output stability.
- Reward Shaping: Encourages the model to perform effective self-correction in subsequent attempts by designing a reward function.
- Policy Initialization: In the first stage of training, improves the model's self-correction ability through specific policy initialization.
- Avoiding Distribution Mismatch: SCoRe avoids the mismatch between training data and the model's actual response distribution by training on self-generated data.
- Incremental Learning: The model attempts to improve based on previous outputs in each attempt, enabling incremental learning.
SCoRe Project Address
- arXiv Technical Paper：https://arxiv.org/pdf/2409.12917
Application Scenarios of SCoRe
- Mathematical Problem Solving: In the field of mathematics, models need to perform complex calculations and logical reasoning. SCoRe helps the model self-correct after giving an incorrect answer, improving solution accuracy.
- Programming and Code Generation: In programming tasks, code correctness is crucial. SCoRe can guide the model to correct errors in code, improving code reliability.
- Legal Document Analysis: Document analysis in the legal field requires extremely high accuracy. SCoRe helps the model self-correct when interpreting legal provisions and cases.
- Financial Report Generation: Errors in financial reports can lead to serious consequences. SCoRe ensures the model's accuracy when generating reports.
- Medical Diagnosis Assistance: In the medical field, the model's self-correction capability helps improve diagnostic accuracy and reduce the risk of misdiagnosis.