What is Skywork-Reward-V2
Skywork-Reward-V2 is Kunlun Wanwei's second-generation open-source reward model series, including 8 models based on different base models and sizes, with parameter scales ranging from 600 million to 8 billion. The Skywork-Reward-V2 series models have comprehensively topped seven major mainstream reward model evaluation leaderboards, demonstrating outstanding performance. The models' success benefits from the Skywork-SynPref-40M dataset, a mixed dataset containing 40 million preference sample pairs, carefully screened and filtered based on a two-stage human-machine collaborative process. Skywork-Reward-V2 performs excellently in general preference alignment, objective correctness, safety, and other aspects, and demonstrates strong generalization ability in Best-of-N scaling capability and style bias resistance.
Main features of Skywork-Reward-V2
- General preference alignment: Accurately determines which of different responses better matches general human preferences, making model outputs closer to human subjective judgments, for example, selecting more natural and appropriate replies in chat scenarios.
- Objective correctness evaluation: Effectively identifies the objective accuracy of responses. For questions with clear factual basis, it can screen out correct answers, such as determining whether responses are accurate in tasks like mathematical calculations and factual queries.
- Safety judgment: Has the ability to identify whether responses are safe, avoiding the generation of replies containing harmful or inappropriate content, and ensuring model outputs comply with moral and safety standards, for example, filtering out responses involving violence, discrimination, and other harmful content.
- Best-of-N scaling capability: When faced with multiple candidate responses, it can efficiently select the optimal answer from them, improving the model's decision-making ability in multiple-choice scenarios, such as providing users with the best solution in multi-turn dialogues.
- Style bias resistance: Has strong adaptability and fairness toward responses in different styles, and will not develop bias due to differences in response style, ensuring the model maintains objective evaluation across diverse expressions, for example, making reasonable judgments in texts of different styles such as literary creation and professional exposition.
Technical principles of Skywork-Reward-V2
- Large-scale, high-quality dataset Skywork-SynPref-40M: Contains 40 million preference sample pairs, providing a rich data foundation for model training. Based on a two-stage human-machine collaborative process, combining the high quality of manual annotation and the scalable processing capability of models, 26 million high-quality preference data entries are screened from a large amount of data, ensuring the diversity and accuracy of the data.
- Training based on the Bradley-Terry model: Uses the classic Bradley-Terry model as the foundation, calculating relative preference scores between different responses to train the reward model. During training, the model continuously learns how to rank responses according to human preferences, optimizes reward signals, and enables the model to better capture human preference characteristics.
- Multi-round iterative training and optimization: Based on multi-round iterative training, the model identifies weak links according to current performance in each round, automatically annotates based on retrieving similar samples and using a multi-model consistency mechanism, and further expands and enhances the training data. The iterative optimization process continuously improves the model's understanding and discrimination of preferences, showing outstanding performance on different evaluation benchmarks.
- Model architecture and parameter adjustment: Trained based on the Qwen3 and LLaMA3 series models, providing models of different parameter scales to meet the needs of different scenarios. By adjusting model parameters and training strategies, such as learning rate and batch size, the model performance is optimized to ensure efficient convergence and stable performance under large-scale data training.
Project address of Skywork-Reward-V2
- GitHub repository：https://github.com/SkyworkAI/Skywork-Reward-V2
- HuggingFace model library：https://huggingface.co/collections/Skywork/skywork-reward-v2-685cc86ce5d9c9e4be500c84
- arXiv technical paper：https://arxiv.org/pdf/2507.01352
Application scenarios of Skywork-Reward-V2
- Dialogue system optimization: Evaluate the responses generated by the dialogue system, select the answers that better conform to human language habits and preferences, and improve the interaction quality and user experience of systems such as intelligent customer service and chatbots.
- Content recommendation optimization: In content recommendation systems, evaluate the attractiveness and matching degree of different content items, and provide users with more accurate and personalized recommendation content based on their historical behavior and preferences, improving the accuracy of the recommendation system and user satisfaction.
- Educational tutoring assistance: In the field of education, evaluate the answers submitted by students, judge their accuracy and completeness, provide targeted feedback and guidance, and assist teachers in teaching evaluation.
- Content moderation: Detect and filter harmful, inappropriate, or violating content, such as violence, pornography, discrimination, etc., to ensure the safe and compliant operation of platforms such as social media and forums.
- Game optimization: In game development, evaluate and optimize textual content such as game plots, character dialogues, and task designs to enhance the immersion and fun of the game.