What is Skywork-VL Reward
Skywork-VL Reward is an open-source multimodal reward model from Skywork AI that provides reliable reward signals for multimodal understanding and reasoning tasks. The model is based on the Qwen2.5-VL-7B-Instruct architecture, with an added reward head structure, trained on pairwise preference data, and outputs scalar reward scores aligned with human preferences. The model achieved a SOTA score of 73.1 on VL-RewardBench and performed excellently on RewardBench, reaching a high score of 90.1. Skywork-VL Reward significantly improves multimodal reasoning capabilities based on Mixed Preference Optimization (MPO), bringing new breakthroughs to the field of multimodal reinforcement learning.
Main Features of Skywork-VL Reward
- Evaluating Multimodal Outputs: Assesses the quality of outputs generated by vision-language models (VLMs) to determine whether they align with human preferences.
- Providing Reward Signals: Outputs scalar reward scores that reflect the quality of generated content or its alignment with human preferences.
- Supporting Multimodal Tasks: Applicable to a variety of multimodal tasks, such as image captioning and complex reasoning, with broad applicability.
- Improving Model Performance: Based on generating high-quality preference data, it supports Mixed Preference Optimization (MPO), significantly enhancing multimodal reasoning capabilities.
Technical Principles of Skywork-VL Reward
- Model Architecture: Based on the Qwen2.5-VL-7B-Instruct architecture, which includes a vision encoder (Vision Transformer), a vision-language adapter, and a language model decoder. On top of the base model, a reward head structure is added to output scalar reward scores. The reward head processes the final hidden state through a fully connected layer to generate reward scores.
- Dataset Construction: Integrates multiple open-source preference datasets (such as LLaVA-Critic-113k, Skywork-Reward-Preference-80K-v0.2, RLAIF-V-Dataset) and internally annotated complex reasoning task data. Through steps such as deduplication, similarity filtering, and preference judgment filtering, it ensures the high quality and consistency of the data. It uses advanced VLM reasoners to generate high-quality preference data, enhancing the model's generalization ability.
- Training Method: Based on a pairwise preference loss function, the model is trained by comparing the pros and cons of two candidate responses, enabling the model to learn relative rankings. Two-stage fine-tuning: the first stage uses multimodal preference data for training, and the second stage incorporates pure text preference data to further improve the model's performance in pure text scenarios.
Project Address for Skywork-VL Reward
- HuggingFace Model Repository：https://huggingface.co/Skywork/Skywork-VL-Reward
- arXiv Technical Paper：https://arxiv.org/pdf/2505.07263
Application Scenarios of Skywork-VL Reward
- Content Generation Evaluation: Evaluates the quality of multimodal content generation, such as image captioning and video subtitles, to determine whether the generated content is accurate and aligns with human preferences.
- Reasoning Task Optimization: In complex multimodal reasoning tasks, such as visual question answering and geometry problems, it evaluates the rationality of the reasoning process and results, helping to optimize reasoning models.
- Model Alignment: Ensures that the outputs of multimodal models align with human values and ethical standards, avoiding the generation of harmful or misleading content.
- Mixed Preference Optimization (MPO): As a key component of MPO training, it provides high-quality preference data, improving the reasoning ability and generalization performance of multimodal models.
- Benchmark Testing: As a benchmarking tool for multimodal tasks, it evaluates and compares the performance of different models, driving the development of multimodal technology.