WorldPM - A series of preference modeling models jointly launched by Alibaba's Qwen team and Fudan University

What is WorldPM

WorldPM (World Preference Modeling) is a series of preference modeling models launched by Alibaba Group's Qwen team and Fudan University. Based on large-scale training, it reveals the scalability of preference models. The model is trained on 15 million preference data entries, and it is found that preference models show a clear power-law decline trend in objective domains, while in subjective domains it is difficult to present a single scaling trend due to multi-dimensional characteristics. The project provides a 72B-scale base model and multiple versions fine-tuned for specific datasets. WorldPM-72B-HelpSteer2 is suitable for specific tasks requiring precise preference judgment, WorldPM-72B-UltraFeedback is suitable for handling large-scale user feedback scenarios, and WorldPM-72B-RLHFLow is specifically designed for low-resource preference modeling, suitable for scenarios with small amounts of data. It supports quick use based on Hugging Face. WorldPM provides powerful preference modeling capabilities for tasks such as dialogue systems and recommendation systems in natural language processing, helping the development of related fields.

Main features of WorldPM

• Preference modeling: Learns human preference patterns and forms a unified preference representation.
• Improves generalization ability: Helps the model perform better on different datasets.
• As a base model: Used for preference fine-tuning to improve model performance.
• Enhances robustness: Identifies and handles erroneous or incomplete information.

Technical principles of WorldPM

• Preference data collection and processing: WorldPM collects user-generated preference data from public forums (such as StackExchange, Reddit, etc.). The data naturally forms preference pairs based on users' voting mechanisms for different answers.
• Large-scale training and scaling laws: WorldPM draws on scaling laws in language modeling, where model performance improves with increases in model parameter scale and training data volume. By training on models of different scales (from 1.5B to 72B parameters), WorldPM can explore the scaling potential of preference modeling. Experiments show that in adversarial and objective evaluation tasks, model performance significantly improves with increases in training data and model scale.
• Preference modeling framework: WorldPM is based on a modeling framework of binary preference pairs. For each pair of preference samples, the model calculates a reward score for each answer and optimizes the BT loss function based on the Bradley-Terry model to learn preference patterns.
• Style preference analysis and control: To address the issue of style preference in subjective preference evaluation, style preference analysis and control mechanisms are introduced. By separating style features (such as text length, Markdown format, etc.) and content features, the model can more accurately evaluate preferences and reduce the interference of style factors on evaluation results.
• Model expansion and fine-tuning: WorldPM can be directly used for preference evaluation, or as an initialization basis for other models for further preference fine-tuning. By fine-tuning on human preference datasets of different scales, WorldPM can further improve model performance, especially showing significant performance improvements when data is limited.

WorldPM project address

• GitHub repository：https://github.com/QwenLM/WorldPM
• HuggingFace model library：https://huggingface.co/Qwen/WorldPM-72B
• arXiv technical paper：https://arxiv.org/pdf/2505.10527

Application scenarios of WorldPM

• Language generation optimization: Makes AI-generated text more natural and more aligned with human preferences, for example, improving the response quality of chatbots.
• Personalized recommendation: Recommend content based on user preferences, such as articles, videos, or music, to improve recommendation accuracy and satisfaction.
• Intelligent Customer Service Improvement: Help intelligent customer service better understand user needs, generate replies that better meet user expectations, and improve user experience.
• Content Moderation and Security: Identify and filter out erroneous or harmful information to ensure the security and reliability of content.
• Multimodal Applications: Extend preference modeling to multimodal content such as images and videos, and optimize content generation and moderation.