What is SRPO
SRPO (Semantic Relative Preference Optimization) is atext-to-imagegeneration model launched by Tencent Hunyuan. By designing the reward signal as a text-conditioned signal, it enables online adjustment of rewards and reduces reliance on offline reward fine-tuning. SRPO introduces Direct-Align technology, which restores the original image directly from any time step through a predefined noise prior, avoiding the over-optimization problem at later time steps. Experiments on the FLUX.1.dev model show that SRPO can significantly improve the human-evaluated realism and aesthetic quality of generated images, and its training efficiency is extremely high, requiring only 10 minutes to complete optimization.
Main Features of SRPO
-
Improve Image Generation Quality: By optimizing the diffusion model, the generated images are significantly improved in realism, detail richness, and aesthetic quality.
-
Online Reward Adjustment: Supports users in dynamically adjusting reward signals through text prompts, changing the style and preferences of image generation in real time, without the need for offline fine-tuning of the reward model.
-
Enhance Model Adaptability: Enables the diffusion model to better adapt to different task requirements and human preferences, for example, optimizing under different lighting conditions, styles, or levels of detail.
-
Improve Training Efficiency: By optimizing the early stages of the diffusion process, SRPO can complete model training and optimization in a short time (such as 10 minutes), significantly improving training efficiency.
Technical Principles of SRPO
-
Direct-Align Technology：During training, SRPO injects Gaussian noise into clean images and restores the original image through a single-step denoising operation.In this way, SRPO can effectively avoid the over-optimization problem that occurs in the later time steps of the diffusion process and reduce reward hacking behavior (such as the model using the bias of the reward model to generate low-quality images).Compared with traditional methods, SRPO can perform optimization at early time steps, which helps improve training efficiency and generation quality.
-
Semantic Relative Preference Optimization（SRPO）：The reward signal is designed as a text-conditioned signal, and the reward signal is adjusted through positive and negative prompt pairs.The model is optimized by calculating the reward difference between positive and negative prompt pairs.SRPO supports dynamically adjusting the reward signal during training, enabling the model to adjust its generation strategy in real time according to different task requirements.
-
Reward Aggregation Framework：To improve the stability of optimization, SRPO injects noise multiple times during training, generating a series of intermediate images and performing denoising and restoration operations on each image.By aggregating intermediate rewards using a decay discount factor, SRPO can effectively reduce reward hacking behavior that occurs in later time steps and improve the overall quality of generated images.
Project Address of SRPO
- Project Official Website：https://tencent.github.io/srpo-project-page/
- GitHub Repository：https://github.com/Tencent-Hunyuan/SRPO
- HuggingFace Model Library：https://huggingface.co/tencent/SRPO
- arXiv Technical Paper：https://arxiv.org/pdf/2509.06942v2
Application Scenarios of SRPO
- Digital Art Creation: Artists and designers generate high-quality digital artworks, dynamically adjusting image styles through text prompts, achieving rapid iteration from concept sketches to final works.
- Advertising and Marketing: Advertising agencies generate images that match specific brand styles and market positioning, quickly generate multiple design options, and improve creative efficiency.
- Game Development: Game developers generate high-quality game textures, character designs, and scene backgrounds to enhance the game's visual effects and player experience.
- Film and television production: In the production of films and TV series, it is used to generate realistic special effects scenes, backgrounds, and characters, reducing the time and cost of post-production.
- Virtual reality (VR) and augmented reality (AR): The model can generate high-quality virtual environments and objects, enhancing the immersion and realism of VR and AR applications.