What is SWEET-RL
SWEET-RL is a multi-turn reinforcement learning framework introduced by Meta, specifically used to train large language model (LLM) agents for collaborative reasoning tasks. SWEET-R optimizes a "critic" model based on extra information available during training (such as reference solutions). The model provides rewards for each step, helping the "actor" model better allocate credit and optimize its policy. SWEET-RL performs excellently on the ColBench benchmark. Compared with other advanced algorithms, it improves the success rate and win rate by 6% on backend programming and frontend design tasks, enabling the Llama-3.1-8B model's performance to be comparable toGPT-4oand even surpass top models such as these.
Main Features of SWEET-RL
- Optimize multi-turn interaction tasks: SWEET-RL is specifically optimized for complex tasks requiring multi-turn interaction, such as backend programming and frontend design.
- Effectively allocate credit: Based on the introduction of extra information available during training (such as reference solutions), it provides rewards for each step, accurately evaluates the value of each action, and solves the credit assignment problem in multi-turn tasks.
- Support multiple task types: It supports handling complex frontend design tasks, demonstrating generality and adaptability across different types of tasks.
Technical Principles of SWEET-RL
- Extra information during training: SWEET-RL optimizes the "critic" model based on extra information available during training (such as reference solutions). The critic model provides rewards for each step, helping the "actor" model better allocate credit.
- Bradley-Terry objective: SWEET-RL uses the Bradley-Terry objective function to directly train the advantage function. The advantage function evaluates the effectiveness of each action under the current state. It avoids first training a value function to predict the expected utility of the current state and action, and better aligns with the pretrained LLM.
- Asymmetric information structure: Based on an asymmetric actor-critic structure, in which the critic model accesses extra information available during training, while the actor model accesses the interaction history. This allows the critic to more accurately evaluate the value of actions, and the actor to optimize its policy based on the evaluations.
- Parameterized advantage function: The advantage function is parameterized as the average log probability of each action, and trained based on a trajectory-level Bradley-Terry objective. The parameterization is more consistent with the pretraining objective of LLMs, improving the model's generalization ability.
SWEET-RL Project Address
- GitHub repository：https://github.com/facebookresearch/sweet_rl
- HuggingFace model library：https://huggingface.co/datasets/facebook/collaborative_agent_bench
- arXiv technical paper：https://arxiv.org/pdf/2503.15478
Application Scenarios of SWEET-RL
- Text proofreading: Helps authors and editors quickly correct typos and sensitive content in articles.
- Social media moderation: Ensures that content published on social media is compliant and protects personal or corporate reputation.
- Advertising compliance: Reviews advertising copy to avoid legal and market risks caused by content errors.
- Academic publishing: Ensures the accuracy and rigor of textbooks and academic works.
- Multimedia content detection: Reviews videos, audio, and images to ensure that multimedia content is legal and compliant.