What Are Self-Taught Evaluators
Self-Taught Evaluators is a new model evaluation method that improves the evaluation capabilities of large language models (LLMs) through self-training, without the need for manually annotated data. It starts from unlabeled instructions and uses an iterative self-improvement scheme to generate comparative model outputs. An LLM is used as a judge to generate reasoning traces and final judgments. This is repeated in each iteration, training the model with improved predictions. In experiments, Self-Taught Evaluators improved the evaluation accuracy based on the Llama3-70B-Instruct model from 75.4 to 88.3, and reached 88.7 with majority voting, surpassing commonly used LLM judges such as GPT-4 and achieving performance comparable to top reward models trained with manually annotated data.
Main Features of Self-Taught Evaluators
- Generate comparative model outputs: Starting from unlabeled instructions, generate pairs of model responses of different quality based on prompts.
- Train an LLM as a judge: Use an LLM to generate reasoning traces and final judgments to evaluate which response is better.
- Iterative self-improvement: In each iteration, use the current model's judgments to annotate training data, fine-tune the model, and achieve self-improvement.
- Evaluate model performance: Evaluate the model's accuracy on standard evaluation protocols such as RewardBench and compare it with human evaluation results.
Technical Principles of Self-Taught Evaluators
- Initialization: Assume access to a large number of human-written user instructions and an initial seed LLM.
- Instruction selection: Classify instructions based on the LLM and select a subset of instructions that are challenging and balanced in distribution.
- Response pair construction: Generate preference data for each selected instruction, including two responses (preferred and non-preferred), generated based on prompts, ensuring that the quality of the non-preferred response is lower than that of the preferred response.
- Iterative training: Includes two steps: judgment annotation and model fine-tuning. Use the current model to generate reasoning traces and judgments, and if the judgment is correct, add the example to the training set. Fine-tune the model with the data to provide an updated model for the next iteration.
Project Address of Self-Taught Evaluators
- GitHub Repository：https://github.com/facebookresearch/RAM/tree/main/projects/self_taught_evaluator
- HuggingFace Model Library：https://huggingface.co/datasets/facebook/Self-taught-evaluator-DPO-data
- arXiv Technical Paper：https://arxiv.org/pdf/2408.02666
Application Scenarios of Self-Taught Evaluators
- Language model development: When developing new large language models (LLMs), Self-Taught Evaluators can evaluate and optimize the output quality of the model, ensuring that the text generated by the model meets the expected standards.
- Automated content evaluation: In content production fields such as news organizations, publishing, or social media platforms, it can be used to automatically evaluate the quality and accuracy of content and improve the efficiency of content moderation.
- Education and academic research: In the field of education, Self-Taught Evaluators can serve as an auxiliary tool to help evaluate students' writing assignments or research papers and provide feedback and improvement suggestions.
- Customer service and technical support: In the customer service field, it can be used to evaluate the quality of automated response systems, ensuring that responses are both accurate and helpful and improving customer satisfaction.
- Programming and code generation: For scenarios requiring code generation and evaluation, Self-Taught Evaluators can evaluate the quality of generated code snippets and help developers improve code.