What is Verifier Engineering
Verifier Engineering is a new post-training paradigm jointly launched by the Chinese Academy of Sciences, Alibaba, and Xiaohongshu. It is designed for foundation models to address the challenge of providing effective supervision signals. Verifier Engineering performs verification tasks based on automated verifiers and provides feedback to foundation models. It is divided into three stages: search, verification, and feedback, to optimize model performance. Based on a closed-loop feedback mechanism, it strengthens the model's generalization ability and is an important step toward achieving artificial general intelligence.
Main Features of Verifier Engineering
- Search: Based on a given instruction, sample representative or potentially problematic samples from the model output distribution.
- Verify: Use various verifiers (such as rule detection, evaluation metrics, or manual annotation) to evaluate the generated candidate responses.
- Feedback: Enhance model performance through methods such as supervised fine-tuning or in-context learning based on verification results.
- Improve Model Performance: Continuously improve the model's accuracy and reliability based on a closed-loop feedback cycle.
Technical Principles of Verifier Engineering
- Goal-Conditioned Markov Decision Process (GC-MDP): Formalize verifier engineering as a GC-MDP, including state space, action space, transition function, goal space, goal distribution, and reward function.
- Search Structure: Linear search and tree search, navigating the state-action space and balancing exploration and exploitation.
- Verifier Classification: Classify verifiers according to verification form (binary, score, ranking, textual feedback), verification granularity (token level, thought level, trajectory level), verifier source (program-based, model-based), and whether additional training is required.
- Feedback Methods：
- Training-based feedback: Update model parameters efficiently based on data.
- Inference-based feedback: Modify the input or policy during inference without changing model parameters.
Verifier Engineering Project Address
- GitHub Repository：https://github.com/icip-cas/Verifier-Engineering
- HuggingFace Model Library：https://huggingface.co/papers/2411.11504
- arXiv Technical Paper：https://arxiv.org/pdf/2411.11504
Application Scenarios of Verifier Engineering
- Natural Language Processing (NLP): Machine translation, text summarization, sentiment analysis, question-answering systems, dialogue systems, etc., using verifier engineering to improve the model's language understanding and generation capabilities.
- Code Generation and Programming Assistance: Assist developers in writing and debugging code, automatically generate code snippets, and optimize code quality.
- Education and Academic Research: Build intelligent educational assistants, provide personalized learning suggestions, and automatically evaluate student assignments and papers.
- Content Moderation and Security: Detect and filter inappropriate content, such as hate speech, false information, and policy-violating content, ensuring the safety of platform content.
- Knowledge Question Answering and Search Engines: Improve the accuracy of search engines and provide more relevant and accurate information retrieval results.