What is Aligner
Aligner is a large language model alignment technology proposed by a team at Peking University, which improves model performance by learning the correction residual between aligned answers and unaligned answers. It uses an autoregressive seq2seq model and is trained on a Query-Answer-Correction (Q-A-C) dataset, without relying on the complex Reinforcement Learning from Human Feedback (RLHF) process. The core advantages of Aligner lie in its efficiency and flexibility. As a plug-and-play module, it can be directly applied to various open-source and API-based models, without access to model parameters.
Main Features of Aligner
- Correction Residual Learning: As an autoregressive seq2seq model, Aligner is trained on a Query-Answer-Correction (Q-A-C) dataset to learn the difference between aligned and unaligned answers, achieving more precise model alignment.
- Weak-to-Strong Generalization: Using a small-parameter Aligner model to fine-tune large-parameter LLMs can significantly improve the performance of strong models.
- Plug-and-Play: Aligner can align models whose parameters cannot be accessed, such as GPT3.5, GPT4, and Claude2.
- Training Process：
- Data Collection: Obtain queries from various open-source datasets and generate original answers.
- Answer Correction: Use GPT-4, Llama2-70B-Chat, and human annotation to correct the original answers in line with human values.
- Model Training: Based on the corrected dataset, train the Aligner model, which can correct original answers into aligned answers.
Aligner's Performance
- Improving Helpfulness and Safety: Experiments show that using Aligner-7B can improve the helpfulness and safety of GPT-4, increasing them by 17.5% and 26.9%, respectively.
- Weak-to-Strong Generalization: Fine-tuning Llama2-70B under the supervision of Aligner-13B improved helpfulness and safety by 8.2% and 61.6%, respectively.
- Multi-Model Compatibility: Aligner-7B alignment improved the helpfulness and safety of 11 models, including closed-source, open-source, and safely/unsafely aligned models.
Aligner's Project Address
- Project Official Website：https://pku-aligner.github.io/
- GitHub Repository：https://github.com/PKU-Alignment/aligner
- HuggingFace Model Library：https://huggingface.co/aligner/aligner-7b-v1.0
- arXiv Technical Paper：https://arxiv.org/pdf/2402.02416
Aligner's Application Scenarios
- Multi-Turn Dialogue Scenarios: In multi-turn dialogues, Aligner can improve the alignment effect of conversations under the challenge of sparse rewards.
- Alignment of Reward Models with Human Values: Aligner can be trained with specific corpora to correct the output of the preceding model to reflect specific values.
- Streaming and Parallel Processing of MoE-Aligner: By specializing and integrating Aligner processing, a more powerful and comprehensive Mixture of Experts (MoE) Aligner can be created.