Purpose
Effective LLM Alignment Toolkit is a toolkit for training and aligning large language models, with the goal of providing a highly customizable, concise, easy-to-use, and efficient solution for LLM alignment. Users only need to define a YAML configuration, incorporating Hugging Face TrainingArguments parameters as well as method-specific parameters, to start training.
Main Capabilities and Supported Methods
The toolkit supports multiple training and alignment methods:
- SFT: can disable loss on unnecessary message roles.
- DPO and ORPO: support all TRL options (such as IPO, SLic-HF, RPO, etc.).
- CPO and SimPO: support all TRL options.
- SMPO: a self-developed stable alignment method.
- GPO: a self-developed offline implementation of GRPO.
- Distillation: comes with scripts and supports multiple distillation losses.
- Gradient-based prompt training: uses the Gumbel-Softmax trick to train prompts using real tokens from the tokenizer.
- Reward modeling and classification: Bradley-Terry reward training (supports margins and rewards centring); rejection sampling using vLLM and RM for asynchronous preference dataset generation; scoring datasets with RM to compare models; supports multi-class/multi-label/binary classification.
In addition, the toolkit supports mixing any number of datasets for training (the duplicate column names must be consistent), and all datasets follow the JSON lines format and conform to the Hugging Face standard (the message format is [{'role': ..., 'content': ...}]). It also provides a prompts basket (vi-basket) format to generate conversations with follow-up questions and system prompts, supports asynchronous batch generation based on the OpenAI client, and provides a FAISS-based Map-Reduce deduplication tool that can process dense embedding deduplication for extremely large datasets in parallel. The toolkit supports freezing specific model modules without LoRA (see unfreeze_layers_patterns in common args).
Technical Foundation
- Core: PyTorch, Transformers, TRL
- Distributed training: Accelerate, FSDP, DeepSpeed (Zero 2/3)
- Acceleration: vLLM, Flash Attention, SDPA, Liger Kernel (used for fused CrossEntropy in SFT)
- Build and installation: Poetry
- Result logging: optional wandb or clearml
Notes on SMPO and GPO
SMPO (Simple Margin Preference Optimization) is a self-developed alignment method inspired by methods such as IPO, SimPO, and C-RLFT, and introduces its own loss function to separate chosen and rejected pairs. Its core idea is to smoothly reach the target margin level, while avoiding forcing the model to retrain by adding balanced SFT losses to chosen and rejected; it also provides an independent margin scheduler (use_margin_schedule in the configuration) to improve stability in the early stages of training and prevent reward hacking.
GPO is a self-developed offline version of GRPO, and does not use vLLM during training. It is similar to KTO in not requiring pairwise comparisons, but adopts the GRPO formulation, the reward can be any floating-point number, and the number of completions per prompt (group size) is unrestricted. The required dataset contains the columns [prompt, completions, rewards], where prompt is List[Dict] (OpenAI format), completions is List[List[Dict]] of size G, and rewards is List[float] of precomputed rewards.
Usage
Run the following commands inside the project folder: Install Poetry: pip install poetry Install project dependencies: poetry install, verify: poetry show (Optional) Set the environment variable HF_HOME: export HF_HOME=/mnt/hf/ (Optional) Log in to the Hugging Face CLI: poetry run huggingface-cli login (Optional) Log in to Weights & Biases: poetry run wandb login Check the configuration settings in the accelerate/ folder (number of GPUs, etc.). Make sure the necessary Linux development libraries are installed,Includes GCC and G++ 8 or higher. Ensure the CUDA version is 11.8 or higher (12.1 recommended), and confirm that all GPUs are detected. After the first installation step, confirm Poetry version 1.8+, and Python 3.10.16 is recommended. After the second installation step, confirm that poetry run ds_report returns meaningful text, and verify the Torch version and NVIDIA packages. If DeepSpeed and fused_adam related errors are encountered during training, DeepSpeed must be removed from the environment and reinstalled.
Script running example: the user needs to select an accelerate configuration + a training configuration + the script itself. For example, start SFT training using a YAML configuration. The official documentation also provides YAML configuration examples, such as training SFT Llama 3.1, using the Liger kernel, assistant responses only, modifying the chat template, LoRA, and generating examples during evaluation.
Applicable scenarios
This toolkit is suitable for researchers and developers who need to train and align large language models, especially scenarios where they want to flexibly combine multiple alignment methods, distributed training, and acceleration techniques in a configuration-driven way, or need to develop their own alignment methods (SMPO, GPO) and complete workflows such as prompt optimization, reward modeling, and data deduplication.
Limitations
The official text does not provide specific performance metrics, a list of supported models, license information, a roadmap, or pricing information; some features and configuration details need to be referenced from the repository code. Before use, prerequisites such as a Linux development environment, CUDA 11.8+, and Python 3.10.16 must be met.
