Purpose
Vision-LLM-Alignment aims to implement alignment training for Vision Large Language Models (Vision-LLMs), covering Supervised Fine-Tuning (SFT), Reward Model training, and PPO/DPO alignment training. The project targets vision-based large language models, including the LLaVA series models and the LLaMA-3.2-Vision model.
Main Capabilities
- Supports SFT training, reward model training, DPO training, and RLHF (PPO) training.
- Supports multi-image instruction fine-tuning (for LLaMA-3.2-Vision), added on 2024/10/03.
- Supports LLaMA-3.2-Vision training: set the model_architecture and template parameters to "llama-3.2-vision", and specify the LLaMA-Vision model path via from_checkpoint (2024/09/28).
- Supports LLaVA-NeXT (LLaVA-1.6) training: set model_architecture to "llava_next", and specify the model path via from_checkpoint (2024/08/19).
- Supports directly loading LLaVA models in all training stages (SFT, RM, PPO/DPO) (2024/07/10); support for directly loading LLaVA models in the SFT stage was added earlier, on 2024/07/07.
- Released RoVRM: a robust visual reward model optimized with auxiliary text preference data, trained and applied based on this repository (2024/08/21).
- Provides large-scale visual feedback datasets, located at wangclnlp/vision-feedback-mix-binarized and wangclnlp/vision-feedback-mix-binarized-cleaned (2024/07/18).
Usage
Anaconda/miniconda can be used to install the dependency packages required by the project. Vision-LLM requires a vision encoder and a language model; its architecture is shown in the architecture diagram in the repository; you can also directly use an SFT-trained vision LLM (such as LLaVA-1.5/-NeXT and LLaMA-3.2-Vision-Instruction) as the actor model. The project implements all alignment training based on the LLaVA dataset format, and example samples can be viewed in the data folder. The training pipeline includes: Supervised Fine-Tuning (SFT), reward model training, Direct Preference Optimization (DPO), and Reinforcement Learning from Human Feedback (RLHF). The repository provides a list of models that support training Vision-LLM from scratch, as well as a list of Vision-LLMs that support reward model training, PPO training, and DPO training. Other LLMs with similar architectures are also supported; custom model architectures can be integrated by modifying training/utils/model/build_model.py (loading models) and training/utils/data/DST.py (templates).
Experiments and Evaluation
A series of benchmark tests were conducted during system development. The preference dataset used RLAIF-V, and the input instructions for RLHF training used LLaVA-Instruct-150K. In the model evaluation stage, benchmarks such as MMHalBench, Object HalBench, AMBER, LLaVA-Benchmark, and MMinstruct were used to compare the differences in trustworthiness and helpfulness of vision LLMs before and after alignment. Reward model training used LLaVA-1.5-7B; Best-of-n sampling and RLHF alignment training were performed separately for the two models LLaVA-1.5-7B and LLaVA-1.5-13B, and DPO training was carried out for these two models, with results shown in the figures and tables in the repository.
Applicable Scenarios
Suitable for scenarios such as alignment research on vision large language models, SFT/RLHF/DPO training experiments, reward model construction, and evaluation of the trustworthiness and helpfulness of multimodal models.
Limitations and Notes
The repository does not specify the specific environment versions, hardware requirements, training time, or cost in the provided materials. Custom model architectures require modifying the specified code files. More alignment algorithm integrations or issue feedback can be submitted via issues.
