MiniGPT-4

MiniGPT-4 is an open-source multimodal vision-language model project by the Vision-CAIR team at King Abdullah University of Science and Technology (KAUST). The repository releases the source code for both MiniGPT-4 and MiniGPT-v2 models.

1. Purpose The goal of MiniGPT-4 is to enhance vision-language understanding with the help of advanced large language models (Enhancing Vision-language Understanding with Advanced Large Language Models). MiniGPT-v2 uses large language models as a unified interface for vision-language multi-task learning (Large Language Model as a Unified Interface for Vision-Language Multi-task Learning). Both can perform conversational understanding around user-provided images.

2. Related Papers The authors of the MiniGPT-4 paper are Deyao Zhu*, Jun Chen*, Xiaoqian Shen, Xiang Li, Mohamed Elhoseiny (* indicates equal contribution). The authors of the MiniGPT-v2 paper are Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong☨, Mohamed Elhoseiny☨ (☨ indicates equal last author).

3. Model Foundation and Versions MiniGPT-v2 is based on Llama2 Chat 7B. MiniGPT-4 provides two versions: Vicuna V0 and Llama 2. The project architecture follows BLIP-2, and the code is built on top of Lavis.

4. Main Update Log On October 31, 2023, the evaluation code for MiniGPT-v2 was released; on October 24, the fine-tuning code for MiniGPT-v2 was released; on October 13, the first major update, MiniGPT-v2, was released; on August 28, M was providedthe Llama 2 version of iniGPT-4.

5. Usage

6. Prepare the code and environment: clone the repository, create a Python environment, and activate it.

7. Prepare the pretrained LLM weights: clone and download the corresponding LLM weights from the corresponding Hugging Face space via git-lfs, then set the llama_model variable in the model configuration file to the LLM weight path. For MiniGPT-v2, set it on line 14 of the model configuration; for MiniGPT-4 (Llama2), set it on line 15; for MiniGPT-4 (Vicuna), set it on line 18.

8. Prepare the pretrained model checkpoint: after downloading the pretrained checkpoint, for MiniGPT-v2 set the path on line 8 of the evaluation configuration eval_configs/minigptv2_eval.yaml; for the MiniGPT-4 Vicuna version, set it on line 8 of eval_configs/minigpt4_eval.yaml, and for the Llama2 version, set it in eval_configs/minigpt4_llama2_eval.yaml.

9. Launch the demo locally: run the corresponding commands for MiniGPT-v2, MiniGPT-4 (Vicuna version), and MiniGPT-4 (Llama2 version) respectively. To save GPU memory, the LLM is loaded in 8 bit by default, and the beam search width is 1. Under this configuration, a 13B LLM requires about 23G GPU memory, and a 7B LLM requires about 11.5G GPU memory. On more powerful GPUs, you can set low_resource to False in the relevant configuration files to run the model in 16 bit: for MiniGPT-v2, this corresponds to minigptv2_eval.yaml; for MiniGPT-4 (Llama2), it corresponds to minigpt4_llama2_eval.yaml; for MiniGPT-4 (Vicuna), it corresponds to minigpt4_eval.yaml. In addition, the project thanks @WangRongsheng, and MiniGPT-4 can also be run on Colab. The training details of MiniGPT-4 and the fine-tuning details of MiniGPT-v2 are both explained in dedicated documentation.

10. Community Derivative Work Community results based on MiniGPT-4 include: InstructionGPT-4 (200 instruction paradigms for fine-tuning MiniGPT-4), PatFig (generating short and long captions for patent figures), SkinGPT-4 (an interactive dermatology diagnosis system based on a vision large language model), and ArtGPT-4 (enhancing MiniGPT-4's artistic vision-language understanding with adapters).

11. Open Source and Citation The repository uses the BSD 3-Clause license, and much of the code is based on Lavis, which also uses the BSD 3-Clause license. If you use MiniGPT-4/MiniGPT-v2 in research or applications, you must cite it according to the BibTeX provided in the repository. The project also provides a code of conduct and a security policy.