MPP-LLaVA

MPP-LLaVA is a personal open-source project, including MPP-Qwen14B and MPP-Qwen-Next (Multimodal Pipeline Parallel based on Qwen-LM). The project is positioned to enable developers to train their own 8B/14B-scale LLaVA-like multimodal large language models on RTX3090/4090 24GB VRAM, supporting three input types—video, image, and multi-image—and two modes—sft and dialogue.

Main capabilities and features:

• Supports single-turn image question answering, multi-turn image dialogue, and video dialogue.
• Multi-image dialogue: without multi-image sft, but this capability emerged after video sft.
• Supports multi-GPU inference, using transformers' device_map="auto" to load the LLM part.
• Supports single-image inference, multi-image inference, and video inference.
• Supports multi-turn dialogue sft, multi-image sft, and video sft.
• Supports DeepSpeed pipeline parallelism (PP+DP).
• Has added huggingface-transformers implementation and pushed to huggingface.
• Open-sources processed pretrain and sft dataset JSON files.
• Open-sources sft weights (15GB), providing modelscope links and Baidu Netdisk links.

Tech stack and sources:

• Built on Lavis, using its BLIP2 ViT and Q-former.
• The language model uses Qwen7B-Chat (for the License, see the project LICENSE, supports commercial, research, and development use).
• The training paradigm follows LLaVA (pretrain + sft), using its pretraining and instruction tuning data.
• Video sft uses VideoChatGPT's 100k data.
• Uses DeepSpeedExamples.
• Some code is based on Lavis, under the BSD 3-Clause License.

Version evolution (according to repository update records):

• 2023/10: MiniGPT4Qwen, a personal bilingual MLLM with single-stage training using 18.8k high-quality bilingual instruction tuning data.
• 2024/2: MiniGPT4Qwen-14B, Scaling Up to 14B, using DeepSpeed Pipeline Parallel, the entire process uses only 2 RTX 4090 GPUs.
• 2024/3: MPPQwen-14B, extending MiniGPT4Qwen-14B to MPP-Qwen14B, with data and training paradigm referencing LLaVA (pretrain + sft), enabling the LLM during instruction fine-tuning, and the entire training process completed on 6 RTX4090 GPUs.
• 2024/4: Support for multi-GPU inference, fixing the chat template to achieve better dialogue results.
• 2024/5: Code supports multi-turn dialogue sft, video sft, and multi-image sft.
• 2024/6: MPP-Qwen-Next adds LLaVA multi-turn dialogue sft data and VideoChatGPT's 100k sft data, supports multi-turn image dialogue and video dialogue, and exhibits emergent multi-image dialogue capability; at the same time, open-sources the sft weights (15GB).

Usage (based on the information provided by the repository):

• Weight and data preparation: For model weights, refer to WEIGHT.md; for training data, refer to DATA.md; they need to be placed in the cache directory, with the structure described in the repository. The sft model weights (15GB) can be downloaded from modelscope or Baidu Netdisk.
• Inference: Supports single-GPU inference; for multi-GPU inference, the LLM is loaded using device_map="auto", allowing the LLM part of the model to be loaded across multiple GPUs. After running, input the image path; multiple images can be input, and after ending image path input with :f, you enter the dialogue; :clh clears the dialogue history (but the image input will not be changed). A gradio webui demo is provided.
• Training: Supports DeepSpeed pipeline parallelism, with example configurations nproc_per_node: 8 dp: 4 pp: 2 (nproc_per_node = pp * dp), and nproc_per_node: 8 dp: 1 pp: 8. Supports conversion of pipeline parallel weights to a pth file (only the linear projection layer is converted; if you need to convert the projection layer and all LLM parameters, the converted model file name is unfreeze_llm_model.pth), and the converted model file is stored under ckpt_dir, named model.pth.
• Custom data format (if you want to continue training): for a single turn, instruction and output are str; for multiple turns, instruction and output are lists of equal length.

Applicable scenarios:

• Individuals or small teams training 8B/14B multimodal large language models on consumer-grade GPUs (such as RTX3090/4090 24GB).
• Research, development, and experimentation on multimodal models that require image, multi-image, and video dialogue capabilities.
• Multimodal model projects that require pipeline parallelism (PP+DP) training and multi-GPU inference.

Limitations and notes:

• The multi-image dialogue capability was not specifically trained with multi-image sft; it emerged after video sft.
• If paths often cause errors, you can change all paths (including dataset configs) to absolute paths.
• Weight and data preparation require configuring the weights according to WEIGHT.md first.
• Other features, fees, prices, platforms, effects, etc. not described in the materials are not provided for now.