lmms-finetune is a minimal codebase for fine-tuning large multimodal models (LMMs). As multimodal models continue to be released, their fine-tuning process is often not simple, and this codebase aims to provide a unified, streamlined structure to lower the barrier to fine-tuning.
Purpose
This codebase is mainly used to fine-tune various large multimodal models, supporting single-image models (such as LLaVA-1.5, LLaVA-1.6/NeXT, Phi-3-Vision, Llama-3.2-Vision), multi-image/interleaved-image models (such as Qwen-VL-Chat, Qwen2-VL-Instruct, LLaVA-NeXT-Interleave, Qwen2.5-VL-Instruct), video models (such as LLaVA-NeXT-Video), and unified models (such as LLaVA-Onevision). For the full support list, see supported_models.md or run python supported_models.py.
Main Capabilities
- Supports multiple training strategies: full fine-tuning, LoRA, and Q-LoRA for the LLM component; full fine-tuning and LoRA for the vision encoder/backbone.
- Uses the official Hugging Face implementation, and inference after fine-tuning is exactly the same as with the original HF model.
- Abstracts the components of the fine-tuning process (such as model loading and data collation), making it easy to integrate the latest LMMs.
- Keeps the codebase concise and lightweight, easy to understand and modify.
- Supports mixed text-image/video data, as well as text-only samples.
- Provides a gradio webui interface for fine-tuning (run python webui.py).
- Provides the merge_lora_weights.py script to conveniently merge LoRA weights to obtain a standalone model.
Usage
- Data preparation: the data must be a JSON file containing a list of dictionaries, each dictionary being one sample. Image and video markers are <image> and <video> respectively, supporting multi-sample, multi-image/video input, and text-only samples. Actual images/videos are by default stored in the corresponding folder, and paths in the JSON are relative to the root folder or use absolute paths.
- Modify the example bash script (example_video.sh or example_image.sh), specify the target model, data path, and other parameters, then run bash example_scripts/example_video.sh or bash example_scripts/example_image.sh to start training. Note that running the provided example_video.sh requires downloading video clips from ShareGPT4Video.
- If you prefer a graphical interface, run python webui.py to launch the gradio interface for fine-tuning.
- Inference: after correctly loading the fine-tuned model, the inference method is the same as the corresponding Hugging Face model. Refer to the inference documentation and merge_lora_weights.py to obtain a standalone model.
Applicable Scenarios
Suitable for researchers or developers who need rapid experimentation, flexible modification, and integration of new multimodal models. Compared with large-scale, highly optimized frameworks, this codebase is more transparent and easier to get started with, making it convenient to understand the details of the training process.
Limitations
- If the training data contains text-only entries and per_device_batch_size is 1, or the number of text-only instances is far greater than the number of multimodal instances, training may fail due to limitations/bugs in deepspeed.
- The supported model list is constantly updated, and some new models may require installing the latest transformers from GitHub (for example, the Qwen2.5 family requires the transformers-4.49.0.dev0 branch).
- When using LLaVA-Onevision, pay attention to the relevant notes.
Update Log
- 2025/01/27: Support for the Qwen2.5 family (requires installing the latest transformers from GitHub).
- 2024/12/16: Support for Llama-3.2-Vision training, added merge_lora_weights.py.
- 2024/10/16: Added LLaVA-Onevision support, updated collators to adapt to the new handling of LLaVA models in transformers.
- 2024/08/28: Support for fine-tuning via the gradio webui interface.
- 2024/07/30: Support for fine-tuning the vision encoder and projector.
- 2024/07/25: Released a Colab notebook demonstrating full training of LLaVA-NeXT-Video-7B; support for pure text in the training setText sample.
- 2024/07/20: Initial version released.
Active maintainer: Yuqian Hong; initial maintainers: Jingyang Zhang, Yueqian Lin. If used in research/projects, stars, shares, or citations are welcome.
