finetune-Qwen2-VL

Purpose

finetune-Qwen2-VL is an open-source code repository designed to provide a quick-start solution for fine-tuning or continued pre-training of the Qwen2-VL 2B model. Qwen2-VL is a multimodal vision-language model released by the Qwen team on August 29, 2024, and includes 2B, 7B, and 72B versions. Since the Qwen team did not provide simple fine-tuning or continued pre-training code at the time, the repository author wrote concise and transparent fine-tuning scripts to make it easier for developers who prefer to write their own training loops.

Main Capabilities

• Provides both single-GPU and multi-GPU fine-tuning scripts (finetune.py and finetune_distributed.py).
• Supports continued pre-training.
• Supports training on video data (requires installing the av package).
• Supports flash_attention2 to accelerate training and testing.
• Supports mixed-precision training (bfloat16 + float32), which can improve validation loss.
• Uses torchvision.io.VideoReader instead of torchvision.io.read_video to read videos, which can improve speed by about 200x.
• Includes a toy dataset (train_data/data.json) and test scripts (test_on_official_model.py, test_on_trained_model_by_us.py).
• Provides a fine-tuned model for download (for use when GPU resources are insufficient).

Usage

1. Prepare data: prepare your own data by referring to the train_data/data.json format.
2. Single-GPU fine-tuning: run finetune.py.
3. Multi-GPU fine-tuning: run finetune_distributed.py.
4. Test the fine-tuned model: modify model_dir in test_on_trained_model_by_us.py to specify the model path, then run it.
5. Training on video data: you need to run pip install av, and note that video data consumes a lot of GPU memory, so you can set the batch size to 1, or reduce the video-related configuration in util/vision_util.py.

Applicable Scenarios

• Developers who want to continue training on their own dataor fine-tune the Qwen2-VL 2B model.
• Researchers who need simple, transparent training code that is easy to modify and extend themselves.
• Users with limited resources (GPU Poor) who want to try fine-tuning can download the fine-tuned model provided by the author.

Limitations

• The training code does not include an evaluation step; it only prints the training loss. If evaluation is needed, you must refer to the data format yourself and modify the script.
• The repository only provides toy data; in actual use, you need to prepare your own data.
• Video training has high GPU memory requirements.
• This repository mainly targets the Qwen2-VL 2B model; other versions (such as 7B and 72B) are not explicitly supported.
• The 2025/02/08 update mentions that fine-tuning support for Qwen2.5-VL-3B has been moved to another repository (https://github.com/zhangfaen/finetune-Qwen2.5-VL），使用 huggingface/accelerate + deepspeed plugin runs on 4 GPUs.

Update Log

• 2025/02/08: Added Qwen2.5-VL-3B fine-tuning support (see the new repository above).
• 2025/01/24: Added mixed-precision training (bfloat16 + float32) to improve validation loss.
• 2024/10/12: Updated dependencies to huggingface/transformers 4.45.2.
• 2024/09/25: Used torchvision.io.VideoReader instead of read_video to improve video reading speed.
• 2024/09/13: Added video data examples; requires installing the av package.
• 2024/09/12: Supports flash_attention2.

Acknowledgments

This repository is built on https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct/ and https://huggingface.co/zhangfaen/Qwen2-VL-2B-checkpoint/ .