Zero-Qwen-VL is an open-source project on GitHub whose goal is to train a LLaVA-architecture model with better support for Chinese. The project author originally hoped to use a language model that supports Chinese, an image encoder that supports Chinese, and Chinese data to build from scratch an image-text model that supports Chinese. The author believed that the Qwen series is currently among the best open-source models and supports Chinese, so chose to combine Qwen-VL's image encoder with Qwen2's language model. After completing the project, the author also mentioned that Qwen-VL's image encoder plus Qwen2 is fairly similar to Qwen-VL itself, but still open-sourced the entire process so that the image encoders and LLMs of other models can be combined later based on this project.
Main capabilities: The project provides a complete process for building from scratch a LLaVA-architecture Chinese image-text model, including training code, data, and three-stage model weights. Training is divided into three stages, with each stage using different data and frozen parts. The data for stage1 and stage2 comes from LLaVA-CC3M-Pretrain-595K and LLaVA-Instruct-150K in Chinese-LLaVA-Vision-Instructions; stage3 uses translated data from TextOCR-GPT4o and ShareGPT-4o. The project also provides specific steps for environment configuration, weight extraction and integration, training, and testing.
Usage: The project provides detailed environment configuration commands, using conda to create a Python 3.12 environment and install dependencies such as transformers, torch, torchvision, Pillow, accelerate, datasets, deepspeed, and tensorboard. You need to download the model weights of Qwen-VL-Chat and Qwen2-0.5B-Instruct, run extract_weight.py to extract the model weights into the corresponding vision folder, and integrate Qwen-VL-Chat's vision weights and Qwen2-0.5B-Instruct's weights into the ./llava_pro directory, after which the model can be loaded directly from that directory. The data path structure is /data_path containing an images folder and chat.json. After training is complete, replace the original model weights with the generated model weights, orplace the model's .py files and configuration files in the model weights folder and run test.py to test the results.
Applicable scenarios: Suitable for developers and researchers who wish to study multimodal large models, the LLaVA architecture, and Chinese image-text model training. The author mentioned that if better results are desired, a larger language model can be used instead. This project can also serve as a reference process for combining the image encoders and LLMs of other models.
Limitations: The author pointed out that the model after stage3 feels worse than after stage2, possibly because the stage3 training data was only simply translated and not cleaned, so the data quality is not high. The project uses Qwen2-0.5B-Instruct, and the model scale is relatively small, so the results may be limited. The project does not provide information on fees, pricing, or commercial use.
