nanoLLaVA

nanoLLaVA is an open-source project on GitHub (qnguyen3/nanoLLaVA), positioned as "the world's smallest vision language model," and built on LLaVA (Large Language and Vision Assistant). The repository also introduces Bunny: a series of lightweight but relatively capable multimodal models.

Purpose and Positioning Bunny targets multimodal (vision-language) tasks, aiming to achieve competitive multimodal understanding capability at a smaller model scale, and supports replacing different vision encoders and language model backbones.

Main Capabilities and Models

• Provides pluggable vision encoders, such as EVA-CLIP and SigLIP.
• Provides multiple language backbones, including Phi-1.5, StableLM-2, Qwen1.5, and Phi-2.
• Compensates for the impact of reduced model size by selecting and constructing more informative training data from broader data sources.
• According to the materials, Bunny-v1.0-3B based on SigLIP and Phi-2 not only outperforms multimodal models of the same scale, but also outperforms larger 7B multimodal models, and even achieves performance comparable to 13B models.
• On 2024.03.15, Bunny-v1.0-2B-zh, focused on Chinese, was released, based on SigLIP and Qwen1.5-1.8B.

Usage

• Provides HuggingFace transformers code snippets for using Bunny-v1.0-3B and Bunny-v1.0-2B-zh.
• Training is divided into two stages: (1) Pretraining stage: connect a frozen pretrained vision encoder with a frozen LLM, training only the connector; (2) Visual instruction fine-tuning stage: make the model follow multimodal instructions, updating the connector and learnable LLM parameters.
• Training dependencies: CUDA 11.8 and cuDNN 8.7.0 (CUDA 12 is also acceptable), need to create a conda virtual environment (python=3.10), install transformers, torch, torchvision, xformers, apex, flash-attention, etc., and install the bunny package.
• Training scale: uses 8 A100 GPUs; if conditions differ, you can reduce per_device_train_batch_size and correspondingly increaseLarge gradient_accumulation_steps, keeping the global batch size unchanged (global_batch_size = per_device_train_batch_size × gradient_accumulation_steps × num_gpus).
• Supported backbones: vision encoders support CLIP, EVA-CLIP, SigLIP; LLMs support phi-1.5, stablelm-2, qwen1.5-1.8b, phi-2. The code is built and tested based on the specific versions above.
• Pretraining data: uses a high-quality core set of LAION-2B (deduplicated and more informative), randomly samples 2 million image-text pairs and converts them into the training format.
• Visual instruction tuning data: constructs Bunny-695K by modifying SVIT-mix-665K.
• Serving and inference: can launch the Controller, Gradio Web Server, and Model Worker, and can also use CLI inference (including two loading methods for fully parameter-tuned models and LoRA fine-tuned models; LoRA weights can be merged with merge_lora_weights.py).

Applicable scenarios Suitable for users who want to conduct multimodal vision-language research and development at a smaller model scale, including scenarios that require replacing the vision encoder/language backbone for experiments, training multimodal models according to a two-stage process, and performing local inference via Gradio or CLI.

Limitations and license

• The materials do not provide nanoLLaVA's own specific model scale, training data, benchmark scores, or commercial use description.
• The project itself uses the Apache 2.0 license; the datasets and checkpoints used are subject to their respective original licenses, and users must comply with all their terms.
• Training recommends using 8 A100s; other hardware requires adjusting the batch size and gradient accumulation according to the instructions. The materials do not provide specific performance and time consumption under other hardware.