nanoVLM is an open-source repository released by Hugging Face, designed to provide the simplest and fastest solution for training and fine-tuning small vision-language models (VLMs). Its design philosophy is similar to nanoGPT, emphasizing code simplicity and readability, implemented in pure PyTorch, without relying on high-level training frameworks such as transformers.Trainer, accelerate, or deepspeed.
Purpose nanoVLM is mainly used for educational and experimental purposes, helping the community quickly understand and get started with the training process of vision-language models. It does not aim for state-of-the-art performance, but instead provides a lightweight platform that is easy to modify and experiment with, suitable for exploring the capabilities and efficiency of small VLMs.
Main capabilities
- The model definition and training logic total about 750 lines of code, including the vision backbone (models/vision_transformer.py, about 150 lines), language decoder (models/language_model.py, about 250 lines), modality projection (models/modality_projection.py, about 50 lines), VLM main body (models/vision_language_model.py, about 100 lines), and training loop (train.py, about 200 lines).
- The default configuration uses SigLIP-B/16-224-85M and HuggingFaceTB/SmolLM2-135M as backbones to build a 222M-parameter nanoVLM. After training for about 6 hours on a single H100 GPU (about 1.7 million samples), it reaches 35.3% accuracy on the MMStar benchmark.
- Provides training scripts, generation scripts, and evaluation support (via the lmms-eval toolkit).
- Supports loading pretrained weights from the Hugging Face Hub, and can also upload trained models to the Hub, automatically generating config.json, model.safetensors, and README.md model card.
- Supports local saving and loading of models.
- Provides the measure_vram.py script for measuring VRAM requirements under specific hardware and configurations.
Usage Users can clone the repository, set up the environment, and run the scripts, or open it directly in Colab. It is recommended to use uv as the package manager, but manual installation is also possible. Dependencies such as torchvision, pillow, datasets, huggingface-hub, transformers, and wandb. During training, run the provided training script using the default models/config.py; during generation, use the generate script, which can load pretrained models or custom models. Evaluation can be performed via lmms-eval.
Use cases nanoVLM is suitable for researchers, students, and developers to quickly experiment with small VLMs, adjust different settings and configurations, and explore model capabilities and efficiency. It is also suitable as teaching material for understanding the basic components and training process of VLMs.
Limitations
- The repository underwent breaking updates on June 4, 2025 and September 9, 2025. Some supporting scripts (such as notebooks and memory evaluation) may no longer work, and the old 222M model is incompatible with the new workflow.
- Training the default model requires at least about 4.5 GB of VRAM (batch size 1), and about 8 GB of VRAM can support batch size 16. Actual requirements vary with model architecture and sequence length.
- The project does not accept contributions that introduce dependencies such as transformers.Trainer, accelerate, or deepspeed, in order to maintain the simplicity of pure PyTorch.
- The current roadmap includes more evaluation, data packing, multi-GPU training, multi-image support, image segmentation, VLMEvalKit integration, etc., but some features have not yet been implemented.
