What is LLaVA-OneVision-1.5
LLaVA-OneVision-1.5 is an open-source multimodal model that achieves high performance, low cost, and strong reproducibility through efficient training and high-quality data. It uses the self-developed RICE-ViT as the vision encoder, combines 2D rotary position encoding and region-aware attention mechanisms, supports variable input resolutions, and enhances object and OCR capabilities. The language model is based on Qwen3 and is optimized through a three-stage training pipeline (language-image alignment, high-quality knowledge mid-training, and visual instruction alignment). During training, offline parallel data packing and hybrid parallelism strategies are used to improve compute and memory efficiency. On the data side, an 85M pretraining dataset was built using a "concept balancing" strategy, covering multiple sources; 22M instruction data covers eight major categories, after multi-source aggregation and format unification. LLaVA-OneVision-1.5 performs well on multimodal benchmarks, with controllable cost, and the entire pipeline is transparent and open, providing code, data, and model resources to help the community reproduce and extend it at low cost.
Main features of LLaVA-OneVision-1.5
-
Multimodal understanding and generation: Can process and understand information from multiple modalities such as images and text, generate high-quality text descriptions, answer questions, or perform reasoning.
-
Visual question answering (VQA): Answers questions about the content in images, supports a wide range of visual tasks such as object recognition and scene understanding.
-
Image caption generation: Generates accurate and detailed descriptive text for input images, helping users better understand image content.
-
Instruction following and execution: Executes corresponding tasks according to user-provided instructions, such as image editing and information extraction, with good instruction generalization ability.
-
Cross-modal retrieval: Supports retrieving images based on text queries or retrieving text based on image queries, enabling cross-modal information retrieval.
-
Long-tail recognition capability: Can effectively recognize and understand categories or concepts that appear less frequently in the data, improving the model's generalization ability.
-
Multilingual support: Supports input and output in multiple languages, with certain cross-lingual understanding and generation capabilities.
-
Knowledge enhancement: Through pretraining on high-quality knowledge data, the model has richer world knowledge and can better handle complex multimodal tasks.
-
Efficient training and reproduction: Uses optimized training strategies and data packing techniques to achieve an efficient training process, and provides complete code and data resources for the community to reproduce and extend.
Technical principles of LLaVA-OneVision-1.5
-
Vision encoder: Uses the self-developed RICE-ViT (Region-aware Cluster Discrimination Vision Transformer) as the vision backbone, enhancing semantic understanding of local regions in images through a region-aware attention mechanism and a unified region cluster discrimination loss, while also supporting variable input resolutions.
-
Projector design: Maps visual features to the text embedding space of the language model through a multilayer perceptron (MLP), achieving effective alignment between visual features and language features.
-
Language model: Based on Qwen3 as the language backbone, it provides powerful language generation and understanding capabilities, and supports text processing in multimodal tasks.
-
Three-stage training pipeline: including language-image alignment, high-quality knowledge mid-training, and visual instruction alignment, progressively enhancing the model's multimodal alignment capabilities and task generalization abilities.
-
Offline Parallel Data Packing: constructing a pretraining dataset through a feature-driven "concept balancing" strategy, and adopting offline parallel data packing technology to reduce padding waste and improve training efficiency.
-
Hybrid Parallelism and Long-Context Optimization: adopting hybrid parallelism (tensor parallelism, pipeline parallelism, and sequence parallelism) as well as long-context optimization techniques during training to improve compute utilization and memory efficiency.
-
Data Construction and Optimization: constructing large-scale pretraining datasets and instruction fine-tuning datasets, and ensuring high quality and diversity of data through multi-source aggregation, format unification, and safety filtering.
Project Address of LLaVA-OneVision-1.5
-
Github Address：https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5
-
HuggingFace Model Library：https://huggingface.co/collections/lmms-lab/llava-onevision-15-68d385fe73b50bd22de23713
- arXiv Technical Paper：https://arxiv.org/pdf/2509.23661
-
Online Experience Demo：https://huggingface.co/spaces/lmms-lab/LLaVA-OneVision-1.5
Application Scenarios of LLaVA-OneVision-1.5
-
Intelligent Customer Service: by understanding the image or text information uploaded by users, providing automated customer service support, answering questions, and offering solutions.
-
Content Creation: helping creators generate image descriptions, creative copy, or stories, improving creative efficiency and quality.
-
Educational Assistance: in the education field, used to explain the content in images, assist teaching, and help students better understand complex visual information.
-
Medical Imaging Analysis: assisting doctors in interpreting medical images, providing preliminary diagnostic suggestions or generating imaging reports.
-
Intelligent Driving: in autonomous driving systems, used to understand road scenes, assist decision-making, and improve driving safety.
-
Image Editing and Design: performing operations such as editing, cropping, and adding special effects to images according to user instructions, improving the convenience of image processing.