OpenVision - Open-source vision encoder family from the University of California

What is OpenVision

OpenVision is a fully open, efficient, and flexible family of advanced vision encoders launched by the University of California, Santa Cruz (UCSC), focusing on multimodal learning. It provides models of various scales from 5.9M to 632.1M parameters, suitable for different scenarios from edge devices to high-performance servers. OpenVision adopts a progressive multi-stage resolution training strategy, with training efficiency 2 to 3 times faster than similar proprietary models. It performs excellently on multimodal benchmarks, comparable to models such as OpenAI's CLIP and SigLIP. OpenVision supports variable-size patches of 8×8 and 16×16, flexibly meeting the needs for detailed visual understanding or efficient processing.

Main Features of OpenVision

• Fully Open: Datasets, training recipes, and model checkpoints are all publicly available, open-sourced under the Apache 2.0 license, promoting reproducibility and transparency in multimodal research.
• Diverse Model Scales: Provides vision encoders from 5.9M to 632.1M parameters, with a total of 26 different models, covering various deployment needs from edge devices to high-performance servers.
• Outstanding Performance: On multimodal benchmarks, OpenVision's performance is comparable to proprietary vision encoders such as OpenAI's CLIP and SigLIP, and in some cases surpasses them.
• High Training Efficiency: Through the progressive multi-stage resolution training strategy, OpenVision is 2 to 3 times faster in training efficiency than proprietary counterparts.
• Flexible Configuration: Supports variable-size patches of 8×8 and 16×16, supporting detailed visual understanding or efficient processing according to actual needs.

Technical Principles of OpenVision

• Progressive Resolution Training Strategy: OpenVision adopts a training approach that starts from low resolution (e.g., 84×84) and gradually increases to high resolution (e.g., 336×336 or 384×384). This significantly improves training efficiency, being 2 to 3 times faster than CLIP and SigLIP, without sacrificing downstream performance.
• Vision Encoder Pre-training: In the pre-training stage, each encoder of OpenVision is trained in three consecutive resolution stages. Specifically, different model variants are trained at different resolutions according to their scale, such as the Large, SoViT-400M, and Huge variants being trained at 84×84, 224×224, and finally at 336×336 or 384×384. After pre-training is completed, the text tower and decoder are discarded, retaining only the vision backbone.
• Multimodal Learning Architecture: The model architecture of OpenVision mainly consists of a vision encoder and a text encoder. The vision encoder is responsible for extracting features from images, and the text encoder is used to extract features from natural language sequences. During training, the model uses contrastive learning on image-text pairs to maximize the similarity of positive pairs and minimize the similarity of negative pairs.
• Optimizing Lightweight Systems and Edge Computing Applications: OpenVision effectively combines with small language models to build multimodal models with low parameter counts.

OpenVision Project Address

• Project Website：https://ucsc-vlaa.github.io/OpenVision/
• Github Repository：https://github.com/UCSC-VLAA/OpenVision
• HuggingFace Model Library：https://huggingface.co/collections/UCSC-VLAA/openvision
• arXiv Technical Paper：https://arxiv.org/pdf/2505.04601

Application Scenarios of OpenVision

• Multimodal Learning: OpenVision can be integrated into multimodal frameworks such as LLaVA for tasks like image recognition, video analysis, and natural language processing.
• Industrial Inspection: OpenVision's high-resolution image sensors and powerful processing capabilities are suitable for industrial inspection, such as defect detection, dimensional measurement, etc.
• Robot Vision: By integrating high-performance image sensors and processing chips, OpenVision can provide robots with real-time visual perception capabilities, supporting tasks such as path planning and object recognition.
• Autonomous Driving: In the field of autonomous driving, OpenVision can serve as a vehicle-mounted vision system, processing image data from multiple cameras to perform environmental perception and decision-making.
• Scientific Research and Education: Its open-source nature makes OpenVision an ideal platform for researchers and educational institutions to conduct visual computing research and teaching.