SenseNova-Vision - SenseTime's Open-Source Unified Vision Large Model

What is SenseNova-Vision

SenseNova-Vision is SenseTime's open-source unified vision large model for understanding and generation, natively integrating classic vision tasks such as object detection, image segmentation, depth prediction, and 3D reconstruction into the large model foundation. The model is based on a unified multimodal generation framework, requires no task-specific heads, and allows tasks to be defined through natural language instructions, outputting textual symbolic records, dense image predictions, or mixed formats. SenseNova-Vision has zero-shot generalization capability and can handle extreme scenarios such as cross-domain game visuals and mirror reflections, reaching the level of specialized models across four major task families: structured perception, dense geometry, segmentation, and multi-view 3D.

Main Features of SenseNova-Vision

• Structured perception: A single model supports symbolic recording tasks such as object detection, OCR, keypoint detection, and GUI localization, with outputs as coordinates and labels in text format.
• Image segmentation: Covers semantic, instance, panoptic, interactive, reasoning, and GCG segmentation, supporting precise individual separation in ultra-dense overlapping scenes.
• Dense geometry: Natively outputs pixel-aligned spatial predictions such as depth maps and surface normal maps, represented using conditional image generation.
• Multi-view 3D: Achieves 3D reconstruction and camera pose estimation based on multi-image input, supporting point cloud and pose parameter outputs.

Technical Principles of SenseNova-Vision

• Unified generation framework: Based on the UMM architecture, the text and image output spaces are natively unified, requiring no task-specific heads or architectural changes.
• Zero-shot generalization: Has cross-domain adaptability to extreme scenarios such as out-of-training-set game visuals and mirror reflections, with linguistic reasoning and visual capabilities intertwined.
• Language programmable: Natural language instructions can define new visual tasks, supporting combinatorial conditions such as category, color, and region, enabling "visual mind reading."
• Data feedback: Constructs the SenseNova-Vision Corpus (50 million examples), uniformly converting heterogeneous visual annotations into instruction-response pairs.

Follow WeChat and reply "Open Source" to join theAI Open Source Project Exchange Group

How to Use SenseNova-Vision

• Environment preparation: Clone the GitHub repository and configure the Python, CUDA environment, and pretrained model weights.
• Model loading: Load the model based on the Bagel UMM architecture, prepare input images, and write task instructions in natural language.
• Task execution: Invoke model inference and generate responses in the native text, image, or mixed-format space according to the instructions.
• Result processing: Parse text coordinate labels or decode image outputs into depth maps, normal vectors, and segmentation masks, and customize tasks by modifying natural language instructions.

Core Advantages of SenseNova-Vision

• Native unified architecture: Natively integrates classic vision tasks such as detection, segmentation, depth, and 3D reconstruction into the large model foundation, requiring no task-specific heads or architectural changes, breaking the fragmented model of "packaging and encapsulating expert models."
• Bidirectional capability gains: Decades of high-quality data in the vision field feed back into the foundation model's understanding capabilities, and the reasoning ability of large language models in turn empowers vision tasks to achieve mastery, even supporting the direct definition of new vision tasks using language.
• Zero-shot generalization: Possesses strong cross-domain adaptability to scenarios outside the training set, and can simultaneously output normal vectors, segmentation, and keypoints without targeted retraining.
• Ultra-dense segmentation: When faced with highly overlapping, similarly colored dense objects, it can precisely separate each individual object like surgery, solving the problem of traditional models being "clueless."

SenseNova-Vision project address

• GitHub repository：https://github.com/OpenSenseNova/SenseNova-Vision
• HuggingFace model library：https://huggingface.co/collections/sensenova/sensenova-vision
• arXiv technical paper：https://arxiv.org/pdf/2607.06560

Comparison of SenseNova-Vision with similar competitors

Application scenarios of SenseNova-Vision

• Digital content creation: Zero-shot parsing of unknown game visuals, simultaneously outputting normal vectors, instance segmentation, and character keypoints, directly embeddable into film, television, and game workflows.
• Industrial warehousing quality inspection: Ultra-dense segmentation capability precisely separates individuals such as overlapping fish schools and densely packed shelf products, providing new solutions for industrial counting and smart warehousing.
• Autonomous driving and robotics: Filters out mirror reflection interference and restores true spatial geometry, improving the reliability of indoor navigation, AR mapping, and autonomous driving environment understanding.
• Scientific research and open-source ecosystem: The model and the 50 million example dataset are fully open-sourced, supporting academic reproduction, secondary development, and cross-domain vision task research.