What is Sa2VA
Sa2VA is a multimodal large language model jointly launched by ByteDance, the University of California, Merced, Wuhan University, and Peking University. It is a combination of SAM2 and LLaVA, capable of achieving dense, fine-grained understanding of images and videos. Based on a unified task representation, Sa2VA integrates tasks such as referring segmentation in images or videos, visual dialogue, and visual prompt understanding into a single framework, using spatial-temporal prompts generated by the LLM to guide SAM2 in generating precise segmentation masks. Sa2VA adopts a decoupled design, retaining SAM2's perception capabilities and LLaVA's language understanding capabilities, and introduces the Ref-SAV dataset to improve referring segmentation performance in complex video scenarios.
Main Features of Sa2VA
- Image and Video Referring Segmentation: Precisely segment target objects in images or videos based on natural language descriptions.
- Image and Video Dialogue: Supports dialogue with users based on images or videos, answering questions related to visual content.
- Visual Prompt Understanding: Supports processing visual prompts (such as boxes, points in images, etc.) and, combined with language descriptions, generates corresponding segmentation masks or answers.
- Instruction-Based Video Editing: Edits video content according to user instructions,
- Dense Visual Understanding: Sa2VA can understand the overall content of images and videos, and can also analyze and operate on pixel-level visual details, supporting fine-grained tasks in complex scenarios, such as video object segmentation with long text descriptions.
- Zero-Shot Inference: Supports inference on unseen videos, directly generating segmentation masks or answering questions based on language descriptions, without additional training.
Technical Principles of Sa2VA
- Model Architecture: Combines SAM2 and LLaVA. SAM2 is responsible for spatiotemporal segmentation of videos, while LLaVA provides language understanding and generation capabilities. The two are connected based on a special [SEG] token, with LLaVA's output serving as SAM2's input to guide it in generating segmentation masks.
- Unified Task Representation: Unifies multiple tasks (such as referring segmentation, visual dialogue, visual prompt understanding, etc.) into a single instruction tuning process. All inputs (images, videos, text) are encoded as visual tokens, input into the LLM, and output text or segmentation masks.
- Decoupled Design: Based on a decoupled design, freezes SAM2's decoder and memory module, retaining its perception and tracking capabilities.
- Ref-SAV Dataset: Introduces the Ref-SAV dataset, containing over 72k object expressions in complex video scenarios. The dataset is generated based on an automatic annotation pipeline, containing long text descriptions and complex scenes, improving the model's performance in complex environments.
- Spatiotemporal Prompts: Uses the [SEG] token generated by LLaVA as spatiotemporal prompts for SAM2, guiding the generation of precise segmentation masks.
- Joint Training: Conducts joint training on multiple datasets, including image QA, video QA, image segmentation, and video segmentation data.
Sa2VA Project Address
- Project Official Website：https://lxtgh.github.io/project/sa2va/
- GitHub Repository：https://github.com/magic-research/Sa2VA
- HuggingFace Model Library：https://huggingface.co/ByteDance/Sa2VA
- arXiv Technical Paper：https://arxiv.org/pdf/2501.04001
Application Scenarios of Sa2VA
- Video Editing: Quickly remove or replace objects in videos according to language instructions, improving creative efficiency.
- Intelligent Surveillance: Real-time recognition and tracking of targets in surveillance footage based on language descriptions, assisting security monitoring.
- Robot interaction: Understanding instructions and performing operations, such as "pick up the red cup," enhancing interaction between robots and the environment.
- Content creation: Generating descriptions and question-answering for images or videos, assisting education or creative writing.
- Autonomous driving: Recognizing and segmenting pedestrians, vehicles, etc., in road scenes to assist driving decisions.