SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

UniPixel - A Pixel-Level Multimodal Large Model Jointly Launched by Hong Kong Polytechnic University and Tencent

Author: AI小集 · Original source: AI工具集

What is UniPixel

UniPixel is the first unified pixel-level multimodal large model developed by the Hong Kong Polytechnic University and Tencent ARC Lab team, focusing on fine-grained understanding and interaction of images/videos. It can accomplish three major tasks in a single model: object referring, pixel-level segmentation, and region reasoning. Through the innovatively designed "object memory mechanism" and unified visual encoding approach, it achieves precise tracking and semantic understanding of targets in videos. The model is based on the Qwen2.5-VL framework and supports three interaction modes: points, boxes, and masks. It surpasses traditional models with 72B parameters on 9 visual task benchmarks, and the code and online Demo have been open-sourced. The core breakthrough lies in the deep integration of visual segmentation and language reasoning, solving the problem that traditional models cannot handle complex referring and dynamic region understanding.

UniPixel

Main Features of UniPixel

  • Pixel-Level Vision-Language Understanding: Focuses on pixel-level vision-language understanding, can achieve pixel-level alignment between visual signals and language semantics, and supports a variety of fine-grained tasks, including image/video segmentation, region understanding, and PixelQA tasks.
  • Unified Object Referring and Segmentation: The model seamlessly integrates object referring and segmentation capabilities, can generate relevant masks based on visual prompt inputs, and perform subsequent reasoning based on these intermediate pointers during the reasoning process, achieving fine-grained pixel-level reasoning.
  • Multi-Task Support: Performs excellently on multiple benchmarks, including the ReVOS reasoning segmentation benchmark, MeViS, Ref-YouTube-VOS, RefCOCO/+/g and other datasets. It designs a new PixelQA task, requiring the model to jointly perform object referring, segmentation, and question answering.
  • Flexible Visual Prompt Processing: Can flexibly process visual prompt inputs, generate masks and perform reasoning, support single-frame and multi-frame video region understanding, as well as mask-based question answering tasks.

Technical Principles of UniPixel

  • Unified Framework Design: UniPixel adopts a unified framework that integrates object referring and segmentation capabilities into one model, achieving a leap from coarse-grained scene understanding to fine-grained pixel reasoning, providing a foundation for complex visual reasoning.
  • Object Memory Bank: The model includes an object memory bank that stores object features extracted from referring tasks, providing contextual information for subsequent segmentation and reasoning tasks, and enhancing the model's performance on pixel-level tasks.
  • Multi-Stage Training Strategy: Adopts a multi-stage training strategy of pre-training, referring task fine-tuning, and segmentation task fine-tuning, gradually improving the model's performance on pixel-level tasks and adapting to different task requirements.
  • End-to-End Mask Generation: The model can directly generate pixel-level masks based on language descriptions, achieving deep integration of language and vision, and supporting a variety of fine-grained tasks such as image/video segmentation and region understanding.
  • Flexible Visual Prompt Processing: Can flexibly process visual prompt inputs, generate masks and perform reasoning, support single-frame and multi-frame video region understanding, as well as mask-based question answering tasks, adapting to different scenario requirements.
  • Powerful Reasoning Ability: In the VideoRefer-Bench-Q question answering task, the UniPixel-7B model achieved 74.1% accuracy, surpassing multiple strong benchmark models including GPT-4o, demonstrating its powerful ability in complex visual reasoning tasks.
  • Model Weights and Datasets Provided: Provides model weights for two versions, UniPixel-3B and UniPixel-7B, as well as the original images/videos and preprocessed annotations for 23 referring/segmentation/QA datasets, offering rich resources for research and applications.
  • Training and Evaluation Support: The codebase supports training and evaluation on 23 datasets and benchmarks, supports flexible hardware settings, efficient training techniques, custom base LLMs and conversation templates, and monitoring of the training process via Tensorboard/Wandb, making it convenient for users to use and optimize.

UniPixel's Project Address

  • Project Official Website：https://polyu-chenlab.github.io/unipixel/
  • Github Repository：https://github.com/PolyU-ChenLab/UniPixel
  • HuggingFace Data：https://huggingface.co/datasets/PolyU-ChenLab/UniPixel-SFT-1M
  • arXiv Technical Paper：https://arxiv.org/pdf/2509.18094
  • Online Experience Demo：https://huggingface.co/spaces/PolyU-ChenLab/UniPixel

UniPixel's Application Scenarios

  • Image Segmentation: UniPixel can generate pixel-level masks for specific objects in images based on language descriptions, making it suitable for scenarios requiring precise image segmentation, such as medical image analysis and object segmentation in autonomous driving.
  • Video Segmentation: In the field of video processing, UniPixel can perform real-time segmentation of objects in videos, supporting applications such as video editing, video surveillance, and augmented reality.
  • Region Understanding: By understanding language descriptions to identify and segment specific regions in videos, it can be used for video content analysis, intelligent surveillance systems, and background segmentation in video conferencing.
  • Question Answering Tasks: UniPixel supports the PixelQA task and can answer questions based on language descriptions and visual information, making it suitable for scenarios such as education, intelligent customer service, and information retrieval.
  • Multimodal Interaction: In scenarios that require interaction combining visual and language information, such as intelligent assistants, virtual reality, and game development, UniPixel can provide a more natural and accurate interactive experience.
  • Intelligent Surveillance: In the field of security surveillance, UniPixel can identify and segment specific objects or regions in surveillance videos in real time, improving the intelligence level of surveillance systems.