Seg-Zero is the official implementation of the paper "Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement", and this repository is also the official training code for "VisionReasoner: Unified Visual Perception and Reasoning via Reinforcement Learning".
In terms of purpose, Seg-Zero focuses on the task of reasoning-chain-guided image segmentation. Its core feature is to let the model first generate a reasoning chain before generating the final segmentation mask, thereby demonstrating emergent reasoning capability at test time. The model is trained entirely using reinforcement learning, without explicit supervised reasoning data. Compared with supervised fine-tuning, Seg-Zero achieves better performance on both in-domain and out-of-domain data.
Main capabilities include: support for reasoning chain generation and outputting the thought process; adoption of a decoupled architecture, comprising a reasoning model and a segmentation model, and design of a complex reward mechanism that fuses format rewards and accuracy rewards; based on EasyR1 and veRL, support for model splitting during sampling, which saves more GPU memory; support for the Qwen2-VL and Qwen2.5-VL series of models; implementation of commonly used rewards in object detection and segmentation, such as IoU reward and L1 reward. VisionReasoner was subsequently released, supporting multiple targets and multiple tasks.
In terms of usage, users can download pre-trained models from Hugging Face (such as Seg-Zero-7B, VisionReasoner-7B). If there are problems connecting to Hugging Face, they can set export HF_ENDPOINT=https://hf-mirror.com。然后运行推理脚本，默认问题为“What can I have if I'm thirsty?", the command line will output the thinking process, and the mask will be saved in the inference_scripts folder. Users can also customize image_path and text. During evaluation, the --batch_size in the bash script needs to be adjusted according to the GPU, and the command line will display gIoU. For training, it is recommended to use a 4x80G GPU server or an 8x46G GPU server for the 7B model. The training data includes MultiObject-1K and MultiObject-7K, which can be downloaded via scripts. If VRAM is low, you can adjust the image size and recompute the corresponding bbox or point coordinates, while also changing the resize_size in evaluation and inference. The training script can launch GRPO training, and hyperparameters can be adjusted according to VRAM. In addition, it also supports merging checkpoints into Hugging Face format, as well as building custom training data.
Applicable scenarios include visual understanding and reasoning research that requires interpretable segmentation reasoning, as well as multi-object, multi-task visual perception and reasoning experiments.
In terms of limitations: the project notes that some operations are risky, and you need to ensure you have made a backup before modifying the code; training has GPU VRAM requirements, and lower VRAM requires adjusting the configuration; Seg-Zero's best results are evaluated using different checkpoints on different benchmarks, while VisionReasoner is evaluated within a single checkpoint, so users are advised to evaluate all benchmarks on the same model for comparison. Other details such as charges, pricing, and specific platform support are not yet provided.
