What is LLMDet
LLMDet is an open-vocabulary object detector launched by Alibaba Group's Tongyi Lab, the School of Computer Science and Engineering at Sun Yat-sen University, Peng Cheng Laboratory, and other institutions. It is based on collaborative training with large language models (LLMs) to improve object detection performance. LLMDet can collect a dataset containing images, grounding labels, and detailed image-level descriptions (GroundingCap-1M), use the long descriptions generated by the LLM to enrich visual features, and train based on standard grounding loss and description generation loss. LLMDet has achieved excellent zero-shot detection performance on multiple benchmarks. As a powerful visual foundation model, it can further build more powerful multimodal models and achieve mutually beneficial win-win outcomes with LLMs.
Main features of LLMDet
- Open-vocabulary detection: LLMDet can detect objects of arbitrary categories not seen during the training stage, enabling recognition of new categories based on the alignment between text labels and visual features.
- Zero-shot transfer capability: Without annotations for the target categories, it can directly transfer to new datasets for detection, demonstrating strong generalization ability.
- Image understanding and description generation: LLMDet can generate detailed image-level descriptions (captions), including rich detail information such as object types, textures, colors, actions, etc., helping the model better understand image content.
- Improving multimodal model performance: As a visual foundation model, it combines with large language models (LLMs) to build more powerful multimodal models and improve model performance in tasks such as visual question answering and image captioning.
Technical principles of LLMDet
- Dataset construction: It uses the GroundingCap-1M dataset, in which each image is paired with grounding labels and detailed image-level descriptions. The descriptions contain rich details, helping the model better understand the objects in the image and their relationships.
- Model architecture: It consists of a standard open-vocabulary object detector and an LLM. The detector is responsible for extracting image features and localizing objects, while the LLM uses these features to generate detailed image-level descriptions and region-level phrases.
- Collaborative training: LLMDet achieves collaborative optimization with the LLM based on two-stage training. First, a projector is trained to map the detector's features to the LLM's input space. Then the detector, projector, and LLM are fine-tuned as a whole, and the training objectives include standard grounding loss and description generation loss.
- Multi-task learning: LLMDet introduces image-level and region-level description generation tasks. Based on generating detailed descriptions, it enriches visual features and improves the model's overall understanding of images. The multi-task learning approach improves detection performance and enhances the model's open-vocabulary capability.
Project address of LLMDet
- GitHub repository：https://github.com/iSEE-Laboratory/LLMDet
- arXiv technical paper：https://arxiv.org/pdf/2501.18954
Application scenarios of LLMDet
- Intelligent security: Real-time detection of abnormal objects or behaviors in camera feeds, with strong adaptability and no need for retraining.
- Autonomous driving: Helps vehicles recognize various obstacles on the road and unseen scenarios, improving safety and reliability.
- Image content moderation: Automatically reviews image content, identifies violating or inappropriate content, and improves review efficiency.
- Intelligent album management: Automatically classify and annotate photos, making it easy for users to search and manage them, with support for a variety of unseen categories.
- Medical image analysis: Analyze medical images to quickly identify abnormal areas without requiring a large amount of annotated data.