What is Finedefics
Finedefics is a fine-grained multimodal large model launched by Professor Peng Yuxin's team at Peking University, enhancing the performance of Multimodal Large Language Models (MLLMs) in Fine-Grained Visual Recognition (FGVR) tasks. By introducing fine-grained attribute descriptions of objects and aligning the representations of visual objects and category names based on contrastive learning, the model solves the problem in traditional models where visual objects and fine-grained subcategories are not aligned.
Main Features of Finedefics
- Improved Fine-Grained Visual Recognition Capability: Finedefics addresses the misalignment between visual objects and fine-grained subcategories in traditional models by introducing fine-grained attribute descriptions of objects and using contrastive learning to align the representations of visual objects and category names.
- Collaborative Training of Data and Knowledge: The model prompts a large language model to construct fine-grained attribute knowledge of visual objects, aligns this knowledge with images and text, and achieves collaborative training of data and knowledge.
- High Performance: On multiple authoritative fine-grained image classification datasets (such as Stanford Dog-120, Bird-200, FGVC-Aircraft, etc.), Finedefics achieves an average accuracy of 76.84%, a significant improvement compared to other similar models.
- Attribute Description Construction and Alignment: By mining key features that distinguish fine-grained subcategories (such as fur color, fur type, etc.) and converting them into natural language descriptions, Finedefics uses these descriptions as intermediate points to align visual objects with category names in the representation space of a large language model.
Technical Principles of Finedefics
- Attribute Description Construction: Finedefics first constructs attribute descriptions by mining key features that distinguish fine-grained subcategories, such as fur color, fur type, fur texture, etc. These features are extracted as attribute pairs of image objects (e.g., "fur color: brown") and converted into natural language object attribute descriptions (e.g., "the kitten in the picture has brown fur with stripes and a soft texture").
- Attribute-Enhanced Alignment: Through attribute-enhanced alignment, Finedefics uses the constructed object attribute descriptions as a common alignment target for visual objects and fine-grained subcategories. Based on object-attribute, attribute-category, and category-category contrastive learning, the model fully establishes the correspondence between visual objects and fine-grained subcategories.
- Contrastive Learning and Instruction Fine-Tuning: During the training phase, Finedefics employs contrastive learning, inputting the global representations of visual objects, attribute descriptions, and category names into a large language model, and optimizes the alignment effect by introducing hard negative samples. Through instruction fine-tuning, it further improves its performance in fine-grained visual recognition tasks.
Finedefics Project Address
- Github Repository：https://github.com/PKU-ICST-MIPL/Finedefics
- HuggingFace Model Library：https://huggingface.co/StevenHH2000/Finedefics
- arXiv Technical Paper：https://arxiv.org/pdf/2501.15140
Application Scenarios of Finedefics
- Biodiversity Monitoring: Finedefics can be used to automatically identify and classify biological species, for example, distinguishing different kinds of birds, plants, or animals.
- Intelligent Transportation: In the transportation field, Finedefics can be used for vehicle recognition and classification, for example, distinguishing different brands or models of cars (such as BMW, Mercedes-Benz, Audi, etc.), and can further identify different models under the same brand (such as Audi A4, A6, A8).
- Smart Retail: Finedefics can be applied in retail scenarios to help identify and classify products, such as different types of fruits, flowers, or retail products. It can be used for inventory management, product recommendations, and automated checkout systems, improving the operational efficiency of retail enterprises.
- Industrial Inspection and Quality Control: In industrial production, Finedefics can be used to detect and classify fine-grained differences in parts or products, such as identifying different models of mechanical components or detecting product quality issues.