SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

FG-CLIP 2 - 360's Open-Source Bilingual Fine-Grained Vision-Language Alignment Model

Author: AI小集 · Original source: AI工具集

What is FG-CLIP 2

FG-CLIP 2 is an open-source bilingual fine-grained vision-language alignment model launched by 360, designed specifically to solve the problem of precise alignment between vision and language. It has made major breakthroughs in the field of vision-language understanding, especially excelling in Chinese-English bilingual tasks. The model adopts a hierarchical alignment architecture, progressively enhancing the model's ability to understand image details through global semantic alignment and fine-grained vision-language learning. It introduces a dynamic attention mechanism that can intelligently focus on key regions of an image, better handling complex vision-language tasks. FG-CLIP 2 surpasses existing top models such as Google's SigLIP 2 and Meta's MetaCLIP 2 on multiple authoritative benchmarks, becoming one of the world's strongest vision-language models.

FG-CLIP 2

Main Features of FG-CLIP 2

  • Fine-Grained Vision-Language Understanding: It can precisely understand details in images, including object attributes, spatial relationships, etc., solving the shortcomings of traditional models in fine-grained recognition.
  • Bilingual Support: The model performs excellently on both Chinese and English tasks, achieving true native bilingual support.
  • Hierarchical Alignment Architecture: It adopts a hierarchical alignment architecture that simultaneously captures macro scenes and micro details, enhancing the model's ability to understand image details.
  • Dynamic Attention Mechanism: It features a dynamic attention mechanism that can intelligently focus on key regions of an image, better handling complex vision-language tasks.
  • Optimized Bilingual Collaboration Strategy: It addresses the imbalance between Chinese and English understanding, improving the model's overall performance in bilingual tasks.
  • Powerful Performance: Across 29 authoritative public benchmarks, it comprehensively surpasses Google's SigLIP 2 and Meta's MetaCLIP2, becoming the world's strongest vision-language model.
  • High-Concurrency Response Speed: It continues to use an explicit dual-tower structure, allowing image and text features to be precomputed and cached, ensuring millisecond-level response speed in high-concurrency scenarios.
  • Adaptive Input Size: The dynamic resolution mechanism allows the model to adaptively process inputs of different sizes, improving the model's flexibility and adaptability.
  • Rich Open-Source Resources: It provides code, model weights, and detailed training datasets, offering great convenience to researchers and developers.

Technical Principles of FG-CLIP 2

  • Hierarchical Alignment Architecture: Through global semantic alignment and fine-grained vision-language learning, it progressively enhances the model's ability to understand image details.
  • Dynamic Attention Mechanism: It intelligently focuses on key regions of an image, better handling complex vision-language tasks.
  • Bilingual Collaboration Strategy: It optimizes the balance between Chinese and English understanding, improving overall performance in bilingual tasks.
  • Multimodal Data Training: It uses large-scale Chinese-English image-text pairs for training, enhancing the model's bilingual generalization ability.
  • Fine-Grained Supervised Learning: It introduces supervisory signals such as region-text matching and long-description modeling to improve fine-grained vision-language understanding.
  • Intra-Text Modal Contrast: Through intra-text modal contrastive loss, it better distinguishes descriptions with similar semantics.
  • Hard Negative Sample Training: Introduce "hard negative samples" generated by large models to further improve model performance.
  • Dynamic resolution mechanism: Adaptively process inputs of different sizes, improving the model's flexibility and adaptability.

FG-CLIP 2 project address

  • Project website：https://360cvgroup.github.io/FG-CLIP/
  • Github repository：https://github.com/360CVGroup/FG-CLIP
  • arXiv technical paper：https://arxiv.org/pdf/2510.10921

Application scenarios of FG-CLIP 2

  • Home robots: Can accurately understand and execute complex household instructions, such as "pick up the phone with a cracked screen on the coffee table", improving the robot's practicality in home environments.
  • Security monitoring: Quickly locate and identify targets, such as "find suspicious persons wearing black baseball caps", improving the efficiency and accuracy of security systems.
  • E-commerce: Accurately understand product descriptions, improve the precision of "text-to-image search", reduce multilingual annotation and adaptation costs, and optimize user experience.
  • Autonomous driving: Accurately identify objects and scenes in the road environment, such as "identify whether there are obstacles in the lane ahead", improving the safety of autonomous driving systems.
  • Medical imaging: Assist doctors in image diagnosis, such as "identify abnormal areas in X-ray images", improving the accuracy and efficiency of diagnosis.
  • Education: Used in intelligent educational tools, such as "identify objects in images and provide related knowledge", enriching teaching content and formats.