SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

AIMv2 - Apple's Open-Source Multimodal Autoregressive Pretrained Vision Model

Author: AI小集 · Original source: AI工具集

What is AIMv2

AIMv2 is Apple's open-source multimodal autoregressive pretrained vision model, which enhances the performance of vision models through deep fusion of images and text. It adopts an innovative pretraining framework that divides images into non-overlapping patches, decomposes text into subword tokens, and then concatenates the two into a unified sequence for autoregressive pretraining. This simplifies the training process and enhances the model's ability to understand multimodal data. AIMv2 offers versions with various parameter scales (such as 300M, 600M, 1.2B, and 2.7B), suitable for different devices from mobile phones to PCs. In terms of performance, AIMv2 performs excellently in both multimodal tasks and traditional vision tasks.

AIMv2

Main Features of AIMv2

  • Visual Question Answering (VQA): AIMv2 extracts visual features and combines them with the question text, passing them to a large language model (LLM) to generate accurate and contextually appropriate answers.
  • Referring Expression Comprehension: In benchmarks such as RefCOCO and RefCOCO+, AIMv2 can accurately map natural language descriptions to visual regions.
  • Image Captioning: Combined with an LLM, AIMv2 can generate high-quality image descriptions.
  • Multimedia Retrieval:AIMv2's multimodal representation capability can efficiently handle multimedia retrieval tasks, supporting joint retrieval of images and text.
  • Integration with Large Language Models (LLMs):AIMv2's architecture is highly compatible with LLM-driven multimodal applications and can be seamlessly integrated into various multimodal systems.
  • Zero-Shot Adaptability: AIMv2 supports zero-shot recognition adaptability, allowing it to adapt to new visual tasks without additional training.

Technical Principles of AIMv2

  • Multimodal Autoregressive Pretraining Framework: AIMv2 divides images into non-overlapping patches, decomposes text into subword tokens, and then concatenates the two into a multimodal sequence. During the pretraining phase, the model autoregressively predicts the next element in the sequence, whether it is an image patch or a text token. This design enables the model to simultaneously learn the associations between visual and language modalities.
  • Vision Encoder and Multimodal Decoder: AIMv2's architecture consists of a vision encoder and a multimodal decoder. The vision encoder is based on the Vision Transformer (ViT) architecture and is responsible for processing image patches. The multimodal decoder uses a causal self-attention mechanism to predict the next element based on the preceding context.
  • Loss Function Design: AIMv2 defines separate loss functions for the image and text domains. The text loss uses the standard cross-entropy loss, while the image loss uses a pixel-level regression loss to compare the predicted image patches with the ground-truth image patches. The overall objective is to minimize the weighted sum of the text loss and image loss to balance the model's performance on both modalities.
  • Training Data and Scalability: AIMv2 was pretrained using a large number of image-text paired datasets, including the public DFN-2B and COYO datasets. The training process is simple and efficient, without requiring excessively large batch sizes or special cross-batch communication methods. AIMv2's performance improves with increasing data volume and model scale, demonstrating good scalability.
  • Post-Pretraining Optimization Strategies: AIMV2 explores various post-training strategies, such as high-resolution adaptation and native-resolution fine-tuning. These strategies enable the model to better handle images of different resolutions and aspect ratios, further improving its performance on downstream tasks.

AIMv2's project address

AIMv2's application scenarios

  • Image recognition: AIMV2 can serve as a frozen feature extractor for multiple image recognition benchmarks.
  • Object detection and instance segmentation: AIMV2 can be integrated as a backbone network into object detection models (such as Mask R-CNN) for object detection and instance segmentation tasks.
  • Open-vocabulary object detection: AIMV2 performs excellently in open-vocabulary object detection tasks, being able to recognize and locate unseen categories, demonstrating strong generalization ability.