SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

FastVLM - Apple's Efficient Vision Language Model

Author: AI小集 · Original source: AI工具集

What is FastVLM

FastVLM is an efficient Vision Language Model (VLM) introduced by Apple, capable of improving the efficiency and performance of high-resolution image processing. The model introduces the FastViTHD novel hybrid vision encoder, effectively reducing the number of visual tokens and significantly lowering encoding time. While maintaining performance similar to existing VLMs, FastVLM greatly improves processing speed. For example, in the LLaVA-1.5 setup, compared to other models, it shortens the time to first token (TTFT) by 3.2 times. FastVLM performs excellently across various VLM benchmarks, and the model size is smaller, requires less training data, demonstrating efficiency and practicality in multimodal understanding tasks.

FastVLM

FastVLM's Main Features

  • Efficient encoding of high-resolution images: Supports quickly converting high-resolution images into visual tokens, reducing encoding time and the number of tokens.
  • Low-latency response: In the LLaVA-1.5 setup, FastVLM's time to first token (TTFT) is significantly shortened, with fast response speed, suitable for real-time interactive scenarios, such as mobile image-text Q&A assistants, capable of quickly giving answers.
  • Model size optimization: The vision encoder size is greatly reduced, 3.4 times smaller than similar models, making it easy to deploy on mobile devices and edge intelligent devices, lowering hardware requirements and improving the model's portability.
  • Improved VLM performance: While greatly shortening time, time to first token (TTFT), maintains performance comparable to existing advanced models.
  • Simplified model design: No additional token pruning steps are needed, simplifying the design of the vision encoder.

FastVLM's Technical Principles

  • Hybrid vision encoder FastViTHD: FastViTHD is the core component of FastVLM. Compared with traditional pure convolutional encoders or pure Transformer encoders (such as ViT), it combines the advantages of convolutional layers and Transformer blocks. Convolutional layers can effectively process high-resolution images and reduce the number of tokens based on downsampling operations. Transformer blocks can further extract high-quality visual features, providing more accurate visual information for the LLM. The FastViTHD architecture contains multiple stages, each with a specific depth and embedding dimension. For example, its depth is set to [2, 12, 24, 4, 2], and the embedding dimensions are [96, 192, 384, 768, 1536].
  • Optimized architecture design: FastVLM is optimized in its architecture design to achieve efficiency in high-resolution image processing. Unlike simply extending the FastViT architecture, FastVLM introduces an additional stage, and the additional stage performs downsampling before the self-attention layer. The self-attention layer only needs to process tensors that have already been downsampled, reducing computation. For example, in a typical hybrid model, the tensors processed by the self-attention layer are downsampled 16 times in each direction. In FastVLM, the input tensors processed by the widest MLP layer are downsampled 64 times in each direction, significantly reducing visual encoding latency.
  • Collaborative work with the LLM: FastVLM connects the vision encoder with the LLM through a projection layer (also called a connector module). The visual tokens output by the vision encoder are converted through the connector module into a format suitable for LLM processing. The LLM uses visual tokens and text input for fused understanding and generates the corresponding output. This collaborative working method allows visual information to be effectively integrated into the language generation process, realizing the functionality of a vision language model.

FastVLM's Project Address

  • GitHub repository：https://github.com/apple/ml-fastvlm
  • HuggingFace model library：https://huggingface.co/collections/apple/fastvlm-68ac97b9cd5cacefdd04872e
  • arXiv technical paper：https://www.arxiv.org/pdf/2412.13303

FastVLM's Application Scenarios

  • Visual question answering: Quickly understand images and answer related questions.
  • Image-text matching: Determine whether an image and a text description are consistent.
  • Document understanding: Parse the text content in an image and understand its meaning.
  • Image captioning: Automatically generate descriptive text for an image.
  • Multimodal recommendation: Combine image and text information for precise recommendations.