SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Gemma 3n - Google's On-Device Multimodal AI Model

Author: AI小集 · Original source: AI工具集

What is Gemma 3n

Gemma 3n is an on-device multimodal AI model unveiled at Google I/O developer conference. Built on the Gemini Nano architecture, it employs per-layer embeddings technology to compress memory usage to the level of 2-4B parameter models. The model has 5B and 8B parameter versions, with memory usage equivalent to only 2B and 4B models. Gemma 3n supports text, image, short video, and audio inputs, and can generate structured text outputs. The newly added audio processing capabilities enable real-time speech transcription, background sound recognition, or audio sentiment analysis. It can be used directly in the browser through Google AI Studio.

Gemma 3n

Main Features of Gemma 3n

  • Optimized On-Device Performance and Efficiency: Gemma 3n delivers approximately 1.5x faster response on mobile devices compared to Gemma 3 4B, achieving smaller memory usage through technologies such as Per-Layer Embeddings (PLE), KVC sharing, and advanced activation quantization.
  • Multimodal Input Support: Supports understanding and processing of text, images, audio, and video, capable of handling complex multimodal interactions.
  • Privacy-First and Offline Availability: All computations are done locally on the device, no internet connection required, data never leaves the device, ensuring user privacy.
  • Flexible Model Architecture: Based on MatFormer architecture, with an embedded advanced 2B submodel, allowing dynamic switching between performance and quality as needed.
  • Multilingual Support: Supports over 140 languages, with excellent performance in languages such as Japanese, German, Korean, Spanish, and French.
  • 32K Token Context Window: Can process inputs up to 32,000 words or symbols, suitable for handling long texts.

Technical Principles of Gemma 3n

  • Based on Gemini Nano Architecture: Gemma 3n inherits the lightweight architecture of Gemini Nano, optimized for mobile devices. Through knowledge distillation and quantization-aware training (QAT), it significantly reduces resource requirements while maintaining high performance.
  • Per-Layer Embeddings Technology: Employs Per-Layer Embeddings (PLE) technology, significantly reducing the model's memory requirements. The original parameter counts are 5B and 8B respectively, with memory usage equivalent to only 2B and 4B models, requiring only 2GB or 3GB of dynamic memory to run.
  • Multimodal Fusion: Combines Gemini 2.0's tokenizer and enhanced data mixing, supporting text and visual processing in over 140 languages, covering global user needs.
  • Local/Global Layer Interleaving Design: Adopts a 5:1 local/global layer interleaving mechanism, with 1 global layer following every 5 local layers, starting computation with a local layer as the first layer of the model. Helps reduce KV cache explosion issues with long contexts.

Gemma 3n Project Address

Application Scenarios of Gemma 3n

  • Speech Transcription and Sentiment Analysis: Can transcribe speech in real-time, recognize background sounds, or analyze audio sentiment, suitable for voice assistants and accessibility applications.
  • Content Generation: Supports generating image descriptions, video summaries, or speech transcriptions on mobile phones, suitable for content creators to quickly edit short videos or social media materials.
  • Academic Task Customization: Developers can use Gemma 3n's fine-tuning capabilities to customize models for academic tasks on Colab, such as analyzing experimental images or transcribing lecture audio.
  • Low-Resource Devices: Designed specifically for low-resource devices, it only requires 2GB RAM to run smoothly on phones, tablets, and laptops.