Gemma

What is Gemma

Gemma is a series of lightweight, advanced open AI models developed by Google DeepMind and other teams at Google, based on the same technology as the Gemini models, designed to help developers and researchers build responsible AI applications. The Gemma model series includes models at two weight scales: Gemma 2B and Gemma 7B, offering pretrained and instruction-tuned versions, and supporting multiple frameworks such as JAX, PyTorch, and TensorFlow to run efficiently on different devices. On June 28, the second-generation modelGemma 2has been released.

Gemma's official entry points

• Gemma's official homepage:https://ai.google.dev/gemma?hl=zh-cn
• Gemma's Hugging Face model:https://huggingface.co/models?search=google/gemma
• Gemma's Kaggle model address:https://www.kaggle.com/models/google/gemma/code/
• Gemma's technical report:https://storage.googleapis.com/deepmind-media/gemma/gemma-report.pdf
• Official PyTorch implementation GitHub repository:https://github.com/google/gemma_pytorch
• Gemma's Google Colab run address:https://colab.research.google.com/github/google/generative-ai-docs/blob/main/site/en/gemma/docs/lora_tuning.ipynb

Gemma's main features

• Lightweight architecture: The Gemma model is designed to be lightweight, making it easy to run in a variety of computing environments, including personal computers and workstations.
• Open model: The weights of the Gemma model are open, allowing users to use and distribute it commercially while complying with the license agreement.
• Pretraining and instruction tuning: It provides pretrained models and instruction-tuned versions, with the latter using reinforcement learning from human feedback (RLHF) to ensure responsible model behavior.
• Multi-framework support: Gemma supports major AI frameworks such as JAX, PyTorch, and TensorFlow, and provides a toolchain through Keras 3.0, simplifying the inference and supervised fine-tuning (SFT) process.
• Safety and reliability: At design time, Gemma follows Google's AI principles, uses automated technology to filter sensitive information in training data, and conducts a series of safety evaluations, including red teaming and adversarial testing.
• Performance optimization: The Gemma model is optimized for hardware platforms such as NVIDIA GPUs and Google Cloud TPUs, ensuring high performance on different devices.
• Community support: Google provides free resources on platforms such as Kaggle and Colab, as well as Google Cloud credits, encouraging developers and researchers to use Gemma for innovation and research.
• Cross-platform compatibility: The Gemma model can run on a variety of devices, including laptops, desktops, IoT devices, and the cloud, supporting a wide range of AI functions.
• Responsible AI toolkit: Google has also released the Responsible Generative AI Toolkit to help developers build safe and responsible AI applications, including safety classifiers, debugging tools, and application guidelines.

Gemma's technical points

• Model architecture: Gemma is built on the Transformer decoder, which is one of the most advanced model architectures in the current field of natural language processing (NLP). It adopts a multi-head attention mechanism, allowing the model to focus on multiple parts simultaneously when processing text. In addition, Gemma also uses rotary position embeddings (RoPE) instead of absolute position embeddings to reduce model size and improve efficiency. The GeGLU activation function replaces the standard ReLU nonlinear activation, and normalization is applied to the input and output of each Transformer sublayer.
• Training infrastructure: The Gemma model is trained on Google's TPUv5e, a high-performance computing platform designed specifically for machine learning. By sharding the model and replicating data across multiple Pods (chip clusters), Gemma can efficiently utilize distributed computing resources.
• Pretraining Data: The Gemma model is pretrained on a large amount of English data (the 2B model is pretrained on approximately 2 trillion tokens of data, while the 7B model is based on 6 trillion tokens). This data mainly comes from web documents, mathematics, and code. The pretraining data is filtered to reduce unwanted or unsafe content while ensuring the diversity and quality of the data.
• Fine-Tuning Strategy: The Gemma model is fine-tuned through supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). This includes using synthetic text pairs and human-generated prompt-response pairs, as well as a reward model trained on human preference data.
• Safety and Responsibility: Gemma was designed with model safety and responsibility in mind, including filtering the data during the pretraining stage to reduce the risk of sensitive information and harmful content. In addition, Gemma has passed a series of safety evaluations, including automated benchmarks and human evaluations, to ensure the model's safety in practical applications.
• Performance Evaluation: Gemma has undergone extensive performance evaluations in multiple domains, including question answering, commonsense reasoning, mathematics and science problem solving, and coding tasks. The Gemma model was compared in performance with open models of the same or larger scale, and among 18 benchmarks such as MMLU and MBPP, 11 test results surpassed models such as Llama-13B or Mistral-7B.
• Openness and Accessibility: The Gemma model is released in open source form, providing pretrained and fine-tuned checkpoints, as well as open source code libraries for inference and deployment. This enables researchers and developers to access and utilize these advanced language models, promoting innovation in the AI field.

Frequently Asked Questions