OpenELM - Apple's Open-Source Efficient Language Model Series

What is OpenELM

OpenELM is the latest series of efficient open-source language models released by Apple, including OpenELM-270M, OpenELM-450M, OpenELM-1_1B, and OpenELM-3B versions with different parameter scales (divided into pretrained versions and instruction-tuned versions). This large model uses a layer-wise scaling strategy to non-uniformly allocate parameters in each layer of the Transformer model, thereby improving the model's accuracy and efficiency. The model has been pretrained on public datasets and has demonstrated excellent performance on multiple natural language processing tasks. OpenELM's code, pretrained model weights, and training and evaluation processes are all open, aiming to promote open research and further community development.

Basic Information about OpenELM

• Parameter ScaleOpenELM has a total of eight models, four of which are pretrained and four are instruction-tuned, covering different parameter scales ranging from 270 million to 3 billion parameters (270M, 450M, 1.1B, and 3B).
• Technical ArchitectureOpenELM adopts a Transformer-based architecture and uses a layer-wise scaling strategy, achieving non-uniform parameter allocation by adjusting the number of attention heads and the multiplier of the feedforward network (FFN). The model uses Grouped Query Attention (GQA) instead of Multi-Head Attention (MHA), uses the SwiGLU activation function instead of the traditional ReLU, and uses RMSNorm as the normalization layer.
• Pretraining DataOpenELM uses multiple public datasets for pretraining, including RefinedWeb, deduplicated PILE, a subset of RedPajama, and a subset of Dolma v1.6, totaling approximately 1.8 trillion tokens.
• Open-Source LicenseOpenELM's code, pretrained model weights, and training guidelines are all released under an open-source license. In addition, Apple has also released code to convert the model to the MLX library, thereby enabling inference and fine-tuning on Apple devices.

OpenELM Official Website Entry

• arXiv Research Paper:https://arxiv.org/abs/2404.14619
• GitHub Model Weights and Training Configuration:https://github.com/apple/corenet
• Instruction-Tuned Model Hugging Face Address:https://huggingface.co/collections/apple/openelm-instruct-models-6619ad295d7ae9f868b759ca
• Pretrained Model Hugging Face Address:https://huggingface.co/collections/apple/openelm-pretrained-models-6619ac6ca12a10bd0d0df89e

OpenELM's Technical Architecture

• Transformer ArchitectureOpenELM adopts a decoder-only Transformer model architecture, which is a widely used architecture in natural language processing and is especially suitable for processing sequence data.
• Layer-wise ScalingOpenELM effectively allocates parameters in each layer of the model through layer-wise scaling. This means that the model's early layers (close to the input end) use smaller attention dimensions and feedforward network dimensions, while the layers close to the output end gradually increase the size of these dimensions.
• Grouped Query Attention (GQA)OpenELM uses GQA instead of the traditional Multi-Head Attention (MHA). GQA is a variant of the attention mechanism designed to improve the model's ability to handle long-distance dependencies.
• RMSNorm NormalizationOpenELM uses RMSNorm as its normalization layer, a technique that helps stabilize the training process.
• SwiGLU Activation FunctionIn the feedforward network (FFN), OpenELM uses the SwiGLU activation function, a gated activation function that helps the model capture complex patterns.
• RoPE Position Encoding: To encode positional information, OpenELM uses Rotary Positional Embedding (RoPE), an encoding method that can handle the order of elements in a sequence.
• Flash Attention: When computing scaled dot-product attention, OpenELM uses Flash Attention, a fast and memory-efficient attention computation method.

OpenELM's Performance

The researchers compared OpenELM with models such as PyThia, Cerebras-GPT, TinyLlama, OpenLM, MobiLlama, and OLMo. At similar model sizes, OpenELM demonstrated higher accuracy on most of the mainstream task tests such as ARC, BoolQ, HellaSwag, PIQA, SciQ, and WinoGrande. In particular, compared withOLMomodels, OpenELM still achieves higher accuracy with fewer parameters and less pre-training data.