SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Arctic - Snowflake's Open-Source Enterprise-Grade AI Large Model

Author: AI小集 · Original source: AI工具集

What is Arctic

Arctic is an efficient, open-source enterprise-grade large language model developed by the AI research team of cloud computing giant Snowflake. This large model is a Mixture of Experts (MoE) model with 48 billion (480B) parameters, composed of 128 fine-grained experts. It has significant advantages in cost-effectiveness, training efficiency, and inference efficiency, and is particularly suitable for enterprise tasks such as SQL generation, programming, and instruction following. Arctic is released under the Apache 2.0 license, providing open access to model weights and code, and has open-sourced datasets and research insights.

Snowflake Arctic

Main Features of Arctic

  • Parameter Scale: Arctic has 48 billion parameters, but only activates 17 billion parameters during inference to improve efficiency.
  • Technical Architecture: It adopts a hybrid architecture, combining a Dense Transformer and an MoE model with 128 experts, each with 366 million parameters.
  • Context Window: The model is trained with a 4K attention context window and is planned to be extended to 32K to support longer sequence processing.
  • Training Cost: Arctic's training compute budget is approximately under 2 million USD (i.e., less than 3,000 GPU weeks) to achieve top-tier large model capabilities.
  • Model License: Arctic is released under the open Apache 2.0 license, allowing free use and modification.
  • Main Uses: Designed specifically for enterprises, it excels at tasks such as SQL generation, programming, and instruction following, making it suitable for creating customized enterprise models.
Arctic's training efficiency

Arctic's Model Architecture

  • Architecture Composition: Arctic uses a Dense-MoE Hybrid Transformer architecture, which combines the characteristics of a Dense Transformer and a Mixture of Experts (MoE) model.
  • Dense Transformer Scale: The model includes a dense transformer model with 10B (10 billion) parameters.
  • MoE Architecture Scale: In the MoE part, Arctic designs 128 fine-grained experts, each with 3.66B (3.66 billion) parameters. Therefore, the total parameter count of the MoE part is 128×3.66B, i.e., approximately 470B (470 billion) parameters.
  • Total Parameter Count: Combining the dense transformer and the MoE part, Arctic's total parameter count reaches 480B (480 billion) parameters.
  • Active Parameter Selection: During inference, Arctic uses a top-2 gating mechanism to select the two most suitable experts from the 128 experts, which means that at any given time, approximately 17B (17 billion) parameters are active.
  • Communication and Computation Overlap: To improve training efficiency, Arctic's architecture design allows the communication overhead between experts to overlap with computation tasks during training, thereby hiding the communication overhead.
  • Inference Efficiency: In small-batch (e.g., batch size 1) interactive inference, the Arctic model has 4 times fewer memory reads than Code-Llama 70B and 2.5 times fewer than Mixtral 8x22B, which helps improve inference performance.
  • System Optimization: Arctic collaborates with NVIDIA, utilizing technologies from the TensorRT-LLM and vLLM teams to provide a preliminary inference implementation for Arctic, and through FP8 quantization technology, Arctic can run on a single GPU node.
  • Attention Context Window: The attention context window used during Arctic training is 4K. The team is developing a sliding window implementation based on attention-sinks to support unlimited sequence generation capability, with plans to expand to a 32K attention window in the future.

Arctic's performance

Snowflake compared Arctic's benchmark tests with models such as DBRX, Llama, and Mixtral on enterprise metrics and academic metrics. Arctic performs better on enterprise intelligence metrics. Its performance on general knowledge benchmarks such as MMLU may be slightly lower than some of the latest models, but it remains competitive on these benchmarks.

Snowflake benchmark scores

Enterprise metrics: Compared with all other open-source models, Arctic demonstrates top-tier performance on these tasks, including:

  • Coding: Coding ability is benchmarked through HumanEval+ and MBPP+.
  • SQL Generation: SQL query generation ability is benchmarked using Spider.
  • Instruction Following: IFEval benchmarks the ability to follow complex instructions.

How to use Arctic

Snowflake Arctic is now available from platforms such as Hugging Face and Replicate, and will subsequently become available on platforms including Snowflake Cortex, Amazon Web Services (AWS), Microsoft Azure, NVIDIA API Catalog, Lamini, Perplexity, and Together.