SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI developer platformsYuan2.0-M32

Yuan2.0-M32

Yuan2.0-M32 is a large language model using a Mixture-of-Experts (MoE) architecture, with a total parameter count of 40B, activating 2 experts at a time and 3.7B activated parameters. It proposes and adopts a new routing network, Attention Router, to improve the efficiency and accuracy of expert selection, achieving a 3.8% accuracy improvement over models using classic routing networks. The model was trained from scratch on 2000B tokens, and its training compute is only 9.25% of that of a dense model of comparable parameter scale. It is competitive in multiple specialized domains such as coding and mathematics, reaching 55.9% and 95.8% accuracy on the MATH and ARC-Challenge benchmarks respectively, surpassing Llama3-70B. Its forward computation per token is 7.4 GFLOPS, only 1/19 of Llama3-70B. The sequence length supports 16K. The related code and model have been open-sourced on GitHub and released on ModelScope and Hugging Face.

Open website

About this tool

Yuan2.0-M32 is a Mixture-of-Experts (MoE) language model developed and open-sourced by the IEIT-Yuan team.

Uses Yuan2.0-M32 is aimed at tasks such as natural language understanding, knowledge acquisition, mathematical computation and reasoning, and code generation. According to the official repository, it demonstrates competitive capabilities in coding, mathematics, and multiple specialized domains, and can be used for language model inference and related research requiring the above capabilities.

Main capabilities and architecture

  • Model type: Mixture-of-Experts (MoE) language model, with a total of 32 experts, of which 2 are activated at a time.
  • Total parameter count: 40B; activated parameters: 3.7B; pretraining tokens: 2000B; sequence length: 16K.
  • Routing network: proposes and adopts a new Attention Router for more efficient expert selection. Compared with models using classic routing networks, accuracy is improved by 3.8%.
  • Training compute: trained from scratch, and the training compute is only 9.25% of the compute required by a dense model of comparable parameter scale.
  • Forward computation: 7.4 GFLOPS per token, which the official source states is only 1/19 of the requirement of Llama3-70B.
  • Benchmark performance: reaches 55.9% and 95.8% accuracy on the MATH and ARC-Challenge benchmarks respectively, surpassing Llama3-70B. The official source states that on benchmarks such as HumanEval, GSM8K, MMLU, Math, and ARC-Challenge, it shows consistent and significant advantages over models such as Llama3-8B and Mistral-8×7B, with overall performance comparable to the larger Llama3-70B.

Versions and downloads The official repository lists the following versions and release information:

  • 2024-05-28 released Yuan2.0-M32.
  • 2024-06-18 released Yuan2.0-M32-HF-INT4.
  • 2024-06-28 released Yuan2.0-M32-HF-INT8, which the official source states has high performance and lossless generation precision. Model downloads are available via ModelScope and Hugging Face; for quantization and inference methods, refer to the guide provided in the repository.

Usage The official repository provides the following related support:

  • Environment configuration: it is strongly recommended to use the latest released Yuan2.0-M32 Docker image, which can be accessed through the provided Docker commandLaunch the container instance.
  • Data preprocessing: Data preprocessing scripts are provided; see the documentation for details.
  • Model pretraining: Multiple pretraining scripts are provided in the example; see the documentation for details.
  • Inference service: For detailed deployment solutions, refer to vllm; to use Yuan2.0-M32-HF-INT4 inference, refer to "The Method of Quantization and Inference for Yuan2.0-M32".
  • Evaluation: Evaluation scripts for HumanEval, GSM8K, MMLU, Math, and ARC-C are provided to support reproducing the evaluation results.

Applicable scenarios According to the official description, this model can be used for tasks such as coding, mathematics, natural language understanding, knowledge acquisition, and reasoning, and can support commercial use without authorization. It is suitable for research and application scenarios that require a MoE architecture, lower activation parameters and computational overhead, while focusing on mathematics, reasoning, and code generation capabilities.

Limitations and statements

  • Use of the source code in this repository must comply with the Apache 2.0 open source license agreement.
  • The Yuan2.0 model supports commercial use without authorization, but you must understand and comply with the "Yuan2.0 Model License Agreement".
  • The open source model and code, as well as derivatives generated by the open source project, must not be used for any purpose that may cause harm to the country and society, or for services that have not undergone security assessment and filing.
  • The official statement says that although measures have been taken during training to ensure the compliance and accuracy of the data, due to the huge number of model parameters and the influence of probabilistic and random factors, the accuracy of the output content cannot be guaranteed, and the model is easily misled by input instructions. This project assumes no responsibility for any data security, public opinion risks, or model misleading, abuse, dissemination, and other risks and liabilities caused by the open source model and code, as well as improper use. The risks and consequences of using, copying, distributing, and modifying the model of this open source project shall be borne by the user.
  • If you have any questions, you can raise them through an issue or contact air_service@ieisystem.com.
  • Officially, we are recruiting talents in the directions of large model framework development, inference performance optimization, and open source community operations. Resumes can be sent to wushaohua@ieisystem.com, and the email subject format is: [Application of Yuan Team Application] - [Your Name].

Related tools

Browse category