What is Ling-V2
Ling-V2 is a family of large language models based on the MoE architecture launched by the Ant Ling team. The first version, Ling-mini-2.0, has 16 billion total parameters, activating only 1.4 billion parameters per input token. The model is trained on 20 trillion high-quality data tokens and enhanced through multi-stage supervised fine-tuning and reinforcement learning, demonstrating outstanding performance in complex reasoning and instruction following. Ling-mini-2.0 uses a MoE architecture with a 1/32 activation ratio, achieving a 7x equivalent dense performance leverage, with fast generation speed and high training and inference efficiency. It has open-sourced an FP8 efficient training solution and provides multiple pre-training checkpoints, supporting continuous training, making it an ideal starting point for MoE research.
Main Features of Ling-V2
-
Powerful Reasoning Ability: It performs excellently in coding, mathematics, and cross-domain knowledge-intensive reasoning tasks, demonstrating outstanding reasoning ability compared to dense models under 1 billion parameters and larger-scale MoE models.
-
High Efficiency: It adopts a MoE architecture with a 1/32 activation ratio, achieving a 7x equivalent dense performance leverage. Activating 1.4 billion parameters can provide performance equivalent to a 7-8 billion dense model. In simple question-answering scenarios, the generation speed can reach 300+ token/s, and when processing a 128K context length, the relative speed improvement can reach more than 7x.
-
Efficient Training Solution: FP8 mixed-precision training is used throughout the entire training process. The FP8 training solution has been open-sourced, based on tile/blockwise FP8 scaling, and further introduces FP8 optimizers, etc., achieving extreme memory optimization. On 8/16/32 80G GPUs, compared withLLaMA 3.1 8BandQwen3 8B, the training throughput is significantly improved.
-
Open Open-Source Strategy: In addition to releasing the trained version, five pre-training checkpoints have been open-sourced, supporting deeper research and broader applications.
Technical Principles of Ling-V2
-
MoE Architecture: Based on the Mixture of Experts (MoE) architecture, by decomposing the model into multiple expert networks and activating only part of the experts for each input token, it achieves model sparsity, improving computational efficiency while maintaining high performance.
-
Optimized Design: Empirical optimization designs are carried out in aspects such as expert granularity, shared expert ratio, attention ratio, auxiliary-loss-free + sigmoid routing strategy, MTP loss, QK-Norm, and half RoPE, further improving the model's performance and efficiency.
-
FP8 Mixed-Precision Training: FP8 mixed precision is used during training. Compared with BF16, in experiments exceeding 1 trillion training tokens, the loss curve and downstream benchmark performance are almost identical. At the same time, the FP8 training solution is open-sourced, supporting the community in efficient continuous pre-training and fine-tuning under limited computational resources.
-
Multi-Stage Training: The model is trained on more than 20 trillion high-quality data tokens and enhanced through multi-stage supervised fine-tuning and reinforcement learning, achieving significant improvements in complex reasoning and instruction following.
Project Address of Ling-V2
- GitHub Repository：https://github.com/inclusionAI/Ling-V2
- HuggingFace Model Library：https://huggingface.co/collections/inclusionAI/ling-v2-68bf1dd2fc34c306c1fa6f86
Application Scenarios of Ling-V2
- Natural Language Processing (NLP) Tasks: It efficiently handles various natural language processing tasks, such as text classification, sentiment analysis, machine translation, etc., providing accurate and efficient solutions.
- Intelligent Customer Service: As the core engine of intelligent customer service, it quickly responds to user questions, provides precise answers, and improves user experience and customer service efficiency.
- Content Creation: Assists content creation, generates high-quality text content such as news reports, creative writing, advertising copy, etc., helping creators improve creation efficiency and quality.
- Education: Used in the education field, such as intelligent tutoring, automatic homework grading, personalized learning plans, etc., providing personalized educational support for students and teachers.
- Healthcare: Ling-V2 can process medical text data, assist doctors in case analysis, medical literature retrieval, etc., improving the accuracy and efficiency of medical decisions.