Instella-MoE - AMD's Open-Source Mixture-of-Experts Language Model Series

What is Instella-MoE

Instella-MoE is AMD's open-source Mixture-of-Experts language model series, with a total parameter count of 16B and an activated parameter count of 2.8B, using a 27-layer decoder architecture. The model is trained from scratch on AMD Instinct MI300X / MI325X GPUs and the ROCm software stack, offering 6 fully open-source versions in total: pretrained, long-context Base, SFT, DPO, and RL-optimized, achieving efficient synergy between training and inference on AMD hardware.

Main Features of Instella-MoE

• Multi-stage model series: Provides 6 versions in total—pretrained, Mid-training, long-context Base, SFT, DPO, and RL-optimized—covering the full pipeline from base model to dialogue.
• Efficient MoE inference: Inference can be completed by activating only 2.8B parameters, achieving performance close to larger dense models while maintaining relatively low computational overhead.
• Long-context support: The Base version supports a 64K context length, capable of handling long-document understanding and reasoning tasks.
• Instruction following and chain-of-thought: The SFT version achieves instruction following and chain-of-thought reasoning, while the Think version further improves response quality through reinforcement learning.
• AMD-native optimization: Deeply optimized based on the ROCm ecosystem and the SGLang inference framework, achieving efficient training and inference on AMD hardware.

Technical Principles of Instella-MoE

• MoE architecture and attention mechanism: The model adopts a Mixture-of-Experts architecture. Of the 16B total parameters, only 2.8B parameters are activated in each forward pass. Through a gated routing mechanism, tokens are dynamically assigned to different expert networks, decoupling large capacity from low inference cost. At the same time, gated multi-head latent attention is introduced, adding gating control in the latent attention space, enhancing long-sequence modeling capability and reducing KV cache overhead.
• AMD-native training system: Training is entirely based on the AMD ROCm™ software stack, with the Primus training framework and Miles RL framework at the底层. During the pretraining stage, the FarSkip-Collective technology is used to overlap communication and computation under expert parallelism, increasing pretraining speed by 12.7%. The model is trained from scratch on AMD Instinct MI300X / MI325X GPUs, fully leveraging the interconnect bandwidth advantages of AMD hardware.
• Post-training and reinforcement learning: Post-training adopts a four-stage pipeline: SFT→ DPO→ two-stage RL. The RL stage is divided into: Stage 1 conducts specialized reinforcement learning for instruction following; Stage 2 adopts MOPD, routing IF Prompts to the IF-RL Teacher and other Prompts to the DPO Teacher through a Domain Router, constraining the student model's policy updates with token-level reverse KL divergence, strengthening instruction following and reasoning quality while maintaining general capabilities.

Follow WeChat and reply “open source” to join theAI Open-Source Project Communication Group

How to use Instella-MoE

• Obtain model weights: Visit the Hugging Face model collection (amd/instella-moe) or the GitHub repository to download the required version (Base / SFT / DPO / Think).
• Environment preparation: Install the AMD ROCm driver and PyTorch for ROCm, or directly use a Docker image that supports ROCm.
• Load the model: Load the model weights with the Transformers library or the SGLang inference framework. SGLang can fully leverage the inference acceleration capabilities of AMD hardware.
• Run inference: Choose the corresponding version based on the task type—Base for text continuation and embedding, SFT / DPO / Think for dialogue and instruction tasks.
• Fine-tuning and Deployment: Continue training the Base model with your own data, or deploy it as an API service via vLLM / SGLang.

Core Advantages of Instella-MoE

• Fully open-source and transparent: All 6 stage models from pre-training to RL are open-sourced, with transparent and verifiable training data and code.
• Native AMD ecosystem adaptation: Trained on ROCm and self-developed GPUs, with optimal inference efficiency on AMD Instinct series hardware, reducing TTFT by up to 39.2%.
• Efficient activated parameters: Only 2.8B of the 16B total parameters are activated, making inference cost significantly lower than dense models of comparable performance.
• Advanced post-training techniques: MOPD multi-teacher distillation combined with token-level reverse KL, delivering outstanding performance in instruction following and mathematical reasoning.
• Deep communication-computation overlap: FarSkip-Collective and SGLang expert parallelism achieve communication hiding, excelling in both training and inference efficiency.

Instella-MoE Project Address

• Project Website：https://rocm.blogs.amd.com/artificial-intelligence/instella-moe/README.html
• GitHub Repository：https://github.com/AMD-AGI/Instella-MoE
• HuggingFace Model Library：https://huggingface.co/collections/amd/instella-moe

Comparison of Instella-MoE with Similar Competitors

Application Scenarios of Instella-MoE

• Enterprise private deployment: Deploy dialogue and inference services on local GPU clusters based on the AMD ROCm ecosystem to meet data compliance requirements.
• Long document analysis: Use the 64K long-context capability to handle long-text understanding and summarization tasks such as legal contracts, research papers, and financial reports.
• Code-assisted generation: The SFT / Think versions support programming instruction following and can be used in IDE plugins or code review tools.
• Research and teaching: The fully open-source pipeline is suitable for academia to conduct MoE architecture research, reproduce training methods, and teach large model courses.
• Edge and hybrid cloud inference: The low number of activated parameters makes it suitable for efficient inference in resource-constrained environments, or as a foundation model for hybrid cloud AI services.