This visual guide aims to demystify Mixture of Experts (MoE) in large language models (LLMs), systematically explaining this important component through more than 50 visual diagrams. The article begins with the foundation of LLMs—the feedforward neural network (FFNN)—pointing out that traditional FFNNs are dense models in which all parameters are activated, whereas MoE adopts sparse activation, using only some parameters, thereby controlling computational cost while scaling up the model.
MoE consists of two core components: Experts and the Router, or gating network. Each expert is itself usually an FFNN submodel, and is not focused on a specific domain such as "psychology" or "biology," but instead tends to learn token-level syntactic information, that is, the ability to handle specific tokens in specific contexts. The router is responsible for deciding which tokens are sent to which experts.
The article explains the architecture of experts in detail: in a decoder block, the original single FFNN is replaced by multiple FFNNs (i.e., experts), and different tokens may choose different experts, thereby forming different computational paths. The routing mechanism section introduces how the gating network selects experts through a probability distribution output by SoftMax, and how expert outputs are multiplied by gating values and summed.
Because simple routing can easily lead to some experts being over-selected while others are undertrained, the article discusses the load balancing problem and introduces the KeepTopK strategy (by adding trainable Gaussian noise and keeping the top-k experts) and the Token Choice method (Top-1 or Top-k routing). At the same time, the auxiliary loss (load balancing loss) constrains the balance of expert importance by calculating the coefficient of variation (CV) of importance scores. Expert Capacity limits the number of tokens each expert can process, and the excess overflows to the next layer.
The article also introduces the Switch Transformer, which replaces the traditional FFNN with a Switching Layer, adopts Top-1 routing, and introduces a capacity factor and a simplified auxiliary loss, improving training stability. Finally, it explores the application of MoE in vision models, such as V-MoE replacing the dense FFNN in ViT with sparse MoE and using Batch Priority Routing to process image patches; Soft-MoE achieves a transition from discrete to soft assignment by mixing image patches.
