moe

Mixture of Experts (MoE) is the sparse-activation architecture that scales a neural network to trillions of parameters while keeping per-token compute fixed — each input activates only a small subset of "expert" sub-networks selected by a learned router, so total model capacity grows without proportional growth in inference FLOPs. Mixtral 8×7B, Switch Transformer, DeepSeek-V2/V3, and Grok all use MoE layers to achieve frontier accuracy at a fraction of the inference cost of an equivalently capable dense model.

The core idea is conditional computation. In a dense transformer, every token passes through every feed-forward parameter. In an MoE transformer, the standard feed-forward block is replaced by many parallel expert feed-forward networks plus a lightweight router. For each token, the router selects a small number of experts — typically just one or two — and only those experts actually run. With 64 experts and 2 selected per token, the model holds 64 times the parameters of a single expert but spends only roughly 2 times the compute per token of one expert — a parameter-to-FLOP leverage ratio around 32 to 1.

The router works by scoring every expert and keeping only the top few. A learned weight matrix maps each token's embedding to a score for every expert, those scores pass through a softmax to become a probability distribution, and the router keeps only the top-scoring experts — the ones whose scores survive this cutoff are the ones that actually process the token, and their softmax weights become the mixing coefficients used to combine their outputs into the layer's final result.

Without an explicit fix, training collapses onto a handful of popular experts, which is exactly the problem the load-balancing loss exists to prevent. A few experts naturally attract more tokens early in training, receive stronger gradients as a result, and become even more attractive to the router in a self-reinforcing loop — "expert collapse" — unless an auxiliary loss term penalizes uneven load across experts, pushing the router toward distributing tokens more evenly; this auxiliary loss is typically weighted at a small fraction of the main training loss, just enough to discourage collapse without distorting the router's actual job of matching tokens to the experts best suited for them.

Each expert can only process so many tokens before it overflows, and different architectures handle the overflow differently. Every expert has a capacity limit per batch — roughly the number of tokens divided by the number of experts, scaled by a capacity factor — and tokens that exceed that limit either get dropped entirely, the approach used in the original Switch Transformer with a capacity factor around 1.25, or get rerouted to a fallback. DeepSeek-V2 sidesteps the dropping problem differently, adding a shared expert that every single token passes through regardless of routing, alongside the specialized routed experts — guaranteeing a baseline of processing no token ever misses.

Mixture of Experts — Sparse Conditional Compute only activate 2 of N experts per token — get large model capacity at small per-token compute cost MoE FFN Layer (replaces dense FFN in transformer block) token hidden vector Router score every expert softmax → top-k weights: 0.7, 0.3 Expert 0 Expert 1 Expert 2 Expert 3 Expert 4 ... Expert 5-7 active (top-2) active Weighted Sum 0.7×E0 + 0.3×E4 output Why MoE Total params: huge Active params: small DeepSeek-V3: 671B total, 37B active Mixtral 8×7B: 47B total, 13B active = dense-model quality at 3-5x less compute Grok-1: 8 experts 314B total, ~86B active Engineering Challenges Load balancing: some experts get all tokens Fix: auxiliary load-balancing loss Communication: experts on different GPUs Fix: expert parallelism (all-to-all dispatch) Memory: all experts in VRAM even if unused Production MoE Models DeepSeek-V3: 256 experts, top-8, 671B Mixtral 8x7B: 8 experts, top-2, 47B Grok-1: 8 experts, 314B (xAI) Qwen-MoE: 60 experts, top-4 DBRX: 16 experts, top-4 (Databricks) Each expert learns to specialize — the router learns which ones to call for a given token Inference runs at roughly the active-parameter cost, with quality closer to the full total-parameter model MoE hires many specialists but consults only a couple per question — large knowledge, small per-query cost.

Training an MoE model requires a distinct communication pattern that dense models never need, and it's the specific bottleneck that makes MoE training harder to scale than it looks on paper. Because different experts typically live on different GPUs, every forward pass needs an all-to-all exchange: tokens get gathered onto whichever GPU hosts their assigned expert, processed there, then scattered back to continue through the rest of the model. This expert-parallel communication pattern, layered on top of the ordinary data parallelism covered under distributed-training, is the reason MoE training places unusually heavy demands on inter-GPU interconnect bandwidth specifically, on top of everything dense model training already requires.

At inference time, MoE's sparsity helps compute but does nothing for memory, which is the architecture's single most counterintuitive cost. Even though only a couple of experts actually run for any given token, every expert still has to sit in accelerator memory, because the router's selection changes from token to token and there's no way to know in advance which experts a future token will need — a 1.8-trillion-parameter MoE model at 16-bit precision needs on the order of 3.6 terabytes of memory just to hold every expert, far beyond a single accelerator and requiring a multi-node serving setup regardless of how few experts actually fire per token. This is why MoE inference is memory-bandwidth-bound even more severely than dense model inference: compute scales with the small active-parameter slice, but memory footprint scales with the full total parameter count.

Expert count and expert size trade against each other, and the field has moved toward many smaller experts rather than few large ones. Early MoE designs like Switch Transformer and Mixtral used a handful of experts, each roughly the size of a full feed-forward block; DeepSeek-V2 and later designs instead use hundreds of much smaller experts — each a small fraction of a full feed-forward block's width — selecting more of them per token without increasing total compute, which produces smoother routing, less load imbalance, and lets each token's output assemble from a more nuanced combination of specialized pieces rather than just one or two large, coarse ones.

The trade MoE makes is the same one every sparsity technique makes in some form: spend extra total parameters, and extra engineering effort routing tokens to the right subset of them, in exchange for inference compute that scales with only the active slice rather than the whole model — which is exactly why DeepSeek-V3's 671 billion total parameters cost roughly what a 37-billion-parameter dense model would cost to run, while reaching for capability much closer to what the full 671 billion parameters would deliver if every one of them had to run on every token.

Go deeper with CFSGPT

Get AI-powered deep-dives, save terms, and run advanced simulations — free account.

Create Free Account