mixture of experts
Mixture of Experts (MoE) is the model architecture that uses a gating network to dynamically route each input to a sparse subset of specialized "expert" sub-networks — enabling models with dramatically more total parameters (and thus more capacity) while keeping per-input computation constant, allowing models like Mixtral 8x7B and GPT-4 to achieve superior performance without proportionally increasing inference cost.
Core Architecture
- Experts: N parallel feed-forward networks (e.g., N=8 or N=64), each potentially specializing in different input types.
- Router/Gate: A network that assigns each token to the top-K experts (typically K=1 or K=2).
- Sparse Activation: Only K out of N experts process each input → computation scales with K, not N.
Routing (Gating)
$G(x) = TopK(Softmax(W_g \cdot x))$
- Linear layer projects input to N scores (one per expert).
- Softmax normalizes scores to probabilities.
- TopK selects the K highest-scoring experts.
- Output: Weighted sum of selected expert outputs, weighted by gate probabilities.
Parameter vs. Compute Scaling
| Model | Total Params | Active Params/Token | Experts | Top-K |
|---|---|---|---|---|
| Mixtral 8x7B | 47B | ~13B | 8 | 2 |
| Switch Transformer | 1.6T | ~100B | 128 | 1 |
| GPT-4 (rumored) | ~1.8T | ~220B | 16 | 2 |
| DeepSeek-MoE | 145B | ~22B | 64 | 6 |
Load Balancing Challenge
- Without intervention: Router sends most tokens to a few "popular" experts → others idle.
- Auxiliary load balancing loss: Penalty for uneven expert utilization.
- $L_{balance} = N \cdot \sum_{i=1}^N f_i \cdot p_i$ where f_i = fraction of tokens to expert i, p_i = average gate probability.
- Expert capacity: Token buffer per expert — overflow tokens dropped or re-routed.
Training Challenges
- Instability: Routing decisions are discrete → training can be unstable.
- Expert collapse: All experts converge to similar behavior → no specialization.
- Communication overhead: In distributed training, tokens must be sent to the GPU holding each expert (all-to-all communication).
Sparse vs. Dense Trade-offs
- Advantage: More parameters → more knowledge capacity at same inference cost.
- Disadvantage: Higher memory footprint (all experts in memory), communication overhead, less efficient on small batches.
- When to use MoE: Large-scale pretraining where parameter count matters more than memory efficiency.
Mixture of experts is the dominant scaling strategy for frontier language models — by decoupling parameter count from per-token computation, MoE enables models to store more knowledge and handle more diverse tasks while maintaining economically viable inference costs.