capacity factor tuning

Capacity factor tuning is the calibration of per-expert token buffer headroom relative to average expected load in MoE routing - it balances overflow risk, memory use, and communication efficiency.

What Is Capacity factor tuning?

• Definition: Setting a multiplier on average tokens per expert to determine maximum accepted tokens each step.
• Formula Context: Capacity is typically proportional to total routed tokens divided by number of experts times a factor.
• Low Factor Behavior: Minimal headroom improves efficiency but increases token dropping during load spikes.
• High Factor Behavior: More slack reduces drops but increases memory footprint and transfer volume.

Why Capacity factor tuning Matters

• Quality Protection: Excess drops can harm convergence and degrade final model performance.
• Throughput Impact: Overly large capacity wastes bandwidth and may slow dispatch and combine phases.
• Stability Control: Proper headroom prevents frequent overflow oscillations.
• Cost Efficiency: Right-sized buffers reduce unnecessary infrastructure overhead.
• Operational Predictability: Balanced settings produce smoother latency distributions across steps.

How It Is Used in Practice

• Baseline Sweep: Benchmark several factor values under representative sequence and batch regimes.
• Metric Pairing: Track drop rate, utilization skew, and step latency together during tuning.
• Adaptive Policy: Revisit factor after router behavior shifts during later training phases.

Capacity factor tuning is a high-leverage MoE systems parameter - disciplined calibration is necessary to preserve quality while keeping sparse execution efficient.