Mixture of Experts routes each input to a small subset of specialized subnetworks, so a model can be extremely large without activating the full parameter set on every pass.
Mixture of Experts is a sparsely activated architecture. A router network selects which experts handle each token or example. Only those experts compute, so total parameter count can grow while compute per token stays roughly constant.
That routing mechanism is why some of the largest deployed models use MoE. A dense model with the same parameter count would be too expensive to run. MoE keeps the capacity high while preserving the inference budget by ensuring each token touches only a fraction of the total weights. The trade-off is implementation complexity and the risk of imbalanced expert utilization.
Think of it like this. Think of a hospital where every patient does not see every doctor. A triage nurse routes each patient to the right specialist. The hospital can employ many specialists without making every patient visit every office.
The model contains many expert feedforward networks. A router, usually a learned softmax or top-k gate, assigns each token to one or more experts. Only selected experts compute for that token. Experts can be grouped across GPUs or TPUs for efficient batch execution.
"MoE is always better than dense." It increases capacity but adds routing complexity and hardware scheduling challenges. "More experts is always better." Expert utilization can collapse; some experts receive almost no tokens. "MoE eliminates training cost." It usually increases total training compute even if inference is cheaper per token.
Higher capacity with similar inference cost, but routing decisions can be unstable and expert load can become unbalanced. Best for very large models on standardized hardware; harder to debug and serve than dense models.