- MoE activates only a fraction of model parameters per input, cutting compute costs.
- DeepSeek-V3 holds 671 billion parameters but uses just 37 billion per token.
- Jacobs, Jordan, Nowlan, and Hinton proposed the architecture in 1991.
A mixture of experts is a neural network architecture that routes each input to a small subset of specialized sub-networks, activating only the relevant "experts" rather than the entire model. The design allows systems to grow massively in total capacity while keeping the computational cost of any single prediction low.
Why It Matters
Key figure
671B
Total parameters in DeepSeek-V3, with only 37B active per token
The mixture of experts approach has become the dominant architecture behind the most capable AI systems in operation. As of early 2026, virtually all leading frontier models use MoE designs, including DeepSeek-V3, Llama 4, Mistral Large 3, and Google's Gemini family. The pattern is not confined to proprietary labs. Over 60% of open-source AI model releases in 2025 adopted some form of MoE, according to NVIDIA.
The reason is economic. DeepSeek-V3, released in December 2024 by the Chinese AI lab DeepSeek, contains 671 billion total parameters but activates only 37 billion per token. The company trained it on 14.8 trillion tokens using 2.788 million H800 GPU hours, a fraction of what competitors spent on dense models of comparable quality.
That efficiency gap explains why MoE has moved from a research curiosity to an industry default. For anyone following generative AI's trajectory, mixture of experts sits at the center of a basic tension: how to make models smarter without making them proportionally more expensive to run.
How It Works
A standard transformer processes every input through all of its parameters. An MoE transformer replaces certain layers (typically the feed-forward layers) with a set of parallel expert networks, each structurally identical but with different learned weights.
Key figure
1991
Year Jacobs, Jordan, Nowlan, and Hinton proposed MoE
A gating network (also called a router) examines each input token and assigns it to a small number of experts, usually two. Only those selected experts process the token. The rest remain idle. The final output is a weighted combination of the activated experts' outputs, with the weights determined by the router.
This selective activation is what makes MoE efficient. A model with 256 experts and a top-2 routing scheme uses less than 1% of its total parameters for any given token. The remaining experts contribute nothing to that particular computation but stand ready for inputs that match their specialization.
During training, the experts develop distinct capabilities without explicit instruction. One expert might handle mathematical reasoning, another code generation, another conversational language. The router learns which expert suits which input, and load-balancing mechanisms prevent any single expert from becoming a bottleneck. DeepSeek-V3 introduced an auxiliary-loss-free balancing strategy that improved training stability at scale.
Key Context
Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton proposed the original mixture of experts framework in their 1991 paper "Adaptive Mixtures of Local Experts." Their system used a gating network to assign inputs to specialized sub-models, establishing the core idea. Jordan and Jacobs extended it in 1994 with a hierarchical version. The concept remained a niche technique for over two decades.
The modern revival began in 2017, when Noam Shazeer and colleagues at Google introduced the sparsely-gated mixture of experts layer. Their key insight was making the gating function sparse: instead of blending all experts' outputs, the router selects only the top-k. That change made it practical to scale to thousands of experts without a proportional increase in computation. Mistral AI's Mixtral 8x7B (December 2023) brought the approach into open-source prominence, and the architecture has spread rapidly since.
FAQ
What is the difference between a mixture of experts model and a dense model?
A dense model activates all of its parameters for every input. An MoE model activates only a fraction, selected by a routing mechanism. This means an MoE model can have far more total parameters while using roughly the same compute per prediction as a much smaller dense model.
Does mixture of experts only work for language models?
No. The original 1991 proposal predated large language models entirely. MoE has been applied to image recognition, speech processing, recommendation systems, and scientific computing. Its recent prominence in LLMs reflects the architecture's effectiveness at enormous scale.
How does the router decide which experts to use?
The router is a small neural network that takes an input token's representation and produces a probability distribution over the available experts. It selects the top-k (usually 2) with the highest scores. The router trains jointly with the experts, learning to match inputs with the most suitable specialists.
Can mixture of experts models run on smaller hardware?
Partially. While MoE models activate fewer parameters per token, the full set of expert weights must still reside in memory. A 671-billion-parameter MoE model requires substantial memory even if only 37 billion parameters compute at any moment. Techniques like expert offloading and quantization help, but hardware requirements remain significant.
Related Reading
Sources
- Primary Research: Adaptive Mixtures of Local Experts (Jacobs, Jordan, Nowlan, Hinton, 1991)
- Additional Context:
- Hierarchical Mixtures of Experts and the EM Algorithm (Jordan & Jacobs, 1994)
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (Shazeer et al., 2017)
- DeepSeek-V3 Technical Report (DeepSeek-AI, 2024)
- Mixture of Experts Powers the Most Intelligent Frontier AI Models (NVIDIA, 2025)
Fact Check: Claim-by-Claim Verification Verified
All major claims verified against primary sources and cross-checked via Perplexity sonar-pro-search. Key facts about DeepSeek-V3 parameters, the 1991 origin paper authors, and Shazeer's 2017 sparse gating innovation all confirmed.
Sources used for verification
- DeepSeek-V3 Technical Report - arxiv.org
- Sparsely-Gated MoE Layer - arxiv.org
- NVIDIA MoE Blog - nvidia.com
- Adaptive Mixtures of Local Experts - MIT Press
- Meta Llama 4 Blog - meta.com
