Mixture of Experts (MoE)

Simple Definition

Mixture of Experts (MoE) is an architecture design for AI models where the model is divided into many specialized subnetworks (called “experts”), and only a small subset of them are used for any given input. A learned “router” decides which experts are best suited to handle each piece of the input.

The result: a model that has the total capacity of a very large system but only uses a fraction of its parameters at a time, making it faster and more efficient than using all parameters for everything.

A Simple Analogy

Think of a large hospital. When you arrive, a receptionist routes you to the right specialist, a cardiologist, a neurologist, a dermatologist. Each specialist handles what they’re best at; the whole hospital isn’t mobilized for your one appointment. A Mixture of Experts model works the same way, inputs are routed to the specialists best equipped to handle them.

Dense vs. MoE Models

	Dense Model	MoE Model
Parameters used per request	All parameters	A small subset (e.g., 2 of 64 experts)
Total parameters	What you run	Can be much larger than what’s active
Compute per request	High	Lower for same total capacity
Training complexity	Simpler	More complex

Why MoE Matters

MoE lets AI companies build models with massive total capacity while keeping inference costs manageable. A model might have 400 billion total parameters but only activate 50 billion for any given token, matching the output quality of a 400B dense model at the compute cost closer to a 50B one.

Models Using MoE

DeepSeek V4: a large MoE model with 1.6 trillion total parameters
GLM 5.2 (Z.ai): roughly 753 billion total parameters, open-weight
Kimi K3 (Moonshot AI): 2.8 trillion total parameters, open-weight
Poolside Laguna S 2.1: 118 billion total parameters, only about 8 billion active per token, open-weight
Mixtral 8x7B (Mistral): one of the first publicly released MoE language models
Mixtral 8x22B: larger open MoE model from Mistral
Grok (xAI): uses MoE architecture
Gemini (Google): uses MoE elements

The Trade-offs

MoE is not without downsides:

More complex to train and fine-tune
Harder to run on single devices (experts may span multiple GPUs)
Load balancing between experts is a non-trivial engineering challenge

Model Parameters, MoE’s key efficiency comes from not using all parameters at once
LLM, most frontier LLMs are now believed to use MoE
Inference, MoE reduces compute per inference call
Foundation Model, MoE is a common architecture for modern foundation models
Transformer, MoE is built on top of the transformer architecture

For real examples of active-vs-total parameter tradeoffs in current models, see Frontier Open Models Explained.