Mixture of Experts

Big without being slow

DeepSeek-V3 has 671 billion parameters, yet each token it reads passes through only 37 billion of them. Kimi K2 has more than a trillion and uses 32 billion. This is the trick behind most of today's largest open models: keep a huge library of knowledge, but consult only a few shelves for each word.

What is a mixture of experts?

A mixture of experts (MoE) is a layer made of several interchangeable sub-networks, called experts, plus a small router (also called a gate) that decides which experts should process each input. In a modern language model the experts are the feed-forward blocks of the transformer, and the router picks a handful of them for every token, in every layer.

The result is a model whose total parameter count (everything stored) is far larger than its active parameter count (what one token uses). Qwen's naming makes this explicit: Qwen3-235B-A22B has 235 billion parameters, 22 billion of them active, with 8 of 128 experts chosen per token.

Why not just build a bigger dense model?

Because compute, not storage, is the binding constraint. In a dense model every parameter takes part in every token, so training and inference cost grow in lockstep with size: a forward pass costs roughly 2 floating-point operations per parameter per token. Doubling the parameters doubles the bill.

Mixture of experts breaks that link. Capacity (how much the model can store and represent) grows with the total parameter count, but the cost per token grows only with the active count. The Switch Transformer paper reported up to 7x faster pre-training than a dense T5 model using the same compute per token, simply by giving the model more experts to route between.

Why does it matter to you?

If you use a frontier open model today, you are probably using a mixture of experts: Mixtral, DeepSeek-V3 and R1, Qwen3's large models, OpenAI's gpt-oss and Kimi K2 all are. Closed labs rarely publish architectures, but Google has stated that Gemini 1.5 Pro is a sparse mixture-of-experts transformer. Understanding MoE explains why these models are cheap to run per token but hungry for memory, why their parameter counts come in two numbers, and why cost-per-token has fallen faster than model size has grown.

The idea itself is old. It spent decades as a niche technique before hardware and scale made it the default way to build large language models.

Key takeaways

• A mixture of experts splits a layer into many expert sub-networks and uses a router to run only a few of them per token.
• Total parameters set how much the model can store; active parameters set the compute per token. MoE lets the first grow much faster than the second.
• Most large open-weight LLMs released since 2024 are MoE models, from Mixtral 8x7B to DeepSeek-V3, Qwen3, gpt-oss and Kimi K2.

Dense and sparse layers

A transformer block has two halves: attention, which moves information between tokens, and a feed-forward network, which processes each token on its own. In most large models the feed-forward half holds the majority of the parameters, and that is the half MoE replaces.

What does a sparse MoE layer replace?

In a dense transformer, every token's hidden state h goes through the same feed-forward network (FFN): expand to a wider hidden size, apply a nonlinearity, project back. An MoE layer keeps the same interface but holds N separate FFNs, the experts, each with its own weights. A router scores all N experts for the token, the top k are run, and their outputs are added together, weighted by the router's gate values:

The MoE layer. gᵢ(h) is zero for every expert outside the top k, so those experts are never computed. The result is added back to h through the residual connection, exactly as a dense FFN would be.

How do the parameter counts add up?

Attention layers, embeddings and normalisation are shared: every token uses them. Only the FFNs are multiplied. That is why Mixtral 8x7B is not a 56-billion-parameter model despite its name. Working through its published configuration (32 layers, model width 4,096, expert width 14,336, SwiGLU experts with three weight matrices each):

• One expert in one layer: 3 × 4,096 × 14,336 ≈ 176 million parameters. Across 32 layers: about 5.6 billion.
• Eight experts: about 45 billion. Attention and embeddings, shared by all tokens: about 1.6 billion.
• Total: 45 + 1.6 ≈ 47 billion. Active per token (2 experts): 2 × 5.6 + 1.6 ≈ 13 billion.

So the “8x7B” is a family resemblance to Mistral 7B rather than a sum, and the model does the arithmetic of a 13B dense model per token while holding 47B parameters.

Why does sparsity matter so much?

Sparsity is what lets capacity and cost be chosen separately. Adding experts adds knowledge the model can store without adding a single operation per token. That makes MoE an attractive way to spend a fixed compute budget, and it is why the scaling laws for MoE models are usually written in terms of both total and active parameters. The costs appear elsewhere: all those parameters still need memory, the router has to be trained, and experts have to be kept busy. The rest of this lesson is about paying those costs well.

Mixtures are not new, though. The 1991 original by Jacobs, Jordan, Nowlan and Hinton was a soft mixture: every expert ran on every input and the gate blended them. It was about specialisation, not saving compute. The modern, compute-saving form arrived with Shazeer and colleagues' sparsely-gated MoE layer in 2017, which ran only the top few of thousands of experts and grew an LSTM language model to 137 billion parameters.

Key takeaways

• An MoE layer replaces one feed-forward network with N expert FFNs and a router; only the top k experts run for each token.
• Attention and embeddings are shared, so total parameters ≈ shared + N × expert, and active ≈ shared + k × expert (Mixtral: 47B and 13B).
• More experts add capacity without adding compute per token; the price is memory, routing and balancing.

Routers and top-k gating

The router is tiny, usually a single matrix, but it makes the decision that defines the whole model: which few experts, out of dozens or hundreds, see this token.

What does the router compute?

For a token with hidden state h, the router multiplies by a learned matrix W_r to get one logit per expert, then applies a softmax so the scores become probabilities that sum to 1. The top k experts by score are selected. Their probabilities become the gate weights that scale each expert's output.

Softmax router with top-k selection. Designs differ in whether the chosen probabilities are used as they are or renormalised to sum to 1.

How do the main designs differ?

The choice of k is a trade between cost and quality. GShard used top-2. The Switch Transformer showed top-1 works and is simpler and cheaper, contrary to the earlier belief that at least two experts were needed for the router to learn. Mixtral uses top-2 of 8 with renormalised gates; gpt-oss uses top-4 and weights the chosen experts by a softmax over just those four; DeepSeek-V3, Qwen3 and Kimi K2 pick 8.

There is a subtlety with top-1. If you renormalise over a single chosen expert, its gate is always exactly 1, whatever the router thought. Switch therefore uses the raw softmax probability as the gate. The point is the gradient: the router learns only through the gate values it produces.

With k = 1 and Renormalise, the gate column reads 1.000 whatever you do to the logits: the router is flying blind. With Raw probability it tracks the softmax. Two other details are worth noticing. “Probability kept” shows how much of the router's belief the chosen experts cover; when several experts score similarly, top-k throws a lot of it away. And noisy top-k gating, which adds random noise to the scores before choosing, turns close calls into coin flips. That spreads tokens across more experts early in training, one of the first tools used against the imbalance problem in the next section.

Are there other ways to route?

Yes. In expert-choice routing the roles are reversed: each expert picks the tokens it scores highest, up to a fixed number. Load is perfectly balanced by construction, but a token might be picked by many experts or by none, and because an expert's choice depends on the other tokens in the batch it does not fit naturally with generating one token at a time. Most autoregressive LLMs stick with token-choice top-k.

Why does routing matter? Because it is a hard, discrete decision inside a network trained by gradients. Everything that is difficult about MoE training (instability, imbalance, experts that never learn) traces back to the fact that the choice itself has no gradient, only the gate values do.

Key takeaways

• The router is a learned linear map plus softmax; the top-k experts run and their outputs are weighted by gate values.
• The router learns only through the gates. With top-1, Switch uses the raw probability so that gradient does not vanish.
• Common choices: top-1 (Switch), top-2 (GShard, Mixtral), top-4 (gpt-oss), top-8 (DeepSeek-V3, Qwen3, Kimi K2).

Keeping every expert busy

Left to itself, a router tends to fall in love with a few experts. They get most of the tokens, learn fastest, and get chosen even more, while the rest sit idle, wasting both parameters and the hardware they live on.

What is expert collapse?

Collapse is a feedback loop. At the start, the router's preferences are nearly random, but some expert is slightly more useful for some tokens. It gets chosen a little more often, receives more gradient updates (unchosen experts get none), improves, and becomes even more attractive. In the extreme, one expert handles everything and the model is a dense network with a lot of dead weight.

The damage is double. Idle experts are wasted capacity. And in a distributed system, where each GPU hosts a few experts, an overloaded expert makes its GPU the bottleneck while the others wait.

Both models usually classify the points perfectly: the task alone does not need eight experts here. What differs is how the work is shared. Without balancing, one to three colours end up covering the whole plane. With the auxiliary loss, the router spreads points across nearly all eight experts at almost no cost in accuracy. In a real LLM those idle experts would be billions of parameters doing nothing.

How does the auxiliary loss work?

The Switch Transformer adds a small extra term to the training loss. For each expert i, let f_i be the fraction of tokens in the batch routed to it and P_i the average router probability it received:

Switch load-balancing loss (α = 0.01 in the paper). It equals α when routing is perfectly uniform and grows as load concentrates. f has no gradient (it comes from a hard choice); P does, so the gradient lowers the probability of busy experts.

A worked example with 4 experts: suppose f = (0.70, 0.10, 0.10, 0.10) and P = (0.55, 0.15, 0.15, 0.15). Then Σ f·P = 0.385 + 3 × 0.015 = 0.43, and N × 0.43 = 1.72, against 1.00 for perfect balance. The gradient with respect to P₁ is α·N·f₁ = 0.028, seven times larger than for the other experts (0.004), so the router is pushed hardest to move probability away from the overloaded expert.

The weakness is that this term competes with the real objective. Set α too low and collapse returns; too high and the router spreads tokens evenly even when that hurts quality. ST-MoE added a second auxiliary term, the router z-loss, which penalises very large router logits (the squared log-sum-exp) and fixed many of the training instabilities that had plagued sparse models.

Capacity and dropped tokens

Accelerators like fixed shapes, so GShard and Switch give each expert a fixed buffer: capacity = capacity factor × (tokens × k) / N. With 4,096 tokens, 64 experts and top-2 there are 8,192 assignments, 128 per expert if perfectly even. A capacity factor of 1.25 allows 160. If the router sends 220 tokens to one expert, 60 of them overflow and are dropped from that expert: they skip it and carry on through the residual connection, unprocessed by that layer. Raising the factor drops fewer tokens but wastes memory and compute on padding. Later systems avoid the trade-off: MegaBlocks uses block-sparse kernels that accept uneven loads, and DeepSeek-V3 reports dropping no tokens in training or inference.

Balancing without an auxiliary loss

DeepSeek-V3 largely replaced the auxiliary loss with a controller, following Wang and colleagues. Each expert gets a bias that is added to its routing score only when choosing the top k, not when computing gate weights. After each training step, the bias of an overloaded expert is lowered by a small amount γ and that of an underloaded expert raised by γ. Because the bias never enters the loss, balancing no longer tugs against the language-modelling gradient. DeepSeek-V3 keeps only a tiny per-sequence balance loss (α = 0.0001) to stop extreme imbalance within a single sequence. The lab lets you try all three approaches side by side.

Key takeaways

• Without pressure to share, routers collapse onto a few experts: the chosen ones get all the gradient and the rest never learn.
• The Switch auxiliary loss α·N·Σ fᵢPᵢ pushes probability away from busy experts; capacity limits cap each expert and drop the overflow.
• DeepSeek-V3 balances with a per-expert selection bias instead of a loss, and drops no tokens.

Fine-grained and shared experts

Early MoE models had a few big experts. The recent trend is the opposite: many small experts, several chosen at once, plus sometimes an expert that every token uses.

What is fine-grained expert segmentation?

DeepSeekMoE proposed splitting each expert into m smaller ones (each 1/m the width) and routing each token to m times as many. The compute per token is unchanged, because k × size stays the same. What changes is the number of possible combinations. Their example: 16 experts with top-2 gives 120 possible pairs; splitting each into 4 (64 experts, top-8) gives 4,426,165,368 possible combinations. The router can assemble a much more specific mix of skills for each token.

What are shared experts for?

Some knowledge is needed by almost every token: common syntax, frequent words, formatting. With purely routed experts, that common knowledge gets learned redundantly by many experts. DeepSeekMoE therefore also sets aside shared experts that every token passes through, leaving the routed experts free to specialise. DeepSeek-V3 has 1 shared and 256 routed experts in each MoE layer, activates 8 routed experts per token, and keeps its first three layers dense.

Not everyone agrees shared experts are needed: Qwen3's MoE models dropped them, using 128 routed experts with 8 active and a global-batch balancing loss. The design space is still being explored.

Why does this matter?

The ratio of active to total parameters is getting smaller. Mixtral uses about 28% of its weights per token; DeepSeek-V3 about 5.5%; gpt-oss-120b about 4.4%; Kimi K2, with 8 of 384 experts, about 3%. Each step down means more stored knowledge per unit of compute, and more pressure on memory and communication. Fine-grained routing and shared experts are what made these extreme ratios trainable.

Key takeaways

• Fine-grained experts: split experts into smaller ones and pick more of them. Same compute, vastly more combinations.
• Shared experts run for every token and absorb common knowledge, so routed experts can specialise (DeepSeek); Qwen3 shows they are optional.
• Recent open models activate only 3-6% of their parameters per token.

What experts actually learn

The name suggests a panel of specialists: a maths expert, a biology expert, a poetry expert. The evidence says experts specialise, but mostly not like that.

What did researchers find?

The Mixtral team measured which experts were chosen for text from different domains: arXiv papers, biology abstracts (PubMed), philosophy papers, DeepMind maths problems and GitHub code. They found no obvious pattern by topic; the distribution of expert choices looked similar for biology and philosophy. What they did find was structure at the level of syntax and tokens: words like “self” in Python and “Question” in English tended to go to the same expert, indentation tokens in code were consistently routed to the same experts, and consecutive tokens were assigned the same expert far more often than chance (at some layers around 28% of the time, against 12.5% for random choice among 8).

ST-MoE found a similar picture in an encoder-decoder model: encoder experts specialised in recognisable token types, such as punctuation, proper nouns, numbers or verbs, while decoder experts showed little specialisation.

How should you picture an expert, then?

Think of the router as partitioning the space of hidden states, not the space of topics. Hidden states encode a mix of what the token is, where it sits in the sentence and what came before. Tokens that look alike to the layer in those terms (the same kind of punctuation, the same position in a code block) land in the same region and so go to the same experts. That is also why neighbouring tokens often share an expert.

The lab shows a small version of this: on 2D data, experts end up owning regions of the plane, and the region they own is determined by what makes the router's job easy, not by the labels a human would give.

Why does it matter?

It changes what you can do with the experts. You cannot simply cut out the “biology expert” to make a small biology model, because there isn't one. It explains why routing is local: consecutive tokens going to the same experts helps caching when experts are offloaded to slower memory. And it is a reminder that MoE is an efficiency technique first; the specialisation that emerges serves the loss, not our categories.

Key takeaways

• In Mixtral, expert choice showed no clear pattern by topic across domains, but strong patterns by syntax and token type.
• Consecutive tokens often go to the same expert, because the router partitions hidden-state space rather than subject matter.
• Treat "expert" as a routing bucket learned for efficiency, not as a human-style specialist.

Serving MoE models

A mixture of experts is cheap in arithmetic and expensive in memory. Every expert has to be loaded somewhere, even though each token uses only a few. That single fact shapes how these models are deployed.

What limits MoE serving?

When generating text one token at a time, a model is usually limited by how fast it can read weights from memory, not by arithmetic. For a single request an MoE shines: each step reads only the shared weights and the k chosen experts, so it runs close to the speed of a dense model the size of its active parameters. But the server must still hold all the weights. And as more requests are batched together, different tokens pick different experts, and soon every expert is read at every step anyway. The benefit shifts from memory bandwidth to arithmetic.

At batch 1, DeepSeek-V3 at 8-bit reads about 37 GB of weights per token while storing 671 GB. By batch 64, each layer's 8-of-256 routing has touched about 87% of its experts, and a decoding step reads most of the model. Per token that is still cheap, because the cost is shared by 64 tokens, but the per-request memory advantage has gone. This is why MoE models are most economical in large, busy deployments, and why running them at home is mostly a question of fitting the total parameters into memory at all.

How are experts spread across hardware?

A model like DeepSeek-V3 does not fit on one GPU, so experts are distributed with expert parallelism: each GPU holds a subset of experts. For every MoE layer, tokens are sent to whichever GPUs hold their chosen experts (a dispatch step) and the results are sent back (a combine step). Both are all-to-all exchanges, where every GPU talks to every other, and they happen twice per MoE layer.

Communication can easily dominate. DeepSeek-V3 restricts each token to experts on at most 4 nodes so that cross-machine traffic stays bounded, and overlaps communication with computation. Load imbalance hurts here too: the slowest, busiest GPU sets the pace for the whole layer. That is the hardware reason behind all the balancing machinery of the previous sections.

What are the other trade-offs?

• Memory versus compute. An MoE with the compute of a 37B dense model needs the memory of a 671B one. For memory-constrained settings, such as phones or a single consumer GPU, a smaller dense model is often the better choice.
• Batch effects. Routing decisions depend on the token, but capacity limits and load depend on the batch. In systems that drop tokens, whether a token gets its chosen expert can depend on what else is in the batch.
• Fine-tuning. Sparse models have been reported to overfit more easily than dense models during fine-tuning on small datasets, and ST-MoE studied which hyperparameters help.
• Quantisation and offloading. Because each token touches few experts, idle experts can live in slower memory or be quantised more aggressively. gpt-oss, for example, ships its MoE weights in a 4-bit format.

Key takeaways

• All experts must be stored; only active ones are computed. At batch 1 an MoE reads little per token, but large batches touch nearly every expert.
• Expert parallelism spreads experts over GPUs and needs two all-to-all exchanges per MoE layer; limiting how many nodes a token can reach bounds that cost.
• MoE wins where compute is the constraint and memory is plentiful: large shared deployments more than small devices.

Check your understanding

Seven situations you could meet when training, choosing or deploying a mixture-of-experts model.

A startup has a fixed training compute budget and plenty of GPU memory for serving. They want the most knowledgeable model they can get. Which choice does this lesson support?

References

Architecture numbers in this lesson come from the technical reports of open-weight models; closed labs rarely disclose theirs. For a hands-on feel of routing, balancing and collapse, train a mixture yourself in the Mixture of Experts Lab.

Sources

1. [1]

   DeepSeek-V3 Technical Report(opens in a new tab)

   DeepSeek-AI, 2024

   671B-parameter MoE with 37B activated per token: 1 shared and 256 routed experts per layer, 8 routed experts per token, auxiliary-loss-free load balancing, no token dropping.

2. [2]

   Qwen3 Technical Report(opens in a new tab)

   Yang, A. et al. (Qwen Team), 2025

   Qwen3-235B-A22B: 128 experts, 8 activated per token, no shared experts, global-batch load-balancing loss.

3. [3]

   Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity(opens in a new tab)

   Fedus, W., Zoph, B., Shazeer, N., 2021

   JMLR 2022. Top-1 routing, the α·N·Σ f·P load-balancing loss, capacity factors, and up to 7x pre-training speed-ups over dense T5 at equal compute.

4. [4]

   Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context(opens in a new tab)

   Gemini Team, Google, 2024

   States that Gemini 1.5 Pro is a sparse mixture-of-experts transformer; parameter counts are not disclosed.

5. [5]

   Mixtral of Experts(opens in a new tab)

   Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., et al., 2024

   Open-weight sparse MoE: 8 experts per layer, top-2 routing, 47B total and 13B active parameters. Includes a routing analysis across domains.

6. [6]

   Adaptive Mixtures of Local Experts(opens in a new tab)

   Jacobs, R. A., Jordan, M. I., Nowlan, S. J., Hinton, G. E., 1991

   Neural Computation 3(1). The original mixture of experts: several networks plus a gating network that learns which one should handle each case.

7. [7]

   Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer(opens in a new tab)

   Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J., 2017

   ICLR 2017. Noisy top-k gating over thousands of experts inside an LSTM language model, with up to 137 billion parameters, and balancing losses.

8. [8]

   GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding(opens in a new tab)

   Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., Chen, Z., 2020

   A 600-billion-parameter MoE translation transformer with top-2 routing, expert capacity and experts sharded across accelerators.

9. [9]

   gpt-oss-120b & gpt-oss-20b Model Card(opens in a new tab)

   OpenAI, 2025

   Open-weight MoE reasoning models: 116.8B total / 5.1B active with 128 experts, and 20.9B / 3.6B with 32 experts, both top-4.

10. [10]

    Mixture-of-Experts with Expert Choice Routing(opens in a new tab)

    Zhou, Y., Lei, T., Liu, H., Du, N., Huang, Y., Zhao, V., Dai, A., Chen, Z., Le, Q., Laudon, J., 2022

    NeurIPS 2022. Experts pick their top tokens instead of tokens picking experts, which balances load by construction.

11. [11]

    ST-MoE: Designing Stable and Transferable Sparse Expert Models(opens in a new tab)

    Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., Fedus, W., 2022

    Router z-loss for training stability, fine-tuning behaviour of sparse models, and an analysis of what encoder experts specialise in.

12. [12]

    MegaBlocks: Efficient Sparse Training with Mixture-of-Experts(opens in a new tab)

    Gale, T., Narayanan, D., Young, C., Zaharia, M., 2022

    MLSys 2023. Block-sparse GPU kernels that handle uneven expert loads without capacity limits or dropped tokens.

13. [13]

    Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts(opens in a new tab)

    Wang, L., Gao, H., Zhao, C., Sun, X., Dai, D., 2024

    Balances expert load with a per-expert bias on the routing scores instead of an auxiliary loss, so balancing does not interfere with the training gradient.

14. [14]

    DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models(opens in a new tab)

    Dai, D., Deng, C., Zhao, C., Xu, R. X., Gao, H., et al., 2024

    Fine-grained expert segmentation (more, smaller experts) and shared experts that every token uses.

15. [15]

    Kimi K2: Open Agentic Intelligence(opens in a new tab)

    Kimi Team, 2025

    A 1.04-trillion-parameter MoE with 32B activated parameters, routing each token to 8 of 384 experts.