Listen to this lesson
Unlock audio and more
Audio streaming, downloadable PDFs and certificates come with Plus and Pro.
What it means
In a conventional dense model every parameter participates in every token. A mixture-of-experts model instead contains many sub-networks — experts — plus a small router that picks a handful for each token. Total parameter count can be enormous while the parameters actually used per token stays modest.
This decouples two numbers that used to move together: how much a model knows and how much it costs to run. That is why vendors now quote both total and active parameters, and why a headline parameter count means less than it used to.
The costs are real. All experts must be held in memory even though few are used, so memory requirements track total size while compute tracks active size. Training is also harder — routing can collapse onto a few favored experts unless deliberately balanced.
Why it matters
Most frontier models are now mixtures of experts, which is why capability has kept climbing without inference cost climbing proportionally. It also means comparing models by total parameter count across architectures is meaningless — a large sparse model and a smaller dense one can cost the same to run.
What people get wrong
That the experts are domain specialists. The intuitive picture — one expert for code, one for French, one for medicine — is not what happens. Routing is learned to minimize loss, not to be interpretable, and the specializations that emerge are typically subtle statistical patterns with no correspondence to human categories. You cannot open a mixture-of-experts model and find the legal expert.
That active parameters tell you the hardware you need. They tell you the compute per token; they say nothing about memory. Every expert must be resident even though only a few fire for any given token, so memory scales with the total parameter count, while compute scales with the active count. Sizing a machine on the active number is the most common and most expensive mistake when self-hosting one of these models.
That total parameter count is comparable across architectures. It is not. A large sparse model and a considerably smaller dense one can cost about the same to run, so holding a mixture-of-experts headline number against a dense model's is comparing two different quantities. Read both numbers, or neither.
In practice
When evaluating an open-weight mixture-of-experts model for self-hosting, size your hardware on total parameters, not active. The memory bill is set by what must be resident, not by what fires.
Where this shows up
Tools and models in our catalog.
DeepSeek R1First open-source reasoning model matching OpenAI o1. MIT license. R1-0528 adds JSON output and function-calling. Distilled variants 1.5B-70B. Banned on government devices in several countries, and ordered off US federal civilian systems in October 2026.
Kimi K3Moonshot AI's flagship — a 2.8 trillion parameter mixture-of-experts model activating 16 of 896 experts per token, with a 1 million token context window and native vision. Weights published in July 2026 under a custom Kimi K3 License. Kimi K3.1 Pro, a post-trained version released October 23, 2026 with cyber-related training data removed, runs on Moonshot's apps and API, and its weights have been public since November 4, 2026 under the same commercial terms as K3.
Mistral Large 3Mistral's previous flagship open-weight model: 675 billion-parameter MoE (41 billion active), 256K context, multimodal, modified MIT license. Succeeded in October 2026 by the 1 trillion-parameter Mistral Large 4, which ships under a different license; Large 3 remains the smaller option to self-host.
Qwen3.8-MaxAlibaba's flagship from August until Qwen4-Max replaced it on November 16, 2026: a 2.4 trillion-parameter mixture-of-experts model with about 95 billion active per request and roughly a one-million-token context. Its weights are downloadable under a custom Qwen license, which makes it the largest Qwen model you can self-host.