Mixture-of-Experts

In a conventional dense model every parameter participates in every token. A mixture-of-experts model instead contains many sub-networks — experts — plus a small router that picks a handful for each token. Total parameter count can be enormous while the parameters actually used per token stays modest.

This decouples two numbers that used to move together: how much a model knows and how much it costs to run. That is why vendors now quote both total and active parameters, and why a headline parameter count means less than it used to.

The costs are real. All experts must be held in memory even though few are used, so memory requirements track total size while compute tracks active size. Training is also harder — routing can collapse onto a few favored experts unless deliberately balanced.

Most frontier models are now mixtures of experts, which is why capability has kept climbing without inference cost climbing proportionally. It also means comparing models by total parameter count across architectures is meaningless — a large sparse model and a smaller dense one can cost the same to run.

When evaluating an open-weight mixture-of-experts model for self-hosting, size your hardware on TOTAL parameters, not active. The memory bill is set by what must be resident, not by what fires.