Mixture of Experts

One number became two.

In a dense transformer, every parameter participates in every token. Feed it a token, and all 70 billion weights are multiplied against it. A mixture-of-experts model replaces the feed-forward block in each layer with a bank of parallel blocks — the experts — plus a small router that reads the token and picks a handful of them. The rest of the layer's weights sit idle for that token.

So a single parameter count no longer describes the model. You need a pair:

• Total parameters — everything that must be resident in memory, because the router may reach for any expert on the next token.
• Active parameters — what actually gets multiplied per token, and therefore what the arithmetic costs.

The published 2026 open-weight models make the spread obvious. Qwen3-235B-A22B is 235B total with roughly 22B active, routing 8 experts out of a pool of 128. Llama 4 Maverick is around 400B total against roughly 17B active, with 128 routed experts plus one shared expert that every token passes through. Mistral Large 3 is 675B total, 41B active. DeepSeek-V4's Pro configuration is about 1.6T total with 49B active, and its Flash configuration 284B against 13B. Note where they converge: the totals span a factor of seven, and the active counts all land between roughly 13B and 50B, because that band is what a serving fleet can push tokens through economically.

Why the same model is cheap to rent and expensive to own.

Renting and owning charge you for different resources, and MoE separates those resources further than any other architectural choice.

• On an API, you are billed against active parameters. The provider's cost per token is dominated by arithmetic, and a 235B/22B model does roughly the arithmetic of a 22B dense model. That is why open MoE models list at prices — commonly in the region of $0.15–$0.30 per million input tokens across hosters — that look impossible next to their headline size.
• On your own hardware, you are billed against total parameters. Every expert must be resident, because you cannot know which one the next token needs. At bf16, 400B total is roughly 800 GB of weights — ten 80 GB accelerators before you have allocated a single byte of KV cache. Quantizing to 4 bits brings it to about 240 GB, which is still three cards to serve a model that computes like a 17B one.
• The idle capital is the real cost. You are paying for, powering, and cooling memory that is doing nothing on most tokens. That trade is excellent for a provider running thousands of concurrent requests across the whole expert pool, and terrible for a team running one workload on a fixed cluster — the point developed in self-hosted inference for agents.

What batch size does to the bargain.

The subtlety that decides whether MoE works for you is that the saving changes shape with concurrency.

• At batch size one, MoE is a genuine local win — if it fits. Generating a single stream is limited by memory bandwidth: the accelerator reads weights faster than it can be given work. An MoE model only reads the routed experts per token, so a 235B/22B model decodes at roughly the speed of a 22B dense model while carrying the knowledge of something much larger. The condition is unforgiving. Spill the resident weights to system RAM or SSD and the transfer cost dwarfs everything you saved.
• At high batch, the per-token saving erodes and the throughput saving arrives. Sixty concurrent requests route to different experts, so a serving step touches most of the pool anyway — but it amortises that read across sixty tokens instead of one. This is the regime hosted providers actually operate in, and it is why their pricing reflects active-parameter economics that a single-tenant deployment will never reproduce.
• Utilisation is the metric, not tokens per second. An MoE model on an under-loaded cluster is the worst of both bills: total-parameter memory, batch-one throughput. If your traffic is bursty rather than sustained, the arithmetic in provisioned throughput and commitments applies with extra force here.

What it changes for the way you pick and run models.

Almost nothing about prompting or agent loop design changes. What changes is the shape of every comparison you make.

• Stop reading parameter count as capability. A 1.6T MoE model and a 1.6T dense model are not comparable objects and will never exist in the same market. Compare on task evals, price per token, and latency at your concurrency — the checklist in choosing a model — and treat size as a deployment fact rather than a quality signal.
• "Open weights" and "you can run it" have drifted apart. The strongest permissively-licensed models are now MoE models whose total size exceeds any single machine you own. The licence is genuinely open; the hardware bill is the gate. This is the sharpest current version of the distinction in open vs closed models.
• Quantization pays back more here, and hurts differently. Because memory is the binding constraint, dropping to 4-bit converts directly into cards you do not have to buy — but experts are individually small, and aggressive quantization of a rarely-routed expert degrades exactly the inputs that needed it, which an average benchmark will not show. Test on your own distribution; see distillation and quantization.
• Sparsity does not compose with your other speedups the way you expect. Speculative decoding still helps, because it also trades spare compute for latency — and spare compute is precisely what an MoE model has. Prompt caching and batching help as much as ever. What does not transfer is any capacity plan you wrote against dense models.

Related: small and local models for what actually fits on your own machine, prefill and decode for why bandwidth rather than arithmetic sets your token rate, inference providers for who is absorbing the memory bill, and model routing for the system-level version of the same idea — send each request to the smallest thing that can answer it.