Mixture of experts. A memory versus compute trade, not a magic trick

A router reads each token and sends it to only a few experts. Total parameters can grow huge while the compute spent on any single token stays small. Raise the top-k, or flip to dense, and watch active parameters move while the total stays put.

Incoming tokens Router Experts route top-k
Total parameters
46.6 B
Active per token
13.0 B
Active ratio
28%
Load balance
--
k = 2

Numbers follow a Mixtral style setup with 8 experts, a shared attention trunk, and top-2 routing. Total and active parameter counts are illustrative, the ratio is the point.