LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts
Venmugil Elango, Nidhi Bhatia, Roger Waleffe, Rasoul Shafipour, Tomer Asida, Abhinav Khattar, Nave Assaf, Maximilian Golub, Joey Guman, Tiyasa Mitra, Ritchie Zhao, Ritika Borkar, Ran Zilberstein, Mostofa Patwary, Mohammad Shoeybi, Bita Rouhani
cs.LG, cs.AI
2026-01-26
LatentMoE projects MoE experts into a smaller latent space while scaling expert count and top-k by the same ratio, beating standard MoE on accuracy per FLOP at 95B scale; Nvidia adopted it for Nemotron-3.
MoE is now standard in open and proprietary LLMs, but how close standard MoE sits to optimal on accuracy per FLOP and per parameter has not been spelled out. The authors break down three bottlenecks from a hardware-software co-design view. At low latency and small batch size, MoE is bound by the memory bandwidth of loading weights; typical deployments give each expert only a few hundred tokens, far short of making the GPU compute-bound. At high throughput, once experts are compute-bound the bottleneck shifts to cross-device all-to-all communication, with a communication-to-compute ratio around 9 to 1. And cutting compute by trimming top-k or expert width directly hurts model capacity.
LatentMoE decouples expert computation from the hidden dimension d. A down-projection maps activations into a smaller latent space (d divided by ℓ equals α, with α=4 in experiments); experts compute there; an up-projection maps back. The move is not just dimensionality reduction but simultaneous scaling of expert count N and top-k by α: the latent space cuts each expert's bandwidth and communication, while the (αN choose αK) mixing space grows exponentially over (N choose K), raising both the non-linearity budget and expert diversity.
Two variants follow. ℓ-MoEeff scales N, keeps top-k, saves compute, and matches accuracy. ℓ-MoEacc (recommended) scales both N and top-k by α, keeps inference compute flat against standard MoE, and gains accuracy. The theory lines up with Barron function bounds: a one-hidden-layer network's O(1/u) error is independent of input dimension, and the effective non-linearity budget scales with the selected experts' total width.
At 95B total, 8B active, trained on 1T tokens (300B-token checkpoint):
| Variant | Active params | MMLU Pro | MMLU |
| Standard MoE | 8.47B | 29.26 | 58.95 |
| ℓ-MoEacc | 8.44B | 34.91 | 62.23 |
| ℓ-MoEeff | 5.62B | 34.75 | 61.06 |
At matched compute the acc variant wins across all downstream tasks; the eff variant reaches near-matched accuracy with far fewer FLOPs. A 73B Mamba-attention hybrid MoE agrees (acc MMLU Pro 52.87 vs 48.30). Inference on H100 with vLLM and FP8 drops throughput at most 6% at high concurrency. Extrapolated to the trillion regime (against Kimi-K2-1T), LatentMoE is worth about 1.35x effective parameters, and an iso-accuracy standard MoE needs 350B more parameters and runs 1.24 to 3.46x slower across the throughput-latency frontier. Nvidia adopted the architecture for the flagship Nemotron-3 Super and Ultra.
For LLM infrastructure builders, LatentMoE is a path to gains without just stacking parameters: it lifts the accuracy ceiling at near-constant inference cost, or slashes compute at matched accuracy. It targets the real MoE inference pain points of memory bandwidth and all-to-all communication, and it is validated in shipping flagship models, not just on paper. For anyone tracking Kimi K3's Stable LatentMoE, this is a root source for that technical line.
The authors concede up to 6% throughput loss at high concurrency, fixable with separate CUDA streams and specialized small-matrix GEMM kernels, and note the acc variant used the baseline's hyperparameters, so more tuning may help.
Open questions: much of the case rests on design-space exploration and extrapolation; the 95B and 1T-token measured scale is modest relative to today's flagships, and the higher-accuracy-at-matched-compute claim at true trillion scale and long token horizons is backed mainly by Nemotron-3 adoption rather than direct data in this paper. The standard MoE baseline in the inference comparison is described ambiguously, leaving unclear whether it was retrained under matched conditions.