LatentLoop: asymmetric compute allocation cuts looped-MoE training cost by 19%

pmttyji · reddit · 2026-10-23

The LatentLoop paper explores asymmetric compute allocation in looped Mixture-of-Experts models:

Finding: when a shared block is applied twice, learned residual coefficients show the first visit operates at a much smaller scale; fixing coefficients confirms shrinking the first visit preserves or improves performance, while shrinking the second hurts.
Method: compress the first visit's sequence, keep full resolution on the second, and reallocate saved compute to a larger hidden size; MoE weights are shared across visits with separate attention parameters for the different resolutions.
Results: across 15 configurations from 1.3B–7B (largest budget >10^21 FLOPs), LatentLoop consistently lowers validation NLL; at fitted NLL 1.988 it saves 19.0% and 7.9% compute vs. vanilla MoE and the looped baseline, while improving average scores on nine benchmarks at all sizes in 0-shot and 5-shot.

Links to arXiv, GitHub, HuggingFace, and project page included.

More from Infra