LatentLoop: asymmetric compute for looped MoE saves 19% FLOPs
Zefeng He · hf · 2026-10-15
For looped MoE models that reuse the same block multiple times, researchers find the two loop visits are asymmetric: the first visit's residual scale is smaller, and shrinking it preserves or improves performance while the same reduction to the second visit hurts.
They propose LatentLoop:
- Compress the first visit's sequence, keep full resolution in the second, and reallocate saved compute to hidden size
- Share MoE weights across visits but use separate attention parameters for different sequence resolutions
- Across 15 configs (1.3B–7B, up to 10^21 FLOPs), it consistently lowers validation NLL; at NLL 1.988 it saves 19.0% compute vs. vanilla MoE and 7.9% vs. the looped baseline, improving 9-benchmark averages at all sizes
Asymmetric compute allocation looks like a promising direction for looped MoE.
More from Research
- 110-Page Survey From 6 Institutions Maps Memory Mechanisms in Autoregressive Video Generation — jiqizhixin · 2026-10-15
- Tinker ships Dimuon, a LoRA optimizer that beats AdamW in fewer steps on SFT tests — miramurati · 2026-10-15
- UW's COUNT stores n-gram counts on host memory, giving 1B LMs the equivalent of 57% more compute — UW · 2026-10-15
- Scale AI's MA-Bench tests LLMs on military-style mission analysis — the best model scores just 62% — Rodney Lafuente-Mercado · 2026-10-15
- ATAS framework uses world-state memory to keep long-form agentic video generation consistent — zju · 2026-10-15
- HumanoidToolBench benchmarks humanoid tool use from selection to mobile execution on real Unitree G1 — snupilab · 2026-10-15