Looped Model Architecture: 8B Params Outperform 32B in Reasoning
SonglinYang4 · x · 2026-08-01
This research delves into Looped Models, which reuse the same weights across depth to achieve a better compute-parameter trade-off.
The authors conducted rigorous apples-to-apples ablations (matching both training and inference FLOPs) across architectures from Ouro to Huginn. Huginn performs better overall, with major gains attributed to the "loop-in-the-middle" (sandwich) design and input injection.
Experiments show that an 8B-A0.8B Huginn MoE trained on 500B tokens approaches or surpasses a 32B-A3.2B feedforward MoE on reasoning benchmarks like GSM8K (83.6% vs. 80.8%), while using 75% fewer resident parameters under matched compute constraints.
More from Research
- DeepMind wires Antigravity into AlphaFold to speed up science, finds novel antimicrobial peptide — pushmeet · 2026-10-16
- It's not gradient descent: burny_tech corrects the diagnosis to RLHF mode collapse — burny_tech · 2026-10-16
- A beginner-friendly introduction to causal inference kicks off, Part 1 — Nasereliver · 2026-10-16
- FBI method trains for Pass@K with one sample per update at 12.3% of the cost — EmtiyazKhan · 2026-10-16
- Berkeley researchers sample humanoid robot policies directly from weight space with diffusion — berkeley_ai · 2026-10-16
- OneWM: unified causal training shows world and action modeling improve each other — IAAI233 · 2026-10-16