Looped Transformers Done Right: Middle-Layer Recurrence Beats Full Looping in Huginn-Style Design
burny_tech · x · 2026-11-01
A study from IFM, USC and CMU disentangles three design axes of looped language models: recurrence placement, input injection, and recurrent-state organization, training models from scratch at matched scale.
- The Huginn-style "prelude–recurrent block–coda" sandwich, looping only the middle layers, consistently beats looping the entire Transformer, especially on reasoning and context extraction.
- Input injection mainly helps knowledge retrieval; random init and separate H/L recurrent states give mixed or negative returns.
- At 730M params / 336B tokens, Huginn beats Ouro on all 10 benchmarks; the advantage transfers to MoE models at 8B resident params (32B unrolled).
- Under matched FLOPs, Huginn MoE uses 75% fewer resident parameters than a 112-layer feedforward MoE while scoring higher on GSM8K (83.6% vs 80.8%) and DROP, matching on MATH500.
More from Research
- If your AI scores 100%, your eval is wrong — an AI safety researcher's eval playbook — AI Engineer · 2026-11-01
- Control Vectors experiment: letting an agent manage its own activations — sloppenheimer · 2026-11-01
- FlashRSI open-sources RSI research: 12 algorithms, 30+ benchmarks, OOD gains still elusive — HuaxiuYaoML · 2026-11-01
- Claude designed scorpion toxin binders: 17 of 211 lab-tested designs bound — shoumikgoswami · 2026-11-01
- Carbon-X: midtrained Qwen3-14B beats every open LLM on DNA benchmarks — LoubnaBenAllal1 · 2026-11-01
- RL fails without midtraining: Qwen3-14B stays at chance on FrontierVEP — LoubnaBenAllal1 · 2026-11-01