Loop only the middle layers: design detail shows 1.51-1.91x compute-equivalent gain vs 1.24-1.36x for last layers
nrehiew_ · x · 2026-11-21
nrehiew breaks down the architecture of the variable-looping model: only the middle blocks are looped (with attention residuals), while the first and last blocks run once, based on the intuition that the outer layers handle embedding translation rather than reasoning; residuals gate the mixing of initial inputs with each loop pass's output.
Supporting data: Inkling 2 Mini has 24 layers in four blocks of six, with the two middle blocks run three times. Looping the middle 12 layers beat looping the last 12 on 27 of 28 evaluations, and compute-equivalent gains were 1.51-1.91x for middle layers (7-18) versus 1.24-1.36x for last layers (13-24).
More from Research
- Looped transformers: more looping boosts reasoning but with diminishing returns past 3 passes and fewer CoT tokens — nrehiew_ · 2026-11-21
- Looped transformer lifts no-CoT time horizon from 16s to 37s, biggest gains on multi-hop reasoning — nrehiew_ · 2026-11-21
- Team releases TML-d24-varloop-research on Hugging Face to study looping's effect on CoT monitorability — nrehiew_ · 2026-11-21
- Released model is 1/3 of training size: pruned to 128 experts, healed with 1T distilled tokens plus 2.2T-token RL — nrehiew_ · 2026-11-21
- Mathematicians weigh in on Levent's Riemann proof: a wormhole through existing theory, not new theory-building — SamuelAlbanie · 2026-11-21
- New Company Agent Bench: Gemini Neon leads at 87.4%, simple API harnesses beat products — philhchen · 2026-11-21