Loop only the middle layers: design detail shows 1.51-1.91x compute-equivalent gain vs 1.24-1.36x for last layers

nrehiew_ · x · 2026-11-21

nrehiew breaks down the architecture of the variable-looping model: only the middle blocks are looped (with attention residuals), while the first and last blocks run once, based on the intuition that the outer layers handle embedding translation rather than reasoning; residuals gate the mixing of initial inputs with each loop pass's output.

Supporting data: Inkling 2 Mini has 24 layers in four blocks of six, with the two middle blocks run three times. Looping the middle 12 layers beat looping the last 12 on 27 of 28 evaluations, and compute-equivalent gains were 1.51-1.91x for middle layers (7-18) versus 1.24-1.36x for last layers (13-24).

Original post →

More from Research

Research channel →