Middle-layer looping yields 1.51-1.91x compute-equivalent gains, echoing J-Space theory
scaling01 · x · 2026-11-21
scaling01 reviews Thinking Labs' looped transformer experiment and shares his own findings on why which layers you loop matters.
- Thinking Labs' Inkling 2 Mini (24 layers in four blocks) loops the two middle blocks three times; looping the middle 12 layers beat looping the last 12 on 27 of 28 evals, with 1.51-1.91x compute-equivalent gains vs 1.24-1.36x for the last loop.
- His own experiments agree: looping the first or second half sucks, middle looping is solid, and slightly above the middle may be optimal.
- His theory ties this to Anthropic's J-Space work: reasoning happens entirely in the looping workspace layers, while early layers handle sensory/input and late layers handle motor/output — looping those would clutter them. Sonnet 4.5's J-lens CKA analysis supports the sensory/workspace/motor division.
More from Research
- Lean community certifies Riemann proof; honeypot fake keys expose breaches at three AI providers — LeviTurk · 2026-11-22
- Riemann hypothesis 'proven by Turkish mathematician using AI' claim mocked as the 'Organic Turk' — CatAstro_Piyush · 2026-11-22
- Andriy Burkov releases The Hundred-Page Language Models Book, hands-on with PyTorch — burkov · 2026-11-22
- Generative AI meets cardiology: Dagstuhl seminar bridges ML and clinical practice — StephanMandt · 2026-11-22
- SubRoformer hits SOTA music source separation with DSP subband filters — affige_yang · 2026-11-22
- Emily Riehl releases Lean game teaching dependent type theory at intro-to-proofs level — abhi9u · 2026-11-22