Loopback Transformer beats standard baselines with 40-75% fewer Transformer parameters
Crazy-Repeat-2006 · reddit · 2026-11-22
A new architecture paper proposes the Loopback Transformer:
- Motivation: Looped transformers cut parameter count by repeatedly applying shared blocks, but parameter sharing alone doesn't determine how repeated computation is reused. Viewing residuals as depth-memory, the authors note a full block output mixes its input and new update — weighting it also reweights history. Reading individual updates instead, with the shared stack's input on a separate path, gives independent control over input strength, total update weight, and allocation across updates.
- Mechanism: geometric decay sets total update weight and supplies a recency prior; attention learns the allocation.
- Results: On ClimbMix at reference scales from 191M to 3B with matched training tokens and executed depth, Loopback reaches lower validation loss than standard Transformer baselines using only 40-75% of their parameters (excluding embeddings and output head); higher mean accuracy across thirteen downstream tasks at 1.4B and 3B; and the best loss and task mean at 191M/374M/778M against both the standard Transformer and seven looped residual designs.
Code and paper are public.
More from Research
- String theorist Cumrun Vafa says Claude-assisted video may show first concrete evidence for string theory — calabi_and_yau · 2026-11-23
- Loopback Transformer matches 3B model with 75% of parameters, gains mostly from decay — pbaylies · 2026-11-23
- RoboQuest benchmark: best AI agent solves only 23.2% of embodied exploration tasks — omarsar0 · 2026-11-23
- AI math swarm proves a 962-variable quartic is minimal; 4,973-variable version spans 1.62 MB of text — gleech · 2026-11-23
- Even OpenAI's quasi-Riemann result already spawned vast expository work, argues AI-proof skeptic — gleech · 2026-11-23
- CMU's ExploitBench ranks LLM cyber agents on a capability ladder, not pass/fail — goyal__pramod · 2026-11-23