Loopback Transformer matches 3B model with 75% of parameters, gains mostly from decay
pbaylies · x · 2026-11-23
A thread on the Loopback Transformer paper (arXiv 2611.07226, UW-Madison/UCSB/Caltech/Microsoft): with 75% of non-embedding parameters it beats a depth-matched 3B standard Transformer on ClimbMix validation loss (1.9849 vs 2.0005 nats).
- From 191M to 3B, it achieves lower validation loss than standard Transformers and seven looped residual designs, with higher 13-task mean accuracy at 1.4B and 3B
- Strikingly, 93.7–94.6% of its gain over plain looping at 191M–374M comes from parameter-free iteration decay
- The paper decouples geometric decay of total update weight, recency priors, and attention-based allocation in looped architectures
More from Research
- RoboQuest benchmark: best AI agent solves only 23.2% of embodied exploration tasks — omarsar0 · 2026-11-23
- AI math swarm proves a 962-variable quartic is minimal; 4,973-variable version spans 1.62 MB of text — gleech · 2026-11-23
- Even OpenAI's quasi-Riemann result already spawned vast expository work, argues AI-proof skeptic — gleech · 2026-11-23
- CMU's ExploitBench ranks LLM cyber agents on a capability ladder, not pass/fail — goyal__pramod · 2026-11-23
- Harvard paper: higher SWE-Bench scores didn't translate into more merged pull requests — rohanpaul_ai · 2026-11-23
- Princeton's SSoN Benchmark: Best Vision Model Scores 39.7% vs Human 83.3% on Spatial Consistency — jiqizhixin · 2026-11-23