Loopback Transformer matches 3B model with 75% of parameters, gains mostly from decay

pbaylies · x · 2026-11-23

A thread on the Loopback Transformer paper (arXiv 2611.07226, UW-Madison/UCSB/Caltech/Microsoft): with 75% of non-embedding parameters it beats a depth-matched 3B standard Transformer on ClimbMix validation loss (1.9849 vs 2.0005 nats).

Related event: Loopback Transformer Matches Standard Transformers with 40-75% of Parameters(2 posts)→

Original post →

More from Research

Research channel →