Looping acts as a regularizer, delivering 2.2x compute efficiency over standard Transformers
SonglinYang4 · x · 2026-09-18
A shared research finding on looping architectures: in the data-constrained, multi-epoch setting, looping has a regularizing effect. At a fixed model size, validation loss decreases monotonically with more loops, and the optimal number of loops grows with compute — yielding 2.2x compute efficiency over a standard transformer even with tuned weight decay.
More from Infra
- BofA: B200 and H100 GPU rental prices hit new highs, up 47% and 31% YTD — TheZachMueller · 2026-11-02
- vikhyat predicts alternating compute oversupply and undersupply on the road to AGI — TheZachMueller · 2026-11-02
- PyTorch Mac downloads hit 3.29M in October, up 20.5% MoM — LysandreJik · 2026-11-02
- CVSS 9.8 ingress-nginx flaw lets unauthenticated attackers get RCE and steal cluster Secrets — evilsocket · 2026-11-02
- RAM prices tripled, so Tesla cut a third of memory from Optimus's brain — aakashgupta · 2026-11-02
- Liquid AI's 1.7B LFM3 becomes first local model to nail live fixtures lookup on a MacBook — NielsRogge · 2026-11-02