Loopback Transformer: separate controls over input strength and update weight beat baselines with 40–75% fewer parameters
uw-madison · hf · 2026-11-16
A paper from UW–Madison, UCSB, Caltech and Microsoft introduces the Loopback Transformer, unifying the view of residual looped transformers.
- Key idea: parameter sharing in looped transformers doesn't determine how repeated computation is reused. A full block output mixes input and new update, so weighting it also reweights carried history. Keeping the shared stack's input on a separate path and reading individual updates enables separate controls over input strength, total update weight, and per-update allocation.
- Mechanism: geometric decay sets total update weight and supplies a recency prior; attention learns the allocation.
- Results: on ClimBix, Loopback reaches lower validation loss than standard Transformer baselines with 40–75% of their parameters (excluding embeddings and output head), plus higher downstream accuracy.
More from Research
- HKUST's HALO collects whole-body mobile manipulation demos up to 2.9x faster than Mobile ALOHA — Robbyant-Research · 2026-11-16
- Researchers push back on autonomous labs: automating the easiest step isn't the hard problem — arjunrajlab · 2026-11-16
- LLM's self-written retry notes told it to repeat the move that lost the game — shoumikgoswami · 2026-11-16
- Survey maps how LLMs now deploy on edge devices via software optimization and hardware — kalyan_kpl · 2026-11-16
- Lagomics claims designed peptides cut time-to-human aging data from 10 years to months — niloofar_mire · 2026-11-16
- Free online textbook 'Bayes, AI and Deep Learning' spans probability to agents — burkov · 2026-11-16