Looped Transformers revive interest in Deep Equilibrium Models, the 2019 infinite-depth trick
burny_tech · x · 2026-09-03
hayousoufiane notes Looped Transformers echo the Equilibrium Models literature, citing Deep Equilibrium Models (arXiv:1909.01377, NeurIPS 2019 Spotlight). DEQ finds hidden-layer fixed points via root-finding, equivalent to infinite-depth weight-tied feedforward nets, backprops analytically through the equilibrium with implicit differentiation, needs constant memory (up to 88% reduction), and matched SOTA on WikiText-103. Whether looping beats vanilla architectures remains unclear.
Related event: Looped Transformer Debate: Just Deeper Networks or Real Recurrence?(8 posts)→
More from Research
- SimInfinity: open-source robotic simulation benchmark with 10K real-world tasks — joemeno · 2026-10-18
- Auction-based LLM routing paper finds fixed token pricing wastes 10%-71% in costs — _onionesque · 2026-10-18
- Meta paper introduces "agent plasticity": best-performing agent isn't the most efficient learner — rohanpaul_ai · 2026-10-18
- SAE weight editing steers robot behavior at inference time, IIIT Hyderabad paper lands IJCAI Safe Physical AI workshop — ponguru · 2026-10-18
- NTU-led survey reviews nearly 200 papers on affordance learning for embodied AI — jiqizhixin · 2026-10-18
- vega-U: a 14.3M decision engine that cold-attaches to any frozen LLM, 0.811 held-out accuracy — kalyan_kpl · 2026-10-18