Looped transformers' real power: input-dependent iteration depth, not just tied-weight depth
kfountou · x · 2026-09-03
Responding to the claim that looped transformers are just feedforward networks made deeper via tied weights (as in Dehghani et al., 2019), the author argues the key distinction is variable, input-dependent iteration depth:
- A fixed number of unrolled iterations is equivalent to a deeper network with fixed computational depth; a true loop can take an input-dependent number of steps, letting a fixed-parameter model implement iterative algorithms whose runtime scales with the input.
- The hidden state carried across iterations also serves as working memory — more than merely adding a fixed number of layers.
Related event: Looped Transformer Debate: Just Deeper Networks or Real Recurrence?(8 posts)→
More from Research
- SimInfinity: open-source robotic simulation benchmark with 10K real-world tasks — joemeno · 2026-10-18
- Auction-based LLM routing paper finds fixed token pricing wastes 10%-71% in costs — _onionesque · 2026-10-18
- Meta paper introduces "agent plasticity": best-performing agent isn't the most efficient learner — rohanpaul_ai · 2026-10-18
- SAE weight editing steers robot behavior at inference time, IIIT Hyderabad paper lands IJCAI Safe Physical AI workshop — ponguru · 2026-10-18
- NTU-led survey reviews nearly 200 papers on affordance learning for embodied AI — jiqizhixin · 2026-10-18
- vega-U: a 14.3M decision engine that cold-attaches to any frozen LLM, 0.811 held-out accuracy — kalyan_kpl · 2026-10-18