Looped Transformers Stuck at 8% on OOD Multi-hop; DiscoLoop Fix Hits ~100%
jasondeanlee · x · 2026-10-13
A retweet-and-quote thread covering the paper DiscoLoop on implicit reasoning in looped Transformers.
- Problem: Looped Transformers still perform poorly on multi-hop questions when the composition is out of distribution. For instance, a model trained separately on "Alice's son is Bob" and "Bob's wife is Carol" fails on "Who is the wife of Alice's son?"
- Mechanistic finding: The hidden state after the first loop decodes to "Bob" almost perfectly, yet is not very similar to Bob's word embedding (cosine ≈ 0.3).
- Fix experiment: Mixing Bob's embedding into that hidden state, with no retraining, jumps OOD accuracy from 8% to nearly 100%.
- Method: DiscoLoop, a residual design for looped Transformers that improves implicit reasoning.
The retweeter praises the paper for showing not just the proposed method but how the hypothesis was formed and verified with light experiments. The quoted post notes frontier labs are rumored to be adopting looped Transformers for their potential to unlock implicit reasoning.
More from Research
- AI-Assisted Stories Score Higher but All Sound Alike, Science Advances Study Finds — lmoroney · 2026-10-13
- GPT-4 story help boosts novelty 8.1% but homogenizes writing, study finds — lmoroney · 2026-10-13
- Princeton's Vertaix lab recruits fully funded PhD students for autonomous scientific discovery AI — davlanade · 2026-10-13
- One edit relights any number of views, including 3D Gaussian Splatting scenes — anand_bhattad · 2026-10-13
- SyncLight Accepted at NeurIPS: One-Step Generator Relights Multiple Views Without 3D — anand_bhattad · 2026-10-13
- Reward model shows near-perfect progress prediction on bimanual precision robot task — DominiqueCAPaul · 2026-10-13