Princeton & UPenn propose Recurrent Looped Transformer: 100% parity at 256 bits after training on 40
TheTuringPost · x · 2026-10-17
Princeton and UPenn researchers propose the Recurrent Looped Transformer (RLT), which splits its layers between a parallel causal encoder (building shared KV memory) and a recurrent decoder that carries a running state updated at each token, so computation depth grows with sequence length at fixed per-token cost. Trained on at most 40 bits, RLT generalizes parity to 256 bits with 100% accuracy (baseline Transformer stays at chance), reaches 97% on swap-based permutation tracking at 8x training length (vs <1%), and up to 93% on out-of-range modular arithmetic (vs 33%). Ablations show layer splits between reading and updating matter per task; chunking feedback to every 4 tokens preserves 99% parity but tanks permutation tracking from 100% to 20%.
More from Research
- Brute-forcing benchmark-style queries won't train retrievers that generalize — bclavie · 2026-10-17
- Zephon brings elastically deterministic data loading to TorchTitan with identical batches across GPU counts — josh_wills · 2026-10-17
- Can your retriever handle tip-of-the-tongue queries? A test of intent depth — bclavie · 2026-10-17
- Zephon x TorchTitan: bit-identical global batches when resuming training on different GPU counts — PyTorch · 2026-10-17
- Physical confinement-driven cell-cycle regulation study published in Nature Communications — wgilpin0 · 2026-10-17
- TANDEM mixes autonomous TAMP with on-demand teleoperation for 2.9× more demos — tomssilver · 2026-10-17