Princeton/Penn's Recurrent Looped Transformer hits 100% on 256-bit parity extrapolation
TheTuringPost · x · 2026-10-17
Princeton and UPenn researchers propose the Recurrent Looped Transformer (RLT) (arXiv:2610.07591), which gives Transformers a running state carried across tokens.
- Architecture: layers are split between a parallel causal encoder and a recurrent decoder; at each token the decoder merges encoder output with the previous token's final decoder state, so computation depth grows with sequence length at fixed per-token cost.
- Results: trained on ≤40-bit data, some RLT splits extrapolate parity to 256 bits with 100% accuracy across three seeds while a standard Transformer stays at chance; on swap-based S5 permutation tracking at 8x training length, RLT reaches 97% vs <1%; modular arithmetic beyond training lengths hits up to 93% vs 33%.
- Ablations: removing the feedback drops everything to chance; updating feedback once per 4-token chunk keeps 64-bit parity at 99% via intra-chunk parallelism, but permutation tracking needs per-token feedback (chunking drops it from 100% to 20%).
Takeaway: fixed per-token depth is structurally ill-suited for state tracking, and recurrent feedback is an effective fix.
More from Research
- DivAlgo benchmark of 7,623 problems shows 24 LLMs fall short of human algorithmic diversity in code — TuhinChakr · 2026-10-17
- Brute-forcing benchmark-style queries won't train retrievers that generalize — bclavie · 2026-10-17
- Zephon brings elastically deterministic data loading to TorchTitan with identical batches across GPU counts — josh_wills · 2026-10-17
- Can your retriever handle tip-of-the-tongue queries? A test of intent depth — bclavie · 2026-10-17
- Zephon x TorchTitan: bit-identical global batches when resuming training on different GPU counts — PyTorch · 2026-10-17
- Physical confinement-driven cell-cycle regulation study published in Nature Communications — wgilpin0 · 2026-10-17