Burkov: 2026 is putting recurrence back into the Transformer it removed in 2017
burkov · x · 2026-09-06
Andriy Burkov (author of The Hundred-Page Machine Learning Book) highlights an architectural full circle: the 2017 "Attention is All You Need" paper worked by removing recurrence from the then-SOTA LSTM-with-attention architecture, keeping only attention so parallelism enabled much bigger models. Now in 2026, the field is adding recurrence back into the Transformer to make models smarter without making them bigger.
More from Research
- JEPA may keep unpredictable features: new paper gives set-level characterization — hisspikeness · 2026-10-23
- GAFU Uses Generation-Side Representations to Boost Fine-Grained Visual Understanding in Unified Multimodal Models — Tengju Ru · 2026-10-23
- NTU Study: Learned Unmasking Policies Improve Speed-Accuracy Trade-offs in Diffusion Language Models — nanyang-technological-university-singapore · 2026-10-23
- NBER releases 'The Economics of Transformative AI' edited by Agrawal and Brynjolfsson — JMateosGarcia · 2026-10-23
- Swapping only the eval harness moves results 30%: designing agent evals like a scientist — AI Engineer · 2026-10-23
- Jane Street and NVIDIA speed up Mamba-2 with custom cuTile kernels — blelbach · 2026-10-23