2016 D-NTM paper's NOP memory cell anticipated today's attention sinks, says author
caglarml · x · 2026-10-22
Author Çağlar Gülçehre points out that attention sinks are "old is new again": his 2016 Dynamic Neural Turing Machine paper (arXiv:1607.00036, with Sarath Chandar, Kyunghyun Cho, and Yoshua Bengio) introduced a NOP memory cell that lets the controller attend to "nothing" when no meaningful read/write is needed — serving the same purpose of absorbing attention probability mass as today's transformer attention sinks. The D-NTM also introduced trainable content/address addressing and outperformed NTM and LSTM baselines on Facebook's bAbI tasks.
More from Research
- StreamEval: first benchmark for interactive streaming video generation, evaluating 37 models — yshan2u · 2026-10-22
- Nature's anti-portfolio: Fermi, Krebs and the first MRI paper were all rejected — luismbat · 2026-10-22
- Carbon-X training mix revealed: vertebrates dominate, human genomes favored in post-training — LoubnaBenAllal1 · 2026-10-22
- Larochelle amplifies debate: should papers be required to submit their code? — hugo_larochelle · 2026-10-22
- New psycholinguistics preprint dissects the weird emerging 'dialect' of frontier AI models — tallinzen · 2026-10-22
- Carbon-X author admits the DNA model is best kept to already-sequenced organisms — ATinyGreenCell · 2026-10-22