Causal masking isn't in the definition of attention—it's a trick for autoregressive training
antoine_chaffin · x · 2026-10-10
antoinechaffin expands on his earlier claim: the definition of attention in 'Attention Is All You Need' does not include causal masking. Causal masking is just a trick to make autoregressive generation learnable—a definitely weaker version of attention. He maintains that bidirectional attention is decisively better for discriminative tasks.
Related event: Meta researcher: attention was never defined with causal masking(3 posts)→
More from Research
- acceptodds builds a prediction market where researchers bet on ICLR 2027 paper acceptances — prof_kamilov · 2026-10-10
- Surge AI launches sudo L7: a benchmark testing whether coding agents can act like staff engineers — rmcwhorter99 · 2026-10-10
- Live feed shows what images AI agents use while hunting for new planets — BLUECOW009 · 2026-10-10
- Jeremy Avigad's slides on the future of mathematics in the age of AI — ChengleiSi · 2026-10-10
- Tetris RL experiment: pretraining caps what RL can reach — PPO can provably converge to a bad policy — shizhediao · 2026-10-10
- Models trained on different data converge to similar concept geometry, researcher argues — cephaloform · 2026-10-10