Attention's definition doesn't include causal masking, argues researcher: bidirectional wins discriminative tasks
antoine_chaffin · x · 2026-10-10
antoinechaffin argues in a two-part post that the definition of attention from 'Attention Is All You Need' does not include causal masking. Causal masking is merely a trick to make autoregressive generation learnable—and thus a strictly weaker version of attention. His claim: bidirectional attention is decisively superior for discriminative tasks, and that fact should not be denied.
Related event: Meta researcher: attention was never defined with causal masking(3 posts)→
More from Research
- Tree-like procedural 3D data is hard for LLMs to learn — and data is scarce — keenanisalive · 2026-10-10
- From Maxwell's field lines to Jim Blinn's blobby molecular visualization — keenanisalive · 2026-10-10
- Metaballs to marching cubes: a quick history of implicit modeling algorithms — keenanisalive · 2026-10-10
- Blobby modeling traces back to Maxwell's 1873 equipotential drawings — keenanisalive · 2026-10-10
- CMU prof's LLM-built implicit-surface 3D dragon: primitives as geometric prior — keenanisalive · 2026-10-10
- Blobby modeling traces back to Maxwell's 1873 equipotential drawings — keenanisalive · 2026-10-10