Meta researcher: attention was never defined with causal masking
Meta researcher antoinechaffin argues that the original attention definition includes no causal masking—it is merely a technique for autoregressive training—and that bidirectional attention is stronger for discriminative tasks.
2026-10-10 ~ 2026-10-10 · 3 related posts
- Attention's definition doesn't include causal masking, argues researcher: bidirectional wins discriminative tasks — antoine_chaffin · 2026-10-10
- Reminder: attention itself doesn't require causal masking — it's just an autoregressive crutch — antoine_chaffin · 2026-10-10
1 near-duplicate retellings: antoine_chaffin