Reminder: attention itself doesn't require causal masking — it's just an autoregressive crutch
antoine_chaffin · x · 2026-10-10
Meta researcher antoinechaffin reminds people to reread "Attention Is All You Need": the original definition of attention does not include causal masking. Causal masking, he argues, is merely a trick to make autoregressive generation learnable — a strictly weaker use of attention. The observation echoes ongoing industry interest in diffusion LMs and alternatives to the autoregressive paradigm.
Related event: Meta researcher: attention was never defined with causal masking(3 posts)→
More from Models
- An AI answered with just a thumbs-up instead of a verbose reply — users love it — enggirlfriend · 2026-10-10
- Blind test: Astra beats Opus 5.5 80% of the time on hardcore coding — bindureddy · 2026-10-10
- Liquid AI's decision model d1 lands on Vercel AI Gateway with vision support — maximelabonne · 2026-10-10
- Open TTS Leaderboard adds Paradee-8M, a Kokoro-82M distill matching WER at 1/10 params — realmrfakename · 2026-10-10
- Kimi gateway latency test ranks GitHub first, Neon second, ngrok third — mariorod1 · 2026-10-10
- Grok in group chats is 'really nice', but users note messages are no longer private — Angaisb_ · 2026-10-10