Opinion: Masked Language Modeling Was a Detour, Autoregressive Was Inevitable
jxmnop · x · 2026-08-04
An AI researcher argued that in the history of model architectures, Masked Language Modeling (MLM, used by models like BERT) was somewhat silly by construction. They believe that in most alternate universes, the field would have skipped this stage entirely and gone straight to the autoregressive approach that dominates today.
More from Models
- Analyzing Kimi-K3 Open Source: Data Flywheels Are the New Moat — karminski3 · 2026-08-04
- US Frontier Labs Tout 'Efficiency' as Chinese Models Drop at Throwaway Prices — deliprao · 2026-08-04
- Qwen 3.8 Coding Test: Nearly Matches K3 at Half the Price — bindureddy · 2026-08-04
- Fable and Gemini Vision Models Misled by Place Names in Geo-Guessing Fail — HanchungLee · 2026-08-04
- DeepMind Releases DiffusionGemma: Discrete Diffusion for Ultra-Fast Text Generation — deepmind · 2026-08-04
- Tencent bets on 295B Hy3 MoE model with a focus on agentic skills — emmanuelvivier · 2026-08-04