~1,700-model study: diffusion LMs need only 3-8x AR compute, not the folkloric 16x
LucaAmb · x · 2026-10-18
A new paper from Guanghan Wang revisits diffusion language model (DLM) scaling laws using 1,700 models, debunking the folklore that DLMs need 16x the compute of autoregressive models to match likelihood. Under current DLM recipes the gap is 3-8x, and with the best recipes roughly 3x. Their earlier DUEL work plays a key role: using exact likelihood instead of the ELBO closes 30% of the gap. The result significantly improves the training-efficiency outlook for DLMs as AR alternatives.
More from Research
- Moritz Hardt: AI benchmarks are fundamentally broken but fixable — asusarla · 2026-10-18
- Attention head acts as a switch: mechanistic interp case flips model answer from No to Yes — CatAstro_Piyush · 2026-10-18
- Harvard's open-source ML Systems textbook hits 29.1K stars on GitHub — mdancho84 · 2026-10-18
- 25-author working paper examines feasibility of a hardwired pause on frontier AI training — wfithian · 2026-10-18
- AI meets archaeology: data science helps reveal past epidemics — ArtificialOther · 2026-10-18
- Weight decay scaling was in the original AdamW paper — first author Loshchilov is on X — SeunghyunSEO7 · 2026-10-18