arXiv: diffusion LM pretraining is far cheaper than the 16x consensus suggests
volokuleshov · x · 2026-10-16
- Paper 'Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models' (arXiv:2610.17590, Kuleshov et al.) revisits DLM scaling laws.
- The cited 16x used ELBO scoring and a full-sequence decoder-only setup, now out of step with practice.
- New protocol: exact likelihood under the model's own decoding order, block diffusion training, encoder-decoder architecture, practical block sizes; 1,700 models across 10 variants.
- Results: exact likelihood → 12.8x; block diffusion + encoder-decoder → 7.8x at block size 32, 4.6x at 8, 3.0x at 4.
- Takeaway: pretraining DLMs from scratch is far cheaper than believed, with an empirical rule for optimal decoder configuration.
Related event: Cornell-Amazon Study Debunks the 16x Compute Gap Claim for Diffusion LMs(6 posts)→
More from Research
- Open-source 12-chapter guide builds a modern LLM from scratch with every line commented — tom_doerr · 2026-10-16
- John Nash's 1998 Hierarchical Introspective Logics paper recirculates in AI circles — fkasummer · 2026-10-16
- Ex-OpenAI interp lead: most neurons in trillion-param models are now legible — CatAstro_Piyush · 2026-10-16
- AI safety researcher: most technical safety topics echo classic statistics from 20-70 years ago — _onionesque · 2026-10-16
- JevAgentHarness: a 4B decision model lets cheap agents offload tool choices from the LLM — youjiaxuan · 2026-10-16
- Google-led paper uses sequential auctions to organize decentralized LLM agent workflows — _onionesque · 2026-10-16