Cornell and Amazon revisit the 16x pretraining compute gap: diffusion LMs need only 3-8x
LucaAmb · x · 2026-10-16
Diffusion language models (DLMs) are widely said to need 16x the compute of autoregressive models (ARMs) to match likelihood. A Cornell/Amazon study sweeping 1,700 models shows the figure stems from an outdated scaling-law protocol (full-sequence decoder-only MDM scored by ELBO).
Under current practice — exact likelihood evaluation (DUEL), block diffusion with an encoder-decoder architecture, and practical block sizes — the gap shrinks dramatically: 12.8x with exact evaluation, 7.8x at block size 32, 4.6x at block 8, and 3.0x at block 4. The authors also derive an empirical rule for optimal decoder depth depending only on block size, verify the gap holds at 3x larger held-out budgets, and show likelihood gains transfer to downstream benchmark accuracy. DLM pretraining is far cheaper than assumed.
More from Models
- Are the fastest GPU kernels now written by AI? Kernels are hyper-verifiable, argues one dev — fleetwood___ · 2026-10-16
- Scale AI launches muse spark open: 816B-param MoE with 1M context, billed as top US open-weight model — jiayi_pirate · 2026-10-16
- NVIDIA HF listing for Qwen3.8-Flash-Next-FP4-Local leaks unannounced Qwen3.8 model — TheZachMueller · 2026-10-16
- Muse Spark Open, billed as the most capable US open-weight model, lands on Nebius day one — Arindam_1729 · 2026-10-16
- Model at capacity for a straight week drives frustrated dev back to Opus — joonasvirtanen · 2026-10-16
- Sentry CEO on expensing Claude subs: it's allowed, employers just want ZDR — zeeg · 2026-10-16