Autoregressive vs Diffusion Training: The Data-Efficiency Math Behind Non-Causal Architectures
mgostIH · x · 2026-09-11
ThePremiseOfIt explains autoregression's core training advantage: a length-T context yields T input-output pairs, so training scales with total tokens rather than total contexts; non-causal training yields only one pair per context, far less efficient — which is why diffusion models are hard to train and block diffusion took off (T//n pairs, but capped generation length n).
mgostIH counters that papers suggest diffusion models are more data-efficient, needing roughly half the tokens on average under uniform prefix/suffix splitting. The thread also touches on whether DeepSeek's V4.1 architecture could have used a non-causal encoder+decoder, and Premise says solving this is part of their work.
Related event: Debate Flares Over Data Efficiency of Diffusion vs Autoregressive Models(2 posts)→
More from Research
- NVIDIA Open-Sources BioNeMo Inference Runtime to GPU-Accelerate Protein Models — AllThingsApx · 2026-09-12
- 31 million protein complex predictions run on NVIDIA BioNeMo, saving an estimated 1.35 GWh — AllThingsApx · 2026-09-12
- ApprenticeBench: Agents Continually Learn Real Jobs, Surpassing Human Pros — ysu_nlp · 2026-09-12
- Why RL Environments Work Better in 2026: Greenblatt's Two Reasons — dejavucoder · 2026-09-12
- Signal65 launches PINNACLE, an agentic AI benchmark scoring correct work over raw throughput — ryanshrout · 2026-09-12
- EvoHarnessBench: adding tools to agents can silently degrade abilities they already had — mohitban47 · 2026-09-11