Autoregressive vs Diffusion Training: The Data-Efficiency Math Behind Non-Causal Architectures
mgostIH · x · 2026-09-11
ThePremiseOfIt explains autoregression's core training advantage: a length-T context yields T input-output pairs, so training scales with total tokens rather than total contexts; non-causal training yields only one pair per context, far less efficient — which is why diffusion models are hard to train and block diffusion took off (T//n pairs, but capped generation length n).
mgostIH counters that papers suggest diffusion models are more data-efficient, needing roughly half the tokens on average under uniform prefix/suffix splitting. The thread also touches on whether DeepSeek's V4.1 architecture could have used a non-causal encoder+decoder, and Premise says solving this is part of their work.
Related event: Debate Flares Over Autoregressive vs Diffusion Training Efficiency(4 posts)→
More from Research
- ParetoRSI-Gym benchmarks whether AI agents can recursively self-improve LLMs across domains — Haojin Wang · 2026-10-28
- Stanford's Ψ-0.6 is a unified vision-language world model with adaptive reasoning streams — Stanford · 2026-10-28
- Shanghai AI Lab's OpenESM shows energy-steered models can scale as foundation learners — Shanghai-AI-Laboratory · 2026-10-28
- CellTrack-Bench tests VLMs on microscopic cell tracking, exposing reasoning gaps — biohub · 2026-10-28
- Hybrid LLM agent scores 83.2%, beats nine VLA policies on new SO-101 robot arm benchmark — Truman Hickok · 2026-10-28
- DeepMind researcher: all of mathematics will soon be formalizable, with agents hunting proofs — ChrSzegedy · 2026-10-28