Why Non-Causal Training Is Data-Inefficient — And What DeepSeek V4.1 Could Have Done
ThePremiseOfIt · x · 2026-09-11
ThePremiseOfIt explains why non-causal training is so difficult: autoregression converts each length-T context into T input-output pairs, so training scales with total tokens; non-causal training yields only one pair per context, far less efficient. That's a key reason diffusion models are data-inefficient, and why block diffusion gained traction (T//n pairs at the cost of limited generation length n).
He also speculates that DeepSeek V4.1's architecture could have used a non-causal encoder + causal decoder trained only on decoded tokens, potentially making it a much stronger reasoner. He notes overcoming this data-efficiency gap is part of what his company Premise works on.
Related event: Debate Flares Over Autoregressive vs Diffusion Training Efficiency(4 posts)→
More from Research
- HABIT-Bench: Best agent memory system scores just 33.61% at inferring user habits — SJTU · 2026-10-28
- MatAnyone-E brings 7.4x faster 4K video matting via sparse memory propagation — mmlab-ntu · 2026-10-28
- CodeCap-Bench: executable code beats natural language as video captions — Jiafu Tang · 2026-10-28
- NVIDIA's ThinkWorld learns robot skills from unlabeled human videos via chain-of-dynamics reasoning — nvidia · 2026-10-28
- Omni-InteractScope benchmarks proactive interaction in continuous omnimodal streams — Qwen · 2026-10-28
- Memory Tax study: dedicated agent memory systems often lose to naive RAG — UIUC-CS · 2026-10-28