Why Non-Causal Training Is Data-Inefficient — And What DeepSeek V4.1 Could Have Done
ThePremiseOfIt · x · 2026-09-11
ThePremiseOfIt explains why non-causal training is so difficult: autoregression converts each length-T context into T input-output pairs, so training scales with total tokens; non-causal training yields only one pair per context, far less efficient. That's a key reason diffusion models are data-inefficient, and why block diffusion gained traction (T//n pairs at the cost of limited generation length n).
He also speculates that DeepSeek V4.1's architecture could have used a non-causal encoder + causal decoder trained only on decoded tokens, potentially making it a much stronger reasoner. He notes overcoming this data-efficiency gap is part of what his company Premise works on.
Related event: Debate Flares Over Autoregressive vs Diffusion Training Efficiency(4 posts)→
More from Research
- GPT-6 Astra tops DDD benchmark for multi-step retrosynthesis, nearing specialist models — CatAstro_Piyush · 2026-09-12
- John Schulman: distillation is the main force fighting AI centralization — himanshustwts · 2026-09-12
- Talk at Mathematics for the Real World 2026 explores how AI and math enable each other — kuchaev · 2026-09-12
- Berkeley Professor Peng Ding Releases Free 490-Page Causal Inference Textbook on arXiv — udmrzn · 2026-09-12
- 'You're laughing? They're tormenting simulated fruit fly brains' goes viral — VoidStateKate · 2026-09-12
- AI Persistent Memory Is a State Management Problem, Not Just Retrieval — Raza2614 · 2026-09-12