Dev argues DeepSeek V4.1 could have used a non-causal encoder for stronger reasoning
mgostIH · x · 2026-09-11
Developer mgostIH offers an architectural hindsight take: DeepSeek V4.1's architecture could have paired a non-causal encoder with a decoder instead of making both causal — a design he believes would likely yield a much stronger reasoner. The full objective could have trained only on decoded tokens while still remaining causal. This is personal speculation, not official information.
Related event: Debate Flares Over Autoregressive vs Diffusion Training Efficiency(4 posts)→
More from Models
- Omni-InteractScope benchmarks proactive interaction in continuous omnimodal streams — Qwen · 2026-10-28
- Mistral releases Timestral time-series foundation models, tops GIFT-Eval — mistralai · 2026-10-28
- Gemini 4 Argon Hits 1.73% WER on OCR Benchmark, Surpassing Human Experts — gabriberton · 2026-10-28
- Meta's Omnilingual MT covers 1,600 languages, 1B–8B models rival 70B LLMs — kalyan_kpl · 2026-10-28
- Uncensored LFM3-1.7B GGUF weights released on Hugging Face — kalyan_kpl · 2026-10-28
- Token demand on OpenRouter grew 27x to 165T weekly, open models now serve 72% — FinanceYF5 · 2026-10-28