Bitstream diffusion nearly matches autoregressive LMs in new entropy-gated paper
LucaAmb · x · 2026-11-01
Luca Ambrogioni announced a new paper with Sander Dieleman, "Towards Closing the Autoregressive Gap in Language Modeling via Entropy-Gated Continuous Bitstream Diffusion," calling bitstream diffusion "the next big thing."
- Continuous diffusion operating directly on bitstreams achieves strong language modeling performance, beating masked and uniform diffusion baselines
- Under their evaluation settings it essentially matches autoregressive models, closing much of the gap between diffusion and AR paradigms
- The method uses entropy gating to steer the denoising of the continuous bitstream (details in the paper)
More from Models
- Fine-grained MoE is the easiest win: DeepSeek-V3-style 128 experts cut loss by up to 6.2% — ivan_bezdomny · 2026-11-01
- Codex user down to 14% quota agonizes over spending last reset amid rumored reset — huangyun_122 · 2026-11-01
- SAM 3 Flags 25 Cells Experts Called Clean in Malaria Smears, Revealing Label Disputes — MaziyarPanahi · 2026-11-01
- DeepSeek v4.1 on 4x H200: 2,400 tps, $120k for 16 concurrent users — teortaxesTex · 2026-11-01
- Hesamation: Anthropic can compete with frontier rivals without shipping its best model — Hesamation · 2026-11-01
- Mystery Groot-280B-A16B Model Appears on Hugging Face; 19B Version Claims to Beat Qwen3.6-35B — openSourcerer9000 · 2026-11-01