Simplex Diffusion Models: lifting discrete diffusion to the probability simplex to stop information collapse
ArnaudDoucet1 · x · 2026-09-30
Researchers including Valentin De Bortoli and Arnaud Doucet propose Simplex Diffusion Models (SDMs), addressing information collapse in discrete diffusion where categorical sampling discards uncertainty at intermediate steps. SDMs lift the diffusion process onto the probability simplex to carry beliefs over categories.
- Closed-form reverse transitions, trainable with a simple cross-entropy loss
- No ODE integration unlike Dirichlet Flow Matching; includes a DDIM-like sampler with tunable stochasticity
- Carrying uncertainty across denoising steps mitigates information collapse
Results: 17.0 GenPPL on OpenWebText at 5.46 unigram entropy in 64 steps (near real validation data); 49.0% vs 45.8% on TinyGSM code generation without Self-Conditioning; distilled to 8 steps solves 32.1% of GSM8K, beating 128-step distilled discrete diffusion (21.4%).
Related event: Simplex Diffusion Models Tackle Information Collapse in Discrete Diffusion(2 posts)→
More from Research
- Small team's AI model wins ECMWF subseasonal forecast bake-off for weeks 3 and 4 — LesterMackey · 2026-09-30
- Researcher: ML peer review is becoming just LLM judgment, so why wait months — aran_nayebi · 2026-09-30
- LLMs Are Overconfident and Oblivious to Details; Abductive Preference Learning Fixes It — qi2peng2 · 2026-09-30
- HCOMP 2026 honorable mention paper: Bayesian cascade analysis of AI credibility indicators — windx0303 · 2026-09-30
- COLM 2026 keynote panel on Oct 7 to discuss frontier LLM advances in math and science — yuqirose · 2026-09-30
- Microsoft interns studied making computer-use agents robust to misaligned incentives — ZacharyHuang12 · 2026-09-30