Explorative Modeling worsens likelihood but improves low-step text diffusion sampling
dejanseo · x · 2026-08-04
Explorative Modeling improves low-step text diffusion sampling but hurts likelihood
The linked article studies Explorative Modeling (XM) on a 115.7M-parameter masked text diffusion model.
Setup
- Corpus: 579,508 documents, 1.81B characters
- Training split: deduplicated, with 4,000 validation and 4,000 test documents
- Tokenizer: 32,768-token byte-level BPE
- Model: 12 layers, 12 heads, dmodel 768, SwiGLU, RoPE, shared adaLN
- Training: continuous-time absorbing-state MDLM on a linear schedule
Compute
- Baseline: 20,000 steps in 8h 53m on one RTX 4090
- XM: 10,000 steps in 9h 01m at K=3
- The authors say XM costs 2× per step, so the runs are compute-matched
Results
- Test likelihood: baseline 3.844 NELBO vs XM 4.037
- Generation at T=8: XM is 15.2% better in generative perplexity
- By T=256, the gain shrinks to 0.3%
The takeaway is a clear tradeoff: worse likelihood, better low-step sampling.
More from Research
- DeepMind Paper Sparks Debate: LLMs Lack Abductive Leaps, Need World Models — theomitsa · 2026-08-04
- Hugging Face Journal Club: Joint Scaling Laws for Pre-training & RL — Hugging Face · 2026-08-04
- Hugging Face Alzheimer's Agent Challenge surpasses 200 submissions — lvwerra · 2026-08-04
- LFM2.5-2.6B released: full agent training pipeline compressed into 2.6B parameters — SergioPaniego · 2026-08-04
- NeurIPS 2026 Call for Papers: Robot Learning with World Models — shaohua0116 · 2026-08-04
- Profluent's New CRISPR Approach Expands Targetable Mutations by 10X — nathanbenaich · 2026-08-04