New Constrained Decoding Method for Diffusion Language Models
机器之心 · wechat · 2026-07-15
Diffusion language models are prone to syntax errors when generating formal languages like code, JSON, and SMILES. This paper introduces **LAVE (Lookahead-then-Verify)**: it first uses the diffusion model's probability distribution over intermediate `[MASK]` tokens for lookahead completion, then employs a context-free grammar parser to verify if these completions can still be expanded into valid outputs, thereby providing syntactic constraints for the generation process. The paper notes that existing parsers struggle with incomplete prefixes containing `[MASK]`, and brute-force enumeration of completions is infeasible. LAVE's approach is to intercept the current prefix needing verification after each new token is generated, sample several high-probability candidates for the remaining `[MASK]` tokens, and feed them to the parser for parallel checking; if any candidate passes verification, the token is accepted; otherwise, it is rejected and retried. Experiments covering LLaDA-8B, LLaDA-1.5, Dream-7B, and DiffuCoder-7B show significant improvements in syntax correctness across C++, Java, Go, JSON, and SMILES tasks, averaging close to 100% under multiple settings. Functional correctness also improved simultaneously; for example, Dream-7B's C++ functional correctness increased from 25.6% to 33.5%. Additional inference overhead is generally small, with JSON tasks seeing an average time increase of only about 3%; SMILES tasks even experienced a drop in inference time due to the reduction of irrelevant natural language outputs. The paper has been accepted by ISSTA 2026, and the code is open source.
More from Research
- Draft paper uses Markov-chain eigenfunctions to build partitions and speed up sampling — michaelchchoi · 2026-07-21
- Autoresearch proposes packaging ML runs as studies with questions, analysis, and code diffs — morgymcg · 2026-07-21
- GitHub repo adds lightweight ternary QAT for Prism-ML Bonsai models — terminoid_ · 2026-07-21
- Qdrant co-hosts a Munich meetup on search, retrieval, and agentic RAG on July 23 — qdrant_engine · 2026-07-21
- GigaChat Audio targets long-form audio grounding with timestamps across 120-minute inputs — ai-sage · 2026-07-21
- Paper models Transformer components as stochastic geometry and tests five architectures — Zhihua Liang · 2026-07-21