Full-sequence masking during SFT unlocks prompt infilling for diffusion LLMs, COLM 2026 paper finds
kastnerkyle · x · 2026-09-24
A Patronus AI paper accepted at a COLM 2026 workshop shows masked diffusion LLMs like LLaDA and Dream can't infill prompts only because SFT masks responses exclusively. Switching to full-sequence masking—masking prompts and responses jointly—lets models infer task-adapted prompts from few-shot examples that match or beat manual templates and transfer across models. The authors argue training practices, not architecture, are the bottleneck, and call on the community to release full-sequence-masked SFT checkpoints.
More from Research
- MEMOIR benchmark: 117 synthetic oncology patients to test AI clinical memory — jefrankle · 2026-09-24
- 1.7h of intervention data beats 21h of demos: finetuning π0.5 on manufacturing — DominiqueCAPaul · 2026-09-24
- Basecamp Research raises $140M Series C for AI models trained on Trillion Gene Atlas — dr_alphalyrae · 2026-09-24
- π0.5 finetuning on real manufacturing: 1 hour of clean data beat the previous 17 — DominiqueCAPaul · 2026-09-24
- Fine-tuning π0.5 on a real factory task: data diversity beats raw scale, hitting 98% success — DominiqueCAPaul · 2026-09-24
- Inside Anthropic's new molecular biology lab: Claude proposes, scientists verify — AnthropicAI · 2026-09-24