PACT: giving intermediate draft steps of diffusion language models informative RL training signals
burkov · x · 2026-11-12
Diffusion language models generate answers through a sequence of partial drafts, revealing several tokens at a time rather than writing left to right. This speeds up generation but complicates RL training: the probability of a finished answer is hard to compute exactly, and whole-answer rewards can't distinguish which intermediate decisions helped or hurt.
The article introduces PACT, which jointly learns the model's response policy and a score for each unfinished answer. The score estimates the promise of a partial answer, anchored to the completed response's reward. During training, PACT encourages each generation step's preference change relative to the original model to match the anchored reward change, yielding more informative signals for intermediate steps.
More from Research
- Caine Prize investigated its AI-controversy winner — and exposed detector blind spots — paulnovosad · 2026-11-12
- Open Dataset Teaches AI to Design Atomically Precise Diamond NEMS Parts — _Stocko_ · 2026-11-12
- CoRL 2026 SPIN workshop talk: Learning by Trying, physical reasoning from puzzles to contact — leto__jean · 2026-11-12
- Harvard mathematician Dan Freed on QFT through a mathematician's lens, 475 AI-written math papers, and math after AI — TOEwithCurt · 2026-11-12
- Claude scans 126,348 ICLR 2027 reviews: ~25K contain Markdown formatting — NielsRogge · 2026-11-12
- Graph RAG isn't a hype cycle: it fixes vector search's #1 failure mode, multi-hop questions — KarabasBarbados · 2026-11-12