Register Tokens Let Diffusion LLMs Reason Across Cleared Context, Gains Up to 19.5 Pts
sprocket-lab · hf · 2026-09-17
Approach
Masked diffusion LLMs (dLLMs) generate text via iterative denoising with bidirectional attention. Cross-chunk reasoning normally requires keeping earlier output in context. This work instead uses a few register tokens: fixed-position tokens whose continuous hidden states are trained to carry reasoning progress.
Training and results
- Post-trained dLLMs decode a chunk, clear it, and continue from the prompt plus carried register state
- On LLaDA and Dream, registers beat discrete-text carry on every benchmark
- Up to 8.5 points on math and 19.5 points on code
- Especially effective for bounded code generation, where correct programs span multiple chunks
- Registers can be further refined with RL on long-horizon reasoning tasks
Related event: Register Tokens Let Diffusion LLMs Keep Reasoning After Clearing Context(2 posts)→
More from Research
- RoMem uses continuous phase rotation to expire stale agent memories, 2-3x better temporal reasoning — Roger_M_Taylor · 2026-09-17
- New Paper 'Instrumental Choices' Benchmarks When LLM Agents Take Policy-Violating Shortcuts — drbiomass · 2026-09-17
- APABOT: Open-source ParlAI chatbot prototype for autism spectrum conversation practice — AryHHAry · 2026-09-17
- LimiX-2 Debuts With Mechanism-Network Paradigm, Tops TabArena Tabular Benchmarks — stable-ai · 2026-09-17
- The Garden and the Tree: how AI will split mathematics into two distinct communities — yeastsplainer · 2026-09-17
- Why compositional diffusion fails: sampler, not model — MCMC samplers fix it — MarkNeumannnn · 2026-09-17