ORL trains diffusion LMs on their own trajectories, boosting a 4B model's throughput by 6.9x past speculative decoding

burny_tech · x · 2026-10-19

Researchers from Together AI, UIUC, UC Berkeley, Stanford and others introduce ORL (On-Policy Residual Learning), a framework for training diffusion language models (DLMs) directly on their own generated trajectories to learn residual context efficiently.

Key points

The paper argues residual learning is inherently on-policy; ORL uses an entropy-contraction-induced indicator to encourage sparsified predictions, plus co-designed async residual capture, block-diffusion attention, and packed compute-balanced training to scale it.
On a 4B DLM converted from Qwen3-4B, it improves tokens per forward pass by 6.2x and end-to-end throughput by 6.9x with the DLMRT engine at comparable or better quality.
In single-request H100 serving, it beats autoregressive speculative decoding (EAGLE3, DFlash) on the speed-quality Pareto frontier, running 1.74x faster than DFlash at matched accuracy under strict relaxed verification.
It transfers to existing DLMs: on LLaDA2.1-mini, 36 H100-hours and 4.2M self-generated tokens yield up to 1.5x higher TPF and 3x+ throughput over prior recipes; on DiffusionGemma it adds another 1.1-1.2x TPS.
Optimized models decode at 5,046 tokens/s on B200s, 7,708 tokens/s on eight B200s, and 2,788 tokens/s on a consumer RTX 5090.

Original post →

ORL trains diffusion LMs on their own trajectories, boosting a 4B model's throughput by 6.9x past speculative decoding

More from Infra