ORL hits 2,788 tokens/s on one RTX 5090, making diffusion LLMs faster than speculative decoding

arty_photography · reddit · 2026-10-17

Researchers from Together AI, UIUC, UC Berkeley, Stanford and others introduce ORL (On-Policy Residual Learning), an open-source method that dramatically speeds up diffusion language models, plus a custom inference engine dlmrt. Key results (single-request, batch=1):

2,788 tokens/s on a single RTX 5090 (LLaDA-2.1-mini + ORL, NVFP4)
7,708 tokens/s on 8×B200 (DiffusionGemma 26B MoE)
Up to 6.9× end-to-end throughput on QwenDLM-4B, with HumanEval pass@1 rising from 82.9% to 89.6%
On one H100: 1,298 tokens/s vs DFlash at 746 and EAGLE3 at 405 at comparable accuracy

How it works:

On-policy residual learning — the model carries forward previous predictions as residual context and refines them, trained on fresh trajectories from the current model to avoid stale-residual distribution mismatch; LLaDA-2.1-mini gains 3× throughput with only 4.2M self-generated tokens and 36 HGPU-hours of post-training.
Adjacent Entropy Contraction — each denoising step is trained to reduce uncertainty relative to the previous step, with loss gating for confident tokens, so more tokens finalize per forward pass.

dlmrt captures the whole denoising loop in a CUDA graph, supports NVFP4 on Blackwell consumer GPUs, and runs 1.2–1.6× faster than SGLang/vLLM in matched comparisons. Models, training code, eval pipeline and engine are all open-sourced.

Original post →

ORL hits 2,788 tokens/s on one RTX 5090, making diffusion LLMs faster than speculative decoding

More from Infra