GWU/Duke Paper Decomposes Training-Inference Mismatch in LLM RL, Proposes DLC Fix
George-Washington-University · hf · 2026-11-02
A George Washington University and Duke paper systematically analyzes training-inference mismatch in on-policy RL for LLMs: an inference engine samples responses while a separate trainer computes gradients, making training data off-policy.
- Decomposition: Evaluating the trainer at the sampler's weights splits each sampled-token error exactly into an engine's numerical discrepancy and a policy lag of the trainer, both tracked through training.
- Two sources, two fixes: The numerical discrepancy stays roughly constant—centering the score at each prefix removes its leading bias. The policy lag is created by the trainer's own updates, forming a feedback loop; importance weighting only helps while the lag ratio stays concentrated—on Qwen3-1.7B it fails once the ratio's effective sample size drops below one half.
- DLC: The derived discrepancy-lag correction centers the numerical discrepancy, clips importance ratios, and resynchronizes sampling weights, stabilizing training where single-source fixes break down.
More from Research
- Synedrion: a multi-agent LLM framework translating tax law into executable code — burkov · 2026-11-02
- RUC open-sources FlashRSI: one framework unifying 13 agent self-improvement algorithms — aigclink · 2026-11-02
- Evals want step-function tasks, training wants smooth pass@k curves — why the two goals conflict — JoshPurtell · 2026-11-02
- SemiAnalysis releases GDPx agentic benchmark; best model scores just 12%, Pichai nods — sundarpichai · 2026-11-02
- VisMatch v1.4.0 adds RaCo, UPAL, REDI-Match: test 50+ image matchers in 5 lines — gabriberton · 2026-11-02
- FrameForge Reuses Video Model Priors for Unified, Adaptive Conditional Image Generation — FrameXAI · 2026-11-02