Together AI paper: score centering cancels drift in off-policy RL under train-inference mismatch
PandaAshwinee · x · 2026-09-18
A new paper from Together AI researchers (Martin Marek & Max Ryabinin) targets RL instability caused by train-inference mismatch (TIM).
- The authors show instability stems primarily from drift: a persistent bias between training and inference engines that accumulates every step
- They derive an additive "score centering" correction that cancels this drift
- Tested on models from 0.6B to 30B parameters: score centering alone matches or beats importance-sampling-based methods, with the gap widening as mismatch grows
- Being additive, it composes with importance sampling, outperforming pure IS baselines in staleness experiments
More from Research
- Anthropic says it may imminently hit RSP threshold for AI-accelerated R&D, releases provable inference work — tobyordoxford · 2026-11-04
- Google scanned 1,033 real household products, packed into a 345 MB GLB for three.js — victormustar · 2026-11-04
- These 10 GPU optimization papers are why your model runs fast today — NandoDF · 2026-11-04
- INT8 isn't the problem: I benchmarked samplers on INT8 flow models — YakovSava · 2026-11-04
- Karpathy Boosts Open Hacker News Dataset: 10M+ Posts Usable as a Personal Knowledge Base — huangyun_122 · 2026-11-04
- PixelCraft: A Multi-Agent Framework Targets Multimodal AI's Chart-Reading Failures — burkov · 2026-11-04