TRL makes score centering the default for async GRPO: FP8 samplers now train like bf16 (0.72)
_lewtun · x · 2026-10-27
Hugging Face TRL merged PR #7760, making score centering + masked importance sampling the default objective in the async GRPO trainer, matching the approaches in miles and Modal's mixed-precision runs.
The problem: in async GRPO, old log-probs come from the vLLM sampler, so the importance ratio mixes staleness with numerical mismatch between server and trainer. With a lower-precision sampler, the clipped objective collapses — with an FP8 vLLM sampler and bf16 trainer, reward fell from 0.60 to 0.34 (FP8 weights) or 0.10 (FP8 weights + KV cache) by step 300, vs 0.71 with a bf16 sampler; capping or masking the ratio alone didn't help.
With score centering enabled, the FP8 sampler trains exactly like bf16 (0.72), with strong stability across staleness and quantization settings. Based on paper 2609.20807.
More from Infra
- AMD MI355X hits 2,100+ tokens/sec per user on DeepSeek V4.1 Flash inference — JiaZhihao · 2026-10-28
- Strata DualGPU runs two mismatched NVIDIA GPUs for 5.6x faster local decoding — GabGarrett · 2026-10-28
- NVIDIA's first Rubin NVL72 Kimi K3 submission hits 850 tok/s/user at 2.8T params — scaling01 · 2026-10-28
- Oracle may now offer the most compute diversity of any cloud, says Bajarin — BenBajarin · 2026-10-28
- Local LLM power user's 12M tokens/day humbled by one Cursor profile burning 1.5T/month — perelmanych · 2026-10-28
- 96.8% of eval labels came from the model itself — fixing bias cut costs 86% — AWS ML Blog · 2026-10-28