bf16 rounding silently discards most RL policy updates — a one-line fix gains 6.1 points
_AndrewZhao · x · 2026-11-02
A Stanford preprint reveals a hidden training-inference mismatch in LLM RL post-training: fp32 master weights are exported to bf16 via round-to-nearest, and at common learning rates most per-step updates are smaller than half a bf16 spacing and get discarded. On Qwen3-1.7B only 0.4–0.5% of served weights change per step, and the sampling policy lags the optimizer by 50–150 steps. The proposed Fair Cast fix uses unbiased rounding (stochastic rounding or error feedback) at export and computes training loss on the served copy — a one-line change, no new hyperparameters, costing 22–38ms per 1.7B export. Across 25 matched seed-pairs on Qwen3-0.6B–4B with AdamW and Muon it wins 18, ties 6, loses 1, with a mean gain of 6.1 points; on zebra puzzles it reaches 0.57±0.16 vs 0.33±0.10 for round-to-nearest. Code is open-sourced.
More from Research
- Scalable gradient attribution finds deceptively harmful training data, with a 0.6B model as monitor — PeterHndrsn · 2026-11-03
- Dynamical releases SDL-2: frontier agents make correct materials-science calls in only ~half of experiments — BenBlaiszik · 2026-11-03
- Follow-up: Astra's weight edits only worked on Qwen's visible flaw, not on Gemma 270m — teortaxesTex · 2026-11-03
- Zuckerberg's Biohub bets $1.8B on virtual cells: the bottleneck is data, not AI — HarperSCarroll · 2026-11-03
- EMBER: pretrained gene-centric foundation model predicts multigene perturbation effects — jmuiuc · 2026-11-03
- New preprint RLCE: reinforcement learning with comparative evidence for social intelligence — pliang279 · 2026-11-03