Towards Full Pipeline FP8 Reinforcement Learning for LLMs
Fanchao Chen, Ziheng Jiang, Ziyun Wei, Zheng Zhong, Du Li, Chi Zhang, Haibin Lin, Shivaram Venkataraman
cs.LG, cs.AI
2026-09-19
FP8 full-pipeline RL over-clips negative tokens via ratio noise. Calibrated Clipping matches BF16 clip quantiles, lifting Qwen3-8B from 46.1 to 55.9 avg at up to 1.5× throughput.
PPO-style RL already stretches LLM reasoning. The efficiency bet now is to run both rollout and training in FP8. FlashRL showed FP8 sampling is about 30% faster, and truncated importance sampling (TIS) patches the probability gap between the sampler and the trainer. Jet-RL and Unified FP8 then quantized the trainer as well, shrinking the precision gap so the setup looks more on-policy.
Once the training backend is FP8, that story breaks. UW-Madison and ByteDance Seed train Qwen3-8B-Base on DeepScaleR and watch entropy explode after roughly 100 steps, with garbled, fragmented completions. BF16 training stays calm, including when rollout is FP8. Tensorwise, rowwise, and blockwise scaling all fail; finer grains only soften the spike. PPO clip fractions sit at 6.68×, 5.48×, and 3.94× the BF16 baseline.
TIS is on the whole time. The train-inference mismatch is not the remaining bug. The trust region itself warps in quantized space.
PPO and GRPO clip on the importance ratio r = πθ / πold. Positive-advantage tokens lose their gradient once r ≥ 1+ε; negative-advantage tokens lose it once r ≤ 1-ε, with ε usually 0.2. Negative updates are how the policy unlearns bad tokens.
In a full FP8 pipeline both πθ and πold come from quantized forwards. Per-probability error versus BF16 is small. Dividing them into a ratio inflates that error by 1.7× to 2.9× (2.90× / 2.08× / 1.65× for tensorwise / rowwise / blockwise). Late in training about 70% of over-clipping hits the lower bound 1-ε, and nearly 90% of those tokens sit in responses with entropy above 1.0. Only 6.64% of those garbled samples carry a positive advantage. They should be penalized. FP8 often drops their ratio below 0.8, zeros the gradient, and lets garbage accumulate until entropy blows up.
Relaxing the lower bound from 0.8 to 0.6 stops the surge. It also over-penalizes: entropy saturates early, and both reward and response length lag BF16. A looser floor by itself is the wrong fix.
Calibrated Clipping rebuilds the trust region in two steps:
Every 20 steps, two forward-only BF16 shadow passes run on master weights. Search ranges are [0.5, 0.9] for L and [1.2, 2.0] for H, step 0.02. A smooth update caps per-step moves at δL=0.05 and δH=0.1. Bounds start at [0.6, 1.8]; GRPO uses reference bounds [0.8, 1.24] to avoid entropy collapse. Once training is stable the calibrated bounds barely move, so a BF16 pass every step is wasted work.
Stack: VeRL, vLLM, TorchAO, FlashRL's FP8-rollout patch, TIS on. Reported checkpoint is the best average.
GRPO on DeepScaleR, eight reasoning benches (AIME24/25, AMC23/24, MATH-500, Gaokao, Minerva Math, OlympiadBench):
| Setup | Qwen3-8B avg | Qwen2.5-32B avg |
| BF16 train+rollout | 57.6 | 51.9 |
| BF16 train, FP8 rollout | 58.2 | 51.2 |
| tensorwise FP8 | 46.1 → 55.9 after calibration | 49.1 → 51.1 |
| rowwise FP8 | 47.0 → 56.5 | 49.0 → 51.7 |
| blockwise FP8 | 54.1 → 58.6 | 50.8 → 53.4 |
Vanilla FP8 costs the 8B model 11.5 / 10.6 / 3.5 points. After calibration, blockwise reaches 58.6 versus BF16's 57.6; tensorwise and rowwise still trail by 1.7 and 1.1. The 32B run has shorter responses and milder entropy spikes; calibrated blockwise hits 53.4, 1.5 above BF16.
DAPO on Qwen3-14B-Base, AIME24 Avg@32: BF16 is 50.9. Vanilla FP8 falls to 35.7 / 38.1 / 41.6; calibration recovers 47.9 / 46.5 / 47.4 (+12.2 / +8.4 / +5.8 vs vanilla, still 3.0 to 4.4 below BF16). DAPO's higher ceiling (reference [0.8, 1.28]) lets even BF16 entropy drift up, so FP8 blows earlier.
Training-phase throughput on an offline TorchAO benchmark, excluding the periodic BF16 reference passes: tensorwise up to about 1.5× BF16, blockwise about 10%-20%, on top of the 30% generation speedup prior FP8-rollout work reported. The 32B / 16K setting OOMs in that throughput test.
In the appendix, CISPO on the same rowwise-FP8 setup collapses around step 300 with reward at 0. BAPO drifts its bounds to [0.9, 3.0] after about 90 steps and also dies. Recalibrating every 10 / 20 / 40 steps scores 58.31 / 56.51 / 57.91 on 8B; init [0.6, 1.8] versus [0.8, 1.2] barely matters. On the Eurus coding split, calibrated averages rise from 45.77 / 46.00 / 46.38 to 47.88 / 48.58 / 48.75 against BF16's 49.61.
FP8 post-training RL is not finished once TIS closes the sampler-trainer gap. After both sides are FP8, the clip bounds themselves drift. Calibrated Clipping leaves the objective alone and retunes two numbers, using an occasional BF16 forward as a ruler.
Integration is cheap: VeRL plus TorchAO, a calibration every 20 steps. Tensorwise buys the most throughput and still trails BF16 by about 1.7 points on 8B; blockwise is the score-first choice. This is an engineering repair that makes a crashing low-precision path usable, not a new RL algorithm.
The throughput numbers omit BF16 shadow passes, so real end-to-end speedup is smaller, and the paper never reports a wall-clock that includes calibration. Calibrated DAPO still sits 3 to 4.4 points under BF16; 8B tensorwise and rowwise never fully catch up. Hyperparameters were "not explicitly tuned." Only token-level GRPO and DAPO are covered; whether GSPO-style sequence clipping suffers the same ratio noise is left as future work. CISPO and BAPO were compared only in one rowwise-FP8 setting. Milder 32B entropy spikes are blamed on shorter responses, with no rerun on a long-response large model. Calibration needs BF16 forwards on master weights, so a pure-FP8 checkpoint with no high-precision copy cannot use it.