Weight-space blend of base and RL checkpoints restores the diversity RL removed: α=0.25 keeps 64-80% of pass@1 gain

Perfect_Power815 · reddit · 2026-11-19

The author observed that RL-tuned reasoning models win at pass@1, but when sampling many times the base model actually solves more problems — RL removed output diversity. The fix: a one-knob weight interpolation, W = Wbase + α(WRL − Wbase).

Tested on Qwen2.5-7B ↔ SimpleRL-Zoo-7B across six math benchmarks with up to 128 samples per problem:

It's a trade-off knob, not a free win; validated on one 7B math pair only. Key gotcha: mix in fp32 and save fp16/fp32 — the RL update is about one bf16 step of the base weights, so saving in bf16 throws most of it away. A 7B pair needs 60GB RAM (a streaming script in the repo uses much less). Paper, code, and project page are public; the author asks for replications on other pairs.

Original post →

More from Models

Models channel →