Weight-space blend of base and RL checkpoints restores the diversity RL removed: α=0.25 keeps 64-80% of pass@1 gain
Perfect_Power815 · reddit · 2026-11-19
The author observed that RL-tuned reasoning models win at pass@1, but when sampling many times the base model actually solves more problems — RL removed output diversity. The fix: a one-knob weight interpolation, W = Wbase + α(WRL − Wbase).
Tested on Qwen2.5-7B ↔ SimpleRL-Zoo-7B across six math benchmarks with up to 128 samples per problem:
- α=0.25: matches the base model's best-of-128 coverage while keeping 64–80% of RL's pass@1 gain, reaching RL's best-of-128 with 2.3–7.1× fewer samples.
- α=0.75: matches RL's pass@1, slightly better at large k.
- Majority voting without a verifier: stick with RL or α=0.75.
It's a trade-off knob, not a free win; validated on one 7B math pair only. Key gotcha: mix in fp32 and save fp16/fp32 — the RL update is about one bf16 step of the base weights, so saving in bf16 throws most of it away. A 7B pair needs 60GB RAM (a streaming script in the repo uses much less). Paper, code, and project page are public; the author asks for replications on other pairs.
More from Models
- Google's enviable position: Gemini on par with Opus plus a 13.7% Anthropic stake — NielsRogge · 2026-11-19
- Grok Bot tests in-chat restaurant reservations with a single booking card — XFreeze · 2026-11-19
- Tencent's open-source Hunyuan Hy4 flash: 12B active params, 1M context, 50T tokens day one — HeyAmit_ · 2026-11-19
- Gemini Neon adapts well across thinking effort levels, from quick mid-tier to frontier genius — intellectronica · 2026-11-19
- Anthropic widens lead to 65% of closed-weight spend on Vercel AI Gateway, open-weight models jump to 29% — cramforce · 2026-11-19
- Tencent Hy4 flash open-weights test: full data-to-deck workflow at $0.07 per 1M input tokens — JaynitMakwana · 2026-11-19