Study: Higher pre-RL accuracy from distillation doesn't guarantee higher post-RL accuracy

teortaxesTex · x · 2026-09-24

Base Labs published a study, "When does distillation help reinforcement learning?", examining how teacher-generated data warm-starts RL training.

Distillation is often treated as a crutch for open-source labs: training on a stronger model's traces gives a non-zero starting reward, unlocking the value of RL environments.
Key finding: higher accuracy before RL (via distillation) does not necessarily yield higher accuracy after RL.
The benefit of distillation depends on student model size, the amount of distillation data, and task category; the paper also analyzes starting-policy differences, next-token entropy shifts, and RL steps needed to hit target accuracy.
Includes a full technical appendix covering teacher data prep, task difficulty selection, RL objectives, and comparisons at equal numbers of RL updates.
A quoted comment argues that the most consequential domains—RSI, cyber, bio—won't need distillation at all, since kernel engineering and CVE search are natural to automate.

More from Research