Google DeepMind trains Gemini models to self-correct with self-generated RL data

heghbalz · x · 2026-07-27

Google DeepMind proposes SCoRe for self-correction

Google DeepMind’s paper “Training Language Models to Self-Correct via Reinforcement Learning” introduces SCoRe, a multi-turn online RL method that trains on the model’s own self-generated correction traces instead of offline human or model-generated correction data.

Key points:

The authors argue that standard SFT on correction traces often fails because of distribution mismatch or behavior collapse.
SCoRe uses the model’s own data distribution during training to better match test-time self-correction.
Stage I stabilizes training with multi-turn RL and KL regularization so the first attempt stays close to the base model while the second attempt learns more.
Stage II adds a reward bonus for the improvement between attempts.
With Gemini 1.0 Pro and Gemini 1.5 Flash, the method reportedly improves self-correction by 15.6% and 9.1% on MATH and HumanEval respectively.

Original post →

Google DeepMind trains Gemini models to self-correct with self-generated RL data

More from Research