Google DeepMind trains Gemini models to self-correct with self-generated RL data
heghbalz · x · 2026-07-27
Google DeepMind proposes SCoRe for self-correction
Google DeepMind’s paper “Training Language Models to Self-Correct via Reinforcement Learning” introduces SCoRe, a multi-turn online RL method that trains on the model’s own self-generated correction traces instead of offline human or model-generated correction data.
Key points:
- The authors argue that standard SFT on correction traces often fails because of distribution mismatch or behavior collapse.
- SCoRe uses the model’s own data distribution during training to better match test-time self-correction.
- Stage I stabilizes training with multi-turn RL and KL regularization so the first attempt stays close to the base model while the second attempt learns more.
- Stage II adds a reward bonus for the improvement between attempts.
- With Gemini 1.0 Pro and Gemini 1.5 Flash, the method reportedly improves self-correction by 15.6% and 9.1% on MATH and HumanEval respectively.
More from Research
- Genome language models uncover new class of reverse-transcriptase mechanisms — BrianHie · 2026-09-23
- Mathematician shares a cheap 4-step heuristic for hyperparameter tuning — dejanseo · 2026-09-23
- Burkov skew AI hype: 'deterministic LLMs' and 'first agents' are old tricks rebranded — burkov · 2026-09-23
- Continuous diffusion beats discrete on random k-SAT, proposed as standard benchmark — ArashVahdat · 2026-09-23
- Grady Booch: Contemporary AI Still Lacks Abductive Reasoning, Just 'Next-Token Prediction' — Grady_Booch · 2026-09-23
- AI solves Navier-Stokes-related problem as machines upend mathematics, New Scientist reports — burny_tech · 2026-09-23