Agon trains reasoning by having models grade each other
Vladislav Beliaev · hf · 2026-07-20
Agon: rival-grading RL for reasoning
The paper argues that verifiable-reward RL methods like GRPO only grade the final answer, which can push models to write more without improving thinking quality. To address that, it introduces Agon, a competitive cross-model RL setup where two models act as each other's graders.
How it works
- Both models solve the same problem.
- In alternating roles, one model drafts a solution while the other reads it and solves it.
- Each model is rewarded for out-reasoning the other, so reasoning quality is judged implicitly during training.
- No process labels or reward model are required.
Why it matters
Because both models are being optimized, each faces a progressively stronger rival, something single-model RL cannot provide. At inference time, the pair is used as a two-stage cascade: one model drafts, the other answers after reading the draft.
Reported results
- On the hard split of DeepMath with Qwen3, Agon roughly doubles GRPO's pass@1.
- The gain is about 8× larger than an untrained Mixture-of-Agents pass over the same base.
- The pattern also holds on competitive-programming code and across model families including Qwen3.5 and Gemma 4.
The authors say the next step is to let the models reason together in latent space instead of text.
More from Research
- Linear Digressions returns with a new season of audio essays on AI agents — ChrisGPotts · 2026-07-21
- ARISE study tested 45 AI clinical tools in 1,100 consult cases — HealthcareAIGuy · 2026-07-21
- Async OPD distillation doubles throughput while matching synchronous math accuracy — _lewtun · 2026-07-21
- A forecasting lesson on why R-squared alone led to overfitting and worse predictions — mdancho84 · 2026-07-21
- Google DeepMind’s Project Genie talk shows how creatives feed into model research — alexanderchen · 2026-07-21
- Nat Lambert says RL distillation does not use the strongest models as teachers — natolambert · 2026-07-21