Agon trains reasoning by having models grade each other

Vladislav Beliaev · hf · 2026-07-20

Agon: rival-grading RL for reasoning

The paper argues that verifiable-reward RL methods like GRPO only grade the final answer, which can push models to write more without improving thinking quality. To address that, it introduces Agon, a competitive cross-model RL setup where two models act as each other's graders.

How it works

Both models solve the same problem.
In alternating roles, one model drafts a solution while the other reads it and solves it.
Each model is rewarded for out-reasoning the other, so reasoning quality is judged implicitly during training.
No process labels or reward model are required.

Why it matters

Because both models are being optimized, each faces a progressively stronger rival, something single-model RL cannot provide. At inference time, the pair is used as a two-stage cascade: one model drafts, the other answers after reading the draft.

Reported results

On the hard split of DeepMath with Qwen3, Agon roughly doubles GRPO's pass@1.
The gain is about 8× larger than an untrained Mixture-of-Agents pass over the same base.
The pattern also holds on competitive-programming code and across model families including Qwen3.5 and Gemma 4.

The authors say the next step is to let the models reason together in latent space instead of text.

Original post →

Agon trains reasoning by having models grade each other

More from Research