J-Zero: Unified Challenger--Solver--Judge Co-Evolution from Zero Data
Gyouk Chu, Myeongho Jeon, Eunho Yang
cs.LG, cs.AI, cs.CL
2026-08-27
J-Zero co-evolves Challenger, Solver, and Judge from zero data, beating R-Zero/G-Zero by 4.2 (verifiable) and 8.0 (open-ended) points, still rising at 10 iterations.
Zero-data self-play already works on math and code, where an executor or majority vote can score an answer. One model plays Challenger (harder questions) and Solver (better answers). Open-ended writing has no checker, so that loop dies.
The usual patch is a frozen reward model. The Solver saturates whatever distinctions that Judge already knows; after two iterations the signal flattens and scores fall. R-Zero never leaves verifiable domains. G-Zero builds preference pairs from Challenger hints and trains the Solver with DPO, skipping a Judge entirely. Both treat evaluation as a constant. J-Zero’s claim is sharper: a self-evolving model can improve only as far as its evaluator can see, so the evaluator has to move too.
Each KAIST iteration has three stages. Challenger and Solver update with GRPO (group-relative policy optimization). The Judge updates with a Bradley-Terry loss, which learns pairwise “A beats B” rather than an absolute score.
The constraint is the point: if labels came from the Judge, the loop would amplify its own bias. An external LLM (Claude Opus 4.8) audited the self-made pairs. Role-asymmetry chosen answers win more than 60% of comparisons at every iteration, falling from 87.9% to about 66% as tasks get harder and both policies struggle. Amplification pairs win only 21.1% at iteration 1, cross 50% from iteration 4, and later sit around 70–80%. The two curves cross mid-training, so the Judge is never left without a usable signal.
Policies start from Qwen3-4B-Base and Qwen3-8B-Base. The Judge starts from Skywork-Reward-V2-Llama-3.1-8B. Each iteration trains the Challenger for 5 steps, the Solver for 15, and the Judge for 8. Baselines are the untrained base, R-Zero, and G-Zero.
| Setup | Verifiable overall | Unverifiable overall |
| Qwen3-4B base | 44.91 | 9.58 |
| R-Zero | 49.64 | 12.66 |
| G-Zero | 47.41 | 10.89 |
| J-Zero | 54.38 | 20.81 |
| Qwen3-8B base | 50.67 | 13.23 |
| R-Zero | 54.99 | 15.54 |
| G-Zero | 53.07 | 15.31 |
| J-Zero | 58.55 | 23.41 |
On 4B, AIME25 moves from 6.67 to 15.83, BBH from 50.88 to 70.85, AlpacaEval 2.0 from 6.22 to 28.56. The abstract’s +4.2 / +8.0 versus baselines lines up with the +4.74 / +8.15 versus R-Zero in the tables. Freezing the Judge drops the 4B run to 52.72 / 16.37. Dropping amplification costs 1.64 overall; dropping role pairs costs 0.97. R-Zero and G-Zero peak at iteration 2 and then decline. J-Zero still rises at iteration 10. On RM-Bench, an off-loop reward-model test, Judge accuracy goes from 92.61 to 93.95, with Hard pairs from 85.08 to 89.85.
On open-ended tasks, a frozen reward model is the ceiling of self-play, not an implementation detail. Putting the Judge in the loop only works if its labels come from role structure and decomposition, not from its own scores. What transfers is that rule, not the three character names. Code and weights are public. The run stops at 8B base models, so this is not a drop-in post-training recipe.
Compute caps Challenger and Solver at 8B, with an 8B classifier reward model as Judge. Long-CoT post-trained reasoners are untested. The three roles do not share one generative initialization; how an LLM-as-a-judge would co-adapt inside the loop is left open.
A few numbers need a discount. Unverifiable eval itself uses a stronger LLM judge, a different standard from the training Judge, so some of the gain may be “better at pleasing the eval judge.” Role-asymmetry win rate falls as tasks get harder; late training leans on amplification, which fails if decomposition is sloppy. The G-Zero baseline keeps the original LoRA setup and a larger Challenger batch; the paper says full-parameter or a smaller batch hurt, but fairness is still arguable. There is no human preference audit, only Claude Opus 4.8.