Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL
Yunhao Yang, Yuexin Bian, Yunjie Tian, Di Fu, Tianjin Huang, Yuanyuan Shi, Ziang Xiao, Nuno Vasconcelos, Yijiang Li
cs.LG, cs.AI, cs.CV
2026-08-18
Decoupled models reward each other with peer majority votes. Across seven text benchmarks Co-RL gains 3.0–8.6 points on average and matches or beats labeled GRPO.
Reinforcement learning with verifiable rewards improved reasoning, but the reward still needs a ground-truth answer. Harder problems make those answers expensive, and humans stop being reliable graders. Self-rewarding methods drop the labels: TTRL uses a model's own majority vote, Intuitor uses confidence, RENT uses entropy. The signal still comes from the policy being optimized. Systematic errors get locked in. Train long enough and diversity collapses, then the reward does too.
Co-RL asks a narrower question: without labels, where does a learning signal come from that is independent enough not to amplify the model's own bias.
The method takes the 1998 co-training idea into GRPO. Two or more models share no parameters and no gradients. They sample in parallel on the same unlabeled prompts. Each agent rolls out K completions, extracts final answers, and majority-votes a pseudo-label. That pseudo-label grades a different agent: match the peer's majority vote, get 1, otherwise 0. With two agents they grade each other. With three or more, votes travel on a directed ring.
Diversity is stacked on purpose, in three layers.
The theory collapses answers to correct versus incorrect. Under self-rewarding, the update follows whichever answer is currently more likely: if the chance of being right sits below one half, training pushes the correct answer down further. In Co-RL, agent A's update direction is set by B's majority vote. If the two success probabilities sum to more than 1, both drift toward the correct answer. A peer that is right on the problems you miss can pull you back.
Text runs train on unlabeled MATH levels 3 to 5 and evaluate seven math, code, and science benchmarks.
| Model | Base avg | TTRL | Supervised GT-Reward | Best Co-RL |
| Qwen2.5-3B | 40.7 | 47.3 | 47.4 | 49.3 (cross-family + rewrite) |
| Llama-3.2-3B-Instruct | 38.7 | 43.1 | 43.0 | 43.9 (cross-family + rewrite) |
Same-family twins already help: +8.0 for Qwen2.5-3B, +4.0 for Llama. Crossing families and rewriting prompts takes Qwen to 49.3, above the supervised 47.4. The same pattern holds at 7B/8B: Qwen2.5-7B goes from 49.0 to 53.6, Llama-3.1-8B from 44.7 to 47.7, the latter also beating its supervised 47.1. Average gains across four language models sit between 3.0 and 8.6 points.
Under the CoMAS protocol, Co-RL averages 62.97 against CoMAS at 58.94, a 4.0 point gap with half as many agents and no extra LLM judge. Jointly training three heterogeneous models lifts each by 7.8, 6.0, and 8.2 points, matching or beating each model's own supervised run.
On vision-language models from 2B to 12B, four multimodal math benchmarks improve by 2.3 to 7.2 points on average. Gemma-3-12B reaches 47.56 without labels versus 45.17 with them. Training curves for self-rewarding show collapsing reward variance, length degeneration, or divergence. Co-RL keeps group reward spread and completion length more stable.
Label-free reasoning RL now has a recipe that other labs can copy: stop grading yourself, pair with a peer whose mistakes do not match yours. Same-family copies already work. Crossing families and rewriting prompts is more stable. The cost is training two or three models at once; the paper uses one node of eight H100s, four per agent. Teams already running TTRL or GRPO can change the reward source without changing the optimizer. Pseudo-labels only check the final answer, so the recipe fits math and code. Open-ended writing is still out of reach.
The reward still asks whether the final answer matches a peer majority vote. Intermediate chain quality is ungraded. If both peers share an error, the pseudo-label is wrong. The theory treats answers as binary; real error modes are messier. Compute nearly doubles, and the paper does not sweep agent count or topology. On the three-agent HumanEval split, Qwen3-1.7B Co-RL scores 64.2 against supervised 70.1, so complementarity is not free. InternVL3.5-2B on MMR1 averages 45.15 for Co-RL, a shade under TTRL at 45.30. Cross-modal gains are not uniform across every table.