From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement
Qinsi Wang, Jing Shi, Huazheng Wang, Kun Wan, Yiran Wu, Bo Liu, Qingyun Wu, Hai Helen Li, Yiran Chen, Handong Zhao, Wentian Zhao
COLM 2026
cs.AI
2026-07-27
SpyRL turns open-ended tasks into a 'Who Is the Spy' game with an environment-set spy, making votes verifiable judge-free, and beats self-evolution baselines on summaries, writing, and math.
RLVR (reinforcement learning with verifiable rewards) is the engine behind the recent reasoning-model boom; OpenAI o1 and DeepSeek-R1 both rely on it. The idea is simple. Math and coding have automatic correctness checks, and that verifier supplies unbiased, unlimited, nearly free supervision, so training can scale.
But RLVR works only where correctness is checkable. Open-ended tasks like summarization and creative writing have no verifier for true quality. The usual fix is to plug in a reward model or LLM judge, which reintroduces evaluation bias, caps the policy at the judge's competence, and adds inference cost on every rollout.
The authors borrow an old trick from self-supervised learning. Self-supervision does not chase human labels; it transforms the task into a pretext (masked-token prediction, say) whose labels are generated automatically from the data. RLSVR moves this trick from labels to rewards. It transforms an open-ended task into a proxy environment that secretly injects a latent variable z (say, who the spy is), known only to the environment. The rules pose a question about z that can be answered only from the task outputs, and the rule-based check on that answer is the verifiable reward. Because z is sampled by the environment itself, ground truth exists by construction, exactly like checking a math answer.
SpyRL instantiates this, inspired by the social-deduction game Who Is the Spy. Each round, n−1 civilians get full information and one spy gets corrupted information (say, 20% of the source masked), and all produce outputs on the same task. Players then vote on who the spy is. Because the spy identity is environment-assigned, the vote is verifiable. The performing reward is the inverse of suspicion votes received (better outputs draw less suspicion), and the detection reward is whether the vote was correct. The two stages alternate, with GRPO-style group-relative advantage within a group.
One crucial detail is RAE (role-advantage estimation). The spy is structurally disadvantaged, so its raw reward is not comparable to a civilian's; each role subtracts its own baseline before comparison. An ablation shows that without RAE the model regresses (average 50.4 drops to 37.5, worse than not training).
Backbones are Qwen3-4B and 8B, across three domains: summarization (GovReport, 20% masked), creative writing (WritingPrompts, 20% masked), and math (Nemotron-CC-Math, 40% masked), group size n=5, 100 epochs. Baselines are the R-Zero and Absolute Zero self-evolution frameworks. SpyRL takes the highest ROUGE-L on every summarization benchmark and both backbones (GovReport 36.7 vs 33.2 for Absolute Zero on Qwen3-4B), and on creative writing it leads every fine-grained dimension, with the largest gains in novelty and emotion, confirmed by a blinded human evaluation. On math and reasoning it is best across five math and two reasoning benchmarks, with AIME25 at 20.0 vs 13.4 and GPQA-D at 41.3 vs 35.3.
The paper also validates reward alignment: players who receive more votes have worse GPT-4o-ranked quality, so the vote proxy tracks true quality without an external judge. Against rubric-as-reward methods, SpyRL needs no judge and is cheaper (baselines spend roughly 200 USD with Qwen3.5-27B and 900 USD with GPT-4o as judges) while matching or beating them.
The verifiability boundary is exactly what has kept RLVR out of open-ended tasks. This is a clean, generalizable workaround: instead of approximating the missing quality judge, construct a new problem whose ground truth exists by construction. For anyone working on LLM self-improvement and post-training it is a direction worth following, and SpyRL's multi-player group competition suppresses judge bias better than a single proposer-solver.
The performing reward is still the vote proxy; the paper shows it correlates with quality but does not verify quality directly, so a cautious objection stands. Transfer is asymmetric: a math-trained model transfers negatively to writing (summarization and writing transfer positively to each other). Each task still needs a hand-specified degradation operator g, though an ablation shows 20% and 40% masking are nearly indistinguishable. Much of the evaluation relies on GPT-4o A/B judgments (alongside human eval), which carries some circularity risk.