SPICE: Self-Play In Corpus Environments Improves Reasoning
Bo Liu, Chuanyang Jin, Seungone Kim, Weizhe Yuan, Wenting Zhao, Ilia Kulikov, Xian Li, Sainbayar Sukhbaatar, Jack Lanchantin, Jason Weston
cs.CL
2025-10-29
Meta FAIR's SPICE trains one model as Challenger and Reasoner on corpus documents; Qwen3-4B-Base rises from 35.8% to 44.9%, beating ungrounded R-Zero.
Self-play wants a model to write its own problems, solve them, and get stronger. Close the loop and the trick fails in two ways. The question writer and the solver share the same weights, so items shrink toward what both already know. Hallucinations also compound, because neither side can check the world. R-Zero, a fully ungrounded cousin, sees pseudo-label accuracy fall from about 79% to 63% after a few iterations. Absolute Zero verifies with a Python interpreter, which pins the method to code.
SPICE treats a document corpus as the outside world. Questions must come with answers extracted from text. The solver never sees that text. The information gap is the corpus, not a story the model told itself.
One set of weights plays two roles.
The Challenger reads a document and emits a question plus a gold answer taken from that document. Formats are multiple choice or free-form (integer, expression, string). The pool is about 20,000 documents: Nemotron-CC-Math for math, NaturalReasoning (a DCLM slice) for general reasoning, each clipped to 5,992 tokens. Invalid questions get a -0.1 penalty.
The Reasoner sees only the question, thinks step by step, and boxes an answer. Math-Verify checks equivalence.
The Challenger is not paid for raw difficulty. Eight Reasoner samples produce a variance over correctness. Reward peaks at 1.0 when variance is 0.25, i.e. roughly a 50% pass rate, and decays for items that are trivial or impossible. The curriculum sits on the frontier, not past it.
Both roles update with DrGRPO: advantages are centered per role, without standard-deviation scaling. Training runs 640 steps, batch 128, temperature 1.0, on Oat actors with vLLM.
SPICE wins on four bases:
| Model | Base | R-Zero | Absolute Zero | Strong Challenger | SPICE |
| Qwen3-4B-Base | 35.8 | 39.5 | 40.7 | 43.0 | 44.9 |
| Qwen3-8B-Base | 43.0 | 46.3 | 46.5 | 45.6 | 48.7 |
| OctoThinker-3B | 14.7 | 20.3 | 21.7 | 21.0 | 25.2 |
| OctoThinker-8B | 20.5 | 29.9 | 29.4 | 28.2 | 32.4 |
Math rises about 8.9 points on average, general reasoning about 9.8. A Strong Challenger that freezes Qwen3-32B-Instruct as the question writer still loses to training both roles. On Qwen3-4B, AIME25 goes from 6.7 to 19.1, MATH-500 from 68.2 to 78.0, GSM8K from 72.6 to 92.7.
Hold the Reasoner at step 200 and later Challengers drive pass rate from 55% down to 35%. Hold the Challenger and later Reasoners climb from 55% to 85%. The two roles are tightening the screw together.
Ablations on Qwen3-4B-Base: NaturalReasoning alone 41.7, math corpus alone 43.2, both 44.9. MCQ only 42.0, free-form only 43.7, mixed 44.9. Swap the Challenger reward for Absolute Zero's "harder is better" and the score falls to 40.7. Drop corpus grounding and the run sits near 40.7; with documents, about 43.9.
The environment is now a pile of web pages, not a hand-built game or a code sandbox. Teams already doing RLVR can steal three pieces: pay the writer for a 50% pass rate, require answers that a document can verify, and estimate advantages separately per role.
This is still an incremental recipe. Corpus quality sets the ceiling, and 20,000 docs are not an infinite world.
There is no Limitations section. Two sources of documents leave the Challenger free to promote errors in the text to gold labels. String matching on free-form answers will miss paraphrases. Evaluation uses GPT-4o as an equivalence judge, and AIME is 32-sample while most other sets are greedy. OctoThinker moves the most; that may be a mid-training quirk, not a promise of +10 on every family. The same group wrote SPIRAL. This paper swaps zero-sum games for a corpus and never pits the two methods against each other.