SPADE: Self-Play in Adaptive Synthetic Executable Environments
Bo Liu, Simon Yu, Yiding Jiang, Ao Qu, Andrew Zhao, Zichen Liu, Junsu Kim, Zijian Zhou, Seungone Kim, Tongzheng Ren, Mickel Liu, Hanfei Yu, Zhaorun Chen, Weiyan Shi, Paul Pu Liang, Luke Zettlemoyer, Yejin Choi, Natasha Jaques
cs.CL, cs.AI
2026-08-20
One LLM writes Gym environments and learns in them. At 30B, SPADE beats the best fixed pool by +5.3 on eight reasoning benches and +13.9 on ACEBench-Agent.
The bottleneck for stronger language agents has moved from pretraining text to verifiable training environments. Hand-written pools do not scale. Frozen synthetic generators do not either: once the model exhausts a fixed goal distribution, improvement stops. Most LLM self-play still emits a single problem with a terminal reward, not a full multi-turn MDP with state transitions, a reward function, and verification code.
SPADE turns environment design into a learnable post-training component. One LLM alternates as Environment Designer and Reasoning Agent. The designer writes executable Python with a Gym-style reset()/step() interface; the agent acts in those environments for verifiable reward.
Both roles share weights and update together with GRPO.
Each round the Environment Designer emits a complete environment (transitions and rewards live inside step()) plus a privileged hint: a short strategy note that must not give the exact answer. The Reasoning Agent plays each environment twice, with and without the hint. The gap in mean return is hint-based regret, the designer's reward. High regret means solvable with the hint and hard without it, i.e. on the frontier. High returns on both arms mean mastery. Low returns even with the hint mean the environment is intractable.
A pure adversary can make tasks unsolvable; a cooperative designer can inflate scores. Hint-based regret sits between those failure modes. Training also mixes in a difficulty anchor: full credit when the agent's win rate lands in [0.4, 0.6], weight 0.6, with floored regret at weight 0.4.
Stabilizers include per-role advantage normalization and a delayed designer update, corrected with truncated importance sampling. Candidates must pass syntax and execution checks before entering the pool.
Ungrounded self-play collapses onto the model's own favorites. Every round the designer conditions on freshly sampled pretraining documents: 10k math plus 5k science from DCLM and MegaScience for games, 15k Nemotron code documents for tool use. An environment memory stores past regret scores and skill tags, so high-regret items become seeds and too-easy or too-hard items become negatives.
Three Qwen3 backbones, 400 training steps each. The games setting compares against two matched-budget fixed pools: official RLVE, and a GPT-5.5 pool generated once and frozen. Eight held-out benchmarks cover contest math, science, code, and four Reasoning-Gym procedural skills.
| Model | 8-bench avg | vs base |
| Qwen3-30B-A3B base | 50.2 | 0 |
| Fixed-env GRPO | 51.4 | +1.2 |
| Fixed-env RLVE | 53.0 | +2.8 |
| SPADE | 58.3 | +8.1 |
At 30B, SPADE beats the strongest fixed pool by 5.3 points. Gains concentrate on procedural reasoning: Reasoning-Gym math 45.0 to 63.3, algorithmic 18.0 to 32.1, cognition 23.0 to 37.7. GPQA-Diamond 70.4 to 75.8, LiveCodeBench-v6 43.2 to 47.3. AIME is roughly preserved. Fixed-env RLVE drops AIME 2025 from 61.5 to 56.9 at 30B.
The same recipe on tool-use environments lifts BFCL v4 multi-turn 49.0 to 54.7 (+5.7), ACEBench-Agent 62.0 to 75.9 (+13.9), and τ²-bench by +3.6 at 30B. The size of the lift tracks how closely a benchmark matches the generated pattern of stateful multi-step tool calls; ACEBench is the closest.
Ablations on the 30B games suite: drop memory 53.2, drop corpus 53.5, freeze the designer and drop memory 40.5 (9.7 below the untrained base), swap in a frozen GPT-5.5 designer 53.0, replace hint-based regret with an EMA learning-potential signal 55.9. Without the corpus, diversity collapses (Vendi/n 0.68 to 0.04) and one stretch emits the same rotating-maze task 41 times. A six-skill curriculum scores 58.3; a two-skill run only 53.7.
Training environments are becoming expensive and scarce. SPADE shows the generator need not be frozen: the designer can be RL-trained with the solver, and the loop runs at 30B scale. Fixed pools get fitted by larger models; an adaptive curriculum keeps writing tasks at the frontier. Skills practiced in synthetic games transfer to held-out math, science, and code. On tool use, the benchmarks that share the generated interaction structure move the most.
This is an engineering recipe, not a new optimizer. GRPO is still human-written.
The authors list three. The designer cannot write environments more complex than the base model can express in context, so complexity tracks scale and generation budget. The learning rule itself does not evolve. Hint-based regret borrows PAIRED's minimax-regret intuition without a proof of optimal curricula, and evaluation is still fixed-task.
A few other gaps. At 4B and 8B the regret estimate stays negative for long stretches, so hints can mislead the current policy, even though those backbones still gain +5.2 and +5.7 over their bases. Tool-use comparisons to AgentScaler and EnvScaler are transcribed from other papers, with mismatched data and protocols. A large share of the games-setting held-out gain sits in Reasoning-Gym, which is closer to the training skill taxonomy than AIME is. A frozen GPT-5.5 designer recovers only about 35% of SPADE's gain and does not help code, so a stronger frozen generator does not substitute for co-evolution.