Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills
Siyuan Huang, Pengyu Cheng, Haotian Liu, Tao Chen, Yihao Liu, Jingwei Ni, Shijie Zhou, Ziyi Yang, Gangwei Jiang, Mengyu Zhou, Yu Cheng, Xiaoxi Jiang, Guanjun Jiang
cs.CL
2026-07-25
Skill Self-Play turns agent skills into a co-evolving library that a proposer, solver, and skill controller refine in an RL loop, reconciling task diversity with verifiable rewards and lifting weak 8B backbones by up to 42.9 points.
LLM self-evolution wants models to generate their own tasks, solve them, and score them without human labels. Existing methods hit a wall. Environment-bound approaches (code executors, game simulators, retrieval engines) give precise rewards but lock learning inside that one environment. Open-ended generation widens the task space but has no reliable verification, so wrong rewards pollute training and synthetic data collapses after a few rounds. The question is how to stay open-ended and verifiable at once. The paper's bet is agent skills, modular bundles of know-how for a specific scenario.
Skill-SP splits self-play into three co-evolving roles: a proposer makes tasks, a solver solves them, and a skill controller manages the library. The library is the centerpiece. A skill is a structured interface carrying routing metadata, generation rules, hints, few-shot examples, executable validators, and usage statistics. The proposer does not invent tasks from nothing; it samples a skill by statistics and generates under that skill's constraints, paired with a skill-free exploration stream to prevent mode collapse.
A task is a pair (x, c): x is the prompt the solver sees, c is a hidden machine-readable contract (unit tests, reference answers) only the environment evaluates. The proposer reward is a gated medium-difficulty score: R = 1{valid} times (1 minus 2|successrate minus 0.5|). Medium difficulty keeps tasks at the solver's learning frontier; the validity gate blocks reward hacking, so the proposer cannot fake difficulty by emitting an unsolvable task. Validity requires schema compliance, contract checks from the skill's validators, and probe consistency, where K rollouts converge to the proposer's reference answer.
All three roles train under GRPO. Each round fixes the current solver as judge, updates the proposer to track the frontier, builds a solver curriculum from valid tasks, and the controller refines old skills, prunes saturated ones, and induces new skills from the exploration stream. Over five rounds the library gains about 20 new packages and grows to 86 active skills.
On tool calling (API-Bank L1-L3, BFCL across JS/Py/Java/live), already-competent models (Qwen3, Granite) gain 2.8 to 6.5 points. The headline is the turnaround on misaligned models: Ministral-3-8B jumps 42.9 points (20.7 to 63.6 average), Ministral-3-14B 42.3 points. Qwen3-4B-Instruct goes from 60.2 to 66.7. Unguided self-play gains less and sometimes degrades the model on subtasks.
| Model | Base | +Unguided SP | +Skill-SP |
| Qwen3-4B-Ins | 60.2 | 64.1 (+3.9) | 66.7 (+6.5) |
| Ministral-3-8B | 20.7 | 20.8 (+0.1) | 63.6 (+42.9) |
On logical reasoning (ZebraLogic), Unguided SP cannot bootstrap at all because it cannot synthesize valid puzzles, so there is nothing to compare. Skill-SP lifts all five models; Ministral-3-14B gains 12.0 points overall and over 35 on small puzzles.
Ablations are clean: dropping skill guidance loses 2.6 points; skill-only pools, uniform routing, and frozen libraries all hurt; freezing the proposer, the judging solver, or both loses 2.1, 3.0, and 3.2 points. Every co-evolving part earns its keep.
For anyone training agents, this is a workable open-ended self-play recipe that needs no bespoke environment per domain and does not bet on post-hoc filtering surviving the noise. The skill library is both structural prior and verification interface, and it grows itself. It is especially valuable for weak models whose tool-use is near zero, which ordinary self-play cannot move. It also extends Anthropic's agent-skill idea from an inference-time scaffold to a training-time curriculum engine.
The caveat is the word open-ended. Both test domains are highly structured and verifiable; whether this holds on fuzzier tasks like open writing or long-horizon planning is untested. The study covers five iterations and five 3B to 14B models, so library bloat and skill-quality drift at scale are open questions.
The authors concede that on the hardest Large/X-Large puzzles, weak models barely move, because pure self-play needs a minimum capability floor to bootstrap any learning signal, the same reason Unguided SP fails entirely on reasoning.
Open questions after reading: with only a few thousand tasks per round and five iterations, cost and stability at industrial post-training scale are unstudied. Skill induction uses the backbone itself (no stronger teacher); how well weak models induce skills is not isolated. All results use avg@8, and the benchmark set is narrow.