Prompt2Skill: Unsupervised Skill Optimization From Natural Language Instructions
Bo Ni, Li Li, Ryan A. Rossi, Franck Dernoncourt, Tyler Derr
cs.CL, cs.AI
2026-09-30
Given only a task description, Prompt2Skill finds or creates training data and refines a skill against the target model's failures, +10.8 points over direct prompting on average.
Skills are markdown artifacts an LLM reads at inference time to pick up procedures and conventions it doesn't reliably hold in its weights, no weight updates required. They're mostly written by human experts: expensive, and never tuned for the model that consumes them, so a skill that helps one model can hurt another. Automated alternatives, Trace2Skill and SkillOpt, both assume a curated, in-distribution labeled training set, and SkillOpt's own results show that shrinking that set to a single example collapses the gains to a data-blind draft. Prompt2Skill targets the setting users are actually in: no samples from the task distribution, just a sentence like "a model that reads a passage and answers questions about it."
Two agents, no weight updates.
The Data Agent builds the data. It turns the description into a task specification: input structure, answer style, and an executable proxy metric such as normalized exact match. It then searches the Hugging Face dataset catalog and Wikipedia, and one neat trick is generating a hypothetical dataset card first and deriving search terms from it. Retrieved sources rarely fit the requested interface, so the agent writes a conversion plan that maps records into input-reference pairs, or synthesizes new items from retrieved material when nothing fits. Every item passes three checks: structural integrity, compatibility with the spec, and an answerability screen that discards an entire source if the seed-equipped model fails every sampled item. What survives is deduplicated into a pool.
The Skill Agent refines. Each round it runs the incumbent skill on a reflection set, hands the failures to a reflector model (GPT-5.5 in the experiments), and collects candidate rewrites. Each candidate is compared with the incumbent on the same fresh acceptance batch: items only the candidate solves count as wins w, items only the incumbent solves as losses l, and a rewrite is accepted when w > l and z = (|w−l|−1)/√(w+l) ≥ 1, taking the largest net gain. Fresh batches keep acceptance separate from the data the edits were proposed on. A frozen selection set is consulted exactly once, at the end, to choose among the initial and all previously accepted skills, so a late edit that generalizes worse can't throw away an early win. A theorem in Appendix A decomposes the gap between the exported skill and the best retained one into distribution mismatch, evaluator mismatch, and sampling error.
| Model | Avg relative gain | One highlight |
| Qwen3-8B | +29.1% | SearchQA 0.431 to 0.640 |
| Qwen3-32B | +26.1% | best on all four benchmarks |
| Llama-3.2-1B | +93.5% | SQuAD 0.207 to 0.492 |
| Claude Haiku 4.5 | +22.0% | SQuAD 0.416 to 0.793 |
| GPT-5.5 | +24.3% | SQuAD 0.651 to 0.852 |
Baselines are direct prompting, public off-the-shelf skills, and a skill authored once by an LLM. Across 19 model-benchmark pairs, Prompt2Skill takes the top mean score in 10, including SQuAD for all five solvers, and never significantly regresses. The paper also quantifies the risk of fixed skills: an off-the-shelf skill drops Llama-3.2-1B's SQuAD from 0.207 to 0.048, while Prompt2Skill lifts it to 0.492.
It doesn't win everywhere. On GPT-5.5's AIME, a public math skill scores 0.914 against Prompt2Skill's 0.861; on Claude Haiku's SearchQA, direct prompting edges it 0.857 to 0.839. Ablations: seeding with the off-the-shelf skill shows no consistent effect, and going from one to four user demonstrations moves SearchQA from 61.1% to 60.9%, essentially nothing. Under a matched budget of 7,400 solver evaluations with identical proxy data and seed, it beats SkillOpt on SearchQA EM 48.71 to 45.79, plus 1.20/0.61 points on SQuAD EM/F1.
Mind the units: the 10.8 average is percentage points, while the headline 36.1% over direct prompting and 83.0% over off-the-shelf are equal-weighted averages of relative changes, which low-baseline pairs inflate. The paper says so itself in E.4.
Skill authoring drops from "expert plus labeled data" to one sentence. The overlap audit in Appendix C found no lexical or workbook-hash matches against the target tests, and the SQuAD gains came from sciq and HotpotQA, so the skill transfers procedure rather than memorizing answers. Appendix B tries it as an AutoML substitute: scores comparable to a Prompt2Model variant that generates 1,000 examples and fine-tunes, which makes skills a cheap alternative to fine-tuning. The cost is real, though: thousands of solver calls per skill, which adds up when the target is an API model.