SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation
Zelin Tan, Yiqun Zhang, Hao Li, Zhiyao Cui, Hejia Geng, Shao Zhang, Hangfan Zhang, Yang Chen, Xiaosong Wang, Lilong Wang, Zhenfei Yin, Shuyue Hu, Chen Zhang, Lei Bai
cs.AI
2026-08-03
From 2,000 public skills SKT synthesizes 27,164 verified skill-use trajectories; one-epoch SFT lifts Qwen3.5-9B and Gemma 4 by 3.2-18.9 points across four agent benchmarks.
Agent Skills, an open format that packages reusable procedural knowledge as instructions, metadata, and optional scripts or templates, has crossed 600,000 public entries since October 2025, with the explicit goal of letting an LLM agent load and run them on demand. The uncomfortable starting finding is that handing skills to current models does not make them reliably identify when a skill applies, follow its operational constraints, or coordinate several skills into one coherent workflow. Most prior work tackles skill retrieval, internalization, or self-evolution; almost none ask how a model actually learns to use external skills. SKT treats this as a data problem.
SKT is a three-stage, verified synthesis pipeline.
Stage I, skill curation: a rubric LLM keeps skills from skills.sh that can ground executable, objectively gradeable tasks. Skills are sampled at cardinality k in {1, 2, 3}; for k>1 a composition judge requires the skills to form a coherent workflow with distinct roles, otherwise the set is resampled.
Stage II, task synthesis: TaskGen fills a fixed template to produce a complete task package (instruction, isolated runtime, execution settings, evaluator, reference solution). Three serial gates follow. Rule-based: package completeness, valid paths, reference solution must execute in a clean workspace and earn full evaluator credit, and no verbatim leakage of answers or skill rules into solver-visible material. Agent-based: a semantic review plus a paired skill-dependence test that runs with-skill and without-skill rollouts and requires r+ > r-. Difficulty control: a fixed solver attempts the task N=5 times, and if the pass rate is at least 0.6 the task is sent back as too easy. Any failing gate returns a structured diagnosis to TaskRepair, which rewrites the package and restarts at the rule-based gate.
Stage III, trajectory synthesis: four teachers (MiniMax-M2.5, GLM-5, Qwen3.5-397B-A17B, DeepSeek V4 Pro) crossed with two harnesses (DeepAgents, OpenCode) solve 32,000 task-teacher-harness combinations; 27,164 trajectories survive a deterministic validator (full credit, clean termination, well-formed tool trace, explicit skill access) and an LLM validator (each skill consulted before the action it should guide and substantively affecting decisions). Failed rollouts are resampled without feedback. Training is masked autoregressive SFT over the complete trajectory.
2,000 skills produced 4,000 task packages (1,520 single, 1,295 two-skill, 1,185 three-skill) and 27,164 trajectories. Full-parameter SFT on Qwen3.5-9B and Gemma 4 E4B-IT. All 16 model-harness-benchmark comparisons improve, by 3.20 to 18.91 points.
Representative scores (mean, 0-100):
| Setting | Original | SKT |
| Qwen3.5-9B / OpenCode / SkillEval | 55.24 | 72.48 |
| Qwen3.5-9B / DeepAgents / SkillEval | 51.62 | 70.53 |
| Qwen3.5-9B / OpenCode / SkillsBench | 5.80 | 15.79 |
| Gemma 4 / OpenCode / MolBench-Bind | 34.46 | 45.27 |
Ablations: withholding skills at inference shrinks the gain from 8.68-18.91 to 0.53-5.69, so SKT teaches skill use rather than internalizing skills. SFT on unverified raw synthesis instead lowers all four benchmarks (-1.9 to -19.5), an 11.91-24.61 point gap to SKT. Cross-harness SFT keeps 49.1-58.1% of the matched-harness gain. A single mixed-harness checkpoint lands within 2.71 points of each specialist. Scaling the training pool from 100 to 2,000 skills lifts SkillEval from 55.24 to 72.48 monotonically.
For practitioners building skill-using agents, SKT is a directly reusable recipe. Base models do not pick up skill libraries on their own, and unverified synthetic trajectories make things worse rather than better. Get the verification right (rules, paired skill dependence, difficulty) and add feedback repair, and a one-epoch SFT on a 9B backbone approaches harness-specialist performance while transferring across harnesses and continuing to climb with broader skill coverage. The released SkillEval (100 tasks, held-out skill pool) also gives the field a shared evaluation surface.
The authors note the gains are harness-specific and only partly mitigated by mixed training. Additional concerns: skill dependence is checked only at the bundle level (r+ greater than r-), not per skill via leave-one-out, so a multi-skill task may be admitted even when one bundled skill is not strictly necessary. SkillEval is built with the same pipeline as the training tasks, so even with disjoint skill pools the task-distribution bias likely favors the method. Training is SFT-only, with no RL stage and no larger backbones tested. Only 2,000 of 600,000 public skills were sampled, leaving long-tail coverage uncharacterized. Both teachers and verifiers rely on closed-source frontier models, raising reproduction cost and potential bias.