SKILLER: Language-Level Reinforcement Learning for Reusable Skill Extraction in Small Language Models
Chenhao Dang, Siyuan Xiong, Conghui He, Weijia Li
AAAI 2027
cs.AI
2026-08-11
SKILLER optimizes skill text via natural-language RL: a frontier model is actor/critic, the small model is the environment; a 9B model then beats a frontier model 167x its price.
Agent skills have become a basic primitive in modern agent systems: a standardized text artifact that packages procedural knowledge, which tools to call, and what the output should look like, then gets loaded into a harness such as Codex, Claude Code, or OpenCode to constrain model behavior and make task execution repeatable. The trouble is that today's skills are tuned for strong closed-source frontier models, whose inference cost makes large-scale deployment unaffordable. Compact open models that run on consumer GPUs (the Qwen3.5 family) are an obvious way to cut that bill, but skills do not port across: instructions written for a frontier model overwhelm a small one, which then fabricates arguments, skips verification steps, and gets derailed by complex instructions. The paper calls this model-mismatch.
The real question is how to automatically generate skills that a specific small model can actually follow. Human authoring is expensive and not tailored to the small model's quirks; existing skill generators (Manus, AutoSkill, EvoSkill, SkillX) are mostly built for larger models. SKILLER targets exactly this gap.
The core move is counterintuitive: SKILLER updates no neural weights. It treats the skill text itself as the policy being optimized, and every state, reward, and update in the loop is carried by structured natural language. That is what language-level reinforcement learning means in the title.
The loop has five parts:
Optimization runs for I steps as a chain of Apply calls. The paper observes that successive edits add progressively finer constraints: first input grounding, then task-local computation, then self-validation, so skills get more exacting as the loop proceeds. A progressive skill disclosure mechanism keeps the small model from being overwhelmed by long context. The whole generation runs offline once with a five-step RL schedule; at evaluation time the small model only does inference, so the generation cost is one-time.
Five benchmarks: SkillsBench (26 single-skill tasks), SWE-Skills-Bench (117 instances, 10 high-difficulty skills, execution-based tests), SkillLearnBench (100 continual-skill-generation instances), GAIA (165 multi-step retrieval tasks), and EarthBench (248 Earth-science and data-processing samples). Baselines are no-skill execution, human-authored skills, Manus (closed source), and three open generators (AutoSkill, EvoSkill, SkillX).
On Qwen3.5-9B, SKILLER wins SkillsBench at 73.91 (next best SkillX 60.87), SWE-Skills-Bench at 82.80 (Manus 62.40), and EarthBench at 76.08, tying SkillX on GAIA at 49.40. On the 4B model the wins are more concentrated: first place on SkillLearnBench (33.00) and SWE-Skills-Bench (66.70), but second to SkillX on SkillsBench (42.03) and GAIA (43.78), and tied with Manus on EarthBench. Tighter capacity makes the payoff from skills more task-dependent.
The headline comparison is cost versus performance (Figure 1). Qwen3.5-9B with SKILLER reaches a 73.91 pass rate on single-skill tasks, beating Claude Opus 4.7 Max with curated skills by 4.4 points while paying 167× less per output token. The 4B model is 71× cheaper than Haiku 4.5 with curated skills. A counterintuitive result: 4B + SKILLER scores 66.70 on SWE-Skills-Bench, higher than the 9B model with human-authored skills (52.00), AutoSkill (44.10), EvoSkill (45.90), SkillX (58.80), or Manus skills (62.40). A smaller model with skills tuned for it can beat a larger model paired with another generator.
Generation cost (Table 4) is reasonable: SKILLER spends $8.95 on average for a 62.86 average score, while SkillX spends $14.55 to reach only 52.60, and EvoSkill is cheapest ($1.95) but lowest (46.39). Structurally (Table 3), SKILLER's skills average 534 words, far shorter than AutoSkill's 1887, with TF-IDF similarity to human skills of 0.07, but carry the most helper scripts (2.96 per task on average, 15,747 total lines of code).
For practitioners, this is a directly usable cost-cutting path: instead of authoring skills for a frontier model, spend a one-time offline cost to generate model-specific skills for a consumer-GPU small model and cut inference spend by two orders of magnitude. SKILLER also turns skill generation from one-shot prompt engineering into an iterable optimization, with a critic comparing against a reference trajectory, an actor constrained to four bounded edits, and a replay memory to prevent regressions. The mechanism ports to other harnesses and other small models (code is open source).
More broadly, it demonstrates language-level RL: optimizing a discrete text object (a skill) by passing signals in natural language, without touching weights, in a way gradient descent cannot. The paradigm extends to any setting where a natural-language artifact serves as a policy.
The authors are explicit about the ceiling: skills constrain procedure, they do not supply factual knowledge or complex mathematical reasoning. On multi-hop retrieval benchmarks like GAIA, the bottleneck is whether the model can retrieve external knowledge, not whether it follows the right procedure, so SKILLER only matches the top baselines there. That is a structural limit of the skill route, not an implementation bug.
The optimization trajectory is nonmonotonic: later edits occasionally narrow instructions useful for other instances, which is why the authors added a snapshot-and-rollback mechanism. That also means the five-step RL schedule is presented without a sensitivity analysis; why five steps, or whether more would help, is left open.
Several concerns stand out on close reading. First, the results depend heavily on a frontier model as actor and critic (GPT-5.4 in the experiments), with no ablation on using a weaker actor. Second, the 4B model actually loses to SkillX on SkillsBench and GAIA, so the abstract's claim of a "1.8 to 13.3 point gain" should be read as an average over baselines rather than a clean sweep. Third, three of the five benchmarks (SkillsBench, SWE-Skills-Bench, SkillLearnBench) and several baselines come from the same cluster of 2026 work, so the evaluation would benefit from more independent external validation.