Progressive Agent Skill Generation via Reinforcement Learning
Junhao Shen, Zhanqiu Zhang, Yiwen Guo, Hong Cheng
cs.LG, cs.CL
2026-08-03
Skill-α frames agent-skill generation as a sequence of edits, each scored by a rollback reward that checks it on an anchored query, lifting tau2-bench by 6.7 points over SkillPro.
To handle complex tasks, agents are often given a "skill": a text rule or procedure telling it which tools to call and in what order. Skills transfer across tasks, so automatically distilling high-quality skills from documents or past experience is valuable.
The hard part is that a skill has no native correctness signal. You cannot tell by reading a rule whether it is good; that only shows up when the agent runs it downstream. Existing generators lean on hand-designed heuristics or pipeline-style consolidation, and the document and experience flavors usually need separate designs, so there is no unified framework.
Skill-α models skill generation as a sequential chain of edits. Starting from an initial skill z₀, the model reads evidence in batches and emits a local edit, restricted to five types: Create (add a missing rule), Update (fix an inaccurate one), Merge (collapse overlap), Prune (cut harmful or redundant content), and Noop (hold). After each edit the skill is updated.
The core idea is the rollback reward. Whether an edit helps is judged not by the edit itself but by an anchored query: run the agent with the pre-edit skill and the post-edit skill, and the edit scores 1 only if the new version does better. Every edit thus gets a clean signal from downstream execution. Training uses GRPO with group sampling, on a Qwen3-8B base, SFT-warmed then RL-optimized, with GPT-4o as the fixed worker.
Main results span document-to-skill (CL-Bench) and experience-to-skill (tau2-bench, SpreadsheetBench).
| Setting | Skill-α | Best baseline | Gain |
| CL-Bench avg | 10.38% | 7.11% (SkillPro) | +3.3 |
| tau2-bench avg | 55.83% | 49.17% (SkillPro) | +6.7 |
| SpreadsheetBench | 27.50% | 26.00% | +1.5 |
Swap in Claude-Sonnet-4.5 as the worker and the skills still transfer: 70.33% on tau2-bench versus 65.83% for Anthropic's own skill creator, a 4.5-point lead. The skills are not overfit to one worker.
The telling ablation is the rollback reward. Drop it and CL-Bench falls from 10.38 to 3.68, tau2-bench from 55.83 to 46.67. Removing Merge and Prune also hurts. Noop is not decoration: it lets the model learn when to leave a good skill alone.
A skill is reusable, so a good one is a one-time investment that keeps paying. Skill-α matters less for the point totals than for giving skill generation a unified, learnable form: whether evidence comes from documents or experience, it is decomposed into individually scored edits, with rewards pulled straight from downstream effect. The rollback reward, a counterfactual way to assign credit to intermediate steps, should travel to other settings that need fine-grained credit assignment.
The authors concede the reward and verifier interface is still benchmark-dependent, the skill representation is text-only, and the reward is a local rollback comparison, so it is short-horizon: a good edit on a long task may not pay off immediately. One open question is how the anchored queries are chosen: GPT-4o is fixed as the worker, and the quality of those queries determines the reward, yet the paper says little about their coverage.