SkillGate: Training In-Policy Skill Selection in Long-Horizon Agents
Qingyao Li, Wenxiang Jiao, Shuai Shao, Kangning Zhang, Yuan Lu, Yi Guo, Weiwen Liu, Weinan Zhang, Yong Yu
cs.AI
2026-08-19
SkillGate gives skill-name tokens their own credit, positive only for a single correct read. A 9B policy rises from 40.8% to 53.2% and cuts misleading reads by two thirds.
Agent frameworks package procedures as skills: a name, a one-line description, and a SKILL.md body. Public libraries now hold thousands of them, far more than a context window admits. Mid-episode the policy has to pick which file to open from the name and the one-liner alone. A wrong read conditions every action after it.
The default recipe hands the candidate slate to the policy and trains with task outcome. This paper measures why that recipe cannot teach the choice on long trajectories, and names the failure selector credit starvation. An offline audit of 12,800 on-policy trajectories from an outcome-only GRPO run shows three facts at once. Tokens that write the skill name carry a median 0.14% of the trajectory's loss weight, and that share dilutes about 7x as trajectories lengthen. About two in five oracle reads inherit a negative advantage because execution later failed; on the longest bin a correct choice is punished more often than rewarded. Inside matched prompt groups, reading the oracle is still worth +11.2 points of success. A decision worth eleven points gets a vanishing, often wrong-signed signal, and all three quantities worsen monotonically with horizon.
SkillGate leaves rollout, reward and optimiser untouched. It only changes where an advantage is allowed to land. Assistant tokens of a trajectory are split into two disjoint credit channels.
Selection tokens are the identity span: the few tokenizer pieces that write the skill name inside the path. Execution tokens are everything left after the entire read tool call is deleted. The tool-call wrapper is trained by neither channel. Skill bodies arrive as observations and are never trained.
The task channel uses the usual group-normalised GRPO outcome advantage, with the whole read call cut out of the mask so a later failure cannot revise the choice. The selector channel scores each read with an action-local utility: 1 if the trajectory issued exactly one read and it was the oracle, otherwise 0. The advantage is centred over all read actions in the prompt group, with no standard-deviation normalisation. Reading the oracle and then three other skills scores 0; reading it twice scores 0. If a group is all clean oracle reads or none, every utility ties and the channel goes silent.
Token weights are then rescaled so both channels sum to the original assistant-mask mass N in a batch. Each credited read gets the same N/M, independent of trajectory length and of how many pieces the name occupies. The objective is clipped GRPO, with the selector term at λ=0.20 plus a KL penalty to a frozen reference. A negative selector advantage means the wrong skill or more than one read. It never means the oracle's content failed.
The standard mixed slate is K=16: one oracle written for the task, five topically adjacent but functionally wrong misleading skills, plus five relevant and five irrelevant skills drawn from 2,045 community skills. Training uses 491 tasks that contain no Claw-Eval item, and training and evaluation oracles are disjoint as identities. Every trained row starts from the same Qwen3.5-9B SFT checkpoint, runs 100 on-policy GRPO steps with 8 rollouts, on 16 H800s. Each configuration is a single run.
| Method | Trial success | Oracle exposure | Misleading exposure |
| SFT init | 40.8% | 37.9% | 61.8% |
| SkillRL (outcome only) | 47.0% | 54.3% | 69.6% |
| SkillGate | 53.2% | 83.9% | 21.8% |
Under the same budget, outcome-only RL learns to read more, not to discriminate. SkillGate cuts misleading exposure by about two thirds and reads fewer skills. Per benchmark: Claw-Eval 60.2% (SFT 50.9 / outcome-only 57.1), SkillsBench 15.6% (6.2 / 3.1), SETA 54.2% (40.0 / 50.0), SWE 65.0% (45.0 / 45.0), Terminal-Bench 2.0 37.5% (21.9 / 31.2). Claw-Eval tasks and evaluation oracle identities were unseen in training. The 9B SkillGate run beats Qwen3.5-397B-A17B at 51.7% overall, and stays below DeepSeek-V4-Flash at 61.0%. None of the frontier models reads the oracle on even half the trials.
A 280-trial ablation drops the selector signal on nowhere, the prompt group, the whole trajectory, the first oracle read, or the trajectory's only read when it is the oracle. Group-level regret and a trajectory bonus stall at 41.8%, against 42.1% for outcome-only. Landing credit on identity tokens reaches 45.0%. Requiring a single read reaches 50.0%, with clean single-oracle rising from 21.4% to 75.4% and reads/trial falling from 1.88 to 1.11.
An external router is not enough. A 27B router advertising top-1 to a frozen SFT executor scores 36.8%, worse than a 9B router at 40.7%. SkillGate choosing from all 16 candidates scores 50.0%, above the oracle-only ceiling of that frozen executor at 48.2%. Telling the model to read exactly one skill barely moves behaviour. Injecting the oracle on top of SkillGate still reaches 52.9%, so selector training did not damage execution.
For agents that already load skills on demand, this is a training patch that can ship: selection and execution are judged by different evidence, and should not share one broadcast advantage. Scale does not grow a selector. Qwen3.5-397B posts 51.7% success while reading the oracle on 16.1% of trials. The cost is that training tasks must know which skill is correct; deployment does not need the label.
This is not a new architecture. It turns a read the policy already emits into a trained decision. Inference is cheaper on the same trials: fewer distinct reads, fewer turns, fewer cumulative input tokens. On-demand bodies load about as many tokens as preloading exactly one skill.
The authors are blunt. Each configuration is a single run; uncertainty is task-level bootstrap, so small gaps are directional. The method needs a known correct skill at train time. An action-local scheme cannot credit an abstention.
The experimental slate is a constructed 16-way mix with hard negatives. Whether the clean single-oracle utility still learns when the real library has no oracle labels and dirtier descriptions is untested. SkillsBench stays at 15.6% even for SkillGate, so a correct read can still fail at execution. λ=0.20 was not swept. One hundred steps on 16 H800s is not a cheap reproduction.