Skill Following: Evaluating Actual Skill Use in Retrieval-Enabled LLM Agents
Seonghyeon Cho, Chanjun Park
EMNLP 2026
cs.CL
2026-09-01
On 17 LLMs, aggregate retrieval lift often reverses under RAE: DeepSeek-V3.2 gains +31.6 pp on MBPP+ overall but drops 9.1 pp on tasks where it actually retrieved a skill.
LLM agents now ship with skill libraries: search a pile of SKILL.md files, stuff the hits into context, then answer. Most evals still ask coarse questions. Did the whole benchmark go up once skills were available? Did retrieved tasks score higher than skipped ones?
The second comparison is biased by construction. The model chooses when to retrieve, so the retrieved set and the skipped set already differ in difficulty and in how well the library matches the prompt. Whole-benchmark averages dilute the signal further, because they mix in tasks that never touched a skill. The narrower question is whether a retrieved skill helped the exact task it was fetched for.
The paper calls that retrieval-to-answer chain Skill Following (SF). The protocol is a paired switch.
The libraries are small and procedural. Coding uses 9 SKILL.md files (binary-search boundaries, hash counting, sliding windows, and similar). Math uses 8 (substitution, case analysis, modular arithmetic). Format follows the AgentSkill frontmatter-plus-body layout.
Three numbers answer different questions:
RAE is protocol-conditional. It is not an unbiased causal effect on a pre-treatment population. Main-text cells require at least 10 retrieval-returned tasks.
Seventeen models, 80-task draws from MBPP+ (three seeds) plus HumanEval+, and 80-task Math500 draws. On MBPP+ seed 42 the two signs often disagree:
| Model | Aggregate lift | RAE |
| DeepSeek-V3.2 | +31.6 | −9.1 |
| Gemini-2.5-Flash-lite | +6.2 | −15.6 |
| Llama-3.3-70B | +69.8 | +4.2 |
| Qwen3-235B | −0.1 | +7.0 |
| Claude-4.5-H | +1.4 | +1.4 |
Gemini-2.5-Flash-lite stays positive on aggregate lift across all three MBPP+ partitions and strictly negative on RAE. HumanEval+ flips the other way for 3 of 13 reportable models: Qwen3-8B, Qwen3.5-9B, and Qwen3-235B show negative aggregate lift with positive RAE. Under the main reporting cutoff, 5 of 13 cells on MBPP+ seed 42 disagree in sign.
Math repeats the pattern. Llama-3.3-70B posts +14.2 pp aggregate lift on Math500 and −39.4 pp RAE (3 helpful, 42 harmful, 99 retrieved tasks). Gemini-2.5-Flash-lite is +13.2 vs −11.0.
High coverage is not following. DeepSeek-V3.2 on MBPP+ seed 42 invokes and receives skills on 100% of tasks, yet structural adherence is 18.8% and effective success 11.2%, with RAE still −9.1. Gemini-2.5-Flash-lite: 82.5% coverage, 13.6% adherence, 10.6% effective, RAE −15.6.
A GPT-5.5 annotator, blinded to SD answers and correctness, labeled harmful flips. For DeepSeek, 64.4% of 45 harms are ignored/independent: a skill came back, the final answer shows no uptake. Another 15.6% leak tool traces into the submission. For Gemini, 72.5% of 40 harms are the same ignore class.
Swapping skill text does not rescue RAE. Filler, random, and corrupted returns still leave pooled RAE between −9.1 and −15.9 for DeepSeek and between −5.0 and −17.9 for Gemini. Empty returns make RAE undefined because the conditioning set is empty.
The harshest coding cell is Mistral-S-24B on MBPP+ partition 3: RAE −45.5 pp (1 help, 31 harms, n=66, McNemar p=1.54×10^{-8}).
A skill library, a high retrieval rate, and a higher benchmark average can all be true while retrieved skills still hurt the tasks they touch. If a product card says "skills on, average up," the model may simply retrieve on easier items.
The cheap audit is a same-task switch: one decode config, tool on vs off, then count helpful versus harmful flips only where retrieval actually returned content. That patch applies to SKILL.md-style products without training a new model.
This is a measurement warning, not a training recipe. Claude-4.5-H keeps matching signs and non-negative RAE on four coding cells, so negative RAE is not inevitable. Most of the panel is still far from "retrieve then gain."
The authors bound the regime: single-pass coding and math, a fixed BM25 interface, tiny static libraries. Long-horizon agents, executable code skills, language-only skills, and memory/planning loops are out of scope.
Because the conditioning set is chosen by SE itself, RAE is not a causal effect on a pre-registered task population and not a model-level skill-use score. Separating retrieval quality from content quality from integration would need oracle retrieval, metadata-only returns, forced full-skill fetch, multiple library constructions, and human causal labels.
A few design choices soften the numbers. Each partition is 80 tasks. Several smaller models never retrieve enough to enter the sign table. Adherence labels come from GPT-5.5 plus a stratified human audit, not full human coding. Nine coding skills plus eight math skills are not the colliding SKILL.md piles in production repos.