Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents
Gen Dong, Yanjie Gao, Liqun Li, Tianyin Xu, Yu Hua, Fan Yang
cs.AI
2026-08-12
Microsoft and collaborators compared 20,000+ paired agent runs and confirmed 307 skill-induced failures, nearly 70% from skills causing incorrect required implementations, not wrong skill choice.
Agent skills, reusable SKILL.md packages loaded into an LLM agent's context, are now the standard way to extend agents with procedural guidance. But existing skill benchmarks only report aggregate pass-rate deltas: SkillsBench found skills dragged down 16 of 84 tasks, and SWE-Skills-Bench saw token overhead spike as high as 451% on some tasks. Those numbers tell you skills can help, hurt, or waste resources, but not which part of a skill's instructions caused a specific run to fail, or why a cheap task suddenly got expensive. Without that attribution, skill marketplaces have no way to screen bad skills before they get reused across many future runs.
The authors borrow differential testing. For each task, they run a target execution with the audited skill loaded, paired against a reference execution that either loads no skill or a semantically matched alternative, retrieved from smithery.ai and skillsmp.com, embedded with all-MiniLM-L6-v2, and kept only above 0.7 cosine similarity. Task, repository state, agent framework (OpenCode 1.15.1), and model (Claude Opus 4.6) stay fixed; only the skill setup varies.
Across SkillsBench (84 tasks) and SWE-Skills-Bench (490 tasks), the augmented comparison space grew from 826 to 20,664 potential pairs. After execution and manual refinement, removing weak evidence, verifier-narrow cases, and duplicates, the final dataset holds 307 confirmed skill-induced failures: 125 functional failures and 182 high-confidence efficiency regressions, each hand-labeled with a root-cause subcategory. The authors then built SKILLTRIAGE, a GPT-5.5-based attribution tool that runs three times per case and takes a 2-of-3 majority vote to predict the root cause automatically.
| Failure type | Subcategory | Count | Share |
| Functional | Task-Implementation Fault (wrong or missing required element) | 86 | 68.8% |
| Functional | Artifact Misplacement (wrong output location) | 24 | 19.2% |
| Functional | Environment Mismatch | 13 | 10.4% |
| Functional | Applicability Mismatch (skill shouldn't apply) | 2 | 1.6% |
| Efficiency | Excessive Procedure (extra exploration/verification/build steps) | 114 | 62.6% |
| Efficiency | Context Bloat (skill text inflates every call) | 46 | 25.3% |
| Efficiency | Dependency Resolution overhead | 22 | 12.1% |
The counterintuitive finding: only 2 of 125 functional failures (1.6%) come from a skill being flatly inapplicable. Well over 80% come from skills that looked relevant but made the agent implement a required element incorrectly (46 cases) or skip it entirely (36 cases). On the efficiency side, 67 of the 114 Excessive Procedure cases are pure over-verification, where a skill turns "run the full test suite and fix everything before stopping" into a mandatory step regardless of task size.
SKILLTRIAGE's automated attribution tracks the manual labels closely: 93.6% high-level category match and 88.8% exact subcategory match for functional failures, 79.7% category match and 72.5% exact subcategory match for efficiency regressions.
If you're building or curating a skill marketplace, this gives a concrete screening target. Don't just check topical relevance; check whether a skill quietly converts optional examples, templates, or checklists into steps the agent treats as mandatory. That is the dominant failure mechanism, not off-topic guidance. If you're writing skill docs, the implicit lesson is to separate hard task requirements from reference examples and optional workflows, and to move long templates or checklists into lazy-loaded supplementary material instead of always-loaded skill bodies. SKILLTRIAGE also suggests this kind of root-cause triage can be automated close to human accuracy, opening the door to continuous, low-cost skill-quality audits instead of one-off manual review.
The study is confined to two benchmarks, one agent framework (OpenCode), and one model (Claude Opus 4.6); the authors don't verify whether the taxonomy transfers to other skill ecosystems or models. Attribution still leans on human judgment calls, such as whether a verifier is narrower than the task or whether an observed divergence is genuinely skill-caused, and while the authors used group consensus to reduce this, they flag it as the main threat to internal validity. SKILLTRIAGE's remaining errors cluster at taxonomy boundaries, such as misclassifying a missing required helper function as an implementation error rather than an omission, showing the category boundaries aren't always sharp and the tool isn't a full replacement for manual review yet.