Paper demystifies agent skills: 65.7% work via procedural anchoring, not knowledge injection
dair_ai · x · 2026-08-17
A new arXiv paper, "Demystifying Agent Skills: Why They Work-Until They Don't," systematically answers when skills help, why they work, and where they fail.
Key findings:
- Across 8,135 normalized trial records, procedural anchoring accounts for 65.7% of cases where a skill helps, versus only 4.5% for explicit knowledge injection—skills stabilize execution rather than supply missing facts.
- Retrieval is a separate bottleneck: as the pool grows from 5 to 100 skills, actual-use precision collapses from 29.6% to 3.3%.
- Skills beat Workflow Memory by 6.06 points in matched comparisons.
- Skills break under brittle assumptions, incompatible contexts, or insufficient guidance.
Essential reading for anyone maintaining skills for their agents.
Related event: Microsoft Study: Agent Skills Can Be Harmful(3 posts)→
More from coding & agent
- Devs say LLM coding mistakes are a different class: overcomplication, inferred limits, worse at higher reasoning — Mu5_ · 2026-10-16
- How do you debug bad agent runs when you already have tracing? — Sensitive-Parsnip-12 · 2026-10-16
- Where does pre-task context live for agents? Devs debate workflows beyond handoff — shishen2026 · 2026-10-16
- No-instruction autonomous agent wakes up, sets own goals, and tries hacking by default — sethlazar · 2026-10-16
- CODESTRUCT reframes codebases as AST action spaces to fix brittle LLM code agents — burkov · 2026-10-16
- Would you install another SDK for agent debugging on top of your tracing stack? — Sensitive-Parsnip-12 · 2026-10-16