Why your agent skills suck: 56% of the time the model never retrieves them
Hydronix2731 · reddit · 2026-09-19
A data-backed breakdown of when AI agent skills actually help:
When they work, gains are huge
- Microsoft's SkillOpt: a spreadsheet skill lifts Claude from 22 to 82 on SpreadsheetBench.
- NVIDIA's with/without eval on 300+ verified skills: 41 points average correctness lift.
- SkillsBench: curated skills add 16 points on average, some domains 50+ — comparable to upgrading a model class.
When they do nothing
- Vercel found the agent never retrieved the skill in 56% of cases; even when triggered, gains can be minimal. The author's own eval of Claude's built-in /code-review found only marginal improvement over a plain "review this code" prompt.
When skills matter
- Only when they encode knowledge the model lacks: your API usage, repo database conventions, MCP call specifics. Biggest gains in specialized domains (e.g. healthcare), smallest in general software engineering; models writing their own skills: zero benefit.
How to evaluate
- Standard method: run with/without the skill on the same task suite. NVIDIA's SkillEvaluator and SkillsBench are reference setups (requiring API keys, agent CLIs, Docker/Modal sandboxes).
- The author open-sourced Terum: SkillEvaluator-style eval with no setup, running on your own harness and subscription, plus live tracking of whether skills trigger at all. Claude skills now; Codex/Cursor planned.
More from coding & agent
- Claude Skill turns an image's buildings into resizable Three.js models — jasonkneen · 2026-09-20
- Cursor Pro users report credits draining fast and degraded Auto mode, weighing $25/month alternatives — ReporterUnusual9295 · 2026-09-20
- Fuzzing vs property-based testing: the terms are splitting — exploration vs assertion — blaizedsouza · 2026-09-20
- Software factory take: LLMs may generate, but never specify or verify — blaizedsouza · 2026-09-20
- Jev as an NSFW prompt filter: 93% on CSAM evals, sub-cent cost, and where thresholds bite — Murky_Ad8671 · 2026-09-20
- "A heuristic cannot certify a heuristic": why software-factory orchestration thinking is wrong — blaizedsouza · 2026-09-20