arXiv: 8,135-trial study shows agent skills stabilize execution, retrieval precision drops to 3.3%
rohanpaul_ai · x · 2026-09-08
The paper "Demystifying Agent Skills: Why They Work—Until They Don't" (arXiv:2608.14036) uses controlled experiments across benchmarks, agent harnesses and LLMs to answer when skills help, why, and where they fail.
Key findings:
- Skills beat Workflow Memory by 6.06 points in matched comparisons. Skills work by turning noisy trajectories into "procedural anchors" that stabilize execution (65.7% of cases), not by injecting missing knowledge (4.5%).
- Retrieval is a separate bottleneck: as the skill pool grows from 5 to 100, actual-use precision falls from 29.6% to 3.3%.
- Built on 8,135 normalized trial records and a taxonomy of three categories and twelve skill-use modes from open coding.
Takeaway: skills stabilize what to do first, which tools to use, and what to verify — and hurt when misapplied or followed rigidly.
Related event: Papers Reveal When Agent Skill Retrieval Helps—and Hurts(4 posts)→
More from coding & agent
- Coding Agent Deleted a Production Database During Code Freeze: Anaconda's Governance Fix — anacondainc · 2026-10-27
- GPT-6.1 Terra Model Slug Spotted in APIs and Codex Changelog Before Launch — testingcatalog · 2026-10-27
- Claude Projects now lets you drag threads into the message box or @ mention them for precise context — dfeinition · 2026-10-27
- Jane Street's Yinsky shares tips for making codebases agent-friendly: types, property tests, formal methods — sh_reya · 2026-10-27
- Celesto launches CI runners nearly 90% cheaper than GitHub Actions, with agent sandboxes — aniketmaurya · 2026-10-27
- Runway launches App Studio: build no-code creative apps that generate image, video and audio — runwayml · 2026-10-27