arXiv: 8,135-trial study shows agent skills stabilize execution, retrieval precision drops to 3.3%
rohanpaul_ai · x · 2026-09-08
The paper "Demystifying Agent Skills: Why They Work—Until They Don't" (arXiv:2608.14036) uses controlled experiments across benchmarks, agent harnesses and LLMs to answer when skills help, why, and where they fail.
Key findings:
- Skills beat Workflow Memory by 6.06 points in matched comparisons. Skills work by turning noisy trajectories into "procedural anchors" that stabilize execution (65.7% of cases), not by injecting missing knowledge (4.5%).
- Retrieval is a separate bottleneck: as the skill pool grows from 5 to 100, actual-use precision falls from 29.6% to 3.3%.
- Built on 8,135 normalized trial records and a taxonomy of three categories and twelve skill-use modes from open coding.
Takeaway: skills stabilize what to do first, which tools to use, and what to verify — and hurt when misapplied or followed rigidly.
Related event: 8,135 Experiments Demystify When Agent Skills Work and Fail(2 posts)→
More from coding & agent
- Sarms paper claims GraphWorkflow runs compiled agent graphs up to 62.5x faster than LangGraph — KyeGomezB · 2026-09-08
- Super Smash Bros Melee fully decompiled after 6 years, with LLMs accelerating the final stretch — banteg · 2026-09-08
- Should you review AI-generated code? A RL discount-factor framework says it depends — sergeykarayev · 2026-09-08
- Miri catches data race in lock-free Rust ring buffer that tests and loom missed — blaizedsouza · 2026-09-08
- AI code review blasted for flagging Carmack's inverse sqrt in 10 lines of working code — ZeeshanZiaML · 2026-09-08
- WSL Manager 2.0 ships a built-in MCP server letting agents create and run Linux distros — bostrot · 2026-09-08