Study of 307 agent failures finds the culprit is often an on-topic, over-trusted skill
alex_verem · x · 2026-08-21
A controlled study examined how loading AI skills affects agent task execution: same task, same model, the only difference being whether a skill was loaded. The team confirmed 307 skill-induced failures (125 functional failures + 182 efficiency regressions).
The key finding: failures almost never came from obviously irrelevant skills (those get ignored) but from skills that looked perfectly on-topic. The agent over-trusts the skill, treating its examples and defaults as the actual task requirements, and confidently builds the wrong thing. About 69% of functional failures followed this pattern — e.g., a task asking for net exports as a percent of GDP led the agent to follow the skill template, divide the numbers, and skip the times-100 step, making every answer 100x too small.
The study ran on Claude Opus 4.6, showing capability wasn't the issue — the skill was steering. Skills help on average, but help and harm sit side by side, and almost nobody measures the harm: a skill isn't neutral documentation the agent consults, it's guidance the agent obeys.
Related event: Study Finds AI Agent Skills Can Cause 307 Task Failures(2 posts)→
More from coding & agent
- AgentRadio Research Enables Mid-Task Communication, Boosting Long-Horizon Task Resolution — import_jmr · 2026-08-21
- Open Source Tool Graft Gives Coding Agents Codebase Memory, Boosting Speed by 60% — alex_verem · 2026-08-21
- Graft: Persistent Codebase Memory Boosts Coding Agents' Speed & Efficiency — alex_verem · 2026-08-21
- Beyond pass/fail: devs debate how to evaluate agent decisions with real costs — KAIT2_1412 · 2026-08-21
- Polsia launches: 9 autonomous agents run full company operations — testingcatalog · 2026-08-21
- MCP Long-Running Tasks Demo: Showcasing Agent Persistence — dsp_ · 2026-08-21