Study of 307 agent failures finds the culprit is often an on-topic, over-trusted skill
alex_verem · x · 2026-08-21
A controlled study examined how loading AI skills affects agent task execution: same task, same model, the only difference being whether a skill was loaded. The team confirmed 307 skill-induced failures (125 functional failures + 182 efficiency regressions).
The key finding: failures almost never came from obviously irrelevant skills (those get ignored) but from skills that looked perfectly on-topic. The agent over-trusts the skill, treating its examples and defaults as the actual task requirements, and confidently builds the wrong thing. About 69% of functional failures followed this pattern — e.g., a task asking for net exports as a percent of GDP led the agent to follow the skill template, divide the numbers, and skip the times-100 step, making every answer 100x too small.
The study ran on Claude Opus 4.6, showing capability wasn't the issue — the skill was steering. Skills help on average, but help and harm sit side by side, and almost nobody measures the harm: a skill isn't neutral documentation the agent consults, it's guidance the agent obeys.
Related event: Study Finds AI Agent Skills Can Cause 307 Task Failures(2 posts)→
More from coding & agent
- Devin Oncall tops SREGym with 100% end-to-end success, auto-resolving production incidents — tianyin_xu · 2026-10-16
- Running a multi-model UI design arena: Paseo orchestrates four design drafts per model — vista8 · 2026-10-16
- Cortex A/B-tests coding-agent skills and rules by replaying developer corrections before keeping them — Samuele Giampieri · 2026-10-16
- Tool calls are the hidden bottleneck killing AI coding speed — Crabbox and 5 fixes — steipete · 2026-10-16
- Agentic sandbox WorldBox doubles GPT-6 Astra's MazeBench score, 23 to 61 gems — ZhitingHu · 2026-10-16
- Two prompts: Gemini 4 Argon turns a single-player game into full multiplayer — chongdashu · 2026-10-16