Matched-comparison study shows helpful agent skills may be hurting the exact tasks that trigger retrieval
dair_ai · x · 2026-09-03
New measurement work finds that agent skills lifting aggregate scores can be hurting every task they touch.
The flaw: Standard comparisons contrast tasks where a skill was retrieved with tasks where none was — different tasks, conflating retrieval's effect with which tasks trigger it.
The fix: Retrieval-Invoked Actual-Use Effect runs the same task twice, with skills enabled and disabled, counting only tasks where the agent actually retrieved something — a matched comparison.
Findings:
- Across 17 LLMs on coding and math, models often show positive aggregate retrieval lift alongside negative same-task effect
- On MBPP+, several models that look beneficial system-wide are hurting themselves on exactly the tasks where retrieval fired
A methodological warning for anyone maintaining an agent skills library.
More from coding & agent
- MCP Platform NitroStack Wins Product Hunt Product of the Day, Focuses on Retention — EstablishmentFun4373 · 2026-10-21
- CodeRabbit Ships 5,000 PRs a Week With a Tiny Team, Hires by Strategy-Game Hours — omojumiller · 2026-10-21
- LukeW maps the five UI problems every AI-native company faces with agents — LukeW · 2026-10-21
- Claude Code over ADB guts a bloatware-laden Android TV, freeing ~90% more RAM — edwinarbus · 2026-10-21
- Developer uses Claude to fix his adware-slowed Xiaomi TV, releases tool for any Android TV — edwinarbus · 2026-10-21
- Open-weights Ornith-1.5-122B debuts with 84.5% SWE-bench Verified, runs on a single DGX card — _akhaliq · 2026-10-21