Skill Following paper: LLMs with skill retrieval often hurt the very tasks they retrieve for
rohanpaul_ai · x · 2026-09-08
A paper accepted to EMNLP 2026 Findings introduces RAE (Retrieval-Invoked Actual-Use Effect), a metric that measures whether retrieving skills actually helps LLM agents on the exact tasks where retrieval was triggered.
Key findings:
- Standard evaluations compare retrieved vs non-retrieved tasks in aggregate, introducing severe selection bias that inflates apparent tool-use proficiency.
- Testing 17 LLMs across coding and math, the authors find a stark paradox: several models show positive aggregate retrieval lift on MBPP+ but negative RAE — they actually perform worse on the very tasks where they retrieved a skill.
- The takeaway: aggregate averages create an illusion of skill use; same-task paired comparisons are needed to see if the retrieval-to-answer pipeline genuinely rescues more outcomes than it harms.
Related event: Papers Reveal When Agent Skill Retrieval Helps—and Hurts(4 posts)→
More from coding & agent
- Coding Agent Deleted a Production Database During Code Freeze: Anaconda's Governance Fix — anacondainc · 2026-10-27
- GPT-6.1 Terra Model Slug Spotted in APIs and Codex Changelog Before Launch — testingcatalog · 2026-10-27
- Claude Projects now lets you drag threads into the message box or @ mention them for precise context — dfeinition · 2026-10-27
- Jane Street's Yinsky shares tips for making codebases agent-friendly: types, property tests, formal methods — sh_reya · 2026-10-27
- Celesto launches CI runners nearly 90% cheaper than GitHub Actions, with agent sandboxes — aniketmaurya · 2026-10-27
- Runway launches App Studio: build no-code creative apps that generate image, video and audio — runwayml · 2026-10-27