Just having a tool available drops model answer rate from 98.2% to 63.5%, paper finds
dair_ai · x · 2026-09-16
A paper shared by dair-ai reveals a counterintuitive finding: giving an assistant extra tools can make it stop answering questions it actually knows.
- Across six models, the answer rate on questions needing no tool fell from 98.2% to 63.5% when a related tool was merely available — the drop comes from the tool's presence, not its use
- The benchmark spans 500 query pairs across 10 domains, each with a tool-unavailable control
- A preceding tool call recovered some lost answers but caused new losses
- A one-sentence system instruction explaining the tool's purpose recovered up to 45.6 points
Directly actionable for agent tool design and prompt engineering.
More from coding & agent
- Engineer uses LLMs to turn knowledge into playable simulations, builds a chip fab game — thehiphopswami · 2026-09-16
- Open-Source Skill Strips AI Slop: 41 Patterns Caught Across 4 Review Gates — Roger_M_Taylor · 2026-09-16
- Muse Code Wins Over Skeptical Dev as Alexandr Wang Touts the Model — alexandr_wang · 2026-09-16
- Figma Links Aren't Enough: What Context to Give AI Coding Agents Before They Touch UI — T07NAD0 · 2026-09-16
- prxhub MCP connector lets AI agents search, cite and publish .prx research bundles — modelcontextprotocol · 2026-09-16
- Building reliable agent harnesses: lessons from 4,000 runs of an open-source deep research agent — Public_Umpire_1099 · 2026-09-16