MCP fault injection study: agent task scores drop 60% under realistic tool failures, scale doesn't help
dair_ai · x · 2026-10-26
A NeurIPS 2026-accepted paper built MCP-Injector, a drop-in proxy that injects faults into MCP tool calls, and stress-tested tool-calling agents under realistic failures:
- The fault model was calibrated on 57,240 issue/PR threads from 1,774 open-source MCP repos, reflecting how MCP servers actually fail
- 1,350 runs across gpt-oss-20B/120B and three agent stacks (custom plan-and-execute loop, OpenAI Agents SDK, LangChain)
- Task-completion score fell 60%, from 8.08 to 3.23 out of 10; the 120B model retained no larger share of its fault-free score (p = 0.96) and its completion rate under faults was slightly lower (91.1% vs 95.6%)
- Differences between agent stacks exceeded differences between model sizes — framework-level error handling matters more than scale
Practical takeaway: capability benchmarks alone mask operational brittleness; evaluate agents with protocol-aware fault injection before production.
More from coding & agent
- SF 'Code Engineering' event picks up where 1,000-person harness engineering meetup left off — LangChain · 2026-10-27
- Prompt turns Claude Code into an iPhone backup forensics tool exposing 30-day 'deleted' photo copies — alex_verem · 2026-10-27
- Prompt turns Claude Code into a forensics tool finding deleted iPhone photos in backups — alex_verem · 2026-10-27
- Phones won't be mobile IDEs but control planes for agent fleets, argues Yuchen Jin — Yuchenj_UW · 2026-10-27
- FrontierCS 2 launches on Kaggle: 48 tasks make agents beat published CS papers — MeganRisdal · 2026-10-27
- Stop remaking old games with AI: a call for more ambitious agentic software — rschu · 2026-10-27