Benchmark: Claude Code calls your MCP tool only 34.5% of the time when it overlaps built-ins
Future_AGI · reddit · 2026-11-03
Analysis of the Sineframe benchmark (3,717 scored trials across Claude Code, OpenCode, Codex) shows MCP tool calls collapse when a server overlaps built-in tools:
- On 11 servers with no built-in equivalent, Claude Code called the server 93.5% of the time; on 10 overlapping ones (WebSearch/WebFetch/Read), that fell to 34.5%. Codex dropped from 92.1% to 17.2%; OpenCode held best at 66.3%
- Many misses still had correct answers — Claude Code answered 378 overlapping trials via its own tools (342 with the expected value), so your server never sees the request or logs anything
- Contributing factor: Claude Code defers MCP tool loading and truncates server instructions at 4,096 characters
Recommended client-side eval practice: 20–30 tagged prompts, run each with built-ins on and off, 30 runs per prompt, rerun after client updates. The author open-sourced an eval library that scores skipped calls as failures.
More from coding & agent
- OpenClaw cut its ~$1M/month CI bill by over 95% with adaptive agentic testing — vincent_koc · 2026-11-04
- ParallACT benchmark: parallel coding agents in shared repos drop from 62% to 16% on conflicting tasks — EliasEskin · 2026-11-04
- Perplexity Computer can now run Meta ads, Klaviyo email and Shopify stores end-to-end — perplexity_ai · 2026-11-04
- Filarise shows finance MCP server letting ChatGPT query your bank via Plaid — ChampionSpecialist60 · 2026-11-04
- Multi-agent messaging barely helps: 61% of agreed runs still hit merge conflicts — EliasEskin · 2026-11-04
- worldinmotion: open-source MCP server for encrypted agent-to-agent messaging across machines, with no approve tool by design — distronode · 2026-11-04