96 tools only cost 9 points: agent failures trace to turn-3 error chains, not tool count
EastVersion1226 · reddit · 2026-09-15
A controlled experiment on a 96-tool commerce MCP surface (175 labelled tasks, 4,000 Claude Sonnet 5 calls) upends standard agent advice:
- Tool count barely matters: 5→96 tools costs 9 points; flat till 30, slipping after 60. Compressing all descriptions by 69% changed nothing.
- Retrieval hurt: BM25-filtered 10-tool shortlists scored below showing all 96, because the retriever dropped needed tools. Conditional on the right tool being present, 10 vs 96 performed identically.
- Zero spurious invocations across 35 no-tool questions.
- The real bottleneck is error chains: in 3-step tasks, turn 3 succeeds only 6% of the time at 96 tools; injecting correct state after each step lifts it to 44%. Late-turn failure is inherited error, not task difficulty or tool organization.
- The author also candidly details pitfalls: 3/4 of first-run "model errors" were broken ground truth; a "non-determinism" scare was just JSON key ordering.
Full spec, tool pool and harness will be open-sourced.
More from coding & agent
- Home Blender render & agent farm built with Omarchy, Tailscale and Syncthing in 2 hours — MaxLenormand · 2026-09-15
- Open-source Harness Terminal unifies Claude Code, Codex and more across machines — dee_hw · 2026-09-15
- Reranking more docs backfires: Recall@10 peaks near 100, then declines, study finds — CShorten30 · 2026-09-15
- Old phone becomes a PC stats HUD with one prompt in 90 seconds via vibecoding — Yamapama · 2026-09-15
- Anthropic says Claude writes 80% of its code as engineers ship 8x more — Grady Booch calls the metric 'profoundly stupid' — blaizedsouza · 2026-09-15
- A 6-step roadmap to becoming a Claude Certified Architect, built on Anthropic's free courses — ZabihullahAtal · 2026-09-15