MCP testing: one question went from 437k to 78k input tokens — cost is driven by round trips
loookashow · reddit · 2026-09-08
A detailed engineering postmortem on agent-testing an MCP server: conformance testing found nothing, while 18 real-agent scenarios found five issues in a week. Key lessons: hosted connectors resend the whole conversation every tool-loop iteration, so cost equals returned payload times remaining turns (one scenario dropped from 437k to 78k input tokens); Anthropic's connector caps at 10 tool iterations and returns pauseturn with HTTP 200, which naive clients log as success; JSON-RPC notifications are defined by a missing id, not the method name; and checks should allow agent exploration — a "no tool errors" assertion wrongly failed scenarios where guessing and recovering was the system working. Page size now defaults to 5, capped at 100, published in the tool schema.
More from coding & agent
- Turn PRs into RL environments at scale — startups already raised seed rounds on this repo — lvwerra · 2026-09-08
- Self-Improving AI's First Payoff Is Cheaper, Not Smarter: Ben Lorica on the Harness Advantage — bigdata · 2026-09-08
- Higgsfield touts GPT-6 Astra agent running full ad post-production across Premiere, Resolve and Ableton — _AustinCalvert_ · 2026-09-08
- Veteran engineer pushes back on AI code quality rants: human code was never clean either — PoorDecisionMaker-69 · 2026-09-08
- x402 is a closed corporate spec, not open source — and a credits system beats it — HotAisle · 2026-09-08
- Vite+ v0.3.1 ships vp env to replace Corepack, slims Windows shim from 214 to 14 KiB — cnakazawa · 2026-09-08