Microsoft Tests: Agent Success Rate Drops from 65% to 25% on Repeats
samueljmcd · x · 2026-08-30
Microsoft tested 507 real MCP workflows in a sandbox called Thinkingbox. The agent completed the job 65% of the time on the first try, but that rate fell to 25% when asked to do the same job twenty times in a row.
Key takeaway: A clean tool call is not a completed job. Reliability is the core challenge.
More from coding & agent
- Simon Willison: code mode is the future of tool calling, if it can be sandboxed — springrod · 2026-10-16
- Agent named 'Ilya' reportedly scores 1.1pt higher on benchmarks — daniel_mac8 · 2026-10-16
- Gemini CLI fix: substring matching inlined unrequested binary files into model context — priyanshu1976 · 2026-10-16
- Building AI agents on real healthcare claims data: the domain work that makes them work — alex_verem · 2026-10-16
- OpenAI stealth-testing mysterious internal model 'Aeon' in Codex desktop app and Dots — cedric_chee · 2026-10-16
- Vibe-coded iPhone Duo sniping bot built with Codex in 17 minutes, 41 tests passing — xiaohu · 2026-10-16