Microsoft Paper: Agents Succeed 91% Once but Are Reliable Only 25% of the Time
rohanpaul_ai · x · 2026-08-23
A new Microsoft paper highlights that "one success isn't reliability" for AI agents. While the best agent solved 91% of business tasks at least once, it consistently succeeded only 25% of the time.
Key Findings:
- Silent Failures: 4 out of 5 failed runs ended politely and appeared to complete successfully (calling database tools), but the database records indicated the job was actually not done.
- Solution (ThinkingBox): Microsoft introduced ThinkingBox, a sandbox that runs agents against real tools, simulated customers, and live backends. It evaluates reliability by checking the database state afterward instead of trusting the agent's reply.
Paper: arxiv.org/abs/2608.19741
More from coding & agent
- Codex Designs Its Own CPU and Runs Doom — Angaisb_ · 2026-08-23
- OpenAI Acquires Instant's Entire Team to Add a Memory Layer for Agents — 新智元 · 2026-08-23
- PostSyncer lets AI Agents post via MCP & REST API — tibo_maker · 2026-08-23
- Fixed 5/5 unseen bugs with one example: generative router in action — No-Program-5087 · 2026-08-23
- Debug LLM features by asking for them directly, fixing Mermaid charts on Remote iOS — Dimillian · 2026-08-23
- Build a Meta-agent: Using 'Agentic Engineer' to audit coding workflows — daniel_mac8 · 2026-08-23