Microsoft Paper: Agents Succeed 91% Once but Are Reliable Only 25% of the Time
rohanpaul_ai · x · 2026-08-23
A new Microsoft paper highlights that "one success isn't reliability" for AI agents. While the best agent solved 91% of business tasks at least once, it consistently succeeded only 25% of the time.
Key Findings:
- Silent Failures: 4 out of 5 failed runs ended politely and appeared to complete successfully (calling database tools), but the database records indicated the job was actually not done.
- Solution (ThinkingBox): Microsoft introduced ThinkingBox, a sandbox that runs agents against real tools, simulated customers, and live backends. It evaluates reliability by checking the database state afterward instead of trusting the agent's reply.
Paper: arxiv.org/abs/2608.19741
Related event: Microsoft Paper: Agents Succeed Often but Rarely Reliably(2 posts)→
More from coding & agent
- YC F26's Praxis launches AI agents to run law firm ops end to end — ycombinator · 2026-10-16
- Developer builds inter-agent messaging for his AI agents — complete with message undo — baditaflorin · 2026-10-16
- iOS 'dots' workflow: one Siri command turns screenshots into a background shopping and saving agent — nickbaumann_ · 2026-10-16
- Your widget tree is already a list of options: where Jev fits in Flutter — rseroter · 2026-10-16
- How OpenAI's Codex went from failed side project to 40M weekly users in under a year — danshipper · 2026-10-16
- Photon 2.0 launches unified messaging API for AI agents across iMessage and more — Scobleizer · 2026-10-16