Microsoft's Thinkingbox: best model hits only 65% pass@1 on real business workflows
dair_ai · x · 2026-08-23
Microsoft released Thinkingbox, a sandbox plus benchmark for evaluating agent reliability in real business workflows:
- The sandbox provides isolated MCP-compatible tool sessions.
- The benchmark spans 507 policy-conditioned workflows across retail, hospitality, auto insurance, neobank IT, and consulting support.
- Grading ignores what the agent says and inspects the backend state it leaves behind: executable checks accept valid trajectories and reject wrong, missing, or extra effects — collateral damage counts against you.
The strongest model reaches just 65.36% pass@1 and 25.25% pass^20. Notably, many failed trials terminate cleanly with valid state-changing tool calls — watching the response or tool call tells you almost nothing about whether the task actually completed.
Related event: Microsoft Paper: Agents Succeed Often but Rarely Reliably(2 posts)→
More from coding & agent
- YC F26's Praxis launches AI agents to run law firm ops end to end — ycombinator · 2026-10-16
- Codex's year-long rise: 40M+ weekly users and its leader now runs all of OpenAI's core products — danshipper · 2026-10-16
- Developer builds inter-agent messaging for his AI agents — complete with message undo — baditaflorin · 2026-10-16
- How Codex Happened: inside OpenAI's comeback in coding — danshipper · 2026-10-16
- iOS 'dots' workflow: one Siri command turns screenshots into a background shopping and saving agent — nickbaumann_ · 2026-10-16
- Your widget tree is already a list of options: where Jev fits in Flutter — rseroter · 2026-10-16