Microsoft's Thinkingbox: best model hits only 65% pass@1 on real business workflows
dair_ai · x · 2026-08-23
Microsoft released Thinkingbox, a sandbox plus benchmark for evaluating agent reliability in real business workflows:
- The sandbox provides isolated MCP-compatible tool sessions.
- The benchmark spans 507 policy-conditioned workflows across retail, hospitality, auto insurance, neobank IT, and consulting support.
- Grading ignores what the agent says and inspects the backend state it leaves behind: executable checks accept valid trajectories and reject wrong, missing, or extra effects — collateral damage counts against you.
The strongest model reaches just 65.36% pass@1 and 25.25% pass^20. Notably, many failed trials terminate cleanly with valid state-changing tool calls — watching the response or tool call tells you almost nothing about whether the task actually completed.
Related event: Microsoft Paper: Agents Succeed Often but Rarely Reliably(2 posts)→
More from coding & agent
- How to hook Claude Code or Codex up to iMessage in 3 steps — jacob_posel · 2026-10-16
- Are the fastest GPU kernels now written by AI? Kernels are hyper-verifiable, argues one dev — fleetwood___ · 2026-10-16
- Anthropic lead engineer's 4-question prompt makes agents more ambitious, now a free skill — daniel_mac8 · 2026-10-16
- ChainForge dev shows 'feature map' for orchestrating subagents on large PRs — IanArawjo · 2026-10-16
- aiDotEngineer benchmarks agent orchestration pipelines, staging humans as Foreman and workers — vikvang1 · 2026-10-16
- Developer mass-produces 200 videos in hours using Claude Code + Hyperframes — mcraddock · 2026-10-16