One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows
Zhuochun Li, Youngmin Ko, Ali Keramati, Nicola Ferri, Susana Palmaz Lopez Pelaez, Liang-Chun Tsai, Calvin Wang, Mirco Milletari, Tuhin Kundu, Vadim Smolyakov, Kjartan Olafsson, Tommy Guy
cs.CL, cs.DB
2026-08-20
Thinkingbox grades 507 business workflows by terminal state. GPT-5.4 hits 65.36% pass@1 and 91.12% pass@20, but only 25.25% pass^20; 84.86% of failures still look finished.
Current agent benchmarks are good at patches that pass tests, function calls with legal arguments, and web clicks that land. Closing a refund, changing a hotel booking, or updating an auto claim is a different job. The agent has to pull missing facts over multiple turns, follow policy, chain dependent tools, and leave the right persistent state without touching the wrong record. A fluent reply or a valid tool call will score many of those failures as done.
Microsoft, with internship coauthors at Pittsburgh, Northwestern, and UC Irvine, built Thinkingbox: resettable MCP tool sessions, full traces, and grading over terminal backend state. Thinkingbox-bench sits on that sandbox: 507 policy-conditioned workflows across retail, travel and hospitality, auto insurance, neobank internal IT, and consulting IT/HR.
Each task is an executable world: initial backend state, user goal, tool set, simulated-user policy, and hidden executable checks. At each turn the agent either talks to the user or calls a tool. Tools run in an isolated session. Two attempts must not share rows, or pass@k becomes contaminated.
Grading ignores gold trajectories. After termination the sandbox extracts side effects and conjunctively applies the checks. Missing, wrong, and extra persistent effects all fail. 477 tasks score database state only. 30 tasks, 15 each in travel and neobank IT, add a binary response rubric such as a confidentiality constraint. The opening request is often incomplete; the simulated user releases disclosable facts only when asked. The agent has to clarify, refuse ineligible asks, and confirm before irreversible writes.
Domain scale differs. Retail has 98 tasks, 11 backend systems, and 16 write / 17 read tools. Travel has the longest policy at 3,684 words and a median 8.8 actions per task. Consulting has 18 backend systems. Tasks are synthetic reconstructions of private enterprise-ticket patterns. The release contains no real customer records; the five company names are fictional. Each case is reviewed for identity, privacy, solvability, and rollout defects. Broken tools and ambiguous goals are dropped.
Twelve proprietary and open-weight models run 20 independent trials per task. The simulated user is fixed as GPT-5.4-mini. pass@1 is single-attempt success. pass@20 is success at least once in 20. pass^20 is success on all 20. The last number measures repeatability; the middle one measures whether retries can stumble onto a working path.
| Model | pass@1 | pass@20 | pass^20 |
| GPT-5.4 | 65.36% | 91.12% | 25.25% |
| Claude Sonnet 4.6 | 58.45% | 88.56% | 20.12% |
| GPT-5.2 | 46.28% | 84.81% | 8.68% |
| DeepSeek-V4-Pro | 43.26% | 84.62% | 3.55% |
| Claude Opus 4.6 | 37.91% | 70.02% | 13.81% |
| Grok-4.3 | 14.38% | 45.96% | 0.00% |
GPT-5.4 scores 76.33% on retail and 54.60% on consulting. Only it and Sonnet 4.6 clear 50% in every domain. Auto insurance is the hardest, about 23% mean pass@1 across models; retail is about 52%. Opus 4.6 hits 74.90% on retail and 14.65% on auto insurance. DeepSeek-V4-Pro is the strongest open-weight row and sits near GPT-5.2. Mistral-Large-3 has 675B total parameters and 4.66% average. Qwen3.6-27B moves from Qwen3.5-9B's 5.41% to 32.94%; parameter count does not rank the table by itself.
Tool usage dominates failures at 77.5%: an error or failed lookup, then no repair, sometimes continuing as if the call had worked. Skipping the required write is rare at 2.5%. Wrong state updates are 12.1%: the mutating call returns clean, the agent confirms to the user, and the database is still wrong. That share is 27.8% of o3-pro's failures. DeepSeek-V4-Pro leaves 24.7% of its failures in an incomplete user-facing close.
The evaluator ablation is the sharper comparison. Among 79,853 failed trials out of 121,680 valid runs, 84.86% terminate cleanly, 80.88% also issued a state-changing tool, and 67.24% ended with no explicit tool error. Executable checks still find a database-hash mismatch on 98.95% of those failures, a wrong field on 77.61%, and an extra unintended effect on 43.30%. Longer traces are not better traces. GLM-5.1 averages 44.86 messages and 11.99 tool calls and still trails GPT-5.4 at 29.80 messages and 11.05 calls.
This is a τ-bench cousin aimed at enterprise assistants, with MCP isolation and stricter collateral-effect checks. The same loop can grade a leaderboard and supply a training reward. For anyone shipping a support agent, retries that find one working path are not reliability. 65% pass@1 looks usable; 25% all-20 success is not a mandate to hand over irreversible writes.
A well-formed tool call and a clean goodbye are not proxies for completed work. Read the database.
On 477 tasks the verdict ignores what the agent told the user, so a correct write paired with a wrong explanation can still pass. Each task has one gold terminal state; workflows with several defensible resolutions were excluded during construction. The simulated user is easier than a person: it never invents facts, never changes the goal, stays cooperative after repeated failures, and answers at most ten follow-ups. It also shares a model family with the top agent, so interaction-style bias is unmeasured. Tasks are synthetic reconstructions of a private ticket collection and are not claimed to sample enterprise work. Scores further depend on harness conventions such as the termination marker and turn or token caps. o3-pro drops 636 system or harness errors, so its denominators are not fully comparable.