Microsoft Open Sources Thinkingbox: Stateful Agent Workflow Benchmark
microsoft · hf · 2026-08-25
Microsoft released Thinkingbox, a sandbox and benchmark for evaluating agents in stateful business workflows. Unlike single-turn benchmarks, it requires multi-turn information gathering, policy adherence, and correct persistent state transitions.
- Scenarios: Covers 507 policy-conditioned workflows across retail, hospitality, auto insurance, neobank IT, and consulting.
- Evaluation: Uses task-specific executable checks to validate trajectories and reject side effects.
- Findings: The top model achieves 65.36% pass@1 but only 25.25% pass@20. Many failures involve valid-looking actions, revealing that single-step signals are poor proxies for end-to-end reliability.
Code and benchmarks are open-sourced on GitHub.
More from coding & agent
- Self Labs launches Agent Vault to let AI agents log in and pay with credit cards — EXM7777 · 2026-10-16
- Plane Agents: triggers, not prompts, are what turn chatbots into workflow agents — JosephJacks_ · 2026-10-16
- Claude eng lead Felix Rieseberg publishes tutorial on being more ambitious with AI — danshipper · 2026-10-16
- Qwen Code Desktop v0.25.2 ships managed-agent fixes and MCP config repair — github-actions[bot] · 2026-10-16
- NVIDIA makes 7 AI agent courses free, from agent basics to securing agents — ifioknkem · 2026-10-16
- Claude runs 9-hour task, then stalls waiting for Drive permission — juliey4 · 2026-10-16