Microsoft's ThinkingBox: Kimi-K3 tops discovery at 93.89% but only 13.41% solve tasks 20/20

tuhin_k · reddit · 2026-10-09

Microsoft researchers released Thinkingbox-Bench: 507 policy-conditioned business workflows across 5 domains, each run 20 times from an identical clean backend, graded on terminal database state (10,140 trials per model).
Key finding: discovery and repeatability rank models in nearly reversed orders — Kimi-K3 solves 93.89% of tasks at least once but only 13.41% on all 20 attempts, while Claude Opus 5 covers fewer (79.09%) but repeats far more (47.53%). Qwen3.8-27B: 89.35% vs 7.50%.
In an ablation over 121,680 trials, 67.24% of failures terminated cleanly with no tool error — a completion-style proxy would have scored them as done; 77.61% involved wrong field values, 43.30% unintended side effects.
Paper, code, and dataset are public; tasks runnable via HF OpenEnv.

More from coding & agent