Microsoft & HF turn database checks into RL rewards, lifting Qwen3.8-27B agents from 51.7% to 60.78%

SergioPaniego · x · 2026-10-24

Microsoft and Hugging Face published a follow-up post on ThinkingBox, a benchmark and training environment where an agent's success is graded by comparing the final database state against the expected outcome — designed to catch agents that claim they're done when a real backend says otherwise (507 tasks, each with a real backend and tool set).

The key insight: those executable checks aren't just graders — they can serve as reward functions. The team built a training pool of ThinkingBox-format tasks outside the benchmark and post-trained Qwen3.8-27B with GRPO.

Results

Average success rate rose from 51.70% to 60.78% after 50 GRPO steps
Tasks completed correctly on all 20 attempts jumped from 38 to 107
The leaderboard shows the RL-trained 27B model surpassing Claude Sonnet 4.6 (59.19) and GPT-6 Astra (58.31), approaching GPT-5.6 Sol (61.91), though still behind Claude Opus 5.5 (67.16)

ThinkingBox-Training is open-sourced on GitHub.

Original post →

Microsoft & HF turn database checks into RL rewards, lifting Qwen3.8-27B agents from 51.7% to 60.78%

More from coding & agent