Study: decision-model controllers raise agent cost 19% with zero success gain
Naman Vats · hf · 2026-10-21
A pre-specified study by Naman Vats (BIT Mesra) on Terminal-Bench 2.1 holds a Claude Sonnet 5.5 terminal agent loop fixed and swaps only the controller for small stop/verify/change-approach decisions.
Key findings:
- The typed decision model (Jev 1.13) raised cost per attempt by 19% (geometric-mean ratio 1.19, 95% CI 1.02–1.36) at an identical pass rate (73.0% vs 73.0%), and was 22% pricier than a placebo controller
- Jev ranked finish requests well offline (AUROC 0.78 vs Haiku 5.5's 0.62, random 0.50) but was poorly calibrated: runs it judged likely to fail passed 87% of the time, triggering useless verification
- No controller ever chose to abandon a run — informative decision signals did not become useful interventions
Code released at github.com/namanvats/decision-model-controllers. A solid empirical counterexample to the popular 'cheap model as judge' scaffold pattern.
More from coding & agent
- Pinekin to open-source: batch-sync conversation context across AI agent harnesses — yangyi · 2026-10-21
- CoEvo-RSI: NUS-led paper tackles brain-body mismatch in self-improving agents — Jiaqi Su · 2026-10-21
- Open-source Hermes video toolchain turns agent requests into finished MP4s — Teknium · 2026-10-21
- Zed tested 25 MCP tool search approaches, picked weighted BM25 over LLMs and embeddings — jobergum · 2026-10-21
- Argon excels as a video director, says Bindu Reddy, orchestrating SeeDance and ElevenLabs — bindureddy · 2026-10-21
- OmniGraph Open-Sourced: Git-Style Branches and Merges for AI Agent Memory — techNmak · 2026-10-21