Good Judgment, Bad Advice: Decision-Model Controllers Raise Agent Cost Without Improving Success
Naman Vats
cs.AI, cs.LG
2026-10-17
A pre-specified controller swap on 315 Terminal-Bench trials: Jev 1.13 raised Sonnet 5.5 cost per attempt 19% while the pass rate stayed 73.0%.
A terminal agent's bill is mostly the reasoning model. The scaffold still has to decide when to stop, when to recheck, and when to change approach. A typed decision model is built for that multiple-choice moment: it reads a state and a fixed menu, then returns one option and a probability. Jev 1.13 is priced at $0.042 per million input tokens. In these runs one decision cost about $0.00008.
A low price does not by itself save money. The score has to say something about how the run will end, and the action has to turn that score into a shorter trace. Judge benchmarks usually measure only the score.
This study freezes Claude Sonnet 5.5 and one loop, and swaps only the controller. The arms are no controller, handwritten rules, Claude Haiku 5.5 answering the same questions, and a placebo. The placebo fires at the same moments but draws from Jev's development-phase choice rates, so the design can separate the content of a decision from the mere interruption.
The benchmark is Terminal-Bench 2.1. Calibration left 21 held-out tasks (14 easy, 3 mid, 4 hard). Eight development tasks supplied the placebo frequencies, and nothing was tuned. Five arms ran three times each: 315 trials, capped at 60 steps and $1.00.
Each step is one Sonnet 5.5 call. The model may run bash or ask to finish. Commands go through Harbor in a Daytona sandbox. Triggers are logged on every arm, including the arm that never calls a controller.
Rules recover on T1 and T2, continue on T4, and verify on T3 unless a check just ran. Abandon is allowed only after four straight failures and 40% of the step budget. Haiku 5.5 has reasoning turned off and reads the same option text as Jev. The placebo ignores the state. On progress questions it continues with probability 0.625, investigates 0.125, recovers 0.25, and never abandons. On finish questions it verifies with probability 0.348. Guards override every controller: an early abandon becomes recover, and the third finish request is accepted after two verifies. The pre-specified confirmatory test is Jev against no controller. Cost is the geometric mean of per-task spend ratios. The pass-rate margin is a 10 percentage point drop.
Jev's geometric-mean cost ratio against no controller is 1.19 (95% CI 1.02 to 1.36, p=0.026). Both arms passed 73.0% of attempts (difference 0.0 pp, CI -11.1 to +12.7). The one-sided lower bound is -9.5 pp, just inside the -10 pp line. That is not a non-inferiority result. The claim that Jev reduces cost is rejected.
| Controller | Pass rate | USD/attempt | USD/solved | Steps | Verify msgs |
| none | 73.0% | 0.177 | 0.243 | 13.3 | – |
| rules | 73.0% | 0.191 | 0.262 | 15.6 | 0.92 |
| Haiku 5.5 | 76.2% | 0.213 | 0.280 | 17.8 | 1.30 |
| Jev 1.13 | 73.0% | 0.206 | 0.283 | 16.0 | 0.51 |
| placebo | 71.4% | 0.156 | 0.218 | 13.2 | 0.33 |
Jev was cheaper than Haiku (ratio 0.81), close to rules (0.96, interval crosses 1), and 22% more expensive than the placebo (1.22, CI 1.08 to 1.39). Corrected pass-rate tests all came back at p=1.00. Against no controller, rules, Haiku, and Jev raised cost by 1.24, 1.47, and 1.19. The placebo ratio was 0.98.
Scored offline, without acting, Jev ranked 53 finish states at AUROC 0.78, against 0.62 for rules and 0.60 for Haiku. Those runs passed 87% of the time, yet Jev's mean probability of finish was 0.50, so it demanded verification on 29 of 53. Argmax accuracy was 0.55, worse than always accepting the finish request (0.87). The ranking still sorts: 23 of 24 runs at probability 0.5 or higher passed, and so did 23 of 29 below 0.5. On progress questions no controller abandoned, and Jev's AUROC was 0.55.
Jev cost $0.029 more per attempt. Of that, $0.011 came after the first verification message; the rest came with longer traces (16.0 steps versus 13.3). Fifteen of 21 tasks got more expensive, and dropping any one task left the ratio between 1.16 and 1.25. Abandon was never chosen in 315 trials. The controller itself was cheap: 0.52 seconds and 0.11% of spend for Jev, 1.46 seconds and 0.47% for Haiku.
A useful ranking did not become a saving. With no controller, finish requests were already right 87% of the time, so verification mostly rechecked correct work. The probabilities sit under the base rate, and acting on the top option therefore interrupts often. The group Jev trusted least still passed 79% of the time. Ten of 17 failures without a controller died at a timeout or the budget cap. Abandon, the one action designed to save money, was never used.
The placebo isolates the cause. Random notes at the same moments matched the pass rate at a cost ratio of 0.98. The extra spend comes from which option Jev picked.
A controller belongs in an end-to-end cost comparison against rules and a placebo. Call overhead is not the constraint. The policy that turns a probability into an action is. Nineteen percent more per attempt is nineteen percent more on every task.
Fourteen of 21 held-out tasks are easy and only three are mid-level, so pass-rate intervals run about ±11 to 13 percentage points. The cost finding is stable. The pass-rate finding is weak either way. The study uses one reasoning model, one minimal loop, and Jev 1.13. Sonnet 5.5 has no temperature, reasoning effort is fixed high, and hosts were not pinned. The pre-specified open-weight rerun was not done, because quantizations differed by host.
Placebo rates come from 8 progress decisions and 23 finish decisions, with abandon fixed at probability 0. They test which option is chosen, not how often a controller speaks. Offline labels are the trial outcome, not the counterfactual of a different action. The abandon guard never blocked a request, which only shows that nobody asked to stop. Whether a policy that actually wants to quit would be stopped by the 40% step gate is untested.