Open 4B decision model cuts agent costs 81-89% and beats the closed model it learned from

alex_verem · x · 2026-10-19

Researchers from UIUC, Princeton and Google released the JevAgentHarness paper: most LLM agent calls are just multiple-choice decisions, so they delegate those to an open 4B decision model (AgentDecision-4B) trained on Jev's answers, calling the big model only when the small one is unsure.

Key findings:

On tasks unlike its training data, the agent solved 91.5% — beating Jev itself (82.0%) at 4x lower cost per decision
On similar tasks: 81-89% cheaper, up to 2.5x faster, success within 3 points
Median response 31ms vs 809ms for a big-model call
Standard benchmarks rank models wrong: the top isolated-test scorer (92.2%) solved only 69.5% inside a real agent; naive delegation without a fitted confidence threshold dropped success by 19 points

The 4B model wins because it trains on states agents reach after their own mistakes, which benchmarks rarely include. Limits: coding agents saved only 7-10.5%; under changed task rules success fell to 3% (retraining lifted decision accuracy from 76.5% to 95.3%). The key training step took 2.2 GPU-hours; models, data and harness will be open-sourced.

Original post →

Open 4B decision model cuts agent costs 81-89% and beats the closed model it learned from

More from coding & agent