Jev benchmarked: the only sub-second model that admits uncertainty across 16 tested

AlexKim · x · 2026-09-19

TypeSafe's Jev returns typed judgments with calibrated probabilities instead of prose: Choice picks an option, Score rates against ordered levels, Noul outputs the probability a yes/no condition holds. Independent questions share one state and run in parallel, so a twelve-check gate is a single call billed once.

The author measured 16 models on 150 passages (from 35 blogs, identifying the rewritten draft) across 2,400 calls, scoring expected calibration error:

In the deployable sub-second tier, Jev was the best calibrated and the only one reliably admitting uncertainty
Only 4 of 16 models admitted uncertainty at all; Jev is the only one under a second
Jev responded in 455ms, fastest of all sixteen; next honest model, Sonnet 5, was 3.7x slower
The takeaway: unless your code branches on the number, Jev and Haiku are the same model — its slot is an affordable gate you can run on everything

The author also released a Claude Code hook to run the check.

Original post →

Jev benchmarked: the only sub-second model that admits uncertainty across 16 tested

More from coding & agent