Jev benchmarked: the only sub-second model that admits uncertainty across 16 tested
AlexKim · x · 2026-09-19
TypeSafe's Jev returns typed judgments with calibrated probabilities instead of prose: Choice picks an option, Score rates against ordered levels, Noul outputs the probability a yes/no condition holds. Independent questions share one state and run in parallel, so a twelve-check gate is a single call billed once.
The author measured 16 models on 150 passages (from 35 blogs, identifying the rewritten draft) across 2,400 calls, scoring expected calibration error:
- In the deployable sub-second tier, Jev was the best calibrated and the only one reliably admitting uncertainty
- Only 4 of 16 models admitted uncertainty at all; Jev is the only one under a second
- Jev responded in 455ms, fastest of all sixteen; next honest model, Sonnet 5, was 3.7x slower
- The takeaway: unless your code branches on the number, Jev and Haiku are the same model — its slot is an affordable gate you can run on everything
The author also released a Claude Code hook to run the check.
More from coding & agent
- AI agent plays Subway Surfers at superhuman speed, 50 games at once, for under a cent — ravithejads · 2026-09-19
- User proposes Codex thread usage panel to track per-thread impact on weekly quota — CtrlAltDwayne · 2026-09-19
- MSP IT Admins Want Agents That Can Patch, Heal and Report — Copilot Isn't It — PerfectReflection155 · 2026-09-19
- Flet 1.0 hits Hacker News front page: Python cross-platform apps spark heated debate — dhruv2038 · 2026-09-19
- jev-sift: open-source MCP tool that lets agents classify before reading — gaganghotra_ · 2026-09-19
- Ex-academic joins Meta to make models good at browser use in Muse Spark CUA — EdwardSun0909 · 2026-09-19