16-model calibration test: Jev fastest honest model but ranks 10th in accuracy

Developer AlexKim ran a systematic test across 16 models to decide whether TypeSafe's Jev deserved a spot in his stack, focusing on comparing Jev with Haiku—and unexpectedly found that "calibration," a metric he never planned to measure, turned out to be the most discriminating one.

Confirmed

• Jev ranks only 10th out of 16 models in accuracy; the author went through several rounds of revised conclusions: AI-generated draft conclusions contradicted each other—one claimed "Jev's confidence is usable, Haiku's isn't," another said "Haiku is more accurate"—and only after correct statistics and additional tasks was it established that Jev and Haiku tie on the key calibration metrics.
• Ambiguity-zone (0.35–0.65) answer rates: Sonnet 5 reaches 41.3%, Fable 5.1 hits 36.7%, Jev 34.7%, and Opus 5 23.3%, with a cliff-edge drop after the top four; open-source models almost never fall in the ambiguity zone—in other words, they almost never say "I'm not sure."
• Jev is the only one of the four models that honestly admits uncertainty while responding in sub-second time: 455 milliseconds, the fastest of all 16; the next-fastest honest model, Sonnet 5, needs 3.7x as long. The author believes this makes Jev suitable as a gate for full-volume request handling.
• Neither price nor release date predicts calibration performance: the most expensive kimi-k3 ranks last in both accuracy and calibration; deepseek-v4-flash ran the entire test suite at $0.003 and actually beat Jev on accuracy.
• Jev returns structured judgments with calibrated probabilities rather than text: Choice picks one from the options, Score rates on an ordinal scale, and Noul outputs the probability that a yes/no condition holds—no prose, no reasoning trace; independent questions can share the same state and be handled in a single request.

Why it matters

• The author stresses that neither price nor "pedigree" determines whether a model will admit uncertainty; model selection should rely on measured calibration data rather than price tags.
• One task does not make a benchmark: drawing on his own experience, the author shows that single-task conclusions (e.g., "Haiku is more accurate") can flip as more tasks are added; evaluation conclusions must repeatedly survive in the face of data.
• The "fast + honest" combo gives Jev a unique niche: it can serve as a low-cost, low-latency gate for full request volume, working in division of labor with other models.

2026-09-19 ~ 2026-09-19 · 8 related posts

• Dev tests 16 models to evaluate TypeSafe's Jev — it ranked 10th on accuracy — AlexKim · 2026-09-19
• Dev runs 16-model eval: Jev and Haiku tie at 0.121/0.122 on calibration error — AlexKim · 2026-09-19
• 16-model eval finds Sonnet 5 hedges most, landing in ambiguous zone 41.3% of the time — AlexKim · 2026-09-19
• [source] I ran 16 models to vet one tool: one task is not a benchmark — AlexKim · 2026-09-19
• [source] 16-model calibration test: open-weight models almost never admit uncertainty — AlexKim · 2026-09-19
• Priciest model finishes last on accuracy and calibration in 16-model test — AlexKim · 2026-09-19
• Jev answers in 455ms — a gate you can afford to run on everything — AlexKim · 2026-09-19
• [source] Jev benchmarked: the only sub-second model that admits uncertainty across 16 tested — AlexKim · 2026-09-19