Dev Tests 16 Models on Calibration: Jev Fast and Honest but Only 10th in Accuracy
To decide whether TypeSafe's Jev deserved a place in his stack, developer AlexKim ran a systematic evaluation across 16 models, focusing on Jev versus Haiku — and unexpectedly found that "calibration," a metric he had barely planned to test, turned out to be the most discriminating.
Confirmed
- Jev ranked only 10th in accuracy among 16 models; the author went through multiple rounds of revised conclusions: AI-generated draft conclusions contradicted each other (one claiming Jev's confidence was usable and Haiku's not, another that Haiku was more accurate), and only after correct statistics and additional tasks did he conclude Jev and Haiku tie on the key calibration metrics.
- On the rate of answering in the ambiguous 0.35–0.65 band: Sonnet 5 hit 41.3%, Fable 5.1 36.7%, Jev 34.7%, Opus 5 23.3%, with a cliff after the top four; open-source models almost never landed in the band, i.e., almost never said "uncertain."
- Jev was the only sub-second model among the four "honest" ones: 455 ms, fastest of all 16; the next-fastest honest model, Sonnet 5, took 3.7x longer. The author sees this enabling Jev as a gate for the full request volume.
- Neither price nor release date predicted calibration: the most expensive model tested, kimi-k3, ranked last in both accuracy and calibration; deepseek-v4-flash completed all tests for $0.003 yet beat Jev in accuracy.
- Jev returns structured judgments with calibrated probabilities rather than text: Choice picks from options, Score rates on an ordinal scale, Noul outputs the probability a yes/no condition holds — no prose, no reasoning trace; independent questions sharing one state can be handled in a single request.
Why it matters
- The author stresses that price and "pedigree" do not determine whether a model admits uncertainty; selection should rely on measured calibration data, not price tags.
- One task does not make a benchmark: his own experience shows single-task conclusions (e.g., "Haiku is more accurate") can flip as tasks are added; evaluation conclusions must repeatedly survive the data.
- The "fast + honest" combination gives Jev a unique niche: a low-cost, low-latency gate for full request volume, complementing other models.
2026-09-19 ~ 2026-09-19 · 8 related posts
- Episode 1: TypeSafe exits stealth with decision model Jev and RLCD training method(2026-09-16, 86 posts)
- Episode 2: TypeSafe AI launches Jev, a dedicated evaluation model showing major speed and cost gains in tests(2026-09-16, 8 posts)
- Episode 3: Vercel fx to adopt Jev safety reviewer, up to 18x faster(2026-09-17, 3 posts)
- Episode 4: OpenJev Open-Source Clone Runs Jev-Style API on One RTX 3090(2026-09-17, 3 posts)
- Episode 5: TypeSafe AI Launches Jev, a Decision-Only Model That Outputs No Text(2026-09-17, 73 posts)
- Episode 6: Mystery decision model Jev sparks wave of community benchmarks(2026-09-18, 6 posts)
- Episode 7: TypeSafe AI's Jev model and RLCD method spark debate(2026-09-18, 3 posts)
- Episode 8: LangChain Publishes Tutorial on Building an Agent Harness with Jev(2026-09-18, 7 posts)
- Episode 9: Jev Ecosystem Explodes with Wave of Popular GitHub Projects(2026-09-18, 4 posts)
- Episode 10: Jev, the Decision-Only AI Model, Goes Viral for Speed and Cost(2026-09-18, 33 posts)
- Episode 11: TypeSafe Jev Decision Models Spark Debate on Breakthrough, Skepticism and Replication(2026-09-18, 7 posts)
- Episode 12: Jev Matches Classifiers at 100x Lower Cost(2026-09-19, 2 posts)
- Episode 13: Jev Positioned as LLM Router, Not a Replacement(2026-09-19, 2 posts)
- Episode 14: Developers Hail Jev as Top Value Model for Batch Labeling(2026-09-19, 2 posts)
- Episode 15: JevBench Launches: Jev 1.13 Holds Narrow Lead as Open-Source Rivals Close In(2026-09-19, 8 posts)
- Episode 16: TypeSafe AI's Decision Model Jev Explodes in Popularity, Setting Vercel AI Gateway Adoption Record(2026-09-19, 14 posts)
- Episode 17: omarsar0 Tests Jev as a New Primitive: Cheap Verifiers for Long-Running Agents(2026-09-19, 10 posts)
- Episode 18: Dev Tests 16 Models on Calibration: Jev Fast and Honest but Only 10th in Accuracy(2026-09-19, 8 posts)
- Episode 19: Independent Tests Confirm Jev Classifier's Accuracy at Fractional Cost(2026-09-19, 4 posts)
Primary sources
- Dev tests 16 models to evaluate TypeSafe's Jev — it ranked 10th on accuracy — AlexKim · 2026-09-19
- Dev runs 16-model eval: Jev and Haiku tie at 0.121/0.122 on calibration error — AlexKim · 2026-09-19
- 16-model eval finds Sonnet 5 hedges most, landing in ambiguous zone 41.3% of the time — AlexKim · 2026-09-19
- [source] I ran 16 models to vet one tool: one task is not a benchmark — AlexKim · 2026-09-19
- 16-model calibration test: open-weight models almost never admit uncertainty — AlexKim · 2026-09-19
- Priciest model finishes last on accuracy and calibration in 16-model test — AlexKim · 2026-09-19
- [source] Jev answers in 455ms — a gate you can afford to run on everything — AlexKim · 2026-09-19
- [source] Jev benchmarked: the only sub-second model that admits uncertainty across 16 tested — AlexKim · 2026-09-19