Dev runs 16-model eval: Jev and Haiku tie at 0.121/0.122 on calibration error
AlexKim · x · 2026-09-19
A developer shares a systematic eval of 16 models, focusing on TypeSafe's Jev vs Haiku:
- Two AI-written draft conclusions contradicted each other: one claimed "Jev's confidence is usable, Haiku's is not," another said "Haiku was more accurate."
- Computing expected calibration error properly gave Jev 0.121 vs Haiku 0.122 — a tie on the very metric the product is built around.
- Multi-task results split: Haiku wins business categories 83.2 to 79.9, Jev wins commit types 50.0 to 42.0, prose dead-ties at 66.0.
- Key lesson: one task is not a benchmark, and AI-drafted eval conclusions shouldn't be trusted without recomputation.
More from Models
- Community asks Moonshot when Kimi K3.1 model weights will drop — bgmshana · 2026-11-04
- Meta's Muse Spark Open license counts distilled models as derivatives — aidenclarke_12 · 2026-11-04
- Tencent Hy4 trails GPT-6.1 on hard tests as Kimi K3.1 Pro beats it on two benchmarks — heyshrutimishra · 2026-11-04
- Tencent's Hy4 agent self-trained a smaller model that beat Tencent's own baseline — heyshrutimishra · 2026-11-04
- Hy4 tops open models in finance eval at 3.04, hallucination rate cut to 5.45% — heyshrutimishra · 2026-11-04
- Tencent Hunyuan teases upcoming LMArena debut — TencentHunyuan · 2026-11-04