Jev 1.14.0 Well-Calibrated Overall but Overconfident by 22 Points on Hard Questions

An evaluation of Jev 1.14.0 on 500 binary questions shows near-perfect overall calibration (ECE 0.023), but splitting by difficulty reveals it is underconfident on easy questions and overconfident by 22 points on hard ones.

2026-10-15 ~ 2026-10-15 · 2 related posts

• Jev 1.14.0 looks calibrated overall but is 22 points overconfident on hard questions — shaun117 · 2026-10-15
• Jev 1.14.0 looks calibrated overall — but is 22 points overconfident on hard questions — shaun117 · 2026-10-15