Jev 1.14.0 looks calibrated overall — but is 22 points overconfident on hard questions
shaun117 · reddit · 2026-10-15
A calibration eval of Jev 1.14.0 on 500 binary questions: overall ECE of 0.023 looks near-perfect, but slicing by difficulty reveals easy questions are under-confident by 7 points, medium is on target, and hard questions are strongly overconfident by 22 points (slice ECE 0.222 vs 0.12 threshold). Overall calibration metrics can hide difficulty-stratified miscalibration.
More from Models
- Daily AI brief: Grok Voice 3.0 ships, Claude Code Desktop adds sessions hub — testingcatalog · 2026-10-15
- StepFun releases Step-5-Preview: 600B/27B-active flagship with 1M context — ResearchCrafty1804 · 2026-10-15
- UltraBERT: open 150M/380M encoders with 32K context and an unusual dual-masking recipe — algo_diver · 2026-10-15
- DeepSeek V4.1 measures its own decode speed at 430-460 tok/s — then the claim falls apart — teortaxesTex · 2026-10-15
- OpenCode offers $60 of Gemini 4 Argon usage for $10 until Nov 30 — m4rkmc · 2026-10-15
- After losing 5 Claude accounts, one user's controlled tests show what actually triggers bans — jiayuan_jy · 2026-10-15