Jev 1.14.0 looks calibrated overall but is 22 points overconfident on hard questions
shaun117 · reddit · 2026-10-15
A Reddit-crossposted eval finds Jev 1.14.0 appears well calibrated on 500 binary questions (ECE 0.023, within chance for a perfectly calibrated model) — but slicing by difficulty reveals a problem:
- Easy: under-confident by 7 points
- Medium: roughly on target
- Hard: overconfident by 22 points, with slice ECE of 0.222 vs a 0.12 threshold
The accompanying chart also compares Jev, subq-snap-preview, and gpt-6-luna; gpt-6-luna shows even stronger overconfidence on hard questions. Takeaway: aggregate calibration metrics can hide systematic overconfidence exactly where it matters most.
More from Models
- Score vs cost chart puts Gemini 4 Argon on the Pareto frontier, Kimi K3 far off it — Hesamation · 2026-10-15
- OpenRouter data shows Anthropic's Opus 5.5 clawing back share from OpenAI within days — menhguin · 2026-10-15
- 189 benchmarks later: how to pick KV quants for Qwen3 27B on 16GB VRAM — Fancy-Snow7 · 2026-10-15
- Follow-up: NVIDIA teases new model ahead of possible GTC reveal — testingcatalog · 2026-10-15
- NVIDIA set to launch Nemotron 3.5 Super VL: 120B params, 12B active, image and video input — testingcatalog · 2026-10-15
- IFM Releases Open-Source Jais 3, Focused on Arabic and Agentic Tasks — valcore93 · 2026-10-15