16-model calibration test: open-weight models almost never admit uncertainty
AlexKim · x · 2026-09-19
A calibration test across 16 models measured how often each puts answers in the ambiguous 0.35–0.65 middle: Sonnet 5 leads at 41.3%, Fable 5.1 at 36.7%, Jev 34.7%, Opus 5 at 23.3%, then a cliff — nothing else clears 7.3%. GPT-5.5 is the best of its family at 11.3%, still below the group, and no open-weight model clears it, with kimi-k2.5 and deepseek-v4-pro at just 0.7%.
More from Models
- Hindsight ships 0.8B model for agent memory decisions: 2.5x cheaper, 1.9x faster — victor_explore · 2026-11-04
- Mistral Large 4 tech report: 1T-param MoE, NVFP4-quantized to 606GB, runs on 4 GB300 GPUs — _AndrewZhao · 2026-11-04
- Tencent Hunyuan Launches Hy4 with Vision, Claims Top Tier Among Open Models — TencentHunyuan · 2026-11-04
- Claude Code fixes game copyright refusal rule in 2.1.317: naming a game now works — apples_jimmy · 2026-11-04
- Microsoft Phi-4-mini: a 3.8B model with 128K context and 52.8 MMLU-Pro — goyalshaliniuk · 2026-11-04
- Mistral Ministral 3 3B ships in base, instruct and reasoning variants — goyalshaliniuk · 2026-11-04