Freshly trained decision models are overconfident: claims 90% sure but only 68% correct
helloiamleonie · x · 2026-11-06
In a technical thread, Liquid AI's Leonie explains problem #2 with decision models: right after training they are overconfident — saying 90% sure while being correct only 68% of the time. Since confidence scores are what make these models useful, calibration is essential (after calibration: says 90%, right 88%).
Related event: Liquid AI: Temperature Scaling Beats RL for Confidence Calibration(2 posts)→
More from Models
- Researcher: Opus over-refuses simple safety-topic questions, even on existing writeups — DimitrisPapail · 2026-11-06
- Apple's GSM-Symbolic paper: renaming numbers sinks SOTA models' math scores — imjustnewatai · 2026-11-06
- Oc-e-Non-v1: a tersified Qwen3-4B that cuts response length by 50%, built with Heretic — -p-e-w- · 2026-11-06
- Robot RSI: testing whether frontier models can set up robot environments — qinzytech · 2026-11-06
- NetEase Youdao's Confucius4-R2T2: Streaming ASR That Never Rewrites Its Output — dr_cintas · 2026-11-06
- Concept Country open-sources Qwen Concept 27B: 54% fewer thinking tokens via weight-space edits — GabGarrett · 2026-11-06