DIY calibration test: 200 items, score accuracy vs self-claimed 90% confidence
colinmcnamara · x · 2026-09-24
colinmcnamara shares a calibration workflow: take 200 of your own items, score each model on both accuracy and whether its '90% sure' claims actually hold up — that gap is the whole test. Full write-up in the reply thread.
More from coding & agent
- Setting Astra's Thinking to 'Extra High' Makes It Over-Engineer Unit Tests — astralmatrix · 2026-09-24
- Free O'Reilly Book Offers a Pragmatic Framework for Scaling AI in Engineering Teams — blaizedsouza · 2026-09-24
- Zilliz CTO: agents make the enterprise data layer impossible to ignore — No_Engineer_1224 · 2026-09-24
- Running Android emulator + Chrome with 60fps streaming in a $0.072/hr cloud VM for always-on agents — cem2ran · 2026-09-24
- 10 agent reruns reached the right neighborhood, none reproduced the key observation — rohanpaul_ai · 2026-09-24
- Using the Jev model for offensive security: OPSC ranking and sensitive file detection in Mythic — dyn___ · 2026-09-24