86 physics questions, 100 runs each: model hits 87.6% accuracy but understates confidence
Ok-Challenge-7810 · reddit · 2026-09-22
A developer ran all 86 GPQA-diamond physics questions through the model Jev 100 times each and plotted confidence vs. actual accuracy. Result: 87.6% actual accuracy versus 72% average stated confidence — the model undersells itself. Asking the same question 100 times in a row produced nearly identical probabilities every time. Charts and code are published in a public notebook.
More from Models
- Xiaomi launches open-source MiMo-2.6 model series, praised as outperforming rivals — arpit_bhayani · 2026-09-22
- Gary Marcus: OpenAI solved the physics problem as posed, but not what physicists actually need — GaryMarcus · 2026-09-22
- Dev questions whether Prime Intellect's livestreamed RL run was a reenactment — giffmana · 2026-09-22
- Thermofluids professor: OpenAI's Clay problem solution is not physically reproducible — GaryMarcus · 2026-09-22
- A local 27B Qwen agent running on 24GB DDR4 started acting human after getting long-term memory — Composer_Fearless · 2026-09-22
- Dev hot take: grok-4.7 is overrated while Xiaomi's mimo-2.6 is underrated — intellectronica · 2026-09-22