Yale-Led Study: Only 12 of 250 Rejected Physics Answers Were Real Model Errors
rohanpaul_ai · x · 2026-09-17
A Yale-led paper finds frontier models are near-saturating today's closed-ended physics benchmarks — and most apparent failures come from broken benchmarks, not the models. Physicists re-audited rejections across 6 popular physics benchmarks: of 250 rejected cases in 4 audited subsets, only 12 were genuine model mistakes; the other 238 stemmed from bad questions, wrong reference answers, or brittle graders rejecting correct answers. After expert re-grading, GPT-5.6-Sol's HLE-Physics score jumped from 47.3% to 78.7%. Caveat: the authors' agents still failed to fully solve any open theoretical-physics problems — benchmark quality is now the bottleneck for measuring frontier AI.
More from Research
- NVIDIA's Kumo Is a Family of Four Tabular Foundation Models, Not Just One — pandeyparul · 2026-11-02
- Aaroth shares Agreement Theorem lecture: round-wise calibration achievable without distributional assumptions — Aaroth · 2026-11-02
- Economist Aaroth: fast agreement dynamics survive without Bayesian updates or accurate priors — Aaroth · 2026-11-02
- ELLIS names 160 new ML Fellows and Scholars for 2025–2026 — JJitsev · 2026-11-02
- Meme turn 50: Aumann's 'Agreeing to Disagree' paper anniversary — Aaroth · 2026-11-02
- Anthropic: analysts unknowingly adopted their agent's secret lean 86% of the time — shoumikgoswami · 2026-11-02