Yale Physicists' Expert Re-Grading Finds Physics Benchmarks Near-Saturated and Broken

A paper by 50+ authors, including Yale physicists and multiple physics professors, titled "How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks," manually re-graded frontier models' rejected answers on physics benchmarks and concluded that the evaluation ecosystem is systematically broken and models' physics ability has been substantially underestimated.

Confirmed

• Physicists manually reviewed answers judged as "failed/rejected" on 6 popular physics benchmarks.
• Across 4 audited benchmark subsets, only 12 of 250 rejected cases were genuine model errors; most failures stemmed from flawed questions or graders.
• After fixing bad items and grading, frontier models approach saturation on several benchmarks; e.g., GPT-5.6 Sol was severely underestimated on HLE-Physics.
• The audit covered mainstream physics benchmarks included in the Artificial Analysis Intelligence Index 2026.

Why it matters

• Community skepticism about LLM evaluation reliability and grader noise is longstanding; blogger zainhas repeatedly argued that most "model errors" in several benchmarks actually originate from the evaluations themselves. The paper provides systematic evidence for this view.
• If benchmarks are broadly distorted, public perception of weak physics reasoning in models could shift dramatically, and benchmark construction and grading need peer-review-style quality control.

2026-09-15 ~ 2026-09-17 · 6 related posts

• Yale physicists re-grade AI benchmarks: most are broken, frontier models near saturation — inductionheads · 2026-09-15
• Most benchmark 'model errors' are actually benchmark or grading errors, analysis finds — zainhas · 2026-09-16
• [source] Expert re-grading shows physics benchmarks are broken: most 'model errors' are benchmark or grading errors — zainhas · 2026-09-16

3 near-duplicate retellings: MikePFrank · rohanpaul_ai · rohanpaul_ai