Yale physicists re-grade AI benchmarks: most are broken, frontier models near saturation

inductionheads · x · 2026-09-15

A new paper, "How Good Are Frontier Models at Physics?", had Yale physicists expertly re-grade existing benchmarks — and found many model "failures" were actually broken benchmark items.

By current benchmarks, frontier models look weak at physics (e.g. GPT5.6-sol at 47.3% on Humanity's Last Exam physics), but this contradicts physicists' hands-on experience
Re-grading the "failed" answers with Yale physicists showed the problems were usually the benchmark, not the model
Once corrected, models almost saturate every benchmark tested, including HLE physics and CritPt

Takeaway: most current physics benchmarks are broken and no longer measure frontier capability.

Original post →

Yale physicists re-grade AI benchmarks: most are broken, frontier models near saturation

More from Models