arXiv: Expert Re-Grading Reveals Broken Physics Evals and Near-Saturated Benchmarks
rohanpaul_ai · x · 2026-09-17
Paper: "How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks" (arXiv:2609.13009), by Ali Ansari and 50+ authors from Yale and other top labs. Key finding: low reported scores on leading physics benchmarks (including those in the Artificial Analysis Intelligence Index 2026) create a false impression that frontier LLMs struggle with advanced physics. Expert audits show most rejected answers were actually failed by flawed questions, reference answers, or graders; frontier models are near-saturating these closed-ended benchmarks, making benchmark quality the new bottleneck for measuring frontier capability.
More from Research
- NVIDIA's Kumo Is a Family of Four Tabular Foundation Models, Not Just One — pandeyparul · 2026-11-02
- Aaroth shares Agreement Theorem lecture: round-wise calibration achievable without distributional assumptions — Aaroth · 2026-11-02
- Economist Aaroth: fast agreement dynamics survive without Bayesian updates or accurate priors — Aaroth · 2026-11-02
- ELLIS names 160 new ML Fellows and Scholars for 2025–2026 — JJitsev · 2026-11-02
- Meme turn 50: Aumann's 'Agreeing to Disagree' paper anniversary — Aaroth · 2026-11-02
- Anthropic: analysts unknowingly adopted their agent's secret lean 86% of the time — shoumikgoswami · 2026-11-02