Yale Physicists Re-Grade AI Physics Benchmarks, Find Frontier Models Massively Underscored
MikePFrank · x · 2026-09-16
A new paper had Yale physicists manually re-grade supposed failures of frontier models on major physics benchmarks and found much of the problem lay in broken questions and grading, not the models. Once corrected, frontier models nearly saturated several benchmarks: GPT-5.6 Sol jumps from 47.3% to 78.7% on HLE-Physics mean@4 and hits 94.4% pass@4 on corrected CritPt. The authors call it a brutal indictment of benchmark indices, echoing long-standing doubts from lab employees about leaderboards like Artificial Analysis ranking GPT-6 Astra below peers.
More from Models
- LightOnOCR-3 Reads 16th-Century Manuscripts and Ancient Greek Without Training — IgorCarron · 2026-11-02
- 'You Can Feel the Gradient Slopes': Dev Contrasts Benchmaxxed RL Models vs General Ones — theteknosaur · 2026-11-02
- LightOnOCR-3 Launches: Tops FRBench, Adds Layout Structure and Chart Extraction — IgorCarron · 2026-11-02
- NVIDIA's Kumo Is a Family of Four Tabular Foundation Models, Not Just One — pandeyparul · 2026-11-02
- SemiAnalysis: safety classifiers are underrated — Meta's Muse tops cyber benchmark at 35.2% — ThomasScialom · 2026-11-02
- DeepSeek V4.1 Flash 1017 wins over a longtime closed-source-only user: fast, cheap, reliable — _bachrc · 2026-11-02