Open-source AI models fail physics research rubrics despite right answers
QuintinPope5 · x · 2026-11-06
Reacting to xDotLI's evaluation, Quintin Pope laments how badly open-source AIs perform. The finding: SOTA agents can get the numbers right but fail to deliver research the way physicists want — falling short on literature review, integrity, and scientific paper writing when judged by rubrics, exposing a gap between solving problems and producing real research.
More from Models
- On METR's new graph: leaked AECI benchmarks are the usual weaksauce, but insiders are freaked out — gleech · 2026-11-06
- On the METR graph: insiders seeing AECI tasks is weak evidence, not zero evidence — gleech · 2026-11-06
- Daily AI brief: GPT-6.2 imminent, Claude Personal Finance beta, METR says Anthropic progress up 1.9x — testingcatalog · 2026-11-06
- METR report researcher quotes lean bullish: Opus 4.5 coding plus faster experiment cycles — gleech · 2026-11-06
- gleech: dropping the three weak benchmarks raises the estimated slope, but the y-axis is meaningless — gleech · 2026-11-06
- Epoch AI review: OSWorld tasks are simple, unstable, and barely test GUI use — gleech · 2026-11-06