AI agents solve 88% of 50 unpublished theoretical physics problems at $3 each, benchmark finds
rbhar90 · x · 2026-10-15
Theoretical physicist Suvrat Raju and collaborators released TheoPhysEval, a preprint benchmark of 50 unpublished, research-level theoretical physics problems contributed by world experts, each with verifiable answers (a human expert would need about a day per problem).
Key findings:
- With realistic tool access (web search, literature, numerical/symbolic scripting), the best agent stack, GPT-6-Astra + Codex, reached 88% final-answer accuracy at roughly $3 per problem.
- Denying web search and background papers changed accuracy only slightly — strong evidence of memorization of the literature, a caveat for benchmark contamination.
- Despite standout problem-solving, agent-led physics breakthroughs remain absent; the authors discuss why.
The authors say the implications kept them up at night: frontier agents can now autonomously execute expert-level "turn-the-crank" physics research calculations.
More from AGI Musings
- Does full RSI actually break the frontier apart? Arguing labs' lead won't widen — panickssery · 2026-10-15
- Amazon's warehouse robot fleet tops 1 million, nearing its 1.5 million human workforce — emmanuelvivier · 2026-10-15
- SciVeri-Bench recruits scientists to build a benchmark for AI agents' open-ended research verification — AkariAsai · 2026-10-15
- NYT Opinion: An A.I. Manifesto to End All A.I. Manifestoes — nytopinion · 2026-10-15
- Gladstone AI report: labs automate R&D pipelines, RSI could start in 3-12 months — sourdub · 2026-10-15
- OpenAI wandered (imagegen, Sora) while Anthropic stayed laser-focused, yet they're neck and neck — voooooogel · 2026-10-15