Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark
Minhui Zhu, Minyang Tian, Xiaocheng Yang, Tianci Zhou, Lifan Yuan, Penghao Zhu, Eli Chertkov, Shengyan Liu, Yufeng Du, Ziming Ji, Indranil Das, Qingzhi Chen, Junyi Cao, Yufeng Du, Jiabin Yu, Peixue Wu, Jinchen He, Yifan Su, Yikun Jiang, Yujie Zhang, Chang Liu, Ze-Min Huang, Weizhen Jia, Yunkai Wang, Farshid Jafarpour, Yong Zhao, Xinan Chen, Jessie Shelton, Aaron W. Young, John Bartolotta, Wenchao Xu, Yue Sun, Anjun Chu, Victor Colussi, Chris Akers, Nathan Brooks, Wenbo Fu, Jinchao Zhao, Marvin Qi, Anqi Mu, Yubo Yang, Allen Zang, Yang Lyu, Peizhi Mai, Christopher Wilson, Xuefei Guo, Juntai Zhou, Daniel Inafuku, Chi Xue, Luyu Gao, Ze Yang, Yaïr Hein, Yonatan Kahn, Kevin Zhou, Di Luo, John Drew Wilson, Jarrod T. Reilly, Dmytro Bandak, Ofir Press, Liang Yang, Xueying Wang, Hao Tong, Nicolas Chia, Eliu Huerta, Hao Peng
cs.AI, cond-mat.other, cs.CL, hep-th, quant-ph
2025-10-01
CritPt tests LLMs on 71 unpublished research-level physics challenges. GPT-5 (high) reaches 5.7% without tools and 10.6% with a code interpreter.
Most LLM benchmarks still live in contest math, coding, and coursework. Physics research asks for something else: unpublished problems, answers that cannot be googled, and error chains that collapse from a single bad assumption. A collaboration led by Argonne and the University of Illinois Urbana-Champaign, with more than 50 working physicists, built CritPt (Complex Research using Integrated Thinking - Physics Test, pronounced "critical point") to test whether models can handle unseen, entry-level research tasks.
Experts pitched the difficulty at a warm-up exercise a PI might give a junior graduate student: the methods are public, but assembling them into a new, verifiable solution is not.
CritPt has two layers. Seventy-one composite challenges mimic small research projects. Each splits into two to four checkpoint questions, 190 in total, so failures can be localized. Coverage spans condensed matter, quantum information, AMO, high-energy, mathematical physics, astrophysics, statistical physics, nuclear, nonlinear dynamics, fluids, and biophysics; nearly half the items cut across more than one area. Theory, experiment, and computation are all represented.
Every problem is newly written and unpublished. Answers are guess-resistant and machine-checkable: high-precision floats, messy symbolic expressions, or Python functions graded on expert test cases. Generation is two-step: free-form derivation first, then a forced extraction into a canonical code block. The autograder applies expert tolerances, SymPy equivalence, and test cases. Solutions to 70 test challenges stay private; a public grading server accepts only full batches, ten per account per day.
Creating one full challenge took more than 40 hours of expert time, with multi-round revision and peer review.
The headline metric is average accuracy over five independent runs. On full challenges, the strongest base model, GPT-5 (high), scores 5.7%. A code interpreter lifts that to 10.6%; adding web search only reaches 12.6%. Search barely helps, which is the point of the leakage-resistant design. Other reasoning models sit at 2% or below. Chat models GPT-4o, Llama-4 Maverick, and GPT-5 (minimal) score 0.
| Model | Challenge | Checkpoint (self-carryover) |
| GPT-5 (high, code & web) | 12.6% | 21.4% |
| GPT-5 (high, code) | 10.6% | 20.0% |
| GPT-5 (high) | 5.7% | 15.3% |
| Gemini 2.5 Pro | 2.0% | 8.1% |
| o3 (high) | 1.4% | 7.4% |
Checkpoints are closer to the local tasks an assistant might actually take. GPT-5 (high) reaches 15.3% when carrying its own prior answers, and 20.0% when given expert answers to earlier checkpoints. Under the stricter "consistently solved" rule (at least four of five runs correct), only the GPT-5 family stably solves any full challenge: 4.3% without tools, 8.6% with code, 10.0% with code and web. Every other model drops to 0.
This is an evaluation built around how physicists actually work, not around a textbook problem set. For anyone shipping a research assistant, the message is blunt: end-to-end research challenges are not ready. Narrow, well-scoped steps plus a code tool start to look useful. Web search barely moves the number, so the bench is testing reasoning rather than retrieval. Physicists in the study already see occasional checkpoint wins, then spend so long verifying plausible traces that doing the work themselves can be faster.
The reliability metric uses only five runs; the authors do not claim statistical sufficiency. Test answers are sealed, so the community can check scores but not independently re-grade each step. Even with expert intermediates, the best checkpoint score just clears 20%. Grading is mostly final-answer, with expert reading of reasoning traces limited to a published example. GPT-5 arrived near the end of data collection and barely beat o3; how long the private set resists the next generation depends on how well those answers stay sealed.